Go in production

Graceful shutdown for a Go HTTP server under systemd

2026-11-09 · 10 min read

Every deploy ends with systemctl restart, and every restart sends your Go process a SIGTERM. If the process ignores it, systemd waits ninety seconds and sends SIGKILL, and whatever was mid-flight is gone: a half-written response, a job that was claimed and never finished, a client that gets a connection reset. If the process exits on the first signal without thinking, you get the same damage a lot faster. Graceful shutdown is the small piece of code that sits between those two failures.

This post builds it up in order: catch the signal, stop accepting traffic, let requests finish, let background workers finish, and make the exit bounded so systemd and your code agree on how long "finish" may take.

Who needs this

Three kinds of service show up again and again on a single VPS.

A JSON API with a few slow endpoints. Most requests take 5 ms, but one report endpoint takes 4 seconds. Every deploy has some chance of landing in the middle of one. Without a drain, that user sees a failed request. With one, the request completes and the new process takes over the next one.

A service with a background worker. It claims rows from a queue (SQLite or Postgres), does work, and marks them done. Kill it at the wrong moment and a row sits in running forever, or the work is done twice. Graceful shutdown decides which of those you get on purpose: finish the current job, or put it back.

A webhook receiver. The sender retries, so dropped requests are mostly recoverable, but a retry storm after every deploy is noise you can avoid, and a request that was acknowledged with 202 and then lost in a kill is not recoverable at all. See receiving webhooks in Go for the acknowledge-then-process pattern that makes the drain matter.

If your service is a static file server with no state, you can skip most of this. Everyone else should have it before the first real user.

What "graceful" has to mean

Write the contract down before you write code, because each line becomes a test:

1. A request already in flight finishes normally.
2. New connections are refused after the signal.
3. Background work in progress either finishes or is put back in the queue.
4. The process exits, with code 0 on a clean drain, inside a bound you chose.

The Go handbook we ship tests exactly this against a real binary with a real SIGTERM. In its recorded run, a 1.5 second job was in flight when the signal arrived. The listener refused new connections 11 ms later, the in-flight long-poll was answered with HTTP 200 about 1.4 seconds after the signal, and the process exited with code 0 at 1.67 seconds. An idle service exits about 2 ms after SIGTERM. Those numbers are from one container on a laptop, so read them as the shape of the thing, not a promise for your hardware.

Step 1: catch the signal with signal.NotifyContext

Since Go 1.16 the simplest correct way to turn a signal into a cancellation is signal.NotifyContext:

sigCtx, stop := signal.NotifyContext(context.Background(), syscall.SIGINT, syscall.SIGTERM)
defer stop()

sigCtx is cancelled when either signal arrives. SIGTERM is what systemd, Docker and kill send by default. SIGINT is what Ctrl-C sends in your terminal. You do not need SIGHUP unless you want to reload config on it, and you cannot catch SIGKILL at all.

One detail is worth copying. After the first signal, call stop(). That restores the default behaviour, so a second Ctrl-C kills a shutdown that has hung. Without it, a stuck drain ignores you until the deadline.

Step 2: separate three things that people lump together

The shutdown code that goes wrong is usually the code that treats "the signal arrived" as one event. There are really three decisions:

• Stop accepting new connections and requests (the HTTP server).
• Stop starting new background jobs (the worker loop).
• Abort what is still running because time is up (the deadline).

The first two happen at the signal. The third happens only if the drain exceeds your limit. So you need two contexts for the work, not one: the signal context, which means "stop claiming", and a work context that you cancel yourself, which means "give up and put things back".

Here is a complete program that does it. It compiles on Go 1.22 and later.

package main

import (
	"context"
	"errors"
	"log/slog"
	"net"
	"net/http"
	"os"
	"os/signal"
	"sync"
	"sync/atomic"
	"syscall"
	"time"
)

const shutdownTimeout = 20 * time.Second

func main() {
	log := slog.New(slog.NewJSONHandler(os.Stdout, nil))
	if err := run(log); err != nil {
		log.Error("exiting with error", "err", err)
		os.Exit(1)
	}
}

func run(log *slog.Logger) error {
	sigCtx, stop := signal.NotifyContext(context.Background(), syscall.SIGINT, syscall.SIGTERM)
	defer stop()

	// workCtx outlives the signal so jobs can finish; we cancel it
	// ourselves only when the drain deadline passes.
	workCtx, abortWork := context.WithCancel(context.Background())
	defer abortWork()

	var draining atomic.Bool

	mux := http.NewServeMux()
	mux.HandleFunc("GET /healthz", func(w http.ResponseWriter, r *http.Request) {
		w.Write([]byte("ok\n"))
	})
	mux.HandleFunc("GET /readyz", func(w http.ResponseWriter, r *http.Request) {
		if draining.Load() {
			http.Error(w, "draining", http.StatusServiceUnavailable)
			return
		}
		w.Write([]byte("ready\n"))
	})

	srv := &http.Server{
		Addr:              "127.0.0.1:8080",
		Handler:           mux,
		ReadHeaderTimeout: 5 * time.Second,
		BaseContext:       func(net.Listener) context.Context { return workCtx },
	}

	var wg sync.WaitGroup
	wg.Add(1)
	go func() {
		defer wg.Done()
		worker(sigCtx.Done(), workCtx, log)
	}()

	serveErr := make(chan error, 1)
	go func() { serveErr <- srv.ListenAndServe() }()

	select {
	case err := <-serveErr: // listener died on its own
		abortWork()
		wg.Wait()
		return err
	case <-sigCtx.Done():
		stop() // a second Ctrl-C now kills the process
		log.Info("shutdown requested")
	}

	draining.Store(true)
	ctx, cancel := context.WithTimeout(context.Background(), shutdownTimeout)
	defer cancel()

	var errs []error
	if err := srv.Shutdown(ctx); err != nil {
		errs = append(errs, err)
		srv.Close() // deadline passed: drop what is left
	}
	if err := <-serveErr; !errors.Is(err, http.ErrServerClosed) {
		errs = append(errs, err)
	}

	done := make(chan struct{})
	go func() { wg.Wait(); close(done) }()
	select {
	case <-done:
	case <-ctx.Done():
		abortWork() // cancel running jobs so they can put themselves back
		<-done
		errs = append(errs, ctx.Err())
	}
	return errors.Join(errs...)
}

// worker stops claiming when stopClaiming closes. A job that is already
// running keeps going until it finishes or jobCtx is cancelled.
func worker(stopClaiming <-chan struct{}, jobCtx context.Context, log *slog.Logger) {
	t := time.NewTicker(time.Second)
	defer t.Stop()
	for {
		select {
		case <-stopClaiming:
			return
		case <-t.C:
			runJob(jobCtx, log)
		}
	}
}

func runJob(ctx context.Context, log *slog.Logger) {
	select {
	case <-time.After(3 * time.Second): // stand-in for real work
		log.Info("job done")
	case <-ctx.Done():
		log.Warn("job aborted; it must be safe to run again")
	}
}

Read the order in run, because the order is the point. Readiness goes red first. Then srv.Shutdown closes the listeners and waits for active requests. Only after that do we wait for the workers. All of it shares one deadline, so the worst case is shutdownTimeout, not the sum of three timeouts.

What http.Server.Shutdown does and does not do

Shutdown closes the listeners first, so new connections are refused. It then closes idle keep-alive connections and waits for active ones to become idle. Three consequences are easy to miss.

It does not cancel request contexts. A handler that waits on something with no bound blocks the shutdown until your deadline. Every wait inside a handler should be bounded, either by a context deadline or by a timeout you set deliberately. Our reference service allows long-polls of at most half the write timeout for this reason.

ListenAndServe returns http.ErrServerClosed immediately. The moment Shutdown is called, ListenAndServe returns, while Shutdown is still draining in another goroutine. If main returns on that error, the process exits and the drain is cut off. That is why the program above reads serveErr after Shutdown returns and checks for http.ErrServerClosed explicitly.

It ignores hijacked connections. WebSockets and anything else taken over with Hijack are not tracked. Register a function with srv.RegisterOnShutdown that tells those connections to close.

BaseContext is the escape hatch. Every request context derives from it, so cancelling workCtx cancels all in-flight requests at once. In the program above it only fires if the deadline passes, which is the right moment to stop being polite.

Step 3: readiness goes red before the listener closes

A health endpoint that reports "not ready" during the drain gives a proxy or a deploy script something to react to. The /readyz route above flips on the draining flag. On one VPS with Caddy in front this matters less than it does behind a load balancer, but it costs four lines and it is what lets a health check that means something tell the difference between "starting", "serving" and "stopping".

If you have a proxy that does active health checks, you can also sleep a second or two between setting the flag and calling Shutdown, so the proxy notices before the listener disappears. Only add that delay if you can see 502s in the proxy log during a restart. Otherwise it is a second of pointless latency on every deploy.

Step 4: background workers, two phases

The worker pattern above has two phases on purpose.

Phase one is "stop claiming". The signal context closes, the loop returns at the next iteration, and no new job starts. A job that is already running continues, because it has the work context, not the signal context.

Phase two exists only if the deadline passes. You cancel the work context, the running job sees ctx.Done(), and it does whatever it must to be safe to run again: roll back its transaction, set the row back to queued, release its lock.

Whatever you do, assume the process can also die with no shutdown code at all (OOM kill, power loss, kill -9). A queue that moves rows through running needs a recovery step at startup that puts orphaned rows back. Jobs then run at least once, never exactly once, so make them idempotent. The retry queue with SQLite post shows that recovery step in a concrete schema.

Step 5: make systemd agree with your code

Your application has a drain deadline. systemd has its own stop timeout. If systemd's is shorter, it kills you in the middle of a clean drain.

# /etc/systemd/system/myapp.service (relevant lines)
[Service]
ExecStart=/opt/myapp/current/myapp
KillSignal=SIGTERM
TimeoutStopSec=30s
Restart=on-failure
RestartSec=1s

The application drains for at most 20 seconds, and systemd waits 30 before it sends SIGKILL. Keep the unit's TimeoutStopSec comfortably above the application's deadline. The default TimeoutStopSec comes from DefaultTimeoutStopSec, which is 90 seconds on most distributions, so if you never set it, a hung process makes every deploy wait a minute and a half.

Rule of thumb: drain deadline below TimeoutStopSec, and TimeoutStopSec below the 90 second default you do not want to wait for.

Restart=on-failure restarts the service after a crash or a non-zero exit, but not after a clean systemctl stop. That ties back to the exit code: when the drain deadline passes, the program above returns an error and exits 1, so you can see it in systemctl status. A clean drain exits 0.

Check what systemd actually has:

systemctl show myapp -p TimeoutStopUSec -p KillSignal -p KillMode

KillMode=control-group is the default and means SIGTERM goes to every process in the unit's cgroup. If your service spawns children, they get the signal too. See why systemctl restart drops requests for what happens at the proxy when the stop and start are not coordinated.

Test it with a real signal

The unit test for shutdown is the integration test: start the real binary, start a slow request, send SIGTERM, and assert the four lines of the contract. In Go that is exec.Command, cmd.Process.Signal(syscall.SIGTERM), a goroutine holding the slow request, and a dial loop that expects connection refused. You can also do it by hand:

./myapp &
curl -s localhost:8080/slow &        # a handler that sleeps 3 seconds
sleep 0.3
kill -TERM %1
curl -sS localhost:8080/healthz      # should fail: connection refused
wait; echo "exit: $?"

The slow curl should still print its response. The second curl should be refused. The exit code should be 0.

Mistakes we see

Calling os.Exit in the signal handler. It skips deferred calls, closes nothing, and drops in-flight requests. The handler should cancel a context and return.

main returning on http.ErrServerClosed. As above, it cuts the drain short. The symptom is that shutdown "works" in dev with no traffic and fails under load.

Shutdown with context.Background() and no deadline. One stuck handler and the process never exits, until systemd gets bored at 90 seconds and kills it anyway.

Database closed before the workers stop. A deferred db.Close() in the wrong place makes in-flight jobs fail with "sql: database is closed". Close the database last, after the HTTP server and the workers are done.

Using the signal context for the job itself. Then every job aborts at the signal and nothing ever finishes gracefully. Keep the "stop claiming" signal and the "abort the job" context separate.

Forgetting that SIGKILL exists. Everything above is best effort. The recovery step at startup is what makes the worst case acceptable.

Choosing a shutdown approach

For most single-VPS apps the third row is the right place to stop. The last row removes the brief window where the port is closed, at the cost of more moving parts; we wrote about that trade in socket activation is not a free lunch.

FAQ

How long should the shutdown timeout be?

Slightly longer than your longest legitimate request or job, and shorter than TimeoutStopSec. For an API with a 30 second write timeout, 20 to 25 seconds is a reasonable drain and 30 to 35 for the unit. If your longest request is 2 seconds, 10 seconds is plenty and deploys stay fast.

Does Shutdown wait for WebSocket or long-lived streaming connections?

No for hijacked connections such as WebSockets, because the server no longer tracks them. Use RegisterOnShutdown to signal them to close, and wait for them with your own sync.WaitGroup. Plain streaming responses are ordinary requests, so Shutdown waits for them until the deadline.

Should I handle SIGHUP or SIGQUIT too?

Handle SIGHUP only if you implement config reload on it. Leave SIGQUIT alone: the default makes the Go runtime dump all goroutine stacks and exit, which is useful when you are debugging a hang.

Why does my service still get killed after 90 seconds?

Your process is not exiting after the signal, so systemd falls back to TimeoutStopSec. Add logging at each step of the drain, with the elapsed time, and the stuck step will show up in journalctl -u myapp. The usual suspects are a handler waiting on something unbounded and a worker that never looks at its context.

Do I need this if I use Docker or Kubernetes instead of systemd?

The Go side is identical. Docker sends SIGTERM and waits 10 seconds by default before SIGKILL, so raise --stop-timeout or the stop_grace_period in Compose if your drain can take longer.

The Go on One VPS handbook has this as a full chapter with the tested Run and shutdown functions, the worker pool, and the SIGTERM integration test, next to the systemd unit that goes with it: Go on One VPS: Production Handbook.

gosystemdhttpdeploys