Webhooks

Receiving signed webhooks in Go with a durable queue

2026-11-09 · 9 min read

A webhook endpoint looks trivial until the sender retries, your handler dies halfway through, or someone replays a captured request. Then you either lose a paid order or ship the same licence key twice. This post walks through a receiver that survives those cases using only the Go standard library and one SQLite file, and shows what each piece prevents.

Who needs this

Three situations we see again and again with small apps on a single server.

A sales notification and licence delivery for a digital product. You sell through Polar (or Stripe) and want an order.paid event to send the buyer a licence key and ping you in a chat channel. If your handler sends the email and then crashes before it records that fact, the retry sends a second email with a second key. If the handler is slow and the provider times out, you get the retry while the first run is still going.

A CI trigger from GitHub. A push to main should start a build on your own box. GitHub does not retry on its own, so a deploy that happened to land while your service restarted is simply gone, and nobody notices until the site is stale. You want the event on disk before you answer, and a way to replay it afterwards.

A subscription backend that grants and revokes access. Stripe sends checkout.session.completed, invoice.paid, customer.subscription.deleted and friends, with no guaranteed order and retries for days. A naive handler that does work inline will process a cancellation before the matching creation and leave a customer locked out or, worse, permanently entitled.

All three have the same shape: authenticate the request, remember that you saw it, answer within a couple of seconds, and do the real work later, once, even though the sender may deliver at least once.

The pipeline in five steps

sender --POST--> /webhook/{source}
                   1. read the body (size limit)
                   2. verify the signature        -> 401 if wrong, nothing stored
                   3. dedupe on the delivery id   -> same event twice = handled once
                   4. INSERT into SQLite          -> 202 Accepted
worker(s) -------> handle(event)  (your code), retry with backoff, dead-letter

Rule of thumb: nothing is written for a request that fails verification, and nothing gets a 2xx unless it is stored.

The order matters. An unauthenticated request must cost you almost nothing and store nothing, so verification comes before any database write. The response is sent only after the event is durably on disk, so a 202 means "I have it" and anything else means "send it again".

1. Read the raw body once, with a cap

The signature covers the exact bytes the sender transmitted. If you decode the JSON into a struct and re-encode it, whitespace, key order and number formatting can change, and the HMAC will no longer match. Read the bytes first and keep them.

body, err := io.ReadAll(http.MaxBytesReader(w, r.Body, 1<<20))
if err != nil {
	var tooBig *http.MaxBytesError
	if errors.As(err, &tooBig) {
		http.Error(w, "payload too large", http.StatusRequestEntityTooLarge)
		return
	}
	http.Error(w, "bad request", http.StatusBadRequest)
	return
}

MaxBytesReader caps memory so a hostile client cannot make you buffer gigabytes. One megabyte is plenty for the provider payloads we have seen; raise it if yours are larger. Pair this with sane server timeouts so a slow client cannot hold the connection open, which we cover in Go HTTP timeouts for slow clients.

2. Verify before you trust anything

Three header formats cover most providers you will meet.

• GitHub style: X-Hub-Signature-256: sha256=<hex>, an HMAC-SHA256 of the body.
• Stripe style: Stripe-Signature: t=<unix>,v1=<hex>, an HMAC over "<t>.<body>", with a timestamp you compare against the clock.
• Standard Webhooks (Polar uses it): webhook-id, webhook-timestamp and webhook-signature: v1,<base64>, an HMAC over "<id>.<timestamp>.<body>".

The differences, and the ways each can go wrong, are the subject of webhook signature pitfalls, and the Standard Webhooks code is in verifying Standard Webhooks signatures in Go. For now the rule is: compare with hmac.Equal, never ==, and return 401 with an empty body on failure.

In the starter, a config object holds every accepted secret, so a rotation needs no downtime:

func (c *Config) verify(r *http.Request, body []byte) (string, error) {
	var lastErr error = signature.ErrInvalid
	for _, secret := range c.Secrets {
		// try the scheme with this secret; the first that verifies wins
		...
	}
	return "", lastErr
}

3. Pick the delivery id and dedupe

Every sender retries, because networks fail and your server restarts. A valid signature proves who sent the request. It does not prove you have not seen it. You need a stable identifier for "this event" and a database constraint that refuses the second copy.

The deeper treatment, including SQLite and Postgres examples and how long to remember ids, is in webhook idempotency and dedupe patterns.

4. Store, then answer

CREATE TABLE IF NOT EXISTS events (
  id          INTEGER PRIMARY KEY AUTOINCREMENT,
  source      TEXT NOT NULL,
  delivery_id TEXT NOT NULL,
  payload     BLOB NOT NULL,
  status      TEXT NOT NULL DEFAULT 'pending', -- pending | processing | done | dead
  attempts    INTEGER NOT NULL DEFAULT 0,
  next_at     INTEGER NOT NULL,
  last_error  TEXT NOT NULL DEFAULT '',
  created_at  INTEGER NOT NULL,
  done_at     INTEGER,
  UNIQUE (source, delivery_id)
);
CREATE INDEX IF NOT EXISTS events_ready ON events (status, next_at);

The insert is one statement, and the unique constraint does the dedupe atomically:

res, err := db.ExecContext(ctx,
	`INSERT OR IGNORE INTO events (source, delivery_id, payload, next_at, created_at)
	 VALUES (?,?,?,?,?)`, source, deliveryID, payload, now.Unix(), now.Unix())
n, _ := res.RowsAffected() // 1 = new event, 0 = we already had it

A duplicate still gets a 202. From the sender's point of view the delivery succeeded, and that is exactly what you want: if you answered with an error for a duplicate, some senders would keep retrying it. If the insert itself fails, answer 503 so the sender retries. Never answer 2xx for something you did not store.

The database opens in WAL mode with a busy timeout and a single connection. A single writer is the simplest correct design for one process, and it keeps you out of the class of bugs we describe in Go SQLite stale reads.

5. Work later, with retries

A worker claims one ready event atomically, runs your handler, and marks it done. On error it schedules a retry with exponential backoff, and after the configured number of attempts it parks the event as dead so a poison message cannot loop forever. The mechanics, including leases and jitter, are in a webhook retry queue in SQLite. Your handler looks like this:

func handle(ctx context.Context, e store.Event) error {
	var ev polar.Event
	if err := json.Unmarshal(e.Payload, &ev); err != nil {
		return worker.Permanent(err) // can never succeed: dead-letter now
	}
	switch ev.Type {
	case "order.paid":
		return fulfil(ctx, ev) // idempotent, keyed on the order id
	}
	return nil // unrelated events are acknowledged and ignored
}

Returning an error means "try again later". Returning a permanent error means "do not bother", which is right for malformed payloads and wrong for anything that a later attempt could fix, like a downstream API being down.

What the handler does for each use case

Sales notification. On order.paid, post one message to your chat webhook. Key the message on the order id: store notified_order_id somewhere with a unique constraint and skip if it exists. Amounts arrive in the currency's smallest unit (cents for USD), so divide before you print them.

Licence issuing. Create the licence row with UNIQUE(order_id) and INSERT ... ON CONFLICT DO NOTHING, then send the email from the stored row. If the email call fails, return an error: the next attempt finds the licence already exists and only retries the send. Do not generate a random key before checking, or retries will mint different keys.

CI trigger. On a GitHub push event, check the ref is refs/heads/main, then start your build script. Make the script take the commit SHA as its argument and refuse to rebuild a SHA that is already deployed. A replayed or duplicated push then costs nothing.

Mistakes we see

Doing the work inside the request. Provider timeouts are short, often ten seconds or fewer, and a slow downstream call turns into a retry storm. Store first, answer 202, process in the background.

Parsing before verifying. A JSON decoder that runs on unauthenticated input is attack surface and also tempts you to verify the re-encoded bytes.

Treating a retry as a bug. Retries are the contract. At-least-once delivery means every handler has to be safe to run twice. Dedupe at the door catches most duplicates, but not the one where your handler crashed after the side effect and before the commit, so the handler needs its own idempotency too.

Logging headers and bodies. Signatures, secrets and customer emails end up in your log files. The starter's access log records a request id, method, path, status, size, duration and an outcome word such as accepted, duplicate or bad_signature, and never headers, query strings or bodies. See structured logging with slog for a setup that stays quiet by default.

Running two processes against one SQLite file to scale out. One process with several workers is fine. Two servers fighting over the single writer is not a queue design.

Running it on the server

The receiver is one binary and one SQLite file, so it fits a single systemd unit behind a TLS reverse proxy such as Caddy.

• Keep /metrics off the public internet or protect it with a bearer token.
• Give the unit a state directory for the database and restart it on failure.
• Let it shut down gracefully so in-flight handlers finish and an interrupted event goes back to the queue without losing an attempt.
• Point the provider at https://hooks.example.com/webhook/polar, using the path segment as the source name that scopes your dedupe key.

Then send one real test delivery from the provider's dashboard and watch the access log for the accepted outcome before you trust it with orders.

How fast is it, honestly

These figures are from the starter's own load test on an Apple Silicon laptop with an SSD, default durability (WAL, synchronous=FULL), 32 connections and 512-byte payloads. Accepting verified events over loopback HTTP ran at roughly 7,200 to 9,400 per second across runs, and end to end with a no-op handler about 3,100 to 3,200 per second. With a handler that sleeps 5 ms to imitate a network call, one worker managed about 163 events per second and 32 workers about 5,264. Extra workers do not help a handler that does no waiting, because every database access shares one connection. Set the worker count near your handler's real concurrency, not higher. A small VPS with different disks will give different numbers, so run the load test on the machine you will deploy to.

For most small products the interesting number is much lower: a few dozen events a day. The reason to build it this way is correctness, not throughput.

FAQ

Should I return 200 or 202 for a webhook?

Either is a success to every provider we know of. 202 says "accepted for processing", which is what happened. Use a 4xx only for requests you will never accept, such as a bad signature, and a 5xx when you want the sender to try again.

How do I test a receiver without the real provider?

Sign a payload yourself with openssl and send it with curl. For a GitHub-style endpoint:

B='{"order":7}'
SIG="sha256=$(printf '%s' "$B" | openssl dgst -sha256 -hmac "$WEBHOOK_SECRET" | awk '{print $NF}')"
curl -i -X POST localhost:8080/webhook/shop -H "X-Hub-Signature-256: $SIG" -H 'X-Delivery-Id: e1' -d "$B"

Send it twice to see the duplicate path, then change one byte of $B and keep the old signature to see the 401.

What if my server is down when the webhook arrives?

It depends on the sender. Stripe retries failed deliveries for up to three days in live mode, Standard Webhooks senders typically retry on a schedule, and GitHub does not retry automatically but lets you redeliver from the repository settings or the API. For senders that do not retry, keep the receiver behind a process supervisor and watch for gaps.

Do I need Redis or a message broker for this?

Not for one server and a modest event rate. A SQLite table gives you durability, atomic claiming and an audit trail you can query with sqlite3. A broker earns its keep when several machines produce or consume events.

Where should the SQLite file live and how do I back it up?

On local disk, not a network share. Back it up like any other database, with a consistent snapshot rather than a raw copy of a live file; see restic backups for SQLite and Postgres.

If you would rather start from working code than assemble it, the Go Webhook Receiver Starter packages this pipeline with tests, the three signature schemes, secret rotation and replay tooling.

gowebhookssqlitereliability