Deploys and releases

Roll back a bad deploy in a minute with release folders

2026-11-09 · 9 min read

A bad release is rarely a disaster by itself. The disaster is the next twenty minutes: finding the last good commit, rebuilding it, hoping the build is the same, restarting while users are hitting errors. If the previous release is still sitting on disk, fully built, a rollback is a pointer change and a restart. This post shows the directory layout that makes that true, the script that does it, how many releases to keep, and what a rollback cannot undo.

Who needs this

• A Go API deployed by scp and systemctl restart. Today the old binary is overwritten by the new one. If the new one is bad, the old one exists only in your CI artifacts or your git history. Release folders keep it next to the new one.
• A Node app deployed with git pull on the server. A rollback means git checkout, npm ci, hope that the lockfile still resolves, and a restart. Immutable release folders with node_modules inside skip all of that.
• A side project with paying users and one maintainer. You are asleep when the bug report arrives, or on a train. A rollback that is one command, even one you run from your phone over SSH, changes what you are willing to ship on a Friday.
• A team of two that shares a server. Two people deploying means two people overwriting each other's work. A deploy lock plus a record of which release is live (and which was before it) stops the confusion.

The layout

/opt/app/
  releases/
    20261105-091200/app
    20261107-143055/app
    20261109-101500/app      # newest
  slots/
    a -> /opt/app/releases/20261107-143055
    b -> /opt/app/releases/20261109-101500
  current -> /opt/app/releases/20261109-101500
  state
  .deploy.lock

Each release is a directory named by timestamp. It holds a complete, runnable build: the Go binary, or the Node source plus node_modules, plus a small note about where it came from. Nothing in a release directory is ever edited after it is created. That is the property that makes rollback safe: the old release is byte for byte what ran yesterday.

Things that change between releases (uploads, the SQLite file, logs) do not live in the release directory. They live in a state directory such as /var/lib/app, or /opt/app/shared. If your app writes next to its own binary, rolling back also rolls back the data, and you will find that out at the worst time.

The kit's make_release creates releases/<timestamp>/app from a binary built in CI or from a server-side build, and writes a RELEASE file with the git revision or the binary's SHA-256. Months later, cat RELEASE tells you what a directory actually is.

The atomic switch

current is a symlink. Switching it must be atomic, so that nothing ever sees it missing:

set_link() {  # $1 target, $2 link path
  ln -sfn "$1" "$2.tmp" && mv -Tf "$2.tmp" "$2"
}
set_link /opt/app/releases/20261109-101500 /opt/app/current

Why two steps? ln -sfn target link on an existing link removes the old link and creates the new one. There is a short moment in between where the path does not exist. Creating the link under a temporary name and then renaming it over the old one uses rename(2), which replaces the destination atomically. mv -T treats the destination as a file, not a directory to move into, which matters because current points at a directory. -T is a GNU option, so this is Linux-only; on macOS you would need a different spelling, and the kit says so in a comment.

One thing to know: a running process does not follow a symlink change. A process that started from current/ has its working directory and its executable open at the resolved path. Re-pointing the link does not move it. That is useful (a deploy cannot pull the rug from under a running instance) and it has a trap, which is the reason blue/green setups start each release from its own directory. We describe that trap for pm2 in zero-downtime deploys on one VPS.

Keeping the previous release runnable

Rollback speed depends on how much work remains when you type the command. From fastest to slowest:

1. The old instance is still running. With blue/green, the previous release can stay up on its port. Rolling back is a proxy switch: one reload of Caddy. Seconds. It costs the memory of a second process.
2. The old release is on disk, the instance is stopped. Rolling back is: start it on the idle port, wait for the health check, switch the proxy, stop the bad one. Roughly the time your app needs to start, plus a couple of seconds of health checks.
3. The old release is not on disk. Rebuild, redeploy. Minutes at best, and only if the build still works.

The kit defaults to the second and offers the first through KEEP_PREVIOUS=1. The decision is memory. A small Go service might use tens of megabytes; keeping a second copy idle is cheap. A Node app with a large heap is a different calculation.

The rollback itself

The kit records the active and the previous release in a plain state file:

ACTIVE_SLOT=b
ACTIVE_RELEASE=/opt/app/releases/20261109-101500
PREV_RELEASE=/opt/app/releases/20261107-143055

deploy.sh rollback reads PREV_RELEASE, refuses if the directory is gone, and then runs the same activation path a normal deploy uses: start on the idle slot, health-check, switch Caddy, switch current, drain, stop the bad instance. Using the same path matters. A rollback script that is a separate, rarely exercised branch is the one that fails when you need it.

If you are not using the kit, the minimum that works for a single systemd service looks like this:

#!/usr/bin/env bash
set -euo pipefail
APP=/opt/app
cur="$(readlink -f "$APP/current")"
# newest release that is not the current one
prev="$(find "$APP/releases" -mindepth 1 -maxdepth 1 -type d | sort -r | grep -vxF "$cur" | head -n 1)"
[ -n "$prev" ] || { echo "no previous release" >&2; exit 1; }
echo "rolling back $(basename "$cur") -> $(basename "$prev")"
ln -sfn "$prev" "$APP/current.tmp" && mv -Tf "$APP/current.tmp" "$APP/current"
systemctl restart app.service
curl -fsS -m 3 --retry 10 --retry-delay 1 --retry-connrefused http://127.0.0.1:3000/health

This one restarts the process, so there is a short interruption. It is a fine floor. Read the readlink -f: it resolves the link to a real directory, which lets the script compare it to the entries in releases/. Sorting by name only works because the names are timestamps; keep that format.

A better rollback has the same property as a good deploy: the target is checked before it receives traffic. If the previous release has a configuration it no longer understands (a renamed environment variable), you want to find out on the idle port, with the bad release still serving, not after you switched.

How many releases to keep

Enough to reach back past the last bad one, not so many that disk becomes the problem. The kit's default is KEEP_RELEASES=5. For a compiled Go binary of 10 to 30 MB that is well under 200 MB. For a Node release that includes node_modules, a release can be hundreds of MB, and five of them on a small VPS disk is a real number; check with du -sh /opt/app/releases/* before you pick one.

Pruning has a safety rule that is easy to forget: never delete what is referenced. The kit's prune keeps the newest N folders and additionally everything that the state file, the slot links and current point at. Imagine a deploy that fails and a rollback that skips back two releases: the live release may be the oldest in the folder. A plain "delete everything but the newest five" would remove the code that is running.

find /opt/app/releases -mindepth 1 -maxdepth 1 -type d | sort -r | tail -n +6
# these are the candidates for deletion; skip anything referenced

Run the prune only after a successful deploy. A failed deploy should leave the folder exactly as it found it, aside from the new directory you can inspect.

The database is the part a symlink cannot roll back

This is the limit of the technique, and it deserves to be stated plainly. A rollback changes which code runs. It does not change the database. If release N+1 ran a migration that dropped a column, release N is back to a schema it does not recognize. If N+1 wrote rows in a new format, N may choke on them.

The way out is not a clever rollback but a deploy discipline: every release must work against the schema of the release before it. That is the expand/contract pattern, and we cover it in migrations during zero-downtime deploys. With it, going back one release is always safe for the schema. Without it, a rollback is a decision you take with a backup (see restic backup for SQLite and Postgres) in your other hand.

Rule of thumb: code can go back one release, data cannot. Keep every release compatible with the schema of the one before it.

Two practical rules:

• Do not ship a destructive migration in the same release as the code change that needs it. Ship the code first. Remove the old column one release later, when you are sure you will not go back.
• When a release must change data in a way the previous release cannot read, treat that deploy as one-way and say so in the deploy note.

Mistakes we see

Overwriting the binary in place. cp app /opt/app/app replaces the file the running process is executing, can fail with "text file busy", and destroys the old build. Always write to a new directory.

Keeping state inside a release directory. The SQLite file, uploads and logs belong in a path outside releases/. Otherwise every deploy starts with an empty database.

Pruning by age instead of by reference. A release that has been live for three weeks is the oldest and the most important.

Rollback that depends on the network. If your rollback runs git clone or npm install, it fails exactly when GitHub or the registry is down. The release folder should be complete.

No lock. Two terminals, two deploys, two state files. The kit takes flock on .deploy.lock and exits immediately with code 75 for the second caller.

Never rehearsing it. Roll back once on a quiet afternoon so that you know what the output looks like and how long it takes. The first time should not be during an incident.

Comparing rollback strategies

FAQ

What is the fastest way to roll back a deploy on a single server?

Keep the previous release running on its own port behind the reverse proxy and switch the proxy back. That is one graceful reload and takes seconds. If you cannot afford the memory, start the previous release from its release directory on the idle port, check it, then switch.

How many old releases should I keep?

Five is a reasonable default for a compiled binary, because the cost is small and it gives you several steps of history. Check the size of one release and multiply. Always keep whatever the active and previous pointers refer to, whatever your count says.

Is replacing a symlink really atomic?

Creating the new link under a temporary name and renaming it over the old one is atomic on the same filesystem, because rename replaces the target in one operation. Plain ln -sf on an existing link is not: it removes and recreates, leaving a brief gap. Use ln -sfn into a temp name plus mv -Tf.

Does a rollback also undo my database migration?

No. It changes which code runs, nothing else. Make each release compatible with the previous schema (add first, remove later), and rollback stays safe. For changes that cannot be made compatible, take a backup before the deploy and treat it as one-way.

Should rollback be automatic?

For the narrow case of a release that fails its own post-switch health check, yes: the old instance is still running, so switching back is cheap and safe. For problems that need human judgement (a subtle bug found by users an hour later), keep it a command someone chooses to run.

The Go + systemd deploy kit ships this layout ready to use: timestamped release folders, the atomic switch, the state file, deploy.sh rollback, pruning that respects references, and the lock.

deploysystemdzero-downtime