Backups and recovery

I destroyed my server on purpose and timed the recovery

2026-11-09 · 8 min read

"We have backups" is a sentence, not a result. The result is a number: how long from a dead server to a working app with the data intact. We scripted the whole thing, destroyed a test server on purpose and timed the rebuild. This post shows how the drill is built so you can build your own, what the numbers did and did not tell us, and five problems it found that a green backup log never would.

Who needs this

A solo founder whose app is also their income. You know the backup runs. You do not know whether, at 02:00 with the site down, you can find the repository password, install the right packages in the right order and bring Postgres back with its roles. A drill replaces hope with a number you can say out loud to a customer.

A small team where only one person set up the server. The recovery steps live in that person's head and shell history. Writing them as a script that runs against a fresh machine forces them onto paper, and the first run will show which steps were never written down.

A project about to take a risky change. Before a major Postgres upgrade, an OS release upgrade or a provider move, a rebuild-from-backup drill is the rehearsal. If you can rebuild from nothing in an hour, a failed upgrade is an annoyance rather than a crisis.

Anyone who has only ever restored one file. Restoring a file proves the repository is readable. It does not prove the server can be reconstructed around it, and that gap is where most of the surprises live.

What the drill does

The source "server" is a clean Debian 12 container with a small Node app, a SQLite database, a Postgres database with a role, an index and rows, and a folder of uploaded files. It is backed up with restic to a volume that stands in for storage outside the server. Then:

1. Destroy the source container. Only the backup volume remains. The script checks that the container is really gone.
2. Start a fresh debian:bookworm with nothing installed.
3. Install packages: restic, PostgreSQL, Node, git.
4. restic restore the uploads and the dumps.
5. Restore the Postgres roles, then the database. Copy the SQLite file back and integrity-check it.
6. Deploy the app from git and wait for its health endpoint.
7. Verify row counts and a SHA-256 over every upload file against what the source had.

The repository password is copied into the fresh container from the host rather than stored on the backup volume, because in a real disaster the password is exactly the thing that lives somewhere else. Docker is a convenient stand-in for "a new server". The same steps run unchanged on a freshly provisioned VPS.

The recovery steps

Each recovery phase is one command, so the drill can time it. The two that carry the Postgres logic look like this.

# restore uploads and dumps back to their original paths
restic restore latest --tag app --target / --quiet

# roles first, then the database with its original owner
runuser -u postgres -- psql -q postgres < /var/backups/dumps/pg-globals.sql
runuser -u postgres -- pg_restore --create --exit-on-error -d postgres < /var/backups/dumps/pg-appdb.dump

The reason for opening the files as root and feeding them on stdin is covered in the findings below. The SQLite phase is shorter:

install -d -o app -g app /var/www/app/data
install -m 644 -o app -g app /var/backups/dumps/sqlite-app.db /var/www/app/data/app.db
[ "$(sqlite3 /var/www/app/data/app.db 'PRAGMA integrity_check')" = ok ]

And the verification step asks the restored app, not the files. The demo app's /health returns JSON with row counts, and the drill compares them and the upload checksum:

body=$(curl -fsS -m 5 http://127.0.0.1:3000/health)
sum=$(cd /var/www/app/uploads && find . -type f -print0 | sort -z | xargs -0 sha256sum | sha256sum | cut -d' ' -f1)
[ "$sum" = "$EXPECT_SUM" ] || { echo "upload checksum mismatch"; exit 1; }

What we measured

Measured on a laptop under Docker Desktop (arm64), a local disk and a warm Docker cache. The repository and the "server" were on the same machine, so no backup data crossed a network.

The package install dominated, and it varied. Across six runs of the small drill it took 21.1, 24.1, 28.2, 31.9, 35.9 and 56.5 seconds, so totals ranged from about 27 to 62 seconds. The steps that did not touch the network were identical to within a tenth of a second. The larger-data column is one run, and we would not extrapolate a line through two points to your data.

Rule of thumb: the scripted steps are the cheap part. Provisioning, DNS and finding the password are what fill the clock.

How to read this honestly: the scripted steps are cheap. What a real incident adds is everything around them. Provisioning a new server at a provider is not in these numbers. Neither is the DNS change and its propagation, the time to find the repository password, or the time you spend debugging before you decide to restore. Add those yourself. The method matters more than our figures: run the drill with data the size of yours and write the result at the top of your runbook.

Five things a drill finds that a backup log does not

None of these shows up in a "backup succeeded" line. Each one surfaced as a failed step in the drill.

1. Roles are not in pg_dump -Fc. Restoring into a fresh cluster failed with role "shop" does not exist. You also need pg_dumpall --globals-only, restored first. It contains password hashes, so keep it mode 600.
2. Permissions on the dump directory. Dumps live in a mode 700 directory, which is right. After restic restore, the postgres user cannot read them, and pg_restore fails with "Permission denied". Root opens the file and the postgres user reads stdin, or you move the dump somewhere readable. You want to meet this on a calm afternoon.
3. Two deploys in the same second can share a release folder. The cleanup after a failed second deploy then deleted the live release. Give every release a unique name. This is a deploy bug rather than a backup bug, but the drill deploys from scratch and shook it out; the pattern is covered in rollback in under a minute with symlink releases.
4. A crash-looping release hits systemd's start limit. The default is 5 starts in 10 seconds, and Restart=on-failure with a short RestartSec gets there while the health check is still waiting. After that systemctl restart app is refused until you run systemctl reset-failed app, so a rollback's restart can fail too.
5. docker exec does not forward the host environment. A "large data" run could silently be the small one because the variables never arrived. The drill now prints the counts it is checking, and so should yours.

The point of the list is not the five items. It is that all five were invisible from the backup side and obvious from the restore side.

What to build into your own drill

• Run it from a script, on a schedule you can keep. A restore you do by hand once a year will not work the year you need it. The cheap restore test runs weekly from cron; the full rebuild a few times a year, and before any big change.
• Verify with counts and checksums, not "the command exited 0". See when a restore succeeds and hands you the wrong data for what a green exit code hides.
• Keep the password and the runbook off the server. A drill that finds the password on the dead machine has found a bug.
• Time every phase, not just the total. The shape of the numbers tells you what to fix. If apt dominates, a base image or a package list at hand helps. If pg_restore grows with your data, that is your real recovery time.
• Turn the steps into a document. The script is the source of truth; write a restore runbook turns it into a page a stressed person can follow.

Mistakes we see

• Drilling on the production server. You will eventually restore over live data. Use a fresh machine, a container or a scratch VM.
• A drill that reuses state from the source. If the "fresh" server still has the packages, the config and the repository password, you tested nothing. Start from an image with none of it.
• Counting only the restic step. The restic restore is the fast part. Package installs, roles, ownership, DNS and TLS certificates are where time goes.
• Never lowering the DNS TTL. A record with a 24-hour TTL turns a ten-minute rebuild into a day of some users hitting the dead address. Lower it ahead of time, not during the emergency.
• Treating the drill as pass or fail only. A pass that took three times longer than last time is a finding.

Drill styles compared

Most of the value comes from the second and third rows. The fourth catches provider-specific surprises the container cannot.

FAQ

How long should disaster recovery of one VPS take?

It depends on data size, network speed and how well you have rehearsed it. The scripted steps in our drill took about 30 seconds for the small data set, while provisioning, DNS and finding credentials add minutes to hours that no container shows. Measure your own and treat it as your recovery time objective.

Do I need Docker to run a recovery drill?

No. Docker makes it repeatable and cheap on a laptop or CI. On a plain VPS you can provision a throwaway server, run the same recovery commands and delete it afterwards. The key requirement is that the machine starts with nothing from the old server.

How often should I do a full disaster-recovery drill?

A few times a year, and before any big change such as a major Postgres upgrade. Run the cheaper automated restore test weekly in between. If your setup changes often, drill more often.

What if the drill fails halfway?

That is the drill working. Write down the failing step, fix the script or the backup, and run it again from a clean machine. Your backups are not trustworthy until it passes, so put the fix ahead of feature work.

The drill script, the recovery steps, config templates and fifteen short chapters are in the Small App on One VPS handbook. One command re-runs every check in Docker.

backupresticrecoverypostgres