VPS operations

Find what filled your VPS disk and keep it from happening

2026-11-09 · 10 min read

The server was fine yesterday. Today the app returns 500s, deploys fail halfway, and the log says no space left on device or, from SQLite, database or disk is full. A 25 GB disk on a small VPS fills slowly enough that nobody watches it, and then quickly enough that it is an incident.

This post is the sequence we use: stop the bleeding, find what is large, understand why it grew, and put limits in place so it does not recur. It assumes Debian or Ubuntu with systemd, and covers journald, logrotate, Docker logs and the other usual suspects.

Who needs this

• A Node app that logs every request to a file. It has been running for eight months with console.log redirected into /var/log/app/app.log. Nobody set up rotation. The file is now the biggest thing on the disk.
• A Docker Compose setup on a single host. The default json-file log driver keeps every line a container has ever printed, with no limit. One chatty container and a few months produce a multi-gigabyte file under /var/lib/docker/containers.
• A Go service with a debug flag left on. Someone turned on verbose logging to chase a bug and never turned it off. Journald stores it, and now /var/log/journal is the biggest directory.
• A box that does nightly backups and deploys. Old release folders, a restic cache, database dumps in a staging directory and WAL files after a failed archive all accumulate, with none of them being "logs".

First, make room (two minutes)

When the disk is at 100% you cannot debug anything, because tools fail to write temp files. Free a safe chunk first:

df -h /                                   # confirm which filesystem is full
sudo journalctl --vacuum-size=200M        # shrink the journal on demand
sudo apt-get clean                        # drop downloaded .deb files

journalctl --vacuum-size deletes the oldest archived journal files until the total is under the target. It is safe: journald is designed to be trimmed. apt-get clean clears /var/cache/apt/archives. Neither touches your data. Now there is room to look around.

If the app is writing a runaway file and you need relief immediately, truncate it rather than deleting it (details in the "deleted but still full" section below):

sudo truncate -s 0 /var/log/app/app.log

Find what is large

Start from the top and descend. The one-liner we keep in our runbook:

sudo du -xh / --max-depth=2 2>/dev/null | sort -h | tail -20

-x keeps du on one filesystem, so it does not wander into /proc or mounted volumes. sort -h sorts human-readable sizes. The result is the twenty biggest directories two levels down. Usually one of /var/log, /var/lib/docker, /var/lib/postgresql, /var/cache or /home stands out. Descend into it:

sudo du -xh /var --max-depth=2 2>/dev/null | sort -h | tail -15

If you can install a tool, ncdu is nicer. It scans once and lets you navigate with the arrow keys and delete from inside:

sudo apt install -y ncdu
sudo ncdu -x /

On a nearly full disk, install it first and run it after the vacuum. Scanning a large tree takes a while; -x is still the right flag.

For single big files, find sorts by size without a UI:

sudo find / -xdev -type f -size +200M -printf '%s %p\n' 2>/dev/null | sort -n | tail

Check inodes too, in case the space is fine and the files are the problem:

df -i /

A filesystem that is out of inodes (millions of tiny files, such as session files or mail queue entries) also reports "no space left", while df -h shows free blocks.

The usual suspects

journald

On systemd machines, anything that logs to stdout under a unit ends up in the journal. Check its size:

journalctl --disk-usage

By default journald caps itself at 10% of the filesystem, with an upper bound of 4 GB, and also tries to leave 15% free. On a 25 GB disk that can still be a couple of gigabytes of logs you never read. Make the limits explicit with a drop-in:

sudo mkdir -p /etc/systemd/journald.conf.d
sudo tee /etc/systemd/journald.conf.d/10-limits.conf >/dev/null <<'EOF'
[Journal]
Storage=persistent
SystemMaxUse=200M
SystemKeepFree=1G
SystemMaxFileSize=25M
MaxRetentionSec=1month
EOF
sudo systemctl restart systemd-journald
journalctl --disk-usage

What the settings do: SystemMaxUse caps the total space the journal may use. SystemKeepFree stops the journal from eating the last of the free space. SystemMaxFileSize bounds each file, which controls how granular deletion is, since journald deletes whole files. MaxRetentionSec drops entries older than a month even if the size cap has not been reached. Pick numbers that fit your disk; 200M is a starting point and not a rule.

You can prove the cap works on a scratch box. We applied the same file with SystemMaxUse=16M and SystemMaxFileSize=2M, sent 300,000 log lines (about 28 MB of text) through systemd-cat, and journalctl --disk-usage reported 15.1M afterwards. The cap held. Do the same with a small value to see it for yourself, then set the real one.

Storage=persistent keeps logs across reboots in /var/log/journal. Debian creates that directory; on systems where it does not exist, the default auto stores logs in /run/log/journal, in memory, and loses them on reboot. If you would rather have no on-disk journal at all, Storage=volatile with RuntimeMaxUse= limits the in-memory one.

Cutting the noise at the source is better than a cap. If one unit logs a hundred lines a second, look at that unit. journalctl -u app --since "1 hour ago" | wc -l tells you the rate. journalctl -p warning filters by priority when you only want problems.

App log files and logrotate

If your app writes its own files (and the best setup is that it does not; log to stdout and let the journal handle it), logrotate keeps them bounded. It runs daily from a systemd timer or cron. Put a file in /etc/logrotate.d/app:

/var/log/app/*.log {
	daily
	maxsize 50M
	rotate 7
	missingok
	notifempty
	compress
	delaycompress
	copytruncate
}

What this says: rotate when a day has passed or the file exceeds 50 MB (maxsize triggers a rotation even before the daily timer is due, but logrotate itself only runs once a day by default, so a file can still outgrow maxsize before the next run), keep seven old files, skip missing or empty files, compress rotated files but leave the newest rotated one uncompressed for a day so tools that were still reading it do not break.

The tricky option is copytruncate. Without it, logrotate renames the file, and your app keeps writing into the renamed file because it holds the old file handle open. With copytruncate, logrotate copies the content and truncates the original in place, so the app keeps writing to the same file without a restart. The cost is that lines written between the copy and the truncate can be lost. If your app can reopen its log on a signal, use postrotate to send it, and drop copytruncate:

	postrotate
		systemctl kill -s HUP app.service
	endscript

Test a configuration without waiting a day:

sudo logrotate -d /etc/logrotate.d/app      # dry run, prints what it would do
sudo logrotate -f /etc/logrotate.d/app      # force a rotation now
ls -lh /var/log/app/

After ten forced rotations the directory should hold the live file, the uncompressed .1, and .2.gz through .7.gz, and never more. That is the point: the total is bounded.

Docker logs

The default json-file logging driver has no size limit. Every line a container prints goes into /var/lib/docker/containers/<id>/<id>-json.log forever. Find the big ones:

sudo du -h /var/lib/docker/containers/*/*-json.log | sort -h | tail
docker system df

Set a global default in /etc/docker/daemon.json:

{
  "log-driver": "json-file",
  "log-opts": { "max-size": "10m", "max-file": "3" }
}

Then sudo systemctl restart docker. The catch: it applies to containers created after the change. Existing containers keep their old settings until they are recreated (docker compose up -d --force-recreate, for example). In Compose you can set it per service, which is easier to review in git:

services:
  app:
    logging:
      driver: json-file
      options:
        max-size: "10m"
        max-file: "3"

Images and build cache are the other Docker growth. docker system df shows them, and docker image prune removes dangling images. Be deliberate with docker system prune -a: it removes every image not used by a running container, which can mean re-pulling on the next deploy. Volumes are data. Do not prune them casually.

The rest of the list

• Old releases. If you deploy into timestamped release folders, prune them. Keep five or so; a deploy script should do this, and never remove the one current points at.
• Package cache and old kernels. apt-get clean and apt-get autoremove --purge handle both. Unattended-upgrades can be told to remove unused dependencies (Unattended-Upgrade::Remove-Unused-Dependencies "true").
• Backup tooling. restic keeps a local cache in ~/.cache/restic of the user running it. Dump directories in a staging area can also keep stale files.
• Core dumps. /var/lib/systemd/coredump can hold large files after crashes. coredumpctl list shows them.
• Postgres WAL. After a failed archive command, WAL files pile up in pg_wal under /var/lib/postgresql. Fix the archive command; do not delete WAL files by hand.
• Uploads and temp files. /tmp and your app's upload directory, if the app never cleans up.

Deleted, but the disk is still full

A classic: you delete a 10 GB log, and df does not change. A process still has the file open, so the kernel keeps the data until the process closes it. Find these:

sudo lsof +L1 | grep -i deleted

The number of links is zero, the file is deleted, and the command column names the holder. Restarting that service (systemctl restart app) releases the space. This is why truncating (truncate -s 0) is safer than rm for a file an app is writing: the space is returned at once and the app's file handle stays valid.

What a full disk looks like from the app

Your app does not get a friendly message. SQLite returns database or disk is full; in our test on a 50 MB filesystem filled past 90%, a write failed with Error: stepping, database or disk is full (13). Postgres can panic and stop if it cannot write WAL. Logging fails first, so you get fewer clues exactly when you need more. This is why you alert at 80%, not at 100%. Monitor one VPS without a stack has a short script for the check, and monitoring backups explains why a backup job that fills the disk is its own failure mode.

Rule of thumb: a vacuum is a one-off, a cap is the fix. Alert at 80%, not 100%.

Prevention checklist

1. Cap journald explicitly (SystemMaxUse, SystemKeepFree).
2. Have the app log to stdout under systemd, not to a file. If it must write files, add a logrotate rule with maxsize and a rotate count.
3. Set Docker log limits globally and in Compose, then recreate old containers.
4. Alert on disk usage at 80% from cron. Look at the trend weekly: df -h / and journalctl --disk-usage.
5. Prune release folders and the package cache during deploys.
6. Know the one-liner (du -xh / --max-depth=2 | sort -h | tail) before you need it.
7. Size the disk for restic's cache, dumps and the next three months of data, not just today's.

Mistakes we see

Deleting a log with rm while the app runs. The space does not come back, and you have lost the file. Truncate, or rotate.

Setting max-size in Docker and expecting old containers to change. They keep the settings they were created with.

Comparing df with du and thinking one is wrong. They measure different things: df counts blocks the filesystem has allocated (including deleted-but-open files and reserved blocks), du walks files. A gap points to deleted-but-open files or another mount hiding files under a mount point.

Running journalctl --vacuum-time and calling it fixed. A vacuum is a one-off. The cap in journald.conf.d is the fix.

Deleting Postgres WAL by hand to free space. This can corrupt the cluster. Fix the cause (usually a failing archive command) and let Postgres recycle its own files.

Alerting at 95%. By then the next deploy has already failed. Pick 80%, and act on it.

FAQ

How much space should journald be allowed to use?

For a small VPS, 200 MB to 1 GB is plenty for one app. The deciding factor is how far back you need to look when something goes wrong. If you ship important logs elsewhere, keep less locally. Set SystemMaxUse and SystemKeepFree so that logs can never be the reason the disk fills.

Is it safe to run journalctl --vacuum-size on a production box?

Yes. It removes the oldest archived journal files to get under the size you give it. You lose old log history and nothing else. If you need that history, copy it out first.

Why does df show the disk full but du shows much less?

Usually a deleted file is still held open by a process (lsof +L1 shows it), or a directory was mounted over files that exist underneath. Reserved blocks on ext4 (5% by default for root) also account for a small gap.

Do I still need logrotate if I use systemd?

Only for apps that write their own log files. Anything logging to stdout under a systemd unit goes into the journal, and journald's size cap handles retention. Many setups need no logrotate rules at all.

Should I delete Docker images to free space?

Dangling images and old versions, yes, with docker image prune. Be careful with system prune -a and never prune volumes unless you know what they hold. It is the container logs that usually grow without bound, not the images.

The VPS Handbook has a chapter on log rotation and journald limits with the tested templates, the check-disk.sh script, and a failure-playbook row for a full disk; see the handbook if you want them ready to drop in.

journald