Go in production

GOMEMLIMIT in an 80 MB container, with the numbers

2026-11-09 · 10 min read

A Go program that keeps 40 MiB of live data should fit comfortably in an 80 MiB container. It does not. Run it with no memory setting and the kernel's OOM killer ends it, because the Go runtime does not know where your container's wall is. It sees a machine with plenty of memory and lets the heap grow to roughly twice the live data before it collects. On a small box that growth is the difference between running and exit code 137.

GOMEMLIMIT is the setting that tells the runtime where to aim. Advice about it usually stops at "set it a bit below your container limit". That is correct, and it leaves the interesting questions open: what does it cost, how soft is "soft", and what should the number be. We built a small program and measured. This post gives the numbers, the program so you can repeat the experiment on your own hardware, and the rules we took from it.

Who needs this

A small VPS shared by several processes. A 1 GB machine running your Go API, Postgres, Caddy and a backup job does not have 1 GB for Go. Memory pressure from one process affects all of them, and the OOM killer picks a victim that may not be the one you wanted.

A container with a hard memory limit. Docker Compose mem_limit, docker run --memory, or a PaaS plan with 256 MB. The cgroup limit is enforced by the kernel with no warning and no shutdown hook. A process killed this way runs none of the graceful shutdown code you wrote.

A service that builds large buffers. PDF or image generation, CSV exports, big JSON responses, a batch that loads a table into memory. Each request allocates a lot and drops it, which is exactly the pattern where the default growth rule overshoots a small limit.

A systemd unit with MemoryMax=. Same enforcement as a container, same kill, and the limit is easy to set and forget. The runtime needs to be told about it.

What the three knobs do

By default, with GOGC=100, Go lets the heap reach about twice the live data before collecting. In a container with a limit, that growth can cross the kernel's line even though the live data fits easily. GOMEMLIMIT adds a second trigger: as the total memory the runtime manages approaches the limit, it collects more often no matter what GOGC says.

You set it as an environment variable with an optional unit suffix, GOMEMLIMIT=56MiB, or in code with debug.SetMemoryLimit. The accepted suffixes are B, KiB, MiB, GiB and TiB. A bare number means bytes.

The experiment

The program keeps 40 MiB of live data, then allocates and drops 3,000 buffers of 1 MiB as fast as it can. That is a crude stand-in for a service building large response buffers. Here is a version that compiles on Go 1.22 and later:

package main

import (
	"fmt"
	"os"
	"runtime"
	"runtime/debug"
	"strings"
	"time"
)

var live [][]byte // keeps 40 MiB reachable

func touch(b []byte) {
	for i := 0; i < len(b); i += 4096 {
		b[i] = 1 // touch each page so it really counts
	}
}

func main() {
	for i := 0; i < 40; i++ {
		b := make([]byte, 1<<20)
		touch(b)
		live = append(live, b)
	}

	start := time.Now()
	for i := 0; i < 3000; i++ {
		b := make([]byte, 1<<20) // garbage: dropped on the next iteration
		touch(b)
	}

	var m runtime.MemStats
	runtime.ReadMemStats(&m)
	peak, _ := os.ReadFile("/sys/fs/cgroup/memory.peak")
	fmt.Printf("elapsed_ms=%d gc_cycles=%d heap_sys_mib=%d memlimit=%d cgroup_peak=%s\n",
		time.Since(start).Milliseconds(), m.NumGC, m.HeapSys>>20,
		debug.SetMemoryLimit(-1), strings.TrimSpace(string(peak)))
	runtime.KeepAlive(live)
}

Build a Linux binary and run it in containers with swap disabled, so the limit is real:

GOOS=linux CGO_ENABLED=0 go build -o memdemo .

# no GOMEMLIMIT
docker run --name md --memory=80m --memory-swap=80m \
  -v "$PWD":/app:ro debian:12-slim /app/memdemo
docker inspect -f 'exit={{.State.ExitCode}} oom={{.State.OOMKilled}}' md
docker rm md

# with GOMEMLIMIT
docker run --rm --memory=80m --memory-swap=80m -e GOMEMLIMIT=56MiB \
  -v "$PWD":/app:ro debian:12-slim /app/memdemo

memory.peak is a cgroup v2 file on recent kernels (Debian 12 has it). If it is missing the program prints an empty value, and everything else still works. --memory-swap equal to --memory means no swap, which turns "slow" into "dead" and makes the result readable.

What happened

These are the numbers from the Go handbook's recorded runs: one Docker Desktop VM with 8 CPUs, three runs per row.

80 MiB container, no GOMEMLIMIT:      killed by the kernel, exit 137, OOMKilled=true   (3 of 3 runs)
80 MiB container, GOMEMLIMIT=56MiB:   survived                                         (3 of 3 runs)
     elapsed 120 to 219 ms, GC cycles 309 to 328, cgroup memory peak 64.6 to 66.6 MiB
512 MiB container, no GOMEMLIMIT:     survived; elapsed 67 to 72 ms, GC cycles 68 to 75,
                                      Go heap reserved 107 to 131 MiB, cgroup peak 112 to 137 MiB
512 MiB container, GOMEMLIMIT=56MiB:  survived; elapsed 101 to 216 ms, GC cycles 280 to 328,
                                      Go heap reserved 59, 59 and 147 MiB, cgroup peak 68.5, 64.3 and 156 MiB

Without a limit, the process was killed in all three runs. With GOMEMLIMIT=56MiB it survived all three, and the container's peak stayed between 64.6 and 66.6 MiB, under the 80 MiB wall. In the roomy 512 MiB container with no limit, the heap grew to 107 to 131 MiB for a workload whose live data is 40 MiB. That is the default growth rule at work: nothing was wrong, the runtime simply had room and used it.

What it costs

Look at the GC cycles. The same workload ran 68 to 75 collections with no limit and 280 to 328 with the limit, roughly four times as many. In this synthetic program, which does nothing except allocate, the elapsed time went from 67 to 72 ms up to 101 to 216 ms, about 1.4 to 3 times as long.

A real service spends most of its time waiting on a database or a network, so the effect on yours will be smaller. The direction is the same, though: a memory limit trades CPU for memory. You can see it in the runtime's own trace without any special tools:

GODEBUG=gctrace=1 GOMEMLIMIT=56MiB ./memdemo 2>&1 | tail -5

Each line is one collection with the heap size before and after. A heap that stays near the limit and a collection rate that climbs is the signature of a limit doing its job, and of a limit that may be too tight.

It is a soft limit

The runtime documentation says it, and one run showed it. In the 512 MiB container, one of three runs with GOMEMLIMIT=56MiB reserved 147 MiB of heap and peaked at 156 MiB of cgroup memory, nearly three times the limit.

The Go runtime caps the share of CPU the collector may use. The garbage collector guide puts it at 50 percent. Under a very high allocation rate the runtime prefers to let the heap grow past the limit rather than starve your program of CPU. We took that explanation from the documentation and did not instrument the run to confirm it was the cause. The practical reading is what matters: GOMEMLIMIT makes overshoot less likely. MemoryMax or --memory is what actually protects the machine.

The other side of "soft" is a live heap larger than the limit. The runtime cannot free data you still reference, so it collects constantly and gets nowhere. We measured it with a 70 MiB live heap and GOMEMLIMIT=56MiB in the 512 MiB container, 300 rounds: 98 GC cycles in 76 ms against 9 cycles in 19 ms without the limit, and the heap still reached 111 MiB reserved. A limit below what you need buys CPU burn and no memory.

Reading the cgroup peak honestly

The container's cgroup peak (64.6 to 66.6 MiB with a 56 MiB limit) is above GOMEMLIMIT. The limit covers memory the Go runtime manages: heap, stacks and its own structures. It does not cover the executable's own pages, thread stacks you did not ask for, or memory allocated outside Go.

That last category deserves attention if you use SQLite through a pure-Go driver, or any cgo library. In the handbook's reference service, after a load run the resident memory was 23 MiB, while the Go runtime accounted for 17.8 MiB of memory obtained from the OS and the live heap was 2.1 MiB. One likely piece of that gap is that the SQLite library inside the driver allocates outside the Go heap, where GOMEMLIMIT cannot see it. That is a hypothesis, and it is one more reason to keep a margin between the limit you set and the wall.

What number to choose

Our rule of thumb is a soft limit at about 80 percent of the hard one:

# /etc/systemd/system/myapp.service.d/limits.conf
[Service]
MemoryMax=512M
Environment=GOMEMLIMIT=400MiB

Rule of thumb: GOMEMLIMIT at about 80 percent of the hard limit, and well above your measured live heap.

That is the pair in the handbook's unit file, and the drop-in template for a smaller box uses 256M and 200MiB. Eighty percent is a rule of thumb, not a measured optimum, and the container experiment above shows the margin can be small when you allocate fast. For the Docker case the equivalent is:

services:
  app:
    image: myapp
    mem_limit: 256m
    environment:
      GOMEMLIMIT: 200MiB

MemoryHigh= is a third option you will see recommended. It does not kill; it throttles the cgroup when it passes the threshold, which pushes the kernel to reclaim memory. It can sit between the soft and the hard limit, but it hides the problem as slowness, so we prefer to see the kill and the metrics rather than a mystery slowdown.

Then watch it. Export go_heap_alloc_bytes, go_sys_bytes and the GC cycle count from your metrics endpoint. A GC rate that keeps rising while the heap stays flat is what a limit that is too tight looks like. See monitoring one VPS without a stack for cheap ways to look at those numbers.

Does your service need any of this?

Often not. The handbook's reference service, a small HTTP API with a SQLite-backed job queue, sat at 13 MiB resident when idle and 23 MiB after a load run (peak 25 MiB), with the Go heap around 2 MiB. Setting GOMEMLIMIT=64MiB changed nothing measurable, because the heap never came close to the limit. Throughput differences between the two configurations were machine noise.

The point of setting it anyway is insurance. You set MemoryMax as a hard ceiling because the kernel needs one. You set GOMEMLIMIT underneath it so that on the day a request builds a large buffer, the collector reacts before the kernel does.

Mistakes we see

GOMEMLIMIT equal to or above the container limit. Then it never acts before the kernel does. It must be below.

Setting it below the live heap. The service spends its time in the collector. Measure the live heap first, under load, then pick a limit comfortably above it.

Expecting it to fix a leak. A limit does nothing about memory that is still referenced. The service gets slower while it leaks and is killed anyway. Use pprof (on a separate loopback port, not a public one) to find what holds memory.

Forgetting non-Go memory. cgo, SQLite, and mmap are invisible to the limit. Leave real headroom.

Testing with swap on. With swap, a process over its memory is slow instead of dead. That is friendlier in production and misleading in a test, so disable swap when you want to see the limit work.

Setting GOGC=off with GOMEMLIMIT. It is a documented pattern for programs that want to use all of a fixed budget, and it makes the collector's behaviour entirely limit-driven. On a shared small box that is usually not what you want.

FAQ

Does Go read the container memory limit automatically?

No. The runtime does not derive GOMEMLIMIT from the cgroup's memory.max. Without the setting, our no-limit run in an 80 MiB container was killed three times out of three. You have to set it yourself, in the environment or with debug.SetMemoryLimit.

What is a good GOMEMLIMIT for a 256 MB container?

Around 200 MiB as a starting point, which is the 80 percent rule. If the service uses cgo or SQLite heavily, go lower, to 160 or 180 MiB, and then measure the container's real peak under load. The right number is the one where the peak stays comfortably below the wall and the GC rate does not climb.

Will GOMEMLIMIT make my service slower?

It can, if the limit is close to what the service uses. In our synthetic worst case, a program that only allocates, GC cycles quadrupled and elapsed time rose by 1.4 to 3 times. In a service that is mostly waiting, the effect is much smaller. If the limit is far above your heap, as in the reference service, you will see no change.

Is GOMEMLIMIT a hard cap?

No. The runtime may exceed it, as our 147 MiB heap against a 56 MiB limit showed, because it limits how much CPU the collector can take. Use the kernel's limit (MemoryMax or --memory) as the real cap.

Should I set it in code or in the environment?

The environment, in the systemd unit or Compose file, next to the memory limit it depends on. Then the two numbers are in one file and nobody changes one without seeing the other. Use debug.SetMemoryLimit only if you compute the value from something at runtime.

The Go on One VPS handbook has this as chapter 14, with the measurement harness, the container commands and the complete results, next to the systemd unit that sets both limits: Go on One VPS: Production Handbook.

gosystemd