v1.7.1: openaudio process leaks anonymous memory to ~20.8 GB and is OOM-killed (~every 2 days) #415

Image: openaudio/go-openaudio:v1.7.1 (@sha256:ccb4268aceed5dbb73899356672a27d55d790d4e308fbca3f4f23fe511904588)
Role: validator / content node (mediorum + core), Ubuntu 24.04, Docker, cgroup v2
Host: 24 GB RAM, no container memory limit

Summary

On a fleet of 20 validator nodes running v1.7.1, the openaudio process's anonymous memory (heap) grows monotonically with no plateau until the host runs out of memory and the kernel OOM-kills it. On 24 GB hosts this happens at ~20.3–20.9 GB anon-RSS, roughly every ~2 days per node from a fresh start. The container's restart: unless-stopped policy (or our autoheal sidecar) then restarts it, memory resets to ~1 GB, and the climb begins again.

It is the anonymous working set that grows, not file cache (file cache is reclaimable and does not drive the OOM).

Evidence

Kernel OOM kills (representative, all constraint=CONSTRAINT_NONE ... global_oom, i.e. host-global, not a cgroup limit):

Out of memory: Killed process 1341 (openaudio) total-vm:22375764kB anon-rss:20902940kB   (node A)
Out of memory: Killed process 1346 (openaudio) total-vm:22271792kB anon-rss:20812784kB   (node B)
Out of memory: Killed process 1335 (openaudio) total-vm:22022560kB anon-rss:20342668kB   (node C)

cgroup v2 memory.stat on a freshly-restarted node, ~8 h after restart:

anon  1046999040      (~1.0 GB  — the leaking working set, still climbing)
file  19987689472     (~20 GB   — reclaimable page cache, NOT the cause)

Anon growth is roughly linear at ~0.25 GB/h; a fresh node reaches the ~20.8 GB kill point in ~2 days. Steady-state healthy nodes show anon at 1–9 GB and rising, never plateauing — there is no workload spike that justifies a 20 GB anonymous heap on an otherwise idle validator.

Reproducible fleet-wide: all 20 identically-configured nodes exhibit the same climb-and-OOM pattern; a synchronized restart makes them all reach the ceiling within the same ~24 h window ~2 days later.

What we've ruled out

• Not file cache — that is separate and reclaimable (~20 GB); the OOM is driven by the unreclaimable anon heap.
• Not a cgroup memory limit — there is no container memory limit; these are host-global OOMs (docker inspect reports OOMKilled=false precisely because the kernel kill is global, not cgroup-scoped).
• Not the host/environment — 20 independent hosts across two subnets show the identical curve.

Questions

1. Is this a known memory leak in v1.7.1? The v1.8.0 / v1.8.1 changelogs don't mention a memory fix — is the leak still present on main / v1.8.x?
2. Is there a bounded cache / mempool / index / goroutine growth path that a config or env var can cap, so anon RSS plateaus instead of growing without bound?
3. Is a pprof heap profile from a high-RSS node useful to you? We can capture and attach inuse_space / alloc_space profiles if the binary exposes /debug/pprof (please confirm the port/flag).

Our current mitigation (so this isn't blocking us)

We've deployed a staggered per-node systemd timer that restarts my-node when its anon RSS crosses a threshold below the OOM point, in a jittered window — this pre-empts the OOM and de-synchronizes the fleet. It's a workaround; a bounded-memory fix upstream is the real solution. Happy to provide any additional diagnostics.