Repository navigation
Shared Codex app-server inherits expendable-agent cgroup and OOM score #26171
Description
Behavior
On Linux, one memory-spiking T3 agent can kill the shared Codex app-server and interrupt every Codex-backed session. Observed in installed T3 0.1.2: the shared server uses the same capped agent ancestry and oom_score_adj=800 as disposable agents. The T3 serve process is already separate at score 200.
Evidence
Read-only observation on 2026-10-27; native-launch excerpts recovered from the installed executable (not an upstream source checkout):
- The Codex app-server spawn path calls AgentScope.wrap with name="codex" and shared=true.
- AgentScope.wrap computes
continuing = shared === true && scope.supportsOomPolicy, using it for OOMPolicy=continue and thread tracking. Placement still selectsslice: survivesRestart === true ? AGENT_SLICE : ownedSlice; shared=true does not select protected ancestry. - The common launch shell contains
RAISE_OOM_SCORE_LINE = '{ echo 800 >/proc/self/oom_score_adj; } 2>/dev/null'. SCOPED_LAUNCH_SCRIPT executes this before execing systemd-run. A later lower-score user-unit drop-in cannot be assumed to undo that raise without privilege. - Shared Codex PID 1504569:
/user.slice/user-1002.slice/user@1002.service/app.slice/app-t3code.slice/app-t3code-agents.slice/app-t3code-agents-a052a83e914f.slice/t3code-codex-cc15d292.scope, score 800. The locally configured agent parent has MemoryHigh=8 GiB / MemoryMax=10 GiB; the shared scope has no separate high/max. - T3 serve PID 3714725: separate
app.slice/t3code.service, score 200. - The cgroup-v2 mount options are
rw,nosuid,nodev,noexec,relatime, without memory_recursiveprot; user@1002.service and app.slice report MemoryLow=0/MemoryMin=0. A nonzero leaf MemoryLow alone would not establish effective protection across these ancestors. - Historical retained host logs attribute shared Codex deaths at score 800 to container-ancestor memory-cgroup OOM. No recent OOM is required to demonstrate the unsafe placement. No forced-spike experiment was run for this draft.
Expected behavior
The shared provider server is infrastructure, not an expendable per-thread agent. Agent memory pressure should remain confined to expendable ancestry. Shared Codex and T3 should stay alive and serving without PID changes when a bounded disposable agent is OOM-killed.
Minimal suggested change
At T3's native scope-creation owner:
- Route shared=true launches into a dedicated shared-server slice outside AGENT_SLICE and ownedSlice, including their capped ancestors. Preserve stdio, signals and the existing single-server lifecycle.
- Select the OOM policy before raising any score: shared launches retain the lower service score (or another supported low score); only expendable launches execute the score-800 shell line. Do not rely on lowering an already-raised score from an unprivileged user manager.
- If declaring MemoryLow/MemoryMin, establish or verify effective ancestor protection; do not assume memory_recursiveprot or advertise an ineffective leaf reservation.
- Add deterministic launch-contract tests for shared/nonshared ancestry, score policy and unchanged lifecycle behavior. Validate operational survival separately with an explicitly scheduled, bounded expendable-agent spike and PID/start-time/readiness readback.
This is a proposed native fix, not a request for an external privileged launcher, runtime binary patch, additional RAM/swap or an uncoordinated restart.
Related
- [Bug]: Codex sub-agents keep failing after the shared App Server is killed by SIGKILL #16502: recovery after the shared Codex app-server is killed (this issue is about why it is killable in the first place).
- fix(server): agents run in their own systemd scopes so an OOM kill spares the server #17662: moved agents into their own scopes so an OOM kill spares the server; the shared Codex app-server still lands in that expendable agent ancestry.
Version: T3 Code 0.1.2 (Linux, systemd user manager, cgroup v2).
Activity
mirkojahn commentedon Oct 28, 2026
Another occurrence, on stable 0.1.3 (headless
t3code.service, Ubuntu 24.04, kernel 6.8, KVM guest with 7.7 GiB RAM, no swap, no systemd-oomd). Here the memory pressure came from Docker containers, not from an agent scope, which supports moving the shared app-server out of the score-800 ancestry.Timeline (UTC, 2026-10-27/28)
docker exec(Playwright e2e, Stryker mutation testing and a full verify run, in parallel). Container processes live undersystem.slice/docker-*.scope, so the agents slice limits (MemoryHigh3.8 GiB,MemoryMax5.8 GiB) never applied to them.constraint=CONSTRAINT_NONE ... global_oom). The kernel killed the sharedcodex app-server(anon-rss:80204kB,oom_score_adj:800).CodexAppServerTransportError: 'read-process-exit-status' ... SIGKILL, the [Bug]: Codex sub-agents keep failing after the shared App Server is killed by SIGKILL #16502 chain that fix(server): Codex works again after its app-server is killed #26188 fixed after 0.1.3. I hard-reset the VM to recover.Same host, 2026-10-26, global OOM: within 84 seconds the kernel first killed
codex(94 MB) and then threedockerCLI clients (about 10 MB each), all at score 800. Only then did it reach the process that was actually using the memory: a 2.39 GB process inside a Docker container, at score 0.Why this matters for this issue: at score 800, the small shared app-server is the first victim of any host-wide OOM, including pressure from processes outside T3's cgroups. Per-thread scopes (#26253) shrink the blast radius. But each kill still lands on the agent rather than on the process using the memory, and it frees almost nothing, so the kernel keeps killing.
benjamin-di commentedon Oct 29, 2026
Another occurrence, on stable 0.1.2. T3 runs headless inside a Docker container that has a 14 GiB memory cgroup limit. There is no systemd user manager, so no agent scopes or slices are involved. The score-800 launch line still applies to every child, including the shared Codex app-server.
memory.eventsshowingmax 485563,oom 26,oom_kill 5.ensureThreadwithCodexAppServerTransportError: 'read-process-exit-status' ... SIGKILL, the [Bug]: Codex sub-agents keep failing after the shared App Server is killed by SIGKILL #16502 chain. Only a T3 server restart recovered it. Disabling and re-enabling the provider did not.dmesgwas not readable from inside the container. Attributing the kill to the cgroup OOM killer rests on theoom_killcounter and the timing, not on a kernel log line.So the kill-order problem in this issue applies outside systemd scopes too: whenever T3 lives inside any memory-capped cgroup (a container), the score-800 raise makes the shared server one of the first OOM victims. Raising the container limit only made it less likely. Running the shared app-server at the lower service score would have avoided it.
benjamin-di commentedon Oct 31, 2026
Source review against stable v0.1.6 confirms that the non-systemd launch path still needs coverage in this issue:
CodexAdapterV2still passesshared: truewhen launching the app-server.AgentScope.tsstill puts the score-800 raise in both launch scripts.agentScopeCommanduses that script even whenscope === undefined.The suggested regression should therefore include the fallback with no systemd scope: a shared launch preserves its inherited OOM score, while a disposable launch raises it. Check PID/stdio/argument preservation too. A change confined to systemd slice placement would leave this path unchanged. Preserving a lower score reduces preference as an OOM victim; it cannot guarantee survival under memory exhaustion. That corrects the stronger “would have avoided it” wording in my earlier comment.
Recovery is separate: merged #26188 is included in v0.1.6 and now forwards transport termination into the adapter event stream so the session manager can retire it. The repeated dead-session failure observed on v0.1.2 is not evidence of a regression in that fix. This follow-up is source analysis; no new OOM injection or v0.1.6 runtime survival test was run.