Skip to content

fix(server): recover Codex child models after metadata lookup fails - #22830

Draft
juliusmarminge wants to merge 8 commits into
mainfrom
fix/issue-15250
Draft

juliusmarminge wants to merge 8 commits into
mainfrom
fix/issue-15250

Conversation

@juliusmarminge

@juliusmarminge juliusmarminge commented Oct 21, 2026 •

Copy link
Copy Markdown
Member

Problem

Fixes #15250.

Change

A native Codex child's first metadata lookup can race an empty rollout. If both read and fallback resume fail, its Lineage model stays “Not reported” even after metadata becomes available. Retry the lookup up to six times with bounded backoff, stopping when the child's actual selection is known or its lifecycle no longer permits recovery. Newer settings/reroute metadata wins over an older lookup; the parent model is never substituted.

Scope and approval

Fixes #15250
GitCafe: https://git.cafe/pingdotgg/t3code/issues/15250
Triaged bug via the automated issue flow.

Evidence index includes correlated native IDs, wire/projection timestamps, exact same-parent task_status errors, card/UI text, completion records, rebuild instructions, temporary fault script/patch, rendering environment and clean restoration proof. Server-only exception: scoped tests and real runtime transcripts replace a pixel matrix. Runtime injection is a published temporary artifact, not committed product code or a committed database seed; the deterministic replay fixture is committed.

Validation: scoped report. Focused adapter suite: 76 passed (prior auditor's run). Scoped formatting passed; lint exited 0 with two unused-variable warnings; server-only typecheck exited 0 with Effect suggestions. Two independent reviews, Codex and Claude, reported MERGE (lead-supplied reports; not rerun by the evidence assembler). At the final lead check, CI reported 21 successful, 2 pending (Collect PR targets and Prepare PR size config), and 0 failing checks. Final proof audit: SUFFICIENT; its timestamp wording finding is corrected in the evidence index.

Verification

Reproduced against the actual base 1688f96568, then verified at 4dd9adf5c0, using Codex 0.163.0 and a real isolated dev server. Parent Sol/low spawned native Luna/high children. Before, a natural empty-rollout read error and a temporarily rejected successful resume result left the child model null through completion. After, attempt 1 hit natural empty-rollout/list_turns errors; attempt 2's successful wire results were rejected by the temporary fault, which then released; attempt 3 read succeeded at 11:35:43.716 UTC. The persisted selection corrected to Luna/high by 43.778 and remained through completion at 11:38:48.129. Healthy control also retained Luna/high through completion. No settings/reroute notification supplied the correction.

Before: failed metadata lookup After: recovered metadata
Before: native child model Not reported After: recovered native child GPT-6-Luna high

Rendering: T3 preview Chromium/Linux, 1280×800 CSS viewport, light, measured DPR 2, no emulation, default font scaling. The after image shows the running recovered card; completion retention is established by the persisted snapshot and final UI text. Across captures, isolated history/titles/timers changed, displayed diff counts included temporary instrumentation, and an update notification was dismissed. Product changes are limited to the server adapter and its tests; existing client renderers and contracts are unchanged.

P/I race and interruption behavior are covered by deterministic replay. B/V retry bounds and invalid-result variants are code-inspected with executable coverage limits disclosed in the index. Native task_status still accepts app-owned tasks only; the same-parent native probe returns task_not_found. Unverified: Claude children, mobile card, remote/relay/tunnel and multi-environment behavior. No pixel-diff or completed-card screenshot claim.

Model: GPT-6.1-Sol. Harness: Codex in T3 Code.


Superseded: the evidence index, validation report and Verification section above were captured at 4dd9adf5c0, before the merge from main. The evidence below replaces them. It was recaptured at aa57d7d3ba, and the implementation is unchanged at head f3117126dc, which adds only the evidence scripts and a knip entry for them.

Evidence

Both sides ran a real t3 start server with an isolated --base-dir, the same driver, and a STUBBED Codex app-server (real Codex was rate-limited). Before is the merge-base 8d3b061121; after is aa57d7d3ba. The stub replaces Codex through providerInstances.codex.config.binaryPath, and its thread/read and thread/resume replies follow a fixture. The client value is the subagent model / modelSelection in orchestration_v2_projection_subagents.payload_json, read read-only. The client shows "Not reported" when it is null.

In the table, R is thread/read and S is thread/resume. Times are seconds after the child registered.

Row Stub fixture Before 8d3b061121 After aa57d7d3ba Client value, before → after
R1 Lookup succeeds first time 1R 0S 1R 0S gpt-6-luna → gpt-6-luna
R2a Attempt 0: R and S fail, later R succeeds 1R 1S, then nothing 2R 1S; retry R at +1.04 succeeds null (Not reported) → gpt-6-luna
R3 R and S always fail 1R 1S 6R 6S at 0 / 1.04 / 3.04 / 7.05 / 15.05 / 31.05, then quiet for 68 s, including after the parent turn completed null → null (stable, no retry storm)
R3b R always hangs 1R, no S 6R, no S (0 → 56 s) null → null
R4a Child interrupted after attempt 0 1R 1S 1R 1S; no retry at +1 s null (interrupted) at both
R5 model/rerouted arrives mid-retry 1R 1S 1R 1S; no further lookups gpt-5.6-sol, one update, at both
R6 R returns an empty model, S succeeds 1R 1S 1R 1S gpt-6-luna at both

Environment: fs-24, Linux 6.1.102 x86_64, node v26.12.1. Ports 127.0.0.1:43832 (before) and 43833 (after). Stubbed Codex app-server, isolated homes under /tmp.

What else changed between before and after: git diff --stat 8d3b061121 f3117126dc lists CodexAdapterV2.ts, CodexAdapterV2.test.ts, the three evidence files and one line in knip.jsonc. Nothing else.

Verification log
  • Audit pass 1 (Claude Opus 5.5): derived the matrix R1–R7 from the diff and named how a running server reaches each row. It raised one design question, R4f: retries stop once a child ends interrupted, failed or cancelled. A child that ends before any lookup succeeds therefore never shows its model. Audit pass 2 calls this unchanged from before. Audit pass 1 reports one race where it differs: when the child is interrupted while a successful lookup is in flight, before applies the model and after drops it. The PR's interruption test asserts the after behaviour on purpose. A maintainer should confirm it's intended.
  • Audit pass 2 (Claude Opus 5.5): SUFFICIENT. It checked every row against the raw results.json and checked both diff claims. It rebuilt R2a and R3 from the committed run.mjs on fresh checkouts at 8d3b061121 and 89d72c1ff2, and both matched within about 20 ms: R2a went from 1R 1S null to 2R 1S gpt-6-luna, and R3 went from 1R 1S to 6R 6S. Caveat: the raw logs from the first pass record checkout paths, not SHAs. The rebuild is what ties those results to SHAs.
  • Tests: vp test run apps/server/src/orchestration-v2/Adapters/CodexAdapterV2.test.ts passed 78/78 at aa57d7d3ba. vp lint and vp fmt --check on the changed files show no errors, only two existing unused-variable warnings in CodexAdapterV2.ts. knip --include files,dependencies and the apps/server exports pass both exit 0 at f3117126dc.
  • CI: at 89d72c1ff2, 25 checks passed and 2 failed: Lint and Check. The local knip run flagged the two evidence scripts as unused files; f3117126dc adds them as knip entries and knip now passes locally. CI is pending on f3117126dc.
  • Not verified:
    • R2b, R2c, R4b–R4e, R6b and R7: not required, and capture ran out of budget. Each is a stub fixture away.
    • R4f and R5b: timing-sensitive races, not attempted on a running server. R5b is covered by an existing unit test.
    • No web or mobile screenshot: no browser use was authorised. The client value comes from the projection row the clients render.
    • Not tested against real Codex, which was rate-limited.
    • Remote, relay and tunnel connections, and multiple environments, were not tested.
  • Model: Claude Opus 5.5 (lead, auditors, capturer). Harness: Claude Code in T3 Code.

@github-actions github-actions Bot added vouch:trusted PR author is trusted by repo permissions or the VOUCHED list. size:L 100-499 changed lines (additions + deletions). labels Oct 21, 2026
@github-actions github-actions Bot added size:M 30-99 changed lines (additions + deletions). and removed size:L 100-499 changed lines (additions + deletions). labels Oct 21, 2026
@github-actions

github-actions Bot commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

Thread transfer impact

✅ Thread transfer remains within every enforced ceiling.

Provider Metric Main baseline This PR Impact PR ceiling
Codex Total thread wire 5.0 KiB 5.0 KiB 0 B (0.0%) 6.8 KiB ✅
Codex Thread snapshot wire 3.8 KiB 3.8 KiB 0 B (0.0%) 4.9 KiB ✅
Codex Live turn WebSocket wire 1.2 KiB 1.2 KiB 0 B (0.0%) 2.0 KiB ✅
Codex Live turn WebSocket decoded 21.6 KiB 21.6 KiB 0 B (0.0%) 24.4 KiB ✅
Codex Live turn messages 2 2 0 (0.0%) 8 ✅
Claude Total thread wire 5.0 KiB 5.0 KiB 0 B (0.0%) 6.8 KiB ✅
Claude Thread snapshot wire 3.8 KiB 3.8 KiB 0 B (0.0%) 4.9 KiB ✅
Claude Live turn WebSocket wire 1.2 KiB 1.2 KiB 0 B (0.0%) 2.0 KiB ✅
Claude Live turn WebSocket decoded 21.9 KiB 21.9 KiB 0 B (0.0%) 24.4 KiB ✅
Claude Live turn messages 2 2 0 (0.0%) 8 ✅

Baseline: f6503f0 · PR result: d3d0823 · Source CI: failure

Scenario and decoded snapshot size

10 historical turns, 5 command tools per turn, 878.9 KiB retained MCP result per historical turn, and a 1.05 MiB retained result in the measured turn.

  • Codex decoded thread snapshot: 108.5 KiB
  • Claude decoded thread snapshot: 108.8 KiB

Updated in place by a trusted workflow. PR artifacts are strictly validated and never executed.

@juliusmarminge juliusmarminge added via-triage Filed through npx t3 triage via-triage:gates-failing Triage fix: a land gate is failing (see PR body) labels Oct 21, 2026
@juliusmarminge juliusmarminge added bugfix Fix PR opened by an issue lead; enters the PR loop pr:gate-failed A land gate failed and removed via-triage Filed through npx t3 triage via-triage:gates-failing Triage fix: a land gate is failing (see PR body) labels Oct 22, 2026
@juliusmarminge juliusmarminge added pr:in-progress A lead is working on this PR and removed pr:gate-failed A land gate failed labels Oct 22, 2026
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions github-actions Bot added size:XL 500-999 changed lines (additions + deletions). and removed size:M 30-99 changed lines (additions + deletions). labels Oct 22, 2026
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@juliusmarminge juliusmarminge added pr:ready Every gate passed; waits for Julius's land label and removed pr:in-progress A lead is working on this PR labels Oct 22, 2026
juliusmarminge and others added 2 commits October 29, 2026 22:47
# Conflicts:
#	apps/server/src/orchestration-v2/Adapters/CodexAdapterV2.test.ts
#	apps/server/src/orchestration-v2/Adapters/CodexAdapterV2.ts
#	knip.jsonc
@juliusmarminge juliusmarminge added pr:needs-julius Needs Julius: route gate, unverifiable surface, or blocked and removed pr:ready Every gate passed; waits for Julius's land label labels Nov 9, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bugfix Fix PR opened by an issue lead; enters the PR loop pr:needs-julius Needs Julius: route gate, unverifiable surface, or blocked size:XL 500-999 changed lines (additions + deletions). vouch:trusted PR author is trusted by repo permissions or the VOUCHED list.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: V2 Codex subagent model stays "Not reported" after metadata lookup fails

1 participant