Review sidecar probes and replays spend the per-key free-tier rate budget the review itself needs

What the evidence now shows

With the catalog fixed (#1939, every run selects 4+4+4 across the three free accounts) and the per-attempt trace landing (#1943/#1944/#1945), the failures that remain on the free pool are not routing failures. Two post-#1939 runs, one hour apart, same shape:

noema-review run 33993637015 (#1687, 21:38Z): preflight ready 1/12 — 429 ×6 (all four OpenRouter routes; deepseek-v4-flash and v4-pro on the primary NVIDIA key while the sibling key's flash was ready), 404 ×4 (gemma-3-12b/4b on both NVIDIA keys), 500 ×1. The one ready route was held 542 s and the caller got 502.
strix run 33992904674 (#1930, 23:48–00:12Z, artifact 9979385164): ready 1/12 — 429 ×6 again, one TimeoutError, gemma-3 ×4 404; the sole ready route (nvidia_nim_sub deepseek-v4-pro) answered 429 rate_limit_exceeded on the first real request and on all five Strix replays; llm_usage.requests = 0.

Corrected post-#1939 tally of noema-review runs that reached the verdict step: 1 success / 3 failures (three run-level "successes" in that window were closure-event runs with the job skipped). Reading: at those hours there was no free capacity to interleave, and each such job still held a runner for ~25 minutes inside a 234-deep queue.

Mechanism

Every sidecar boot probes all twelve catalog routes with a real completion (REVIEW_PREFLIGHT_BASE_TOKENS = 16, contextual_orchestrator_review_launcher.py:404), i.e. four requests per account per boot, then the review makes its own requests, and Strix replays each scan step up to five times. All of that draws on the same per-key budgets (NVIDIA NIM ≈ 40 requests/min per key, community-reported; OpenRouter free tier is per-key and per-day). With N sidecars booting in the same minute across the organization — required workflows run in every repository — the probes alone can consume the budget the review then needs, and a 429 at probe time is frequently caused by the other sidecars' probes. #1947 stops discarding such routes at preflight, which restores failover depth, but cannot create capacity: a route that is rate-limited for the hour fails fast for everyone.

What this is not

Not a routing defect (#1939 measured working), not a failover defect (#1938 thread: 429 is retryable=True and fails over; provider_errors.py:82), not the launcher strip (#1476 thread). The four permanent gemma-3 404s are a separate discovery-side waste of a third of the catalog.

Levers — all policy values, hence the owner's

Probe budget per boot. Stop probing once K routes are ready (or probe one route per account first), instead of twelve unconditional completions; that divides probe spend by up to 3–4 at the cost of a smaller preflight report. This is an amendment to ADR-0005's "bounds request count" contract.
Replay budget. Strix's five replays against an already-429 route multiply the burn; a replay policy that stops on a rate-limit answer is a strix-agent/CO decision.
Concurrency of sidecar boots. The organization ceiling (60 jobs) is not a per-key limiter; an admission rule per provider key (or a shared rate-limit ledger across sidecars — the same cross-process state contextual-orchestrator#1045 needs for its breaker) would be.
Paid headroom. OpenRouter credits lift the free tier's daily cap; NVIDIA's per-key RPM does not change with credits.

What to measure before choosing

With #1944's artifact every failed noema-review now ships contextual-orchestrator-preflight.json and the sanitized stderr. Two numbers decide between (1)–(3): the fraction of probe rejections that are 429 (vs 404/500/timeout), and the number of sidecars whose probes fell in the same minute (created_at of in-progress noema/strix/opencode jobs org-wide). If most 429s cluster at boot storms, (1) and (3) are the levers; if the sole ready route also 429s on the real request in quiet minutes, the key is simply saturated and (4) is the only one.

Datum for lever (3), measured while filing (00:20Z): 25 noema-review/strix workflow runs in_progress in this repository alone — each one a sidecar that probed twelve routes at boot and is now making review requests against the same three free accounts. That is the boot-storm denominator; the organization-wide figure is larger since the required workflows run in every repository. The per-key budgets (≈40 requests/min on NVIDIA NIM, per-key/day on OpenRouter free) are shared by all of them.

Correction to the body (host 1): "held for 542 s (= one route's full 2 × 3 × 90 s budget)" is wrong — that arithmetic belongs to the orchestrated _invoke/chat() path. The review request goes through the passthrough path (proxy_completion → proxy_send_once, orchestrator.py:4137, :2408-2412): one attempt per candidate, stream=False. The artifacts confirm attempt=1/1 on every provider attempt. So 542 s was one candidate whose upstream held a non-streamed request open for nine minutes before answering 502. Nothing else in this issue changes — the probe/replay budget mechanism and the four levers stand — but it adds a fifth datum for the owner: the only ceiling on a held request is a response-start or total deadline at the gateway (§8), and the artifacts' per-candidate timestamps now show how often that class occurs.

Measurement that bears on levers (1) and the "permanent 404s" aside — from the sidecar artifacts, pre- and post-#1939, same repository, same discovery data.

Post-#1939 the served NVIDIA set lost every non-deepseek working route. Discovery (contextual-orchestrator-discovery.json in #1938's Strix artifact 9979583987) lists 21 models per NVIDIA key — deepseek ×2, gemma-3 ×2, gemma-4, llama-3.2 11b/90b, muse-glimmer-30b, minimax-m3, mistral ×2, kimi ×2, nemotron ×4, cosmos, riva, gpt-oss-20b, laguna. The 4+4+4 fill takes each key's first four in catalog order, which is alphabetical by model within the tier: deepseek-ai/deepseek-v4-flash, deepseek-ai/deepseek-v4-pro, google/gemma-3-12b-it, google/gemma-3-4b-it. Two of those four are the permanent 404s, so each NVIDIA key contributes two working routes, both deepseek-v4 — the most-contended models on the key.

Pre-#1939 the same order with the 8+4 fill reached google/gemma-4-31b, meta/llama-3.2-11b-vision-instruct, meta/llama-3.2-90b-vision-instruct, meta/muse-glimmer-30b on the primary key, and three of those were ready:

Strix run (artifact)	created	fill	ready / 12	ready routes
`33979406293` (#790)	16:56Z	8+4+0	6	deepseek flash+pro ×2 keys, llama-3.2-11b, muse-glimmer-30b
`33979153466` (#789)	16:51Z	8+4+0	6	deepseek flash+pro, llama-3.2-11b, llama-3.2-90b, muse-glimmer-30b, sub deepseek-pro
`33978435797` (#1382)	16:37Z	8+4+0	5	deepseek-pro ×2 keys, llama-3.2-11b, llama-3.2-90b, muse-glimmer-30b
`33992904674` (#1930)	23:48Z	4+4+4	1	sub deepseek-pro
`33992736699` (#1938)	23:47Z	4+4+4	3	deepseek-pro, sub deepseek-flash, sub deepseek-pro

So the interleave traded three ready NVIDIA routes (llama/muse, which nobody else was hammering) for four OpenRouter routes that 429 at preflight — the account diversity went up, the served diversity went down, and the ready count fell from 5–6 to 1–3 before the evening load is counted.

Honest post-#1939 verdict tally (every noema-review run in .github created after 17:30Z with a PR attached, by the Prepare Noema model verdict step's own conclusion): 0 successes, 22 failures (verdict-step durations 199–4273 s, median ≈1500 s). The two run-level "successes" (#1902, runs 33982955696 / 33994883456) are draft skips — PR is draft; Noema verdict preparation skipped., verdict step 0–1 s — which corrects the "1 success" in my comment on contextual-orchestrator#1045 and this issue's opening tally. Pre-#1939, 10:00–16:25Z, the same count was 7 / 14. That is not proof #1939 caused the drop (load rose in parallel), but the served-set change above is a mechanism that does not need load to explain part of it.

Lever that addresses both this and (1) at once, for the owner's call: fill each account's slots lazily from its full ranked candidate list until K routes are ready, instead of probing a fixed first-four. A permanent 404 then costs one cheap probe and is replaced by the next candidate (llama, muse, nemotron …) rather than occupying a slot for the whole run; probing stops at K ready, which is lever (1)'s bound; and no model name needs hard-coding. Failing that, the smaller change is to rank a tier's candidates by model family round-robin (deepseek, google, meta, minimax, …) before the per-account cap, so no key's four are two dead gemma-3 plus the two most-contended deepseek routes. Both stay inside contextual_orchestrator_review_policy.py / the launcher preflight, i.e. this lane; I am not opening a competing change.

Generated by Claude Code

Correction to my correction (host 1): the first noema-sidecar-evidence artifact (#1661 run 33995553859, 3873 s) shows the review path is _invoke — per silent agent two rounds × three 90 s timeouts (measured 270.3–271.1 s per round, 541 s per agent), not one passthrough attempt; the attempt=1/1 lines were preflight probes. The 542 s in the body is therefore one agent's two rounds, as first written. The probe/replay-budget mechanism and the levers stand; the response-start deadline still matters, but the dominant cost here was 13 silent rounds on two NVIDIA keys repeated across three internal walks, plus two preflight-ready routes never attempted — see the table on #1939.

Lever taken (host 1): #1949, stacked on #1947. The catalog becomes a candidate list (24, per-account cap 8, #1939's tier-then-round-robin order) and _preflight_review_agents probes it in order, stopping at 8 ready routes or 16 probes, whichever comes first. A dead candidate then costs one probe instead of a served slot, and each account's list continues past its gemma-3 404s to the llama-3.2 / muse-glimmer entries that were ready in the pre-#1939 artifacts. No model name is hard-coded; the 404s are found by their probe. Report gains candidate_count, target_ready, probe_budget; probed_count now counts probes actually sent. ADR-0029 records it and amends ADR-0003 §2's "twelve-route" sentence.

Two honest numbers for this issue's ledger. Worst case (nothing ready) is now 16 probes per stage where it was 12 — a third more against keys that are already exhausted; that is the price of finding routes past the dead ones, and the shared rate ledger above it stays this issue's lever. Healthy case stops at about 8–12 probes instead of always 12, so the common hour spends less. Your family round-robin fallback is recorded in the ADR as the next layer if the post-merge census shows same-family contention as the residual.

Discriminator per boot after merge: probed_count vs candidate_count and the served set's ready_count, against 6/6/5 of 12 before #1939 and 1–3 after.

Slot measurement 2026-09-06T04:30Z (lane jan), for the levers list.

Who held the runners. Of 10 in_progress runs in .github, 7 were strix jobs on the pre-bump pin holding runners since 00:10–03:23Z by job started_at — the five oldest 3 h 45 m–4 h 25 m inside "Run Strix (quick)" (#1231, #1271, #1555 and two push/main scans; #1678 and #1872 since 03:16Z / 03:23Z) — plus 2 noema-review verdict steps (#1872 since 03:22Z, #1280 since 03:47Z) and one OpenCode Review Dispatch (#939). There is no job timeout on either review job by design (strix.yml:345-350, noema-review.yml:261-276), so a job that enters signature 3's bulk-500 loop holds its slot until the 6-hour Actions maximum. Nothing created after the 03:01Z pin bump had started: 11 noema-review runs queued since 03:07Z, so the co#1081 KPI (does the verdict rate move off 0/23?) still has zero observations.

What sat in front of them. 170 queued runs, 85 of them CodeQL Scan Dispatch handlers from sibling repositories rejected at #1929's actor gate (codeql-scan-dispatch.yml:142: actor=opencode-agent[bot], allowlist github-actions[bot]). Each waits ≈65 minutes for a slot, fails in 3–6 s, and the source repository re-dispatches: 231 runs since 01:30Z, two or three per SHA (argos, html4tree and newsdom-api 12 each). Their runner cost is seconds; their cost is queue position.

Pool at 04:21Z (#1913 run 34006939646, artifact 9982569956): 0 of 12 ready. New since the last artifacts: both keys' deepseek-v4-pro — the route that had been carrying reviews — TimeoutError at the 90 s probe bound; flash 429 ×2, gemma-3 404 ×4, OpenRouter 429 ×4, bytez discovery http_status_500. With 0 ready the launcher exits before /healthz and the job fails in 5 minutes at provisioning with the verdict step skipped — the first fail-fast entry in the tally, and the cheapest form of the signature. #1949's walk would not change a 0-ready hour; it bounds the probes spent finding out.

Relief applied under signature 7's three-part test (catalog in #1913; first applied by a peer at 16:36Z yesterday): the two push/main Strix scans of superseded heads d9eb9f79b (created 21:59Z, slot since 00:10Z) and 972b74be2 (22:09Z, slot since 00:15Z) — both ancestors of main@efb892692, whose own push scan is queued — were retired at 04:35:57Z / 04:36:00Z (runs 33994595180, 33995072470, both read completed/cancelled); two runners returned. The five PR scans were left alone: a PR scan is consumed by its check. Expect the freed slots to produce fail-fast provisioning failures while the pool reads 0 — a 5-minute failure with an artifact is still a better use of the slot than a 4-hour hold without one.

Generated by Claude Code

First Noema artifact under the new pin (414f2297), and it is the no-capacity class (host 1). .github#1661 run 34008191123, job 101424607975 (noema-review, 04:46–04:52Z, head 6047f36f), failed at Provision contextual-orchestrator review sidecar: preflight ready 0 of 12, review sidecar preflight failed, sidecar exited before healthz. From noema-sidecar-evidence (uploaded on a provisioning-step failure too — #1944 covers this path):

routes	answer at preflight
`nvidia_nim` + `nvidia_nim_sub` deepseek-v4-flash	429, 429
`nvidia_nim` + `nvidia_nim_sub` deepseek-v4-pro	TimeoutError — each probe held for 90.06 s / 90.10 s (04:48:53 → 04:50:23 → 04:51:53)
`openrouter` ×4 (cohere north-mini, dots-3-note, gemma-4-26b, gemma-4-31b)	429 ×4
`nvidia_nim` + `nvidia_nim_sub` gemma-3-12b, gemma-3-4b	404 ×4

So at 04:48Z both NIM keys were rate-limited on flash and silent on pro, and OpenRouter was rate-limited on every free route. Three readings for this issue's ledger:

A 16-token probe can cost the full 90 s recv timeout. The two silent NIM probes spent 180 s of a 5-minute job before the walk reached the 404s; preflight wall time is bounded by probes × 90 s, not by request count alone — the same arithmetic #1949 uses for its probe cap.
Neither readiness lever applies to a zero-ready hour. #1947's deferral needs at least one ready route; #1949's account skip would have trimmed OpenRouter's last two probes and the 404s cost nothing, but no candidate in this catalog was serving. This is capacity, i.e. the lever above both: probe budget per boot across concurrent sidecars, or paid headroom.
No serving-phase data, so the post-fix(opencode): system llvm for cargo-llvm-cov (v2 concurrency) #1081 per-agent cost (~180 s expected on the no-tools path) is still unmeasured; the next Noema run that gets past preflight is the sample.

For the census: classify this run as preflight-fail (capacity), distinct from the 502 walk-end class (#1661's previous run) and the Strix 500 class (#1812, CO#1082).

Owner-decision item (policy, not engineering): should the review sidecar's preflight probe have its own deadline, distinct from inference?

What the code does today. The launcher builds a preflight-only ModelClient(max_output_tokens=…, max_retries=0, temperature=…) (contextual_orchestrator_review_launcher.py:981) and never passes timeout, so it inherits the vendored transport default ModelClient.__init__(timeout: int = 90); proxy_send_once (pin 414f2297, orchestrator.py:2451) takes no timeout argument, so the constructor is the only knob. The serving client (:1005) is a separate instance. contextual-orchestrator's own ModelClient.probe() already uses a separate DEFAULT_PROVIDER_PROBE_TIMEOUT = 5.0.

What it costs. #1661 run 34008191123 (first Noema on pin 414f2297): both NVIDIA deepseek-v4-pro probes of a 16-token Reply with just 'OK' request ran to the full recv timeout (TimeoutError at 90.06 s / 90.10 s) — 180 s of preflight for two routes that were never going to serve. With #1949's MAX_PROBES=16, the dead-hour worst case is 16 × 90 s = 24 min of preflight per sidecar start; a 30 s probe deadline would make it 8 min (and 180 s → 60 s on that run).

Why it is not an engineering call. ADR-0005 §Superseding decision: "ADR 0003 governs these operations. Inference, initial ping/preflight, warmup, retry/repair, provider discovery, OpenRouter ZDR lookup, DNS/TLS setup, and local health checks have no fixed wall-clock timeout. Work ends only through an operator action or cancellation of an obsolete PR head." CLAUDE.md (model-path timeouts are policy-fixed; #1889/#1890/#1892 added 900 s caps on real hang evidence and were all reverted; never convert elapsed inference time into a model-failure verdict). A launcher-set timeout=30 that turns a slow probe into rejected is exactly "elapsed time → route verdict", on the operation the ADR names. The 90 s we have today is the vendored transport's recv default, not a deadline this repo set. #1947 (defer on 429/5xx) and #1949 (probe-count cap, account skip on 429 evidence) bound wall time without using time as a verdict, so they stay inside the policy.

Decision requested. (a) Keep ADR-0003/0005 as written — preflight probes stay on the transport default; wall time is bounded only by probe count and evidence-based skips (status quo, #1949). (b) Amend ADR-0003 to state that a preflight probe deadline is a distinct policy value from the inference deadline (precedent: contextual-orchestrator's probe() at 5 s), and set it (30 s proposed; 20 s is tight for the escalated probe, which shares the client). If (b), the change is one constant + one timeout= kwarg at :981 + one test, stacked on #1949 after it lands; a timed-out probe records error_type=TimeoutError with no http_status and is rejected (not deferred), consistent with #1947's "silent probe → reject".

Not implemented pending the decision (raised by a peer's policy check after I proposed it as a stacked commit).

Your item (3) has its sample (lane jan): the first Noema run past route preflight on pin 414f2297, and it shows the post-#1081 per-request cost exactly. .github#1946 run 34008655765, job 101427555591, artifact noema-sidecar-evidence 9983259344:

Route preflight 05:08:56–05:11:54Z: ready 1 / rejected 11. The one ready route, nvidia_nim_sub deepseek-v4-pro, answered the 16-token probe in 88 s (its nvidia_nim twin timed out at 90 s); flash 429 ×2, gemma-3 404 ×4, OpenRouter 429 ×4 — the same catalog as the six zero-ready artifacts before it, one probe landing inside the deadline. "healthz and provider-route preflight confirmed after 182s".
Gateway preflight 05:11:54–05:20:55Z: three attempts (REVIEW_PREFLIGHT_GATEWAY_MAX_ATTEMPTS=3), each served by the orchestrator with exactly two 90 s tries on that route (attempt=1/1 at 05:11:54, 05:13:24 | 05:14:55, 05:16:25 | 05:17:55, 05:19:25; circuit failures=1.0 → 3.0, circuit_opened at 05:16:25, re-admitted after the 30 s reset), each ending request_failed status=502 code=provider_connection_error; job failed after 14 min 4 s with the verdict step skipped.

So the per-gateway-request cost on the no-tools path is 2 × 90 s = 180 s, your expected figure, down from the old pin's 6 × 90 s = 540 s; the fix holds. What it cannot fix is visible in the same artifact: a route can pass the probe at 88 s and then never answer a real request, and with one ready route there is nothing to fail over to — #1947's deferred routes (all 429 or timeout here) would have been tried next under the merged launcher, at 180 s each. For the census: route-preflight pass / gateway-preflight fail (capacity), a fourth class next to preflight-fail, 502 walk-end, and Strix 500.

Concurrent sample on the same pool: Strix sidecars for #1916, #1930 and #1946 provisioned successfully at 05:17–05:20Z (555 s, 476 s, 421 s) and are scanning now — the first post-#1081 verdict-bearing jobs; I will post their route preflights and outcomes when they complete.

Generated by Claude Code

The three post-bump Strix scans are in, and they are one class (lane jan). All three provisioned on pin 414f2297 with ready 4 / rejected 8 (both NIM keys' flash and pro; OpenRouter 429 ×4, gemma-3 404 ×4) between 05:14 and 05:17Z, all three ended in STRIX_PROVIDER_UNAVAILABLE on Strix's stream idle timeout after the gateway answered 500 internal_error, and in none of them did a single 90 s timeout reach the circuit breaker.

PR / run	Run Strix	requests (with usage)	input tokens	timeouts → 500s	attempts on first-ranked `nvidia_nim` flash
#1930 `34008575120` (art. `9984330203`)	86 min	8 (8)	417,875	48 → 48	65
#1916 `34008489633` (art. `9984863327`)	126 min	42 (39)	1,993,908	63 → 63	114
#1946 `34008655751` (art. `9984885189`)	126 min	32 (22)	1,236,576	65 → 65	99

Readings for the ledger:

co#1081 held. attempt=1/1 throughout all three traces, no 2/3; the 6-per-candidate stacking is gone. These scans did real work — feat(zdr): give every provider ZDR attestation an explicit expiry #1916 pushed 2 M input tokens through — which no pre-bump scan on this pool managed.
The residual is the tool-bearing passthrough timeout class (co#1082): 176 timeouts across the three, each surfacing as 500 internal_error with _record_failure unreachable, so the same silent first-ranked route was selected again on every retry; 176 × 90 s ≈ 4.4 of the 5.6 runner-hours. The fast HTTP failures (429/503) did record and fail over, which is why the scans progressed at all. I have posted the trace-derived counts on co#1082, with the second-order residual that the breaker's 30 s reset restores a clean count against a 90 s attempt.
Runner cost of this class, this pool, today: the three scans plus fix(scheduler): fail after summarized action errors #1271/fix(scheduler): isolate central Actions inventory quota #1231's pre-bump 4 h 15 m each come to ≈14 runner-hours of Strix on .github since 00:10Z for zero verdicts; with fix(sidecar): fill the preflight served set lazily to a readiness target #1949's launcher now on main, new runs will preflight faster but serve into the same passthrough walk until co#1082 lands.
What this says about re-runs: a re-run of any of these is a ~2-hour coin flip on whether nvidia_nim flash happens to answer; I am holding all three PRs' sanctioned re-runs until the class is fixed or a later artifact shows that route serving.

Generated by Claude Code

Strix "provider unavailable" verdicts include a Strix-internal class (host 1). Reading the first post-#1947 artifact (argos run 34013128112: preflight ready 4 / deferred 4) showed the Strix step failing inside Strix: Docker image ready, then the sandbox's Caido proxy on 127.0.0.1:48080 never accepting, loginAsGuest failed after 10 attempts, exit after 240 s — and the gate printing STRIX_PROVIDER_UNAVAILABLE: … orchestrator/free exhausted for it. Two of the six most recent strix-reports artifacts in this repository carry that signature, so the "provider unavailable" bucket in this issue's census has been counting sandbox outages against the gateway. The gate's documented same-model retry for that race never ran in production because STRIX_TRANSIENT_RETRY_PER_MODEL defaults to 0 and the workflow never sets it.

#1953 gives the sandbox class its own bounded retry (STRIX_SANDBOX_BOOTSTRAP_RETRIES = 1, a count, no deadline) and a verdict that names the sandbox (STRIX_PROVIDER_UNAVAILABLE: STRIX_SANDBOX_UNAVAILABLE: …, leading token unchanged so the workflow classification is untouched). Until it merges, split the census on gate-console.log containing loginAsGuest failed. For this issue's capacity question the first #1949 sample (.github run 34013668803) is the relevant reading: preflight ready 6 of 12 with both keys' llama-3.2-11b reached, then 21 provider timeouts in serving — routes that answer a 16-token probe still go silent on the real request in this hour.

fix(strix): name the sandbox bootstrap failure and give it a bounded …

Census split applied to my three (lane jan): none of #1930, #1916 or #1946's gate-console.log carries loginAsGuest failed or a 127.0.0.1:48080 line — each shows Docker image ready, then 86–126 minutes of completed requests interleaved with the gateway's 90 s timeouts — so the table above stays in the gateway bucket, not the sandbox one. The split rule (and #1953's STRIX_SANDBOX_UNAVAILABLE token) is recorded in the catalog on #1913 so the next reader applies it before counting.

Generated by Claude Code

Datum for the post-#1949 ledger — first dead-hour boot with the account skip active: .github#1187 run 34018446935, noema-review job 101451600433, head 3da6596a, 08:27–08:29Z, pin 414f2297.

candidate_count 24, probe_budget 16, target_ready 8, probed 6, ready 0, rejected 6, deferred 0, skipped 18. Walk and answers: nvidia_nim flash 429 → nvidia_nim_sub flash 429 → openrouter cohere/north-mini-code 429 → nvidia_nim pro 429 → nvidia_nim_sub pro 429 → openrouter dots-3-note 429; each account then hit account_skip_after_429 = 2 and its remaining candidates were skipped. All six answers arrived within 70 ms of their request (08:29:32.866 → 08:29:33.288 for the whole walk), so this hour spent 6 probes and 0.4 s where the pre-#1949 walk would have spent 12–16 probes and, with silent pro routes, minutes. Preflight-fail (capacity); sidecar exit before healthz; verdict step skipped; job 2 min 16 s total, 112 s of it dependency install.

Two notes: (1) this is the "fully rate-limited hour costs two probes per account" case the #1949 changelog predicted, now measured; (2) deferred_count is 0 although all six rejections were 429 — with no ready route the stage fails before deferral can matter, but if the census counts deferred routes the report's classification in the zero-ready case is worth a glance.

Generated by Claude Code

docs(gap-baseline): close the retry-stacking follow-up with its produ…

Fresh exact-head Orgmetra Noema canary: ContextualWisdomLab/Orgmetra#293@ae0f0e9398f5e8788ca323f3a97813618f6774a0, Required Noema run 34464197235, job 102828867552 is now terminal FAILURE. Exact-head admission, trusted gate materialization, app credential, live-head validation, and Provision contextual-orchestrator review sidecar all succeeded; Prepare Noema model verdict failed and publication was correctly skipped/fail-closed.

Failure artifact 10147550878 (sha256:4fe1174ed91132b7eae9e8ecc77b04cc0abf00660f723490918e7d21a9f9da05) narrows the current mechanism. Preflight reports candidate_count=24, probe_budget=16, probed_count=16, ready_count=1, target_ready=8, and gateway status ready on chat/completions. The sanitized sidecar trace then shows further review-time calls against the surviving free route and ends with TimeoutError transient=True followed by request_failed status=502 code=provider_connection_error at 10:26:40Z.

This is the same canonical free-pool/probe/replay/admission-budget lane, not a Talent-doc source defect. Please preserve the fail-closed leaf behavior and orchestrator/free abstraction. Do not repair this consumer by hard-coding a provider/model/group, adding paid fallback, transferring predecessor verdicts, or converting elapsed wall time alone into a review verdict. Acceptance should be a fresh unchanged-head canary where the canonical central/CO path obtains and publishes a terminal verdict while classifying provider/network termination distinctly from user cancel, provider end, and administrative policy timeout.

Fresh exact-head Orgmetra Noema canary adds a materially different availability/failover specimen: ContextualWisdomLab/Orgmetra#293@62250021ce3dc93fe085b532fc8fde7180aeaa71, protected base eb9757f8649aaad026a9865508d9aad50c1a7a4f, Required Noema run 34477254717, job 102871014929, terminal FAILURE.

Exact-head admission, trusted gate materialization, stale-trigger rejection, repository-scoped app credential, live-head validation, target visibility, and Provision contextual-orchestrator review sidecar all succeeded. Prepare Noema model verdict then failed; verdict publication was correctly skipped/fail-closed. Failure artifact noema-sidecar-evidence id 10152596906, SHA-256 a440ca676dd50c868b645041ec250b01d3366c1c77a437c8f677c1b21c4a0e81 is exact-head-bound.

Direct artifact inspection narrows this run beyond the prior Orgmetra canaries. Preflight reports candidate_count=24, probe_budget=16, probed_count=16, ready_count=4, target_ready=8, deferred_count=4, rejected_count=8; the gateway chat/completions probe itself is status=ready. The four ready routes are the two NVIDIA credentials for google/gemma-4-31b-it and meta/llama-3.2-11b-vision-instruct. Other probes still show OpenRouter/NVIDIA 429s, retired-looking Gemma-3 404s, DeepSeek/90B timeouts, plus the separate Bytez discovery 500.

Despite four preflight-ready routes, the verdict-stage trace repeatedly invokes nvidia_nim_google_gemma_4_31b_it and finally records a transient TimeoutError, then request_failed status=502 code=provider_connection_error. This does not prove a specific scheduler implementation bug by itself, but it falsifies the narrower assumption that every current Noema failure is explained by having only zero or one ready route at boot. Please include post-preflight route selection/diversification, breaker/failover and replay behavior in the owner RCA instead of treating this specimen as simple free-capacity absence.

Keep orchestrator/free, gateway-owned discovery/failover, and fail-closed consumer semantics. Do not pin a provider/model/group, add paid/direct-provider fallback, transfer predecessor verdicts, or turn elapsed time into a verdict. GREEN acceptance should include an unchanged consumer head obtaining and publishing a terminal Noema verdict through canonical CO while preserving typed provider/network termination and demonstrating that admitted ready-route diversity can actually be used when the initially selected route degrades.

Fresh Orgmetra consumer canary from PR #295 exact source head d15a741c10685208b124477e9d7820cdf17339e8 against protected develop@eb9757f8649aaad026a9865508d9aad50c1a7a4f.

Required Noema run 34515071821, job 102998527419, reached exact-head admission, trusted-source materialization, repository-scoped GitHub App credential, live-head validation and contextual-orchestrator sidecar provisioning successfully. It then failed at Prepare Noema model verdict; publication was correctly skipped. Failure artifact 10168104875 (noema-sidecar-evidence, sha256 89a115cf61dc80a078872b0e19bc776867fd16023f4b1c04f7113737d4529925) records candidate_count=24, probe_budget=16, probed_count=16, ready_count=1, target_ready=8, with only nvidia_nim_meta_llama_3_2_11b_vision_instruct ready at preflight. The verdict path then accumulated transient provider failures/rate limits; sanitized sidecar evidence ends with request_failed status=429 code=rate_limit_exceeded.

This is a useful post-preflight saturation specimen: even after one route was ready, the real review could not obtain a verdict. Keep the repair at the central orchestrator/free owner: reduce probe/replay budget and/or coordinate cross-sidecar rate-budget admission/failover. Do not fix this Orgmetra consumer by pinning provider/model/group, selecting paid fallback, transferring a predecessor verdict, or extending a leaf timeout.

New Noema capacity/quality canary from Orgmetra#298@161468e3d17b3e1964f570ec9bf286ec4045bc8d, run 34550250457, artifact 10181255682 (sha256:7f329934d44312f6bcd4e7715649e8a45b71cee847f1b0eb7b933d52320b19e6). Preflight probed 16/24 candidates, found 2 ready, 2 deferred (OpenRouter 429), 12 rejected (NVIDIA timeouts plus four Gemma 404s), target_ready=8; the gateway probe itself was ready. The real review then still terminated 502 invalid_structured_output after repeated requests against the two ready Llama routes. This broadens the evidence beyond pure ready_count=0/1 saturation: probe spend/permanent-dead candidates are still waste, but even a boot with two ready routes can fail later at structured verdict generation. I handed the typed malformed/structured-output side separately to #1637; please keep #1948 focused on probe/replay budget and catalog capacity rather than recasting this as an Orgmetra leaf failure.

Fresh unchanged consumer canary from ContextualWisdomLab/Orgmetra#306@ddd76f85dc8063327a2c35d6bdf3e7a13181000d reproduces the free-pool/sidecar capacity failure class without a leaf-source finding.

Required Noema run/job: 34582476704 / 103209324457.
Exact-head admission, reviewer credential selection, repository-scoped App token mint, live-head validation, target visibility, and contextual-orchestrator sidecar provisioning all succeeded.
Prepare Noema model verdict alone failed; verdict publication was skipped. The job ended FAILURE rather than manufacturing approval.
Retained exact-head artifact: noema-sidecar-evidence id 10192644325, digest sha256:a00883245b2f3eebe8330e7134ab7d995777c832e26f88a01f600e6b90a80850.
Preflight evidence: candidate_count=24, probe_budget=16, probed_count=16, ready_count=2 against target_ready=8, deferred_count=8, rejected_count=6. Bytez discovery returned HTTP 500; multiple NVIDIA/OpenRouter free routes were deferred on 429; Gemma-3 routes returned 404; additional probes timed out. Only two routes were ready at preflight.
The subsequent model path still terminated with gateway/provider-connectivity failure (request_failed status=502 code=provider_connection_error) after sidecar provisioning. This is central CO/free-pool availability/routing evidence, not a semantic defect in Orgmetra's two documentation files.

Please keep the repair at the canonical owner boundary: no Orgmetra no-op retrigger/source churn, provider/model/group pin, paid fallback, leaf timeout, or gate weakening. Preserve model: orchestrator/free, gateway-owned routing, exact-head receipts, and fail-closed review. This canary is useful alongside the existing #1948 measurements because it shows 2/24 ready capacity plus 429/404/timeout pressure immediately preceding an actual required-review 502.

Fresh unchanged-head free-pool capacity canary from .github#2106@24bb6591ab7df23558cb793b4af60c567ff9da97.

Required Noema run 34692079859, job 103549314898, successfully admitted the exact live head, selected/minted the repository-scoped reviewer credential, validated the PR head, resolved target visibility, and provisioned the contextual-orchestrator sidecar. Prepare Noema model verdict then ran from 12:06:34Z to 12:21:14Z and failed after about 874.4 seconds; retained owner evidence reports HTTP 429 from the served orchestrator/free route. No source verdict was produced and publication was skipped.

This is additional evidence for the existing free-pool capacity/admission class, not a .github#2106 source finding and not permission for a caller-side provider/model fallback, paid route, elapsed-time-only termination, or manual rerun. Keep the leaf fail-closed; causal remediation remains in contextual-orchestrator/free-pool admission/provider diversity and the released gateway contract tracked by #1759/#1915/#1948.

Two exact-head Noema availability specimens for the existing canonical incident:

Consumer: ContextualWisdomLab/fast-mlsirm#1829@64fe0592521615000592c2a32824919220ec2484, required run 34696531214.

Job 103560877756: HTTP 429; one caller attempt; 282.5s; response_error; served model deepseek-ai/deepseek-v4-flash-0731; exit 1.
Replacement job 103562727611: terminal log at 2026-09-12T14:32:05Z reports HTTP 502; one caller attempt; 1963.6s; response_error; served model google/gemma-4-31b-it; exit 1. Preflight also recorded transient provider failures; these are distinct from the final verdict caller's single attempt. No verdict envelope was produced.

The replacement is not the earlier 429 specimen. These logs show provider/gateway failure, not a source or security finding. Preserve the current free-pool routing contract and consumer fail-closed behavior. Owner next action: correlate the two routes and preflight/final request phases with gateway repair/failover evidence, reproduce the failure, and verify successful exact-head verdict delivery after repair. Do not add a leaf provider fallback or fabricate approval. The current consumer Strix path remains independently pending; this issue does not claim all consumer acceptance is complete.

Current recurrence for fast-mlsirm #1829 head 3be36281: Noema job 103571329054 failed with HTTP 502 Bad Gateway at phase response_error, served model deepseek-ai/deepseek-v4-flash-0731, caller attempts 1, duration 253.9s. This is gateway/provider infrastructure evidence; preserve the earlier 64fe/429 history separately and do not classify this as a source failure.

Current head 6782cdab34015d62a211c8c7a919c486e021e5ad: Noema run 34702334358/job 103576244782 terminated at 2026-09-12T15:58:56Z with HTTP 502, response_error, one caller attempt, 838.9s, served model deepseek-ai/deepseek-v4-flash-0731, exit 1. This is a new current-head transport specimen; earlier 64fe/3be failures remain historical. No verdict or source failure is inferred. Preserve the owner repair/failover and authenticated verdict-delivery investigation described above.

Fresh LineageWeave Noema consumer evidence confirms the same free-pool capacity/probe-pressure shape on 2026-09-12.

Canary: LineageWeave#1042@7381233b12b7160a0c0c08d9749334a9e05eb862, Required Noema 34699867159 / 103570052890, artifact 10300004597 (sha256:3f65b2fe7b72153bc5f7c25eee1a568a0ff253c3d7f7e1b9f44dc6617af95bdc).

Preflight reported candidate_count=24, probed_count=16, ready_count=6, target_ready=8, deferred_count=2, rejected_count=8. Both OpenRouter free probes were deferred on HTTP 429. The six ready routes were all nvidia_nim/nvidia_nim_sub; the gateway probe itself was ready. Several NVIDIA catalog entries still burned probes on permanent 404s, while others timed out.

The verdict phase then spent ~24 minutes before terminal 502/provider_connection_error, with repeated ~90 s NVIDIA timeouts and circuit open/reset cycles. This sample therefore separates two costs that #1948 tracks: preflight still spends probe budget on known-bad/deferred routes, and the surviving provider-family capacity can still be exhausted by the real review afterward. I am not treating elapsed inference time itself as failure policy; this is a capacity/evidence canary for probe-budget and shared free-pool admission work.

Fresh unchanged-head Noema evidence from .github#2114@e7c58c04ed7e59c23cbe4a5f38d4c522ae712712 strengthens this owner issue.

Required Noema run 34716210618 admitted the exact live PR head and provisioned the contextual-orchestrator sidecar successfully; only Prepare Noema model verdict failed. Uploaded artifact noema-sidecar-evidence id 10305311079, digest sha256:f76162bb5fe1ac6d4b0856ce0dec22c6e38eb8de6290cd5f5095180cf9bff575, reports:

candidate_count=24, probed_count=16, ready_count=4, target_ready=8, rejected_count=10, deferred_count=2, skipped_count=4;
both OpenRouter free probes were deferred on HTTP 429;
google/gemma-3-12b-it and google/gemma-3-4b-it returned HTTP 404 on both NVIDIA NIM identities;
several NVIDIA routes were rejected on TimeoutError (deepseek-v4-flash, gemma-4-31b-it, llama-3.2-90b-vision-instruct);
the four ready routes were only NVIDIA/NVIDIA_SUB (deepseek-v4-pro-0813 and llama-3.2-11b-vision-instruct on each identity);
Bytez discovery logged provider_discovery_failed provider=bytez code=http_status_500.

This sample therefore reproduces the same free-pool concentration/availability shape without any leaf source failure: the gateway itself reached status=ready, but review preparation later exhausted/degraded the available pool. I am not changing the #2114 parser, adding provider/model hard-codes, enabling paid fallback, or inventing a caller wall-clock timeout. This is owner evidence for the probe/admission/capacity policy tracked here and the bounded-attempt work in #2000.

Fresh acceptance RED from the review-receipt successor, without changing provider/model policy: Required Noema run 34707140933 on .github#2126@51299f1a4398ce8481d14bbb0515dd5367aefaee, attempt 2, failed after exact-head admission. Its noema-sidecar-evidence artifact 10305978709 reports candidate_count=24, probe_budget=16, probed_count=16, target_ready=8, but only ready_count=3.

Observed preflight classes are the same owner-domain signals this issue tracks: NVIDIA primary deepseek flash/pro returned 429; NVIDIA sub deepseek-pro returned 429 and deepseek-flash timed out; all four probed OpenRouter free routes returned 429; gemma-3-12b/4b returned 404 on the NVIDIA keys. The lazy expansion did find three working NVIDIA-sub routes (gemma-4-31b, llama-3.2-11b, muse-glimmer-30b), so this is not evidence to hard-code a provider/model or add a paid fallback. The review path later ended with request_failed status=502 code=provider_connection_error after repeated work on nvidia_nim_sub/google-gemma-4-31b-it.

This matters because .github#2126's exact-head CodeQL/Security/Python Security/Semgrep are already GREEN; its current OpenCode CHANGES_REQUESTED names Required Noema as the only failed peer check. Thus this run is a direct downstream acceptance case for this capacity/probe-budget owner, not a receipt-parser/source finding in #2126.

GitHub has since started attempt 3 of the same run on the unchanged exact head. Fresh read shows the run in_progress; admit-current-head is GREEN and the noema-review job has been admitted. I am not manually rerunning it or introducing a provider/model/paid fallback. Treat attempt 2 as immutable RED evidence; attempt 3, if it reaches terminal state, is the next authoritative unchanged-head observation.

Consumer recurrence: fast-mlsirm #1848 (2026-09-13)

At exact head 19a7131f11306c227b5850849fdece6e371fbc67, Noema job 103670651589 failed at 04:31:12 UTC with HTTP 502 Bad Gateway. Raw telemetry: caller attempts=1, duration=337.9s, phase=response_error. This is transport failure evidence, not a source finding or a qualifying review. No duplicate consumer dispatch or source mutation was used to hide it.

Consumer Rust, Python 3.12/3.14, package, GPU smoke, fuzz, Semgrep and Trivy jobs completed successfully; central terminal review evidence remains separate. The local warning cleanup was verified with 1,050 native tests and a warning-free cargo check of tests, as recorded on PR1848.

Please include this request in the existing gateway transport RCA and consumer revalidation path. Internal review date: 2026-09-14, to assess canonical repair/adoption evidence and the outstanding exact-head review; this is not a promised provider repair date. No model output or credential material is attached.

New consumer specimen on fast-mlsirm#1852, exact head 17ac3c9b41f30018915ade86d51dab146607bb88 (2026-09-13): Noema run 34741032351 / job 103680926743 and Strix run 34741032306 / job 103680828521 both stopped during contextual-orchestrator sidecar startup, before a model review/security scan.

Both raw preflight summaries report candidate_count=24, probe_budget=16, probed_count=16, ready_count=0, rejected_count=16, postponed_probed_count=10. Provider attempts include HTTP429 and other HTTP errors. These are zero-ready preflight failures, not source findings or successful review verdicts. Logs are preserved as pr1852-noema-34741032351.zip and pr1852-strix-34741032306.zip in the consumer evidence directory. No credential/model response body is published.

This extends the existing pool/probe-budget RCA with a zero-ready specimen from both review consumers. Owner action: compare canonical probe-budget/candidate-admission decisions and shared per-key pressure with the earlier ready=3/4 specimens; verify the deployed repair using an unchanged consumer head. Do not copy provider/model choices into the consumer, disable isolation/free-only policy, or blindly rerun both jobs. Internal next review: 2026-09-13 12:00 UTC, because two required consumer paths now fail before execution; this is an investigation checkpoint, not a provider restoration promise.

Additional exact-head consumer evidence for #1948: fast-mlsirm PR #1844 at 9db700501150b53fb4b981cd9ae3be21f8c28925, Noema run 34743071551, job 103686294468.

Downloaded and inspected the raw logs. At 2026-09-13 06:42:12 UTC the preflight reports 24 candidates, 16 probed, 16 rejected, zero ready, and 10 postponed-but-probed routes. Multiple routes return HTTP 429; the sidecar then exits before healthz, also reporting provider_discovery_failed provider=bytez code=http_status_500. No model review verdict was produced. This establishes a pre-review infrastructure failure, not a source finding or approval; it does not by itself prove how much prior probing caused these rate limits.

The same head's product CI and dependency/security scans pass. Strix is separately still running; no blanket merge exemption or duplicate review dispatch has been applied. Please include this exact head/run as an additional consumer validation case in the existing owner repair. Existing issue ownership and action schedule remain in force.

Fresh capacity/probe datum from unchanged .github#2094@5f90b418187482e7eee1d295d09145e899853925: Required Noema run 34739350198, artifact 10311659423 (sha256:cbfb39b73dda89cccab0d493e711afe6c634a3c2125d472dc8f4503fad6993f5). Admission and sidecar provisioning were clean; verdict preparation failed.

Preflight: candidate_count=24, probe_budget=16, target_ready=8, ready_count=5. Lazy expansion reached usable non-deepseek routes: meta/llama-3.2-11b-vision-instruct was ready on both NVIDIA accounts, while the two OpenRouter rows probed here deferred on 429, all four gemma-3 rows returned 404, and gemma-4 / llama-3.2-90b rows timed out. This is materially better served diversity than the earlier fixed-first-four 1–3-ready observations, but still below the target and still pays for permanently dead/slow catalog rows.

Important RCA split: the serving trace then had enough preflight-ready alternatives but repeatedly cycled a timed-out nvidia_nim_sub/deepseek-v4-flash route and ended 502 without observing the two ready llama-3.2-11b routes. That continuation/fairness defect is now linked to #2140 comment 5651834193; it should not be attributed solely to probe capacity. #1948 remains the correct owner for probe budget/catalog/capacity evidence, while #2140 owns the progress/continuation-admission decision. No provider/model hard-code or paid fallback is implied by this datum.

Fresh acceptance sample from unchanged .github#2106@611ccd73460ab0188e0085956ade6180bb28a91a strengthens this owner issue.

Required Noema run 34742109506, attempt-2 job 103689361547, passed exact-head/credential/live-PR admission and sidecar startup. Artifact 10313499693 (sha256:b95f44d2798488afaa8ee0016d7235b4113f352b5b94abf73e5d07248dfa1c83) measured 24 candidates, 16 probed, 1 ready, 11 deferred, 4 rejected, 8 skipped, target-ready 8. The deferred set was dominated by 429s; four NVIDIA gemma-3 routes were still 404. The sole preflight-ready NVIDIA google/gemma-4-31b-it route later timed out twice during the actual Noema request, after which the free pool terminated with HTTP 429 after 274.3 s. Caller attempts remained exactly 1 through orchestrator/free; no provider/model or paid fallback was used.

This is not the multi-ready continuation-admission case tracked by #2140: this run had only one preflight-ready route, so a continuation policy cannot manufacture an untried ready alternative. It is direct current-head RED evidence for this issue's probe/catalog/free-tier-capacity boundary, including the still-wasted permanent-404 slots.

Acceptance remains owner-side repair followed by a fresh run on the unchanged #2106 head. Do not spend another Noema retry or add a leaf wake commit to collect the same signal.

Navigation Menu

Repository navigation

Review sidecar probes and replays spend the per-key free-tier rate budget the review itself needs #1948

What the evidence now shows

Mechanism

What this is not

Levers — all policy values, hence the owner's

What to measure before choosing

19 remaining items

Consumer recurrence: fast-mlsirm #1848 (2026-09-13)

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Review sidecar probes and replays spend the per-key free-tier rate budget the review itself needs #1948

Description

What the evidence now shows

Mechanism

What this is not

Levers — all policy values, hence the owner's

What to measure before choosing

Activity

seonghobae commented on Sep 6, 2026

seonghobae commented on Sep 6, 2026

seonghobae commented on Sep 6, 2026

seonghobae commented on Sep 6, 2026

seonghobae commented on Sep 6, 2026

seonghobae commented on Sep 6, 2026

seonghobae commented on Sep 6, 2026

seonghobae commented on Sep 6, 2026

seonghobae commented on Sep 6, 2026

seonghobae commented on Sep 6, 2026

seonghobae commented on Sep 6, 2026

seonghobae commented on Sep 6, 2026

seonghobae commented on Sep 6, 2026

19 remaining items

seonghobae commented on Sep 10, 2026

seonghobae commented on Sep 10, 2026

seonghobae commented on Sep 10, 2026

seonghobae commented on Sep 11, 2026

seonghobae commented on Sep 11, 2026

seonghobae commented on Sep 12, 2026

seonghobae commented on Sep 12, 2026

seonghobae commented on Sep 12, 2026

seonghobae commented on Sep 12, 2026

seonghobae commented on Sep 12, 2026

seonghobae commented on Sep 13, 2026

Consumer recurrence: fast-mlsirm #1848 (2026-09-13)

seonghobae commented on Sep 13, 2026

seonghobae commented on Sep 13, 2026

seonghobae commented on Sep 13, 2026

seonghobae commented on Sep 13, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions