Repository navigation
Scheduler pre-review update-branch discards in-flight checks under queue saturation: 76/77 merges since 09-04 had 0/12 required contexts at merge time #1935
Description
Summary
Under queue saturation, the scheduler's pre-review update_branch path discards in-flight checks. A scheduler run waits 2h+ for a runner, executes after main has moved, picks one behind PR from its scan — its own PR (#1926) or another one (a codex/originweave-mv3-owner run updated #1857) — and merges main into that head before any review is dispatched. Every completing run therefore has one near-random victim, and the scheduler's completion rate is the PR cancellation rate. The push cancels every queued check on the old head and puts the new head at the back of the queue. main moves roughly every 28 minutes (77 merges into this repository in ~36h), so no PR head survives long enough for its 12 required contexts to complete.
Measured on #1926 (2026-09-05):
| item | value |
|---|---|
| scheduler run for the PR | 33963177129, pull_request_target, created 11:22:53Z, completed 13:44:41Z (success) |
bot push c730ae0b (Merge branch 'main' into …) |
13:44:37Z |
| scheduler log line | PR #1926: update_branch: current head has no OpenCode approval; branch is outdated before review dispatch; branch update requested with OpenCode app token inside GitHub Actions as OpenCode GitHub App |
check-runs on the replaced head 7920a70b |
28 total: 22 cancelled within seconds of the push (queued since 11:22:53Z), 3 success, 3 skipped |
Same mechanism observed on #1812 (c9244ceb, 13:31:29Z) and, after the #1926 merge, on #1484 (777ee3ed, 13:58:42Z: 21 of 30 check-runs on the old head cancelled at the push, 8 successes orphaned, new head starts from zero). The pushes are not a burst caused by the merge: they are a steady drip, one per PR scheduler run that finishes its queue wait, and with main moving every ~28 minutes almost every run finds its PR behind. The 5–12 bot-pushed branches seen in the 20 minutes after 6f8c51d7 are the scheduler runs that happened to complete in that window.
Consequence on merge governance
Reconstructing the check state at merged_at for every PR merged into .github since 2026-09-04 (77 PRs, REST commits/{sha}/check-runs, only runs completed before merged_at counted, skipped counted as satisfied):
| required contexts satisfied at merge | PRs |
|---|---|
| 0 / 12 | 76 |
| 7 / 12 | 1 |
| 12 / 12 | 0 |
Depth ≥ 2 jobs (opencode-review, coverage-evidence, trivy-fs, scorecard, osv-scan, dependency-review, both CodeQL compatibility analysis (...)) had no check-run at all before merge on 75–77 heads: their upstream gate job never got a runner. Every merge since 09-04 was therefore an admin bypass with the required review evidence absent. merged_by is populated on these, so it is not a bypass signal; the check state at merged_at is.
Where
.github/workflows/pr-review-merge-scheduler.yml:139—UPDATE_BRANCHESistrueforpull_request_targetruns, so every PR's own event run performs the update.scripts/ci/pr_review_merge_scheduler_core.py:4774-4785—if behind_by and trigger_reviews:→update_branch; there is no check for required check-runs still queued/in progress on the current head.CLAUDE.mdstates the scheduler "updates a PR branch only when the latest review is approved, no current-head check has failed, and GitHub reports the PR as behind". The pre-review path above is a second, undocumented update path.
Proposed fix (smallest root-cause change)
- In
pr_review_merge_scheduler_core.py, before the pre-reviewupdate_branch, decidewaitwhen the current head still has required check-runs inqueued/in_progress— do not discard evidence in flight. One helper, one guard, one contract test. - Optionally, set
UPDATE_BRANCHEStofalseforpull_request_targetruns and leave the refresh to theschedule/pushsweeps, which are rate-limited byBRANCH_UPDATE_LIMIT. - Correct the
CLAUDE.mdsentence to describe both update paths.
Related: #1931 (queue delay invalidates the dispatch side via base_sha drift — same family, other half), #1929/#1927 (dispatch actor), #1925 (CodeQL dispatch never succeeds).
Instrument notes for whoever re-measures: the pulls list endpoint omits merged_by; a check-run's started_at is null while queued; the closed event after merge creates fresh check-runs on the same head, so "latest check-run per name" reads post-merge state unless filtered by merged_at.
Activity
seonghobae commentedon Sep 5, 2026
Rate measured after the #1926 merge (
3f88e13a, 13:57:04Z) — peer session's readings at +0/+5/+10 min,.githubonly:Bot pushes arrived at 13:58:46, 14:03:11, 14:03:17, 14:05:18, 14:05:40, 14:06:31Z — roughly one every 100 s, spread evenly, not a burst at the merge. Each push spawns ~10 runs (60/6), so the drip injects ~6 runs/min while the queue's 10-minute net change is +1: cancellations leave at the same rate. The queue is a treadmill, which is why depth has oscillated between ~500 and ~600 all day without moving in either direction.
One scheduler completion ≈ one bot push (victim chosen by the run's scan, not necessarily its own PR: a
codex/originweave-mv3-ownerrun updated #1857) ≈ ~10 new runs + every queued check on that PR cancelled. On #1484 the push also cancelled the PR's own queued scheduler run 6 s later.Consequence for #1926's behavioural evidence (#1730's two CodeQL shards re-running):
codeql-scan-dispatch.ymlhas to survive this treadmill first, so that evidence may not appear until the guard lands. Guard PR follows.seonghobae commentedon Sep 5, 2026
독립 검증 + 속도 실측 (host2 세션)
이 이슈의 기전을 peer1 세션이 찾았고, 아래는 제가 별도 검색어와 별도 사례로 확인한 것입니다. 인용이 아니라 실행입니다.
소스 (origin/main
3f88e13a)현재 head의 check 상태를 보지 않고 update합니다.
pull_request_target에서도 이 경로가 열려 있습니다.사례 1 — #1926 (
7920a70b→ 봇 pushc730ae0b)사례 2 — #1484 (
13c444fe→ 봇 push777ee3ed, #1926 머지 +102초)희생 PR은 유발 PR이 아닙니다
33965444839(codex/originweave-mv3-owner의 pull_request_target, 14:03:11Z success)의 scan 로그(##[endgroup]이후):originweave의 run이 #1857을 밀었습니다. 완주 1건 = push 1건, 희생 PR은 스캔이 고른 것. 그래서 #1484처럼 자기 run이 push에 취소되는 사례가 생깁니다.
속도 (#1926 머지 후 t0/t5/t10,
gh --jq끝까지)push당 run ~10개, 유입 분당 ~6 run, 10분 순증 +1. 취소가 유입과 같은 속도로 빠집니다. 큐는 트레드밀입니다 — 오늘 큐 깊이가 500~600 사이에서 회전만 하고 어느 방향으로도 안 간 이유입니다. 머지 직후 버스트가 아니라 스케줄러 완주율에 묶인 꾸준한 방울이며, 머지는 P(behind)를 1로 만들 뿐입니다.
한계
.github한 저장소, 10분 창입니다.수정 방향(현재 head에 대기·진행 중인 필수 check가 있으면
update_branch대신wait)에 동의합니다. 트리거 형태와 무관하게 그 가드가 막습니다.seonghobae commentedon Sep 5, 2026
Fix PR: #1937 —
has_in_flight_check_runs+ awaitguard in the pre-review update path, no age cap, contract test with positive/negative/empty cases,CLAUDE.mdnow names both update paths. Gate on the PR head: 2893 passed / 100% coverage / 100% docstrings; the new test fails againstorigin/main's core.fix(scheduler): hold pre-review branch updates while current-head che…
seonghobae commentedon Sep 5, 2026
Residual old-code drip removed: cancelled 40 queued scheduler runs created before the #1937 merge.
pr-review-merge-scheduler.ymlhas noactions/checkout; it fetches the central scripts as a tarball atTRUSTED_SOURCE_REF= the run'sworkflow_sha(:313-352), which is fixed when the run is created. So every scheduler run created before7f4c5e3elanded (14:17:16Z) would still execute the guard-lesscore.pywhen it finally got a runner — one victim each, for another 2–3 h. At 14:2xZ all 40 queued scheduler runs in this repository were pre-merge (oldest 10:52:16Z; 37pull_request_target, 1push, 2repository_dispatch); none were in progress. Cancelling a queued run costs no runner time, and each affected PR is rescanned by the hourlyschedulesweep and its next event with the new code, so nothing durable is lost..githubqueue depth went 608 → 568.From here on every scheduler run in this repository carries the guard, so the post-merge predictions (bot pushes → ~0,
cancelled_at_push= 0, queue trending down) are tested cleanly. The same residual exists in every sibling repository where the ruleset injects this workflow; counting it there is in progress before any decision.Cancelled run ids:
33971122915,33971046733,33971024024,33970822494,33970780040,33970761084,33970662388,33970657142,33970601476,33970578666,33970439323,33970363093,33970215310,33970209470,33969870992,33969842287,33969835753,33969784791,33969715681,33969691083,33969631593,33969604212,33969537936,33969389572,33969329111,33969197240,33969142435,33969140347,33969062391,33968270218,33968197172,33967973427,33967682868,33967556296,33967512492,33967260496,33966978881,33966845220,33966317530,33961798794seonghobae commentedon Sep 5, 2026
Baseline correction (peer measurement): the evidence discarded per push is ~10–11 checks, not 21–22.
Check-runs for jobs behind
needs:do not exist until their upstream completes; when a push cancels the workflow they are created and cancelled in the same instant. Counting only check-runs withstarted_atbefore the push:7920a70b13c444feThe mechanism and the target (0 real in-flight cancellations per bot push after #1937) are unchanged; the numbers in the issue body, the #1937 description and its CHANGELOG entry ("22/28", "21/30") are the cancelled-at-push totals and overstate the discarded evidence by about 2×. Post-merge readings use the
started_at < pushfilter with a separatephantom=counter.Merge origin/main (#1937 in-flight-check hold) into contextual-orches…
seonghobae commentedon Sep 5, 2026
Org-wide residual removed too, and the first post-merge readings.
The ruleset injects the same scheduler into sibling repositories, with the same creation-time
workflow_shatarball, so their queued pre-merge runs carried the oldcore.pyas well. A peer counted 43 across 19 repositories; my own per-repository query (statusqueued,pathending inpr-review-merge-scheduler.yml,created_at< 14:17:16Z) found 42 (clearfolio 2 vs 3 — one presumably left the queue in between). 38 cancelled; 4 cancel calls were refused — exactly the four zombies queued since 2026-07-13 (naruon, disksage), 08-19 (inkspan) and 09-02 (TEPP), which the API no longer accepts a cancel for and which are unlikely ever to run.One straggler explained. At 14:21:00Z a bot merge landed on #1911 (
ci/main-full-suite-gate), cancelling 12 real in-flight checks (+14 phantom). Its producer was scheduler run33966317530(created 12:32:11Z,workflow_sha19f8ff5f, i.e. old core): it was assigned a runner at 14:20:34Z — 26 s before the.githubcancellation sweep reached it — issued the update-branch, and was cancelled at 14:21:03Z. Thein_progress = 0check preceding the sweep was true when taken; sequential cancels took ~40 s. Not a guard defect. A peer also enumerated every update-branch call site onmain: the scheduler core is the only one that can actually update (opencode-review-dispatch.ymlcalls it with--no-update-branches,agent_mention_router.pyhard-codesupdate_branches: False), so there is no second pusher.Independent post-merge count (surviving-head instrument, 14:25:00Z): of 137 open PRs, exactly 1 head is a bot merge-commit created after 14:17:16Z — #1911 at 14:21:00Z, the straggler above. Under the pre-merge regime the same 8 minutes produced ~5. Predictions ① and ② therefore hold from 14:21:03Z onward; ③ (queue trend) is read against 568.
Also observed live in the same window:
opencode-review-dispatchrun33966200868still rejectedactor=opencode-agent[bot]at 14:20:09Z — the variable behind #1929 is unchanged.Sibling cancellations by repository — ContextualWisdomLab.github.io:
33971329739,33970961103; Orgmetra:33971182649; TEPP:33971132327,33970743395,33648251504; appguardrail:33971327056,33971019552,33968624177; argos:33971093164,33970864336,33967872388,33966466101; bandscope:33971280631,33970256883,33969894325,33968011439; clearfolio:33970280235,33967679022; contextual-orchestrator:33970774990,33970678974,33967327860,33967146342,33967113270; disksage:29257025640; fast-mlsirm:33970736655,33967986067; html4tree:33970255670,33969794800,33969382246; inkspan:33969215494,32214389134; linux-cluster-ops:33970388678; naruon:33969245374,33968577535,29254301754; nonnest2:33970374725; pg-erd-cloud:33970777607,33970152562; scopeweave:33971173311; seedream_evasepic:33968018597; xtrmLLMBatchPython:33970799642seonghobae commentedon Sep 5, 2026
⑥ after #1937 (
7f4c5e3e, merged 14:17:16Z) — t5/t10/t15Instrument: per bot-pushed branch since merge, count the previous head's check-runs
cancelledwithin ±60 s of the push withstarted_at < push(real in-flight); check-runs created by the cancellation itself (started_at ≥ push) are reported separately asphantom(see the phantom split in the comment above). Pre-guard references: #1926 = 10 real (+12 phantom), #1484 = 11 real (+10 phantom).ci/main-full-suite-gate(#1911) @14:21:00claude/profile-catalog-reconciliation-v2(#1687) @14:27:14Both pushes were made by scheduler runs created before the merge, executing the old core. The scheduler fetches its source at
TRUSTED_SOURCE_REF = workflow_sha, fixed at run creation, so a run queued before 14:17:16Z runs the unguarded core whenever it finally gets a runner.33966317530, created 12:32:11Z, scan job got a runner at 14:20:34Z (10 steps), sent the async update-branch, then was cancelled at 14:21:03Z by the pre-merge-run sweep — 3 s too late. The other cancelled runs sampled hadsteps=0 runner=0.33966913388, created 12:44:53Z, scan job 14:26:34→14:27:17Zsuccess(push 3 s before completion — the fix(codeql): serialise the dispatched scan matrix with toJSON #1926 signature). Its log:TRUSTED_SOURCE_REF: f250638827f8252b0d9e5cb2601f4d333f96162f(main as of 12:44Z),PR #1687: update_branch: … branch update requested with OpenCode app token,counts: {update_branch: 1}. The new core would have loggedholding the update. This run was inside the sweep's created-range and on its first page (only 72 scheduler runs were created after 12:44:53Z), so the sweep's list or its cancel call missed it.Second-pusher hypothesis refuted at source. Every
update-branchcaller onmain:pr_review_merge_scheduler_core.py(the scheduler),opencode-review-dispatch.yml:7788which passes--no-update-branches,agent_mention_router.py:371which hard-codes"update_branches": False, and a test script. Only the scheduler core can update a branch, and #1937 guards that path.State now:
created=<2026-09-05T14:17:16Z&status=queued→total_count=0,in_progress→ 0 (server-side filter, no page trap). Every scheduler run from 14:27:17Z forward executes7f4c5e3e. ⑥ restarts from that instant with a fresh +5/+10/+15 series.Queue delta is not bot churn: of the 98 runs created after 14:22Z, 67 were triggered by session pushes (#1717
14d40755, #1936c7115fe3, #1629) and 28 by the bot, across 8 branches. Prediction 3 (depth trending down) must be read net of session pushes.Cross-check on a different artifact (peer session): filtering the 137 open-PR head commits for bot-authored two-parent commits after 14:17:16Z yields exactly the same two events.
seonghobae commentedon Sep 5, 2026
Second straggler (#1687, 14:27:14Z) explained — and the sweep's blind spot closed.
The producer
33966913388(created 12:44:53Z, old core,TRUSTED_SOURCE_REF f2506388) was not in the cancellation list (40 listed, 40 cancelled, 0 failed), although it shares the sameworkflow_id299236240andpathand its only job waited until 14:26:34Z. Three listing instruments agree with each other now (workflow-runstotal_count9 = paginated 9 = repo-runs path-filtered 9), so this was not an API omission but a status filter: the scheduler's concurrency group is per repository and per PR (central-pr-review-merge-scheduler-{repo}-pr-{n}), so a PR with two pending events has its second run parked in statuspending, whichstatus=queueddoes not return. Every count so far — mine, the 43 across siblings, and "remaining old runs = 0" — read onlyqueued.Re-measured with
status=pending|waiting|requested+created<14:17:16Z+ path across.githuband the 19 sibling repositories: 5 more old-core scheduler runs —.github33969771830,33969684836,33969659836(13:42–13:44Z), naruon33969517187, appguardrail33968100685. All five cancelled (0 failures) at ~14:39Z. A full re-sweep over bothpendingandqueuednow leaves only the four zombies the API refuses to cancel (naruon/disksage 07-13, inkspan 08-19, TEPP 09-02).So the clean observation window for the guard starts at ~14:39Z, not 14:21Z or 14:27Z. Both stragglers so far (#1911 at 14:21:00Z, #1687 at 14:27:14Z) are old-core runs with the same signature (push seconds before the run completes,
update_branchin the log where the new core printsholding the update); neither is a guard defect.Metric note for the queue-trend prediction:
.githubcurrently has 37 runs inpending(any workflow) on top of thequeueddepth. Cancellingqueuedruns can be masked bypendingruns promoting intoqueued, so readqueued + pending, and exclude session-driven pushes (67 of the 98 runs created 14:22–14:32Z wereseonghobae-actor).seonghobae commentedon Sep 5, 2026
Correction to my "state now" paragraph above. The claim "
created=<14:17:16Z&status=queued→ 0 … no page trap" was exact over the page dimension and blind over the state dimension: a scheduler run waiting on its concurrency group (pr-review-merge-scheduler.yml:79-91) sits instatus=pending, notqueued. That is also why the sweep missed33966913388— it waspendingat sweep time, not absent from the API. A peer session re-read withpending|waiting|requestedand found five more pre-merge scheduler runs org-wide:.github×3 (33969771830,33969684836,33969659836, created 13:42–13:44Z), naruon ×1, appguardrail ×1. All three.githubruns are nowcompleted/cancelled; my independent read at 14:41:01Z over all five states (pending,waiting,requested,queued,in_progress) withcreated<14:17:16Zreturns 0 for each.Clean start for ⑥ is therefore 2026-09-05T14:39:13Z (the last of the three cancellations), not 14:27:17Z. Bot pushes between 14:27:17Z and that time are attributable by the pushing run's
created_atexactly as the two above were.Queue-reading rule that falls out of this:
pendingis a separate pool fromqueued(at 14:41:01Z in.github: queued 582, pending 34, in_progress 15). A cancellation that promotes a pending run intoqueuedleaves the queued count flat, so prediction 3 must be read onqueued+pending. The instrument now reports both.General form, for anyone repeating this: a server-side
status=filter is exact only over the states it names — enumeratepending|waiting|requested|queued|in_progressbefore calling a count "remaining".seonghobae commentedon Sep 5, 2026
Guard verified in production (16:18Z). Of the 25 scheduler runs created in
.githubafter7f4c5e3elanded, 10 have executed. Their decisions:33971948808wait— …branch is outdated before review dispatch, but current-head checks are still queued or running; holding the update so their evidence is not discarded33971906521wait— same reason33971691883wait— same reason33971548421review_dispatch— current head has completed Strix evidence; same-head OpenCode dispatched33972657372security_dispatch33971862168security_dispatch33972093234block— 13 unresolved review threads33971711250block— 1 unresolved review thread33971916754skip— draft PR33971360998skip/blockdecisions onlyupdate_branchdecisions: 0 / 10. The three holds are exactly the PRs the old core would have pushed (behind, unreviewed, checks in flight).Queue (
.github,queued+pendingas agreed): 616 at 14:43Z → 509 at 16:18Z (−107 in 95 min), within_progress18 → 24. The treadmill has stopped spinning: cancellations no longer refill the queue, and slots are being spent on runs that finish.Remaining, out of this issue's scope: the post-approval update under
strict=true(bounded, one refresh per approval), the dispatch-actor variable (#1929 — #1911's dispatch above will still be rejected at the gate until it is set), and draft PRs, which none of the gates ever approve (see #1913).seonghobae commentedon Sep 5, 2026
⑥ closed — the guard holds on a clean window
Clean start 14:39:13Z (last pre-merge scheduler run cancelled; see the correction above). Same instrument: per bot-pushed branch, the previous head's check-runs cancelled within ±60 s of the push with
started_at < push.The two pushes in the long window:
automation/sbom-inventory@15:28:30Z is the SBOM automation commit, not an update-branch (0 check-runs on the previous head).fix/agent-mention-dispatch-credential-…@15:46:59Z is the real test: the previous head38b300a3carried 37 check-runs and lost none — nothing was in flight when the scheduler updated it, which is precisely the guard's contract. Under the pre-guard rate (~1 push/100 s, ~10 real in-flight each) this window would have shown ~60 pushes.Independent artifact (peer session, scheduler decision logs, 16:18Z): of the 25 scheduler runs created after
7f4c5e3e, 10 executed —wait×3 with the guard's own reason string (#1936, #1934, #1872),review_dispatch1,security_dispatch2,block2,skip1, push sweep 1,update_branch0/10. Check-run observation and decision-log census agree.Predictions 1–3 hold. What the guard does not fix (peer1's scope-out at the top of this issue) is now the visible next layer: #1911 reached
review_dispatchfor the first time today and will be rejected at the dispatch actor gate until the #1929 variable is set; the CodeQL dispatch pipeline sits at 101 queued / 0 completed since #1926.seonghobae commentedon Sep 5, 2026
Correction to the last line of my ⑥-closed comment: "the CodeQL dispatch pipeline sits at 101 queued / 0 completed since #1926" was a page artifact — with 110 dispatch runs queued, the 100 most recent by creation contain no completed run. Server-side since #1926 (13:57:04Z): 71 completed, all 71 failures, 0 successes; 55/0 in the hour before. Each fails in
validate-dispatchat the actor gate (codeql-scan-dispatch.yml:142-164,vars.OPENCODE_REPOSITORY_DISPATCH_ACTOR) withauthorization rejected actor=opencode-agent[bot] … must match one configured scheduler identity, and the matrix job carrying #1926's fixed step is skipped. #1926's behavioural proof is therefore not "pending" but blocked by #1929: 71 consecutive dispatches have been turned away at that one variable. Same finding as the direct probe run33969658525(rejected 15:07:07Z) and the dispatch run33966200868(14:20:09Z).seonghobae commentedon Sep 5, 2026
Next layer, baselined — "green on the current head AND up to date" (the scope-out at the top of this issue)
Full census of
.github's open PRs at 16:52Z via GraphQLmergeStateStatus+statusCheckRollup(3 pages, 138 rows =totalCount, so complete; a first attempt returned exactly 100 rows from a failed second page and was discarded).Non-draft (105): current-head rollup FAILURE 89 / PENDING 16 / SUCCESS 0. The only
SUCCESSrollups in the repo are on 5 old draft heads, where most gates skip.What fails, by check name, on non-draft heads (each PR counted once per name; 38 PRs sit at the 40-context cap of the query, so these are lower bounds for those):
The top four are the structural classes already attributed on this issue and its neighbours: the CodeQL shards fail by design awaiting a dispatch verdict that #1929's variable blocks (71/71 dispatches rejected since #1926);
noema-reviewis dominated by the gateway 502 class (12 of the last 15; CO#1045);opencode-reviewverdicts are gated by the same variable. The tail (~20 PR-checks) may be real and is the review lane's material.The guard's cohort. 16 non-draft PRs currently carry no failing check on their head — 12
BLOCKED, 4BEHIND— because their heads are fresh (today's conflict resolutions) and their checks are still queued: #789 #1382 #1397 #1545 #1623 #1661 #1678 #1687 and eight more (head SHAs recorded). #1937 guarantees those checks will now complete rather than be cancelled. Prediction, written before observation: none of the 16 reaches rollupSUCCESS; they land inFAILUREon the two CodeQL shards (by design, until #1929) and, for a majority, onnoema-review(502 class). Read at ~19:00Z. If that holds, the queue and the guard are no longer the binding constraint on merges — one repository variable is.17 remaining items
seonghobae commentedon Sep 6, 2026
First #1949 verdict sample (a peer's read of noema run
34013668803, sidecar created afterfb2ae81d): route preflightcandidate_count 24 / probed 16 / skipped 4 / deferred 2 / ready 6 / rejected 8(rejected =gemma-3404 ×4 + four probes that ran into the recvTimeoutError; ready includesllama-3.2-11bon both NVIDIA keys, which the alphabetical 4-per-key slice never reached). Readiness is back at the pre-#1939 level (5–6/12) → no rollback. Serving then walked 1689 s to502 provider_connection_error:provider_attempt×45, failures 28 (TimeoutError21,HTTPError7),circuit_*×26,unexpected_exception0 — the no-tools shape classifies its transport failures, as expected. Class: (5)-like capacity — routes that answer a 16-token probe and then never answer a real request (#1948 / contextual-orchestrator#1045).Correction to my pin note above: after
414f2297(contextual-orchestrator#1081single_attempt_transport) the no-tools path logsattempt=1/1per candidate too (42 of the 45 attempts here; the remaining 3 were1/3), notn/2. So on the new pin the attempt denominator no longer separates the two request shapes; use the HTTP status (500 internal_error= tools/passthrough leak until contextual-orchestrator#1082 lands;502 provider_connection_error= orchestrated walk) and the presence ofunexpected_exception type=(tools only). Per-candidate cost on a silent route is one recv timeout (~90 s) on either shape; the walk length is what differs.seonghobae commentedon Sep 6, 2026
Second before-sample for the tools/500 leak, same shape (#1916 strix run
34008489633, job 05:08–07:24Z, artifactstrix-reports9984863327, pin414f2297): route preflight ready 4/12;provider_attempt×139,provider_attempt_failed error_type=TimeoutError×63 ↔request_failed status=500 code=internal_error×63 (1:1 again),HTTPError×23 →circuit_failure×13 /circuit_cleared×9 (the breaker still reacts only to HTTP errors), strixError code: 500×6 +503×1, fail-closed after 7586 s. With #1930 that is n = 2 for "one 500 per timed-out passthrough attempt, breaker blind to timeouts" on the current pin; contextual-orchestrator#1082 is the fix, and the after-sample is the first Strix run created once the sidecar pin passes it.seonghobae commentedon Sep 6, 2026
n = 3 for the tools/500 leak on pin
414f2297, read independently of lane jan's table on #1948: #1946 strix run34008655751(artifact9984885189) — route preflight ready 4/12;provider_attempt×121,provider_attempt_failed error_type=TimeoutError×65 ↔request_failed status=500 code=internal_error×65;HTTPError×21 →circuit_failure×13 /circuit_reset×3; fail-closed after 7569 s. With #1930 (48 ↔ 48) and #1916 (63 ↔ 63) the three post-bump scans agree with jan's ledger (176 timeouts → 176 × 500, breaker blind to all of them; contextual-orchestrator#1081 held —attempt=1/1throughout, no2/3stacking). Owner decision on the preflight-probe deadline (my item above) is still open.fix(strix): name the sandbox bootstrap failure and give it a bounded …
seonghobae commentedon Sep 6, 2026
Seventh class — recovered transient (found by a peer, confirmed independently on #1689 strix run
34013778497, artifactstrix-reports9985499801, sidecar bound tomain@fb2ae81d): the scan completed —run.jsonstatuscompleted, 423,895 tokens,strix.log07:06:48→08:09:35Z ending inStrix scan … done, SARIF with 0 results, strix's own report rendered — yet the job failed closed withSTRIX_PROVIDER_UNAVAILABLE: … orchestrator/free exhausted. The only failure signal in the artifact is threeWARNING … strix.core.execution: transient model/provider error for <agent>; replaying turn (attempt 1/5 | 2/5 | 3/5, backoff …): InternalServerError(…)lines — strix-agent 1.5.3's own retry, which succeeded (0ERRORlines, exit 0); exhaustion would log a separate ERROR and a non-zero exit. The quick gate's report check (strix_quick_gate.shhas_strix_report_failure_signal, agrep -Ei 'Warn|Warning|WARNING|Timeout'over the report logs) treats any WARNING as a failure signal, and the narrower Python allow-listknown_internal_warningonly knows theforcing tool continuation (n/m)family. Cost: a clean 63-minute scan on a 60-job-plan slot turned into a failure and a re-dispatch of the same head.Fix being drafted (host 2 → host 1 verification → my merge): allow the recovered retry warning explicitly (attempt
n/mwith the scan's completion still required), with the first sanitizer test for that allow-list. Census rule until it lands: aSTRIX_PROVIDER_UNAVAILABLEverdict whoserun.jsonsayscompletedand whosestrix.loghas WARNINGreplaying turnlines but no ERROR is this class, not a gateway failure.Classes now: (1) gateway walk — tools/500, no-tools/502; (2) verdict schema; (3)
gh pr diff300 files; (4) preflight-fail ready 0; (5) route-pass / gateway-fail ready 1; (6) Strix sandbox bootstrap (#1953, merged43024633); (7) recovered transient mislabelled as provider-unavailable.seonghobae commentedon Sep 6, 2026
Class (4) on the post-#1949 launcher (a peer's read of contextual-orchestrator#1078 strix run
34017965035, created 07:00:10Z — afterfb2ae81d, before #1953): route preflightcandidate_count 24 / account_skip_after_429 2, thenpreflight_route_rejected … 429twice on each ofnvidia_nim,nvidia_nim_sub,openrouter→ all three accounts skipped → ready 0 →review sidecar preflight failed, sidecar exited beforehealthz, provisioning step failed 4 minutes in. No sandbox/bootstrap strings in the 1524-line log, so not class (6). The levers behaved exactly as designed (diversified pool, deferral, lazy fill with account skip); the pool was rate-limited end to end — #1948 territory.One measurable benefit is visible in that number: on the pre-#1949 launcher a dead hour cost the full 12 probes (up to 90 s each, ~393 s observed on #1911's old run before the walk even started); with the account-skip rule it costs 6 probes and the job fails closed in ~4 minutes, returning the runner slot instead of holding it. That is the wall-time bound by count and evidence, not by a deadline.
fix(review): name the Strix sandbox class in the failed-check finding (…
merge: main into #1757 head to clear a 35-commit BEHIND
seonghobae commentedon Sep 13, 2026
Live re-verification of the title defect, 2026-09-13 — proposing closure of this issue (the other failure classes recorded in the thread have their own landed fixes: #1944, #1953, #1960):
7f4c5e3e0,scripts/ci/pr_review_merge_scheduler_core.py:4816): an outdated PR is not updated while any newest current-head check is queued or running.34738720917for.github#2147decidedwait: current head has no OpenCode approval; branch is outdated before review dispatch, but current-head checks are still queued or running; holding the update so their results are not discarded— the exact scenario of the title ("update-branch discards in-flight checks under queue saturation") now resolves to a hold instead of an update.If someone still holds an open class in this thread that is not covered by #1944/#1953/#1960/#1937, please split it into its own issue before closing; otherwise this can close as fixed-and-verified.
seonghobae commentedon Sep 17, 2026
Triage (2026-09-18): close-as-done candidate
Unique delta appears on protected
mainvia merged #1937 (hold pre-review update-branch while current-head checks are queued/running). Procedure + core cite this issue; seedocs/pr-review-and-merge-procedure.mdandscripts/ci/pr_review_merge_scheduler_core.py.Ranked backlog: #2251. Spot-check one behind PR with in-flight checks before closing.
seonghobae commentedon Sep 17, 2026
Closing after spot-check: unique valid delta is on protected
main.Merged fix: #1937 (merged 2026-09-05,
7f4c5e3e0efb7bfe29f33b60d4264858effd2996)Tree evidence on
origin/main:scripts/ci/pr_review_merge_scheduler_core.pyhas_in_flight_check_runsholds pre-review branch updates while current-head checks are queued/running (explicit#1935cite)CLAUDE.mdgovernance paragraph documents the same hold (see#1935)Verified with
git greponorigin/main(not redundancy alone).fix(ci): let schedule recovery update despite in-flight checks