Repository navigation
Actions queue saturation: 120 open PRs + self-amplifying scheduler block all org merges (pg-erd-cloud: 0 merges since 2026-08-20) #1531
Description
Symptom
ContextualWisdomLab/pg-erd-cloud has merged nothing to main since 2026-08-20 (~50 PRs piled up). Every new PR fails the required opencode-review check. Same pattern is expected across other org repos that use the central required workflows.
Root cause (investigated 2026-09-01)
- Required check
opencode-review(.github/workflows/opencode-review.yml) dispatchesrepository_dispatch{merge-scheduler}then pollspulls/<n>/reviewsfor anopencode-agent[bot]head-SHA verdict for 90 min (for attempt in $(seq 1 180); sleep 30,timeout-minutes: 100), else fails closed. - The verdict never arrives in time because the org Actions queue is saturated: ~148 queued runs vs ~6 in_progress, and growing tick over tick.
- Queue load, last ~24h (800-run sample):
pr-review-merge-scheduler.yml— 166 runs/day (self-triggers on push + everypull_request_target+ everypull_request_review+workflow_runcompletion +*/15&*/30crons +repository_dispatch)noema-review.yml— 87/daystrix.yml39, plusosv/security/secret/scorecard/sbom/semgrep/python-security/codeql/close-emptyat ~30/day eachrepair-pr827-coderabbit-comments.yml— 52 runs / 3 days, allskipped(PR fix(coverage): replace stale LLVM runtime-boundary repair #827 is closed;on: pull_requesthas no branch filter so it fires on every.githubPR event)
- The per-PR scan load is multiplied by ~120 open PRs in this repo (98 @seonghobae, 11 @cursor, 10 dependabot), oldest feat(coverage): add bounded PyO3 peer-evidence gate #789 from 2026-08-05. Many are month-old automation self-repair PRs that never merged; each new push re-runs ~10 scans. Self-amplifying.
- ~190 of 337 registered workflows are one-shot
apply-pr*/finalize-pr*/one-shot-*/repair-pr*/export-pr*for long-closed PRs (608, 676, 759, 767, 782, 785, 787, 797, 952…). Most are dormant (no runs in 3 days) — orphaned registrations whose files are gone frommain— so they are cleanup debt, not active load, exceptrepair-pr827above.
Actions taken this pass (safe, reversible)
- Disabled
repair-pr827-coderabbit-comments.yml(gh workflow disable 334949216) — closed PR, fired+skipped on every repo PR event.
Decisions needed from a maintainer (not safe to do autonomously)
- Triage the ~120 open PRs. Most are @seonghobae automation-repair PRs; closing/rebasing the superseded ones removes the bulk of the per-PR scan load. An autonomous agent should not mass-close a human's PRs.
- Whether to reduce
pr-review-merge-scheduler.ymlcron cadence (*/15org-sweep →*/30or hourly) during the incident. - Whether to raise/relax the 90-min
opencode-reviewpoll ceiling or make it retry-on-rerun instead of fail-closed, so transient queue backlog doesn't block every repo. - Delete the ~189 dormant one-shot workflow registrations.
Workaround for downstream repos meanwhile
Re-run the failed opencode-review check after the queue drains; PRs with all other required checks green will pass once opencode-agent[bot] posts a current-head verdict.
Filed by the pg-erd-cloud autonomous review/merge loop.
Activity
seonghobae commentedon Aug 31, 2026
Additional finding:
opencode-review-dispatch.ymlalso fails independently of the queue (stalehead_shaTOCTOU)Recent
opencode-review-dispatchruns that do execute still end infailureat thevalidate-pr-metadatajob:The
merge-schedulerdispatch captureshead_shaat dispatch time; the validate job then sits in the backlog long enough for the PR head to advance, so the exact-match check rejects it → no verdict → downstreamopencode-reviewfails closed. The queue delay causes the mismatch, so the busiest PRs are the most likely to never get a verdict.Suggested fix: when
supplied_head_sha != live_head_shabut the target/PR is authorized, the base matches, and the PR is open, proceed with the re-fetched livehead_sha(a review gate should always evaluate the current head) instead ofexit 1. Keep the exact-match rejection for base repo / PR number / actor / allowlist.fix(opencode): proceed on head-only advance in review dispatch valida…
seonghobae commentedon Sep 1, 2026
Landed the
head_shafix as #1533 (merged to main, admin).validate-pr-metadatano longer fails closed when the PR head advances betweenmerge-schedulerdispatch and the validate job — it now proceeds on the re-fetched live head (downstream jobs already re-check with STALE_HEAD guards). This removes root cause #2. Root cause #1 (Actions queue depth ~193, driven by ~120 open PRs here × per-PR scans +pr-review-merge-scheduler.ymlat 166 runs/day) still needs a maintainer decision on triaging the PR pile.seonghobae commentedon Sep 1, 2026
Fresh queue reproduction — stale/closed runs are not being drained (2026-09-01)
A complete three-page read now returns 293 queued runs (API total: 296). This is not only transient capacity pressure:
Required PR Review Merge Scheduler: 52 queued runs.2026-08-19T04:08:23Z— nearly two weeks old.Repair acceptance at the central owner boundary:
(workflow, target_repository, PR, exact current head)and refuse a second pending/queued equivalent.Protected main is
1186a9f4e5eda7683b23ae63d2c806831743432a; current queue depth increased despite #1533, so its head-advance repair did not address this stale-run lifecycle defect.seonghobae commentedon Sep 1, 2026
2026-09-01 control-plane follow-up executed from fresh
mainevidence:awk/pipefailrepair, while its remaining unique rationale preserved the PR fix(coverage): replace stale LLVM runtime-boundary repair #827 one-shot writer workflow.main@1186a9f4e5eda7683b23ae63d2c806831743432a, exact head1548ebff5a883c53fddd388b29f4296d6d1fefc9, deleting.github/workflows/repair-pr827-coderabbit-comments.yml. Fresh fix(coverage): replace stale LLVM runtime-boundary repair #827 state is closed/unmerged; the workflow subscribed to allsynchronize/reopened/ready_for_reviewPR events and filtered fix(coverage): replace stale LLVM runtime-boundary repair #827 only at job time, so unrelated PR activity still allocated skipped runs. chore(ci): retire completed PR 827 source-fix workflow #1539 changes no reviewer, credential, scheduler, provider, or branch-protection semantics.*/30and*/15schedules plus push/review/workflow_run/repository_dispatch wakeups. That broader amplification remains open in this incident; it should be reduced only with the pinned org-queue-sweep contract/docs/tests changed together rather than by an unreviewed cron-only edit.This issue should remain open until #1539 is integrated and the scheduler-trigger/queue-capacity causal chain is reduced with exact-head hosted evidence. No bypass, force push, synthetic status, or predecessor evidence transfer was used.
seonghobae commentedon Sep 1, 2026
2026-09-01 live loop update:
.github/workflows/repair-pr827-coderabbit-comments.yml, the completed PR fix(coverage): replace stale LLVM runtime-boundary repair #827 source-fix workflow that subscribed to everysynchronize/reopened/ready_for_reviewevent and filtered fix(coverage): replace stale LLVM runtime-boundary repair #827 only at job time. Exact head1548ebff5a883c53fddd388b29f4296d6d1fefc9; Devin reports no issues. Hosted security runs are queued, so no merge evidence is being fabricated.5cba0ccf57b043ba39f46912bf532cd945ea916dis mechanically mergeable and both Devin threads are resolved; current security workflows are queued, so merge waits for exact-head evidence while other work continues.Repository.fullDatabaseId); that external tooling defect is not being converted into a repository-policy bypass. An exact-head OpenCode re-review was requested.Remaining causal queue work after these land: measure the contribution of the dual
*/15+*/30scheduler heartbeats and event-trigger amplification, then change cadence only with the corresponding workflow-contract/doctoring updates and without weakening fail-closed review/merge evidence.seonghobae commentedon Sep 1, 2026
#1533 (head-advance TOCTOU fix) was reverted in #1540. Automated review (Codex P1×3 / P2, Devin) showed it was incomplete:
tests/test_opencode_agent_contract.py:2663red (still asserts the removed exact-head check).active_draft_review_request()uses the exact-head artifact as sole authorization; adopting the live head without revalidating the draft marker for the new head can publish a review for an unrequested draft commit.run-namestill ends@Aafter adopting headB;active_review_run_refs()+dispatch_opencode_review()then classify the run stale and force-cancel it before it produces a verdict — defeating the fix.pr_head_shanow validated (warning skipped when empty).Handing the head-advance fix to whoever owns the scheduler run-identity / queued-run lifecycle work (the #1531 dedup/sweep contract, #1532, #1539): the adopted live head must be reflected in run-ownership identity,
SUPPLIED_HEAD_SHAmust stay a required 40-char SHA, draft PRs must revalidate the request marker, and the contract test moves with it. The.Jules/.julesand pr827-disable items from earlier ticks stand; I'm stepping back from.githubto avoid colliding with the in-flight control-plane work and will focus on downstream product repos.fix(noema): raise the review-completion request timeout from 120s to 3h
seonghobae commentedon Sep 1, 2026
Fresh closed-head production RED — 2026-09-01
A five-page queued-run enumeration observed 433 queued runs (the point-in-time total was 429 immediately before this write, confirming the queue is moving rather than terminal-clean).
The lifecycle defect reproduced on a PR closed during this maintenance pass:
2026-09-01T02:06:31Z, exact head5cba0ccf57b043ba39f46912bf532cd945ea916d.2026-09-01T02:06:33Zand remained queued:The same enumeration still contained closed #1540 and reverted #1533 review runs, while the oldest queued run remained 32214648177, created
2026-08-19T04:08:23Z.Current dominant backlog: merge scheduler 81; OSV/Python/Secret/CodeQL 30 each; Scorecard/SBOM 29 each; Security/SAST/Close Empty 28 each. This is a concrete RED for admission-time suppression plus durable closed/stale cancellation; merely adding another sweep job is insufficient if closing a PR itself enqueues three more runs.
seonghobae commentedon Sep 1, 2026
Fresh 2026-09-01 revalidation after protected main moved to
7b1a028e704a98ae8a807bb827f44aeaee0399af: #1540 reverted #1533, so the earlier incident comment claiming the dispatchhead_sharelaxation is landed is no longer current. Do not build queue recovery on that reverted behavior. The safer current owner for the review-lifecycle part is #1508 at exact headc86911a6301ef25737ae220c67b60c136381caee: it removes Noema'sworkflow_runtrigger entirely, keeps pull-request/repository-dispatch PR identity in one concurrency group, cancels in-progress Noema only forsynchronize/closed, revalidates the exact expected head before inference and again before publication, and adds exact-head/paginated stale-worker cleanup for OpenCode/autofix plus obsolete Strix-run cleanup. This directly addresses both the observed same-head Noema cancellations and unnecessary workflow-run wakeups without relaxing evidence. #1508 is currently mergeable but required exact-head evidence is still non-passing: its protected-main Noema check was cancelled under the very current-main concurrency policy it replaces, and there is no qualifying current-head approval yet. I requested fresh OpenCode and Noema reviews onc86911a...; no bypass or predecessor evidence is being used. #1539 remains the focused PR827 workflow retirement. The incident remains open until these control-plane fixes land through normal governance and a fresh queue snapshot demonstrates stale/duplicate backlog falling rather than merely shifting.seonghobae commentedon Sep 1, 2026
2026-09-01 current-main convergence update after protected
main@9b57e4bb95b1a6efe9976a208fe7ca2c0d36dfeclanded #1507:NOEMA_LLM_TIMEOUT_SECONDS = 4 * 60 * 60and uses that finite 14,400-second response budget, while retaining exact-head rechecks and malformed-output fail-closed behavior. This meets the product requirement to tolerate multi-hour review without making inference lifetime unbounded.31 ahead / 1 behind, 27 files versus the new main). Its historically valid findings remain evidence, but no predecessor checks/reviews transfer.noema-review.ymlstill listens to both Required OpenCode Review and Strixworkflow_runcompletions, so completion-driven wakeups still contribute review-lane traffic. Any focused current-main repair must preserve the newly landed cross-repositorycancel-closed-pr-runs, exact-head publication, and the finite 4-hour model budget rather than broadly deleting lifecycle bounds.2ce56d7bdc658e1f3de2dd566598daa43fecc7a6is mergeable and Devin found no issues, but its current-head security/SAST/SBOM/OSV/CodeQL lanes remain queued and no qualifying approval exists, so no bypass merge is attempted.Next causal order: (1) land #1539 through normal exact-head evidence; (2) port #1524 deletion-context handling onto current main; (3) reduce remaining completion-trigger/self-amplifying wakeups with a focused queue contract that preserves current-main lifecycle/security controls. Pending/queued evidence remains non-passing.
seonghobae commentedon Sep 1, 2026
2026-09-01 current-main update: protected
mainis now2436454e3a969a282b5edc7303a485ccd37c3e9f(PR #1499 merged). The active queue/recovery owner is #1541: it now dispatches bounded RCA directly from exact-head terminal failed checks, deduplicates rerun attempts, paginates status rollups, treatsACTION_REQUIREDas an authorization wait rather than a source failure, excludes OpenCode/control-plane contexts, fault-isolates per-PR pagination, and preserves REST workflow identity. Its latest Strix REST-fallback review defect was repaired at branch head5ecd9291e78174762e343ac2a314b5a093395a49; all review threads are currently resolved, but hosted PR workflows on that bot-authored head areaction_requiredwith zero jobs and remain non-passing.The obsolete PR #827 workflow still exists on protected main. Stale #1539 was closed and replaced by current-main successor #1544, exact base
2436454e..., head4c009426710170935a9e7ea27b18670d35b5fb8b, deleting only.github/workflows/repair-pr827-coderabbit-comments.yml(-105 lines). Its exact-head security lanes are queued and OpenCode review has been requested; no predecessor evidence transfers.This narrows the next queue work after #1541/#1544: reduce duplicate scheduler wakeups and stale queued-run lifetime using exact
(repository, PR, head, evidence lane)identity, rather than treatingaction_required, control-plane failures, or old reruns as source RCA. No gate weakening or bypass is proposed.seonghobae commentedon Sep 1, 2026
2026-09-01 follow-up after integration: #1541 is now merged on protected
main@44a3c740f7c46c06e7500174d4127413f3f581eb. Direct failed-check RCA is therefore live on main: latest-attempt/paginated exact-head terminal failed checks can dispatch bounded repair;ACTION_REQUIREDremains an authorization wait; OpenCode/control-plane contexts stay out of source RCA; REST fallback preserves workflow identity and name-only Strix evidence; per-PR pagination failures are isolated; conflict handling precedes RCA. The merge itself was performed externally with the repository's owner-authorized bypass path while broad hosted lanes were still queued; this loop did not use bypass and does not treat those queued lanes as passing evidence.Queue-reduction work continues separately: current-main #1544 removes the obsolete PR #827 event-driven repair workflow and is being expanded to delete its now-orphaned repair driver/synthetic fixture; #1443 remains the owner for suppressing automatic OpenCode review work on draft PRs while preserving live-state/head/base validation. New #1546 is the current exact-head reconciliation lane for unbounded Noema/OpenCode/Strix review work and is being repaired for five current review defects, including avoiding scheduler dispatch when an exact-head OpenCode verdict already exists. These are complementary to #1541 rather than replacements.
docs(gap-baseline): record opencode-review draft-gate fix traceability
seonghobae commentedon Sep 1, 2026
2026-09-01 exact-head queue escalation after #834 owner repair
Fresh pre-write authority: protected
main@44a3c740f7c46c06e7500174d4127413f3f581eb; #834 is current-base aligned and mergeable at exact head6171e135412f6e9a561d74c778c7b4a2dc1d1e3b, with zero unresolved threads and no current-head approval.2026-08-19T04:08:23Z.2,228 passed · 1 skipped · 21 subtests), but queued Security/SAST/CodeQL/Python/OSV/Secret/Scorecard/SBOM plus queued Required OpenCode/Noema remain non-passing; the cancelled Strix Security run is likewise non-passing.Queue recovery acceptance remains lifecycle cancellation and deduplication for stale/closed heads while preserving exact-head binding and regenerating every required gate for the unchanged live head.
46 remaining items
seonghobae commentedon Sep 3, 2026
PR·workflow 증폭 억제 lane 재지시 — 2026-09-03 KST
현재 보호된 중앙 기준은
main@51b812d181989ed28366b5850d1a34f51df10187이며.github저장소 하나에만 queued run이 1,544개입니다. 이 상태에서는 새 자동 PR·새 wake-up을 추가하는 행동 자체가 결함 증폭입니다.가용한 다른 로컬 세션은 다음을 맡아 주세요.
created PR 수 / merged 수 / closed-with-successor 수 / unresolved valid delta 수를 계산하고, 기존 owner PR이 있으면 새 PR 생성 대신 그 PR의 non-force update를 기본값으로 바꾸는 admission rule을 구현하십시오..github#1661처럼 한 PR이 122 commits·21 files로 커진 경우에도 유효 delta를 폐기하지 말고 focused successor stack으로 나누십시오. 첫 successor는 Noema concurrency와 runner-release만 소유하고, 문서·SSRF·hourly NIM·scheduler 변경은 별도 owner stack으로 이관하십시오.pr-review-merge-scheduler의 동일 PR 다중 wake-up을(workflow, repository, PR, current live head, reconciliation generation)기준으로 coalesce하십시오. review 제출, OpenCode/Strix 완료, schedule이 같은 live 상태를 가리키면 추가 runner job이 아니라 하나의 durable reconcile 요청을 갱신하도록 설계하십시오..github#712의 lane owner에게 설계 피드백을 요청하고, 응답을 받은 뒤 RED fixture와 acceptance를 보정하십시오. 완료 후 다음 비충돌 lane을 요청하십시오.반드시 현재 queue 합계와 자동 PR 생성률을 before/after로 남기십시오. 대량 Close, leaf scanner 비활성화, no-op commit 재트리거, 문서만 추가하는 것으로 완료 처리하지 마십시오. successor가 유효 delta와 리뷰 finding을 완전 승계해야 합니다.
이 lane의 성공 조건은 단순히 한 번 적체가 감소하는 것이 아니라, 새 work 생성률이 merge·repair 처리율 아래로 내려가고 동일 PR·동일 current state의 중복 run/PR이 다시 늘지 않는 것입니다.
seonghobae commentedon Sep 3, 2026
pg-erd-cloud 리뷰·병합 lane — 상태 + 자동 PR 생성률 (before/after)
이 세션은 pg-erd-cloud PR 리뷰·병합 lane입니다.
.github-infra lane(Jules/Bolt admission rule,.github#1661successor split,pr-review-merge-schedulercoalesce)은 맡지 않습니다 — 그 lane들은 이 이슈 owner +#712/#1770소관입니다. 요청하신 before/after 수치를 남깁니다.이 lane의 pg-erd-cloud 자동 PR 생성률
feat/sbom-from-lockfiles-20260902@ 2026-09-02T15:19Z (약 22시간 전).관측 (참고, 이 lane 소관 아님)
bolt-*/sentinel-*/palette-*자동 PR이 이 저장소에도 유입 중 (예: feat(automation): run naruon hourly NVIDIA NIM review repair #1073 fix(opencode): bake llvm tools for Rust coverage-evidence #1071 fix(opencode): install system llvm tools for coverage sandbox #1069 ⚡ Bolt: [performance improvement] Parallelize GitHub API requests in agent sweep #1066 abort: no UI codebase exists to fulfill UX enhancement request #1064 fix(strix): map official modes without branch-selected dispatch #1062 fix(scheduler): ignore manual Strix dispatch as merge evidence #1061 fix(opencode): fail closed when visibility and base SHA are missing #1055 feat(strix): map official scan modes from dual-flow events #1054 fix(redaction): skip gh run view job/step prefixes #1053 fix(opencode): split review surfaces, give NIM two hours, and remove GitHub Models #1052). 이 lane 산출물 아님.main=8dc74692(2026-08-20 이후 0 병합, ~15일). 어떤 pg-erd-cloud PR에도opencode-review디스패치 없음. fix(opencode): execute asyncio helper on hash-locked quality CI #1059·#1046은 merge 없이 closed..githubqueued (이 세션 관측): 1781 → 1748 → 1629 → 1547 → 1601 → 1668. 진동, 결정적 하락 아님.이 lane의 다음 행동
opencode-review디스패치 또는 첫 pg-erd-cloud 병합이 관측되면 dependency 순서 merge wave 실행.seonghobae commentedon Sep 3, 2026
Fresh ConceptWeave owner-path reproduction separates current review-continuation provenance from leaf source and from runner acquisition.
ContextualWisdomLab/ConceptWeave#1@bba351b77bf5f1ab5cfd55979fbb2bd158f78b81Required OpenCode run33527145720, attempt 1, job100426495473acquired GitHub-hosted runner1001638434, passed required-workflow admission/coverage, and itsRequest OpenCode review jobstep succeeded. Noopencode-agentformal review materialized. The exact-head verdict waiter then occupied the runner from2026-09-03T10:22:37Zuntil GitHub cancellation at2026-09-03T16:17:30Z(~5h55m). This is central review continuation/runtime evidence, not a ConceptWeave source finding and not runner-acquisition starvation.Protected
.github/main@72f63c9e32194512fd5358ba4bff6ac4365be8d7now includes the merged #1707 contract:opencode-review.ymlfails closed after 180 minutes (poll_deadline_epoch += 10800) and releases the runner. A single bounded rerun of the historical ConceptWeave run was attempted after central source changed, but attempt 2 proves why old-run reruns are not current-source acceptance:opencode-reviewjob100733416054is queued onubuntu-24.04, while replayed bootstrap/coverage jobs retain predecessorubuntu-latestselectors; current protectedopencode-review.ymlcontains noubuntu-latestselector. Do not repeatedly rerun this historical run and do not require a no-op leaf push/draft toggle solely to retrigger.Owner acceptance for this class should require a legitimately fresh required-workflow materialization on the unchanged leaf head, bound to the then-current protected central workflow identity, runner assignment + exact-head admission, successful review dispatch, and an actual current-head
opencode-agentformal review before the bounded waiter expires. A replay of a predecessor required-workflow revision is diagnostic evidence only, never GREEN.seonghobae commentedon Sep 3, 2026
Naruon adds a current consumer reproduction of the missing-verdict failure class.
ContextualWisdomLab/naruon#1497152d1998c4e8024be9dc7026c8789d343c884fd0opencode-reviewrun/job/check:33462446296/99830468507The required job successfully obtained GitHub OIDC, exchanged the repository-scoped App token and dispatched the central
merge-schedulerrequest. It then polled the Reviews API for an exact-headopencode-agentAPPROVEDorCHANGES_REQUESTEDverdict. No qualifying verdict arrived during the full fail-closed polling window, so the check terminated failure withNo APPROVED or CHANGES_REQUESTED from opencode-agent on the current head.This is not a Naruon source finding. The consumer PR has been returned to Draft rather than receiving a dummy commit, bypass, synthetic status or predecessor review. A valid recovery requires a fresh exact-head review dispatch through current protected central workflow truth after the owner-side queue/dispatch path is healthy.
docs(doctoring): mark the org-queue-sweep rotation record historical
seonghobae commentedon Sep 14, 2026
TEPP measurement: the scheduler is not the queue driver here
Measured on
ContextualWisdomLab/TEPPatorigin/main(a243f18d) on 2026-09-15, viagh api repos/ContextualWisdomLab/TEPP/actions/runs.Queued runs by trigger event, right now:
pull_requestschedulepushworkflow_dispatchEvery queued run in TEPP is pull-request-triggered. Nothing in the backlog comes from a scheduled workflow.
Why zero: TEPP already carries no
scheduletrigger in any workflow.git grep -l 'schedule:' origin/main -- .github/workflowsreturns 0 files. The last one was removed by #490 when hourly development admission moved to the central coordinator. The historical count of 795scheduleruns is entirely pre-#490.Lifetime run counts for context:
pull_request16377,schedule795,push463,workflow_dispatch18. Pull-request events are roughly 93% of all runs TEPP has ever produced, and 100% of what is queued.Implication for this issue
The title attributes saturation to "120 open PRs + self-amplifying scheduler". For TEPP the first half holds and the second does not. TEPP currently has 100+ open PRs, each fanning out to roughly ten required checks. Removing or centralising more schedules will not move this backlog, because the schedule contribution is already zero. Draining open PRs, or reducing per-PR check fan-out, is the lever that applies.
I am not asserting this generalises to the other org repositories. It is one repository's measurement, offered so that scheduler-side work is not sized against TEPP's share of the queue.
Related repair
#490 removed TEPP's local
scheduletrigger but left the paired contract test assertingcron: "47 * * * *", so TEPP's shared Python branch-coverage job has failed on every main-based branch since that merge. Repaired in ContextualWisdomLab/TEPP#517, which realigns the assertion to the property #490 actually established: absence of a localscheduletrigger, with the central coordinator owning cadence viaworkflow_dispatch. No workflow behaviour changes.That coupling shape is worth noting for other repositories: a contract test asserting a literal cron string breaks the moment central admission removes the local schedule. TEPP now has zero literal cron assertions remaining.
🤖 Generated with Claude Code
seonghobae commentedon Sep 14, 2026
Second repository measured, same conclusion as the TEPP comment above — and a correction to a hypothesis I stated to another session earlier.
ContextualWisdomLab/contextual-orchestrator, measured 2026-09-15 by event type:pull_requestpull_request_targetschedulepushworkflow_dispatchrepository_dispatchPull-request events are 12 of 15 queued runs and about 96% of cumulative runs;
scheduleis 2 queued and roughly 1% cumulative. I had suggested to another session that this repository's hourly loop contributes meaningfully to the pressure. That was a hypothesis and the measurement does not support it: retiring hourly schedules here would remove 2 of 15 queued runs and change nothing about the backlog.The lever is the same one the TEPP evidence points at: open-PR count and per-PR check fan-out. Note this repository's own local queue is 15 while the organization-wide queued total was 228-230 when I sampled it yesterday, so saturation is cross-repository and no single repository's schedules explain it.
One consequence worth recording here, because it makes the backlog self-sustaining for review-gated repositories: a verdict comment posted today can describe a run created long before a since-merged repair. Concretely,
ContextualWisdomLab/.githubrun 34784851740 was created 2026-09-13T21:47:07Z and itscoverage-evidencejob still shows the pre-repairlocked VCS source fast-mlsirm has a missing or ambiguous import rootfailure; the repair (#2123) merged at 23:37:49Z, and the resultingCOVERAGE_BLOCKEDcomment landed on contextual-orchestrator#1177 at 2026-09-14T09:25Z, about 11.5 hours after its run started. PRs cannot merge without a current verdict, verdicts arrive stale, so open PRs accumulate and add fan-out. Anyone reading a verdict during the backlog should compare the dispatch run'screated_atagainst the relevant repair merge time before treating it as a regression.🤖 Generated with Claude Code
seonghobae commentedon Sep 14, 2026
Repeating the cancellation measurement in
ContextualWisdomLab/contextual-orchestrator, the same one that localised TEPP's waste to a single draft branch. The pattern reproduces here, but the cause is different, and the largest contributor is my own session.Measured 2026-09-14T19:12Z, last 100 completed runs:
Cancellations by head branch:
feat/message-count-provenance-1157(PR #1188)pr/canonical-immutable-release-current-main(PR #1186)refs/pull/1188/headfix/reference-cases-terminology-1015Half of all recently completed runs were cancelled, and 27 of those 50 belong to one branch I own. So the mechanism the TEPP evidence identified — superseded runs discarded because a head moves faster than CI drains — holds here too.
The difference matters for the fix. PR #1188 is not a draft, so a draft guard would not have covered it. Its head moved four times in about three hours (
cc09e6f003:28Z,ac63794c04:01Z,50b0fec906:14Z,cceef95f06:26Z) because I pushed each reviewed unit as it completed. Every push cancelled the prior run's work. Nothing could merge during that window anyway: the verdict queue was backed up, and the verdict that did arrive described a run created before the repair it was meant to validate (noted in my earlier comment).So there are two distinct sources of discarded runner time, and they need different remedies:
Timestamp note: my earlier comment in this issue said "measured 2026-09-15". That reading was KST; in UTC it was 2026-09-14, the same day as the TEPP measurements. The numbers and conclusions are unchanged. Using UTC from here on.
🤖 Generated with Claude Code
seonghobae commentedon Sep 14, 2026
Correcting my earlier comment: the lever is not open-PR count
Earlier in this thread I measured TEPP's queue and concluded that draining open PRs, or reducing per-PR check fan-out, was the lever. That was incomplete. I had measured what was queued but not what happened to work that actually ran. Measuring completions changes the conclusion.
All timestamps UTC. My earlier comment dated its measurement 2026-09-15; that was KST. Same day as this one.
Throughput, not capacity
At 2026-09-14T19:07Z,
ContextualWisdomLab/TEPP:Conclusions of the 100 most recently completed runs, spanning 14:17Z to 19:06Z:
Six minutes later, at 19:13Z, the same query returned 100 cancelled out of 100, with zero successes.
In both windows every single cancellation belonged to one branch,
codex/rubin-loading-projection-policy, the draft TEPP#506. The concentration rose rather than stabilised.Runners are not scarce. From run
34884569907: created 19:03:33Z, jobs started 19:03:35Z, cancelled 19:06:45Z. Work is picked up in two seconds and discarded three minutes in, having occupied four runners. That branch carries 69 commits over about ten hours, with three head commits inside four minutes at one point.Queued work is spread evenly at nine runs per branch across many branches, so the queue itself is fair. The saturation is not many PRs competing. It is one branch consuming and discarding the throughput.
Two sources, only one fixable in CI
A peer session reproduced this on
contextual-orchestratorand found the same waste with a different cause: their largest contributor is not a draft, and its head moved four times in three hours because each finished review slice was pushed immediately. So:ready_for_reviewso the full gate set runs on the exact head that will merge.contextual-orchestratoralready had this insecurity.yml, which is why no draft appears in its cancellation profile.Circularity worth recording
TEPP#523 is the fix for the first source, and it is itself queued behind the branch it would neutralise. With recent throughput at zero successes, that does not resolve on its own. I am recording the evidence here rather than acting on it: two measurement windows, rising concentration, two-second pickup against three-minute cancellation, and the per-branch queue distribution. Batching pushes on the affected branch would release the throughput immediately and costs nothing, because no intermediate head on a draft can merge.
I have not modified that branch or any other author's work.
seonghobae commentedon Sep 15, 2026
A third source of consumed runner time, distinct from the two already measured here (draft re-push cancellations, ready-PR iteration): a single hung job can hold a runner for hours.
ContextualWisdomLab/contextual-orchestratorPR #1179,noema-reviewrun 34811048383: created 2026-09-14T15:46:01Z, cancelled 2026-09-14T21:46:44Z — six hours on one runner. The log ends atstarting review sidecar on 127.0.0.1:18080with nohealthz and provider-route preflight confirmedline and zeroprovider_attemptentries, so the sidecar never became healthy and no request reached a provider.For scale, this one job consumed more runner time than the entire cancelled-rerun pattern measured in contextual-orchestrator earlier: 50 cancelled runs there were each a few minutes of work.
Comparable runs on the same pinned revision (
767e67fb) that did not hang, for contrast:HTTP Error 429after preflightHTTP Error 502So this is a distinct mode, not the free-pool rate-limit path: those fail in minutes. Whatever stalls before healthz has no overall deadline, so the job runs until the platform cancels it.
Recording it here because it changes the shape of the remedy: the cancellation and batching levers reduce how often work is discarded, while this one is about a single job's unbounded occupancy. A per-job timeout on the review lanes would bound it regardless of the underlying cause, and is independent of whatever fix the stall itself needs (tracked for the noema lane in ContextualWisdomLab/contextual-orchestrator#1106).
🤖 Generated with Claude Code
seonghobae commentedon Sep 19, 2026
2026-09-19 fresh queue-saturation evidence from two metadata/control-plane PRs after their Ready transitions:
Canonical runtime owner #2265 documents the org plan ceiling and unbounded Strix occupancy. #2267's higher scheduled dispatch caps must be evaluated against this live saturation so recovery does not amplify the queue before occupancy is bounded.
seonghobae commentedon Sep 19, 2026
2026-09-19 additional exact-head queue evidence:
.github#2110headb360990b42380169d1bccc9f4b2dc83b7336bc63created five ordinary pull-request runs at 11:07:01Z:3543922240735439222409354392224123543922244135439222460queuedwith no conclusion; their last recorded updates are 11:07:02–11:07:03Z.github#2124heade9acd81064408f0c49b72d7a617c747c96f21244still has four additional pull-request runs35433647206/212/221/237queued since 09:03:22–23ZThis extends the existing organization runner-capacity finding; source owners remain Draft/Ready according to their code state, and current-head settlement is still fail-closed.
seonghobae commentedon Sep 19, 2026
2026-09-19 terminal cancellation increment
Fresh readback shows the previously queued same-repository runs have now terminated as
cancelledwithout source execution.5d0969f53f2ff944c139908efe9e133778e71231: Repository Metadata Reconcile35441664992, CodeQL PR35441664978, Python Security35441664896, SAST Semgrep35441664907, Security Scan35441664995e9acd81064408f0c49b72d7a617c747c96f21244: SAST Semgrep35433647206, Python Security35433647212, CodeQL PR35433647221, Security Scan35433647237Every exposed job is terminal
cancelledand haslogs_url=null; there is no source/test failure log to repair. This is the terminal continuation of the prior prolonged queued/capacity evidence, not GREEN settlement. No duplicate rerun or no-op wake was issued.The metadata PR remains Draft. The sidecar PR retains its earlier same-head GREEN source checks but has no independent current-head approval; the cancelled review-cycle materialization is not treated as approval or fresh settlement.
seonghobae commentedon Sep 19, 2026
2026-09-19 central-owner exact-head extension
Fresh readback adds two owner PRs to the same queue/cancellation incident:
.github#2040@ecc9e1d11149ae44ec4f8389e4ac72a08ba45ba7: Security Scan35436310648, CodeQL PR35436310711, Trusted uv Materializer Quality CI35436310573, Python Security35436310584, Agent Review Runtime Quality CI35436310634, and SAST Semgrep35436310640are all terminalcancelled. The exposed CodeQL jobs are cancelled and provide no source-failure log..github#2279@d1e4380c15e948aaf104d46aa134fa614058782a: the first Security/Python Security/CodeQL/SAST batch35438679364/344/363/346is terminalcancelled; the current Python Security35438799959is pending and CodeQL/SAST/Security/Runtime Quality35438799890/35438800012/35438799983/35438679390remain queued.This is not proof that either source is GREEN. It also gives no leaf defect to patch. No unchanged-head rerun, no-op commit, or bypass was issued. #2040 stays Draft; #2279 remains review-admitted but cannot merge until terminal exact-head evidence exists.