tower: overview cache burns the user's entire GitHub GraphQL quota (3 gh calls × N workspaces every 30s, failures uncached) — every gh command on the machine then fails

[waleedkadous]

Symptom (2026-09-07/08, this machine)

Every gh GraphQL call on the account — gh pr view/list/checks, gh issue view, consult PR search, afx spawn's issue fetch — fails with API rate limit already exceeded for user ID … for most of every hour, while gh api rate_limit misleadingly reports 5000/5000 (the response headers show X-RateLimit-Resource: graphql, Used: 5000, Remaining: 0). Builders cannot run CMAP PR reviews, the architect cannot merge via gh pr merge, and the Tower dashboard/cloud UI "times out" because overview requests wait on hanging gh spawns.

Cause, measured

OverviewCache (servers/overview.ts, TTL 30 s) backs each workspace's overview with three GraphQL-backed CLI calls — gh issue list --limit 200 --json …, gh issue list --state closed --search closed:>…, gh pr list --state merged --search merged:>… — via executeForgeCommand. With 13 active workspaces polled (cloud UI + dashboard + VS Code extension every 60 s) the steady state is 13 × 3 × (3600/30) ≈ 4,700 GraphQL calls/hour: the whole 5,000-point budget, forever.

Two amplifiers once the limit is hit:

1. Failures are not cached. executeForgeCommand returns null on a non-zero exit (lib/forge.ts:336/352) and the cache only stores non-null (if (data !== null) cache.set(...)), so every overview request re-spawns all three commands immediately. Observed: 51 gh spawns from the Tower daemon in 40 s (≈4,600/h) — faster than the TTL allows.
2. No rate-limit awareness. Nothing parses the rate-limit error or the reset header; Tower keeps hammering until the hour rolls over, and the first requests after reset re-exhaust it.

Cost per call is also higher than 1 point: --limit 200 with author/assignees/labels expansions is a multi-hundred-node GraphQL query.

Fix (BUGFIX, prescribed)

1. Negative cache + backoff in OverviewCache: a failed forge fetch stores a null result with its own TTL (start 60 s, double to 15 min) so a failing workspace costs one spawn per window, not one per request.
2. Rate-limit awareness: detect the GraphQL rate-limit error (stderr rate limit already exceeded or graphql_rate_limit), read X-RateLimit-Reset via a single gh api rate_limit --jq .resources.graphql, and suspend ALL GitHub-backed overview fetches until that instant. Surface it in the overview payload (forgeStatus: 'rate-limited', resetAt) so the dashboard shows why data is stale instead of timing out.
3. Raise the positive TTL to 120 s and refresh only workspaces with a live SSE client or an overview request in the last TTL window; a workspace nobody is looking at must cost zero calls.
4. Collapse the three calls into one GraphQL query (gh api graphql) per workspace, or move the two --search calls to REST (gh api search/issues, separate 5,000 budget). One request per workspace per window, budgeted at ≤ 1/13 of the hourly quota for 13 workspaces.
5. Doctor check: codev doctor reports the GraphQL budget (used/limit, reset time) and the projected hourly spend N_workspaces × calls × 3600/TTL, warning when the projection exceeds 50 % of the limit.
6. Acceptance: with 13 workspaces open and the dashboard polling, the Tower daemon spawns ≤ 60 gh processes per hour at steady state (count children of the Tower pid over 10 min); a forced rate-limit error produces zero further spawns until reset; full suite green.

Refs: Spec 0126 (overview cache), #1641 (consult also blocked by the exhausted quota), #1629 (same week's Tower incident).

[waleedkadous]

Measured sample, 60 s of the Tower daemon's gh children (2026-09-08 ~05:50Z, while rate-limited):

workspace (cwd)	gh spawns / 60 s
hobbies	33
codev	6
taqwabench	3
writing	3
ansari-frontend	2
exited before cwd could be read	31
total	78 (≈ 4,700/h)

Two things the table shows: the fan-out is fleet-wide (five workspaces in one minute), and one workspace whose overview is being requested every 2–3 s dominates — exactly the uncached-failure amplifier in the body (every request re-spawns the triple while the quota is exhausted). Fix items 1 and 2 stop the amplifier; 3 and 4 bring the steady state under budget.

[waleedkadous]

Correction to the fix text, item 4.

Moving the two --search calls to REST gh api search/issues for "a separate 5,000
budget" would not help. The GitHub search resource is rate-limited at 30 requests
per minute, not 5,000 per hour — measured on this account while the GraphQL budget was
exhausted:

$ gh api rate_limit --jq .resources.search
{"limit":30,"remaining":30,"reset":1788846476,"used":0}

Collapsing into one gh api graphql query is the right direction; REST search is not.
That collapse is out of scope for this bugfix (new forge concept, provider scripts,
contract types, fallbacks) and is filed as #1647.

Also worth recording: the gh api rate_limit REST endpoint misreports the GraphQL
bucket. While gh api graphql -i returned X-Ratelimit-Used: 5000, X-Ratelimit-Remaining: 0, gh api rate_limit --jq .resources.graphql reported
{"limit":5000,"remaining":5000,"used":0} with a reset instant ~45 minutes later than
the real one. PR #1646 therefore treats that probe as advisory — it trusts the reported
reset only when the probe agrees the budget is gone, and otherwise falls back to a
doubling 60 s → 15 min backoff.

[waleedkadous]

Acceptance criteria, amended (architect decision, 2026-09-08)

The ≤ 60 gh processes/hour across 13 workspaces number in the fix section above was
written against a design that item 4 would deliver — one collapsed GraphQL call per
workspace per window. Item 4 is out of this issue's PR and is tracked as #1647, so that
number belongs there, not here.

This issue's acceptance is therefore:

1. Zero further gh spawns while rate-limited, until reset. A forced rate-limit error
   produces one batch and then nothing until the suspension lifts.
2. ≤ 50 % of the GraphQL budget at 13 watched workspaces. At the shipped 180 s
   positive TTL / 600 s search TTL this is 52 calls/h per watched workspace ≈ 2,028
   points/h at ~3 points/call — 41 % of 5,000/h.
3. Single-flight, so a slow or hanging gh can never fan out across polls or clients.

#1647 owns the one-call-per-workspace goal and the ≤ 60/h-across-13 number that
depends on it.

For the record, the pre-fix baseline measured read-only on the production Tower
(pgrep -P <tower-pid> at 2 Hz for 10 minutes): 1,515 distinct gh processes ≈
9,090/hour.