Skip to content

fix(explorer): stop shedding routine Explorer reads as 503 Explorer busy - #1800

Draft
danieljrc888 wants to merge 1 commit into
v0.121from
fix/v0.121-explorer-busy
Draft

danieljrc888 wants to merge 1 commit into
v0.121from
fix/v0.121-explorer-busy

Conversation

@danieljrc888

Copy link
Copy Markdown
Contributor

Fixes GENLAYER-STUDIO-V1K, GENLAYER-STUDIO-V1M, GENLAYER-STUDIO-V1N (plus the smaller V1V/V1T/VAD/XMA/V2J with the same message).

What

  • ExplorerQueryRunner waits up to 1s for one of its 2 query slots instead of rejecting instantly. Waiters are capped at 8 per process (admission semaphore), so blocked AnyIO workers cannot starve the API threadpool; overflow still sheds immediately with 503. Query concurrency, and so DB pool pressure, is unchanged.
  • /stats gets the same coalesced TTL cache as /stats/counts (10s; counts stays 5s). A cold-cache caller now waits for an in-flight refresh instead of getting 503.
  • Explorer homepage "Recent Transactions" reuses stats.recentTransactions (already computed by /stats, previously unused) instead of calling /transactions?limit=10.
  • Explorer fetchBackend retries 503 up to twice, honouring Retry-After (capped at 1s).
  • Load-shedding 503s raise ExplorerBusyError, listed in Sentry ignore_errors.

Why

Production has been rejecting Explorer reads continuously since the #1766 limiter was promoted on 2026-09-14 (last 24h: /transactions 4.7%, /stats 3.5%, /stats/counts 2.1% of requests). Each jsonrpc pod admits 2 concurrent Explorer queries (4 replicas in studio-prd = 8 total) and rejects the rest instantly, while every homepage SSR fires 3 backend calls at once with no retry. Reproduced on prod: 6 parallel /stats requests give one immediate 503.

  • /transactions?limit=10 from the homepage is 99.99% of /transactions traffic (180k/day) and runs an unfiltered count(*) each time; it also accounts for 8,665 of the 8,845 daily /transactions rejections.
  • The handled 503s are ~23k Sentry events/day, roughly the project's whole accepted error volume. They exhausted the Sentry quota between 2026-09-27 and 2026-10-12, which made the problem look resolved while it was not, and they hide real errors.

Testing done

  • tests/unit/test_explorer_query_runner.py: 25 passed. New cases: waiting acquires a freed slot, wait timeout returns 503 with Retry-After, waiter overflow rejects immediately, cold-cache caller waits for the in-flight refresh, and the TTL/no-shared-mutation test now runs for both counts and stats. Tests that assert instant rejection pass wait_timeout=0.
  • Backend unit suite: 871 passed, 7 skipped.
  • Explorer tsc --noEmit passes. eslint crashes on any file (react/display-name: contextOrFilename.getFilename is not a function, eslint 10 vs eslint-plugin-react); this is pre-existing and reproduces on unchanged files.
  • Pre-commit (black, autoflake) passes.

Decisions made

Checks

  • I have tested this code
  • I have reviewed my own PR
  • I have created an issue for this PR (tracked in Sentry, linked above)
  • I have set a descriptive PR title compliant with conventional commits

Reviewing tips

Start with backend/protocol_rpc/explorer/query_runner.py: run() (admission, then a timed slot wait) and _cached(), which generalises the existing counts cache. After deploy, verify with the Sentry traces query transaction:/api/explorer/* span.status:unavailable. It should drop to near zero, and /transactions?limit=10 traffic should disappear.

Rollback: revert this commit. No schema, config, or infra changes.

User facing release notes

The Explorer no longer intermittently fails to load statistics or recent transactions with "Explorer busy" under normal traffic.

…busy"

Since the #1766 limiter shipped (2026-09-14), production rejects 2-5% of
Explorer reads per endpoint (~23k Sentry events/day) because each jsonrpc pod
admits only 2 concurrent Explorer queries and rejects the rest instantly,
while one homepage render fires /stats, /stats/counts and
/transactions?limit=10 at once and fetchBackend never retries.

- ExplorerQueryRunner waits up to 1s for a slot. Waiters are capped at 8 per
  process so blocked workers cannot starve the API threadpool; overflow still
  sheds immediately. Query concurrency (and DB pool pressure) is unchanged.
- /stats gets the same coalesced TTL cache as /stats/counts (10s).
- The homepage recent-transactions section reuses stats.recentTransactions
  instead of /transactions?limit=10, which was 99.99% of /transactions
  traffic and ran an unfiltered count(*) on every render.
- fetchBackend retries 503 up to twice, honouring Retry-After (capped at 1s).
- 503s raise ExplorerBusyError, which Sentry ignores; traces still mark them
  unavailable, so the reject rate stays measurable.

Fixes GENLAYER-STUDIO-V1K
Fixes GENLAYER-STUDIO-V1M
Fixes GENLAYER-STUDIO-V1N
@danieljrc888 danieljrc888 self-assigned this Oct 26, 2026
@coderabbitai

coderabbitai Bot commented Oct 26, 2026

Copy link
Copy Markdown
Contributor

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

  • Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@sonarqubecloud

Copy link
Copy Markdown

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant