Repository navigation
fix(explorer): stop shedding routine Explorer reads as 503 Explorer busy - #1800
Draft
danieljrc888 wants to merge 1 commit into
Draft
danieljrc888 wants to merge 1 commit into
danieljrc888 wants to merge 1 commit into
Conversation
…busy" Since the #1766 limiter shipped (2026-09-14), production rejects 2-5% of Explorer reads per endpoint (~23k Sentry events/day) because each jsonrpc pod admits only 2 concurrent Explorer queries and rejects the rest instantly, while one homepage render fires /stats, /stats/counts and /transactions?limit=10 at once and fetchBackend never retries. - ExplorerQueryRunner waits up to 1s for a slot. Waiters are capped at 8 per process so blocked workers cannot starve the API threadpool; overflow still sheds immediately. Query concurrency (and DB pool pressure) is unchanged. - /stats gets the same coalesced TTL cache as /stats/counts (10s). - The homepage recent-transactions section reuses stats.recentTransactions instead of /transactions?limit=10, which was 99.99% of /transactions traffic and ran an unfiltered count(*) on every render. - fetchBackend retries 503 up to twice, honouring Retry-After (capped at 1s). - 503s raise ExplorerBusyError, which Sentry ignores; traces still mark them unavailable, so the reject rate stays measurable. Fixes GENLAYER-STUDIO-V1K Fixes GENLAYER-STUDIO-V1M Fixes GENLAYER-STUDIO-V1N
Contributor
|
Important Draft PR not reviewedDraft PRs are not automatically reviewed by default.
To automatically review draft PRs, update your CodeRabbit configuration: reviews:
auto_review:
drafts: true
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.



Fixes GENLAYER-STUDIO-V1K, GENLAYER-STUDIO-V1M, GENLAYER-STUDIO-V1N (plus the smaller V1V/V1T/VAD/XMA/V2J with the same message).
What
ExplorerQueryRunnerwaits up to 1s for one of its 2 query slots instead of rejecting instantly. Waiters are capped at 8 per process (admission semaphore), so blocked AnyIO workers cannot starve the API threadpool; overflow still sheds immediately with 503. Query concurrency, and so DB pool pressure, is unchanged./statsgets the same coalesced TTL cache as/stats/counts(10s; counts stays 5s). A cold-cache caller now waits for an in-flight refresh instead of getting 503.stats.recentTransactions(already computed by/stats, previously unused) instead of calling/transactions?limit=10.fetchBackendretries 503 up to twice, honouringRetry-After(capped at 1s).ExplorerBusyError, listed in Sentryignore_errors.Why
Production has been rejecting Explorer reads continuously since the #1766 limiter was promoted on 2026-09-14 (last 24h:
/transactions4.7%,/stats3.5%,/stats/counts2.1% of requests). Each jsonrpc pod admits 2 concurrent Explorer queries (4 replicas in studio-prd = 8 total) and rejects the rest instantly, while every homepage SSR fires 3 backend calls at once with no retry. Reproduced on prod: 6 parallel/statsrequests give one immediate 503./transactions?limit=10from the homepage is 99.99% of/transactionstraffic (180k/day) and runs an unfilteredcount(*)each time; it also accounts for 8,665 of the 8,845 daily/transactionsrejections.Testing done
tests/unit/test_explorer_query_runner.py: 25 passed. New cases: waiting acquires a freed slot, wait timeout returns 503 withRetry-After, waiter overflow rejects immediately, cold-cache caller waits for the in-flight refresh, and the TTL/no-shared-mutation test now runs for both counts and stats. Tests that assert instant rejection passwait_timeout=0.tsc --noEmitpasses.eslintcrashes on any file (react/display-name: contextOrFilename.getFilename is not a function, eslint 10 vs eslint-plugin-react); this is pre-existing and reproduces on unchanged files.Decisions made
ExplorerBusyErroris dropped from Sentry errors only. Traces keepspan.status:unavailable, so the reject rate stays measurable.v0.123-devdoes not carry fix(rpc): prevent Studio database-pool starvation (v0.121 hotfix) #1766 at all, so this only applies tov0.121. Forward-porting fix(rpc): prevent Studio database-pool starvation (v0.121 hotfix) #1766 is a separate concern.Checks
Reviewing tips
Start with
backend/protocol_rpc/explorer/query_runner.py:run()(admission, then a timed slot wait) and_cached(), which generalises the existing counts cache. After deploy, verify with the Sentry traces querytransaction:/api/explorer/* span.status:unavailable. It should drop to near zero, and/transactions?limit=10traffic should disappear.Rollback: revert this commit. No schema, config, or infra changes.
User facing release notes
The Explorer no longer intermittently fails to load statistics or recent transactions with "Explorer busy" under normal traffic.