Repository navigation
fix(health): detect head-of-queue stalls, not just queue depth - #1621
Conversation
The "orphaned_transactions" metric exposed by /health was producing
false-positive degraded alerts on instances with bursty traffic and
slow per-contract serialization. The dashboard would alert "There are
N pending transactions" even when consensus was healthy, just slowly
draining a backlog.
Two bugs in the previous _check_consensus_health:
1. active_workers was derived from "txs created in the last 1h with
worker_id set". Workers processing OLD txs (a 6h-old head with a
freshly-claimed worker) were invisible. With bursty traffic and no
new tx in the last hour, the query returned zero, falsely
declaring all in-flight work orphaned.
2. total_orphaned_transactions semantics mixed two unrelated things —
"queue depth" and "no workers alive". A long queue with a moving
head is not a problem; the head being stuck is the problem
regardless of queue depth.
New logic, matching the spec from the discussion:
- A contract's head is the oldest non-final tx for that contract.
- The head is "stuck" when:
a) it was created > HEAD_STUCK_AFTER_MINUTES ago, AND
b) no tx for that contract has a fresh blocked_at within the
RECENT_ACTIVITY_WINDOW_MINUTES window (no work in flight).
- total_orphaned_transactions now counts the number of CONTRACTS in
this state. Field name preserved for backwards-compat with the
external dashboard, but semantics changed: long queues with active
heads now read 0.
- active_workers counts distinct worker_ids holding a fresh blocked_at,
regardless of how old the underlying tx is.
Tests cover: long-queue-with-active-head reads 0, stuck head reads 1,
recent head doesn't trip, workers on 6h-old txs count as active,
multi-contract scenarios only flag the genuinely-stuck contract.
This is the metric driving the public Studio Health dashboard. The
existing /health/consensus endpoint still exposes per-contract detail
unchanged.
📝 WalkthroughWalkthroughThe consensus health check is rewritten to provide per-contract diagnostics. A new nested SQL query collects per-contract metrics including processing/pending transaction counts, time-bucket tallies, and oldest transaction ages. Results aggregate across contracts to determine overall health status and orphaned transaction counts. A new test suite validates the per-contract logic across multiple scenarios including stuck heads, expired claims, and threshold boundary conditions. ChangesConsensus Health Per-Contract Rewrite
Sequence DiagramsequenceDiagram
participant API as Health Check<br/>Endpoint
participant HC as _check_consensus<br/>_health()
participant DB as Database
participant Agg as Aggregation<br/>Logic
API->>HC: Request consensus health
HC->>DB: Query active workers<br/>from recent transactions
DB-->>HC: Worker list
HC->>DB: Query per-contract metrics<br/>(nested: processing, pending,<br/>time-buckets, oldest ages)
DB-->>HC: Per-contract rows
HC->>Agg: Build contracts array<br/>with metrics
Agg-->>HC: Contracts with<br/>orphaned counts
HC->>Agg: Aggregate totals &<br/>compute status
Agg-->>HC: status + totals
HC-->>API: Return health response<br/>(status, total_processing,<br/>total_orphaned, contracts[])
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~25 minutes Possibly related PRs
Suggested labels
Suggested reviewers
Poem
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
Actionable comments posted: 3
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@backend/protocol_rpc/health.py`:
- Around line 528-539: The stuck-head detector currently omits PENDING because
IN_FLIGHT_STATUSES_SQL excludes it, so oldest non-final txes in PENDING don't
count toward total_orphaned_transactions; fix by introducing a separate status
constant (e.g. STUCK_HEAD_STATUSES_SQL or HEAD_NON_FINAL_STATUSES) that includes
'PENDING' while keeping IN_FLIGHT_STATUSES_SQL as-is if you want processing
counts narrower, then update the head-CTE filter, the stuck-head query (where
total_orphaned_transactions is computed), and any references at the spots noted
(including uses around lines referenced and functions/queries that reference
IN_FLIGHT_STATUSES_SQL) to use the new STUCK_HEAD_STATUSES_SQL so the detector
treats PENDING as non-final head-of-queue.
- Around line 528-530: _top-level health downgrade uses a hard-coded ">= 3"
while you introduced DEGRADED_AT_STUCK_HEADS, causing services.consensus.status
(from _check_consensus_health) to disagree with the top-level /health status and
issues; update _run_health_checks() to use the DEGRADED_AT_STUCK_HEADS constant
instead of the literal 3 so the same threshold drives both the consensus check
and the top-level status/issues logic (look for DEGRADED_AT_STUCK_HEADS,
_check_consensus_health(), and _run_health_checks() to make the replacement and
ensure the comparison semantics remain the same).
In `@tests/db-sqlalchemy/test_health_orphan_detection.py`:
- Around line 42-59: The autouse fixture _wire_health_module_to_test_engine
should pin the environment keys HEALTH_HEAD_STUCK_AFTER_MINUTES,
TRANSACTION_TIMEOUT_MINUTES, and HEALTH_DEGRADED_AT_STUCK_HEADS so
_check_consensus_health (which reads os.environ each call) is deterministic;
modify the fixture to save os.environ.get(...) for those three keys, set them to
the canonical test values you expect before yield (using os.environ[...] =
"..."), and restore the previous values (or delete if previously absent) after
the yield while keeping the existing session_factory/_rpc_router_ref setup and
teardown.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: 8ebf73f3-5242-40d3-80c8-0f59bc059133
📒 Files selected for processing (2)
backend/protocol_rpc/health.pytests/db-sqlalchemy/test_health_orphan_detection.py
| # Match the dashboard alert threshold so consensus.status and the | ||
| # top-level "issues" tag agree on what counts as degraded. | ||
| DEGRADED_AT_STUCK_HEADS = int(os.environ.get("HEALTH_DEGRADED_AT_STUCK_HEADS", "3")) | ||
|
|
||
| # Statuses considered "in flight". Includes the *_TIMEOUT variants | ||
| # because finalization and appeal claim paths process them. Must | ||
| # match the head-CTE filter and the in-flight-count query for | ||
| # consistency. | ||
| IN_FLIGHT_STATUSES_SQL = ( | ||
| "'ACTIVATED','PROPOSING','COMMITTING','REVEALING'," | ||
| "'ACCEPTED','UNDETERMINED','LEADER_TIMEOUT','VALIDATORS_TIMEOUT'" | ||
| ) |
There was a problem hiding this comment.
PENDING heads are currently invisible to the stuck-head detector.
The docstring here defines the head as the oldest non-final tx, but the shared status set excludes PENDING. That means a contract whose oldest tx has been sitting in PENDING past the stuck threshold with no claim will never contribute to total_orphaned_transactions, which misses one real head-of-queue stall mode. If you want total_processing_transactions to stay narrower than the head detector, this probably needs separate status sets instead of one shared constant.
Also applies to: 579-600, 615-620
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@backend/protocol_rpc/health.py` around lines 528 - 539, The stuck-head
detector currently omits PENDING because IN_FLIGHT_STATUSES_SQL excludes it, so
oldest non-final txes in PENDING don't count toward total_orphaned_transactions;
fix by introducing a separate status constant (e.g. STUCK_HEAD_STATUSES_SQL or
HEAD_NON_FINAL_STATUSES) that includes 'PENDING' while keeping
IN_FLIGHT_STATUSES_SQL as-is if you want processing counts narrower, then update
the head-CTE filter, the stuck-head query (where total_orphaned_transactions is
computed), and any references at the spots noted (including uses around lines
referenced and functions/queries that reference IN_FLIGHT_STATUSES_SQL) to use
the new STUCK_HEAD_STATUSES_SQL so the detector treats PENDING as non-final
head-of-queue.
| # Match the dashboard alert threshold so consensus.status and the | ||
| # top-level "issues" tag agree on what counts as degraded. | ||
| DEGRADED_AT_STUCK_HEADS = int(os.environ.get("HEALTH_DEGRADED_AT_STUCK_HEADS", "3")) |
There was a problem hiding this comment.
DEGRADED_AT_STUCK_HEADS doesn’t fully align /health yet.
This new threshold only affects _check_consensus_health(). _run_health_checks() still hard-codes >= 3 when it downgrades the top-level /health payload, so any env override will make services.consensus.status disagree with top-level status/issues.
Suggested fix
- elif consensus_health.get("total_orphaned_transactions", 0) >= 3:
+ elif consensus_status == "degraded":
if overall_status == "healthy":
overall_status = "degraded"
issues.append("orphaned_transactions")🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@backend/protocol_rpc/health.py` around lines 528 - 530, _top-level health
downgrade uses a hard-coded ">= 3" while you introduced DEGRADED_AT_STUCK_HEADS,
causing services.consensus.status (from _check_consensus_health) to disagree
with the top-level /health status and issues; update _run_health_checks() to use
the DEGRADED_AT_STUCK_HEADS constant instead of the literal 3 so the same
threshold drives both the consensus check and the top-level status/issues logic
(look for DEGRADED_AT_STUCK_HEADS, _check_consensus_health(), and
_run_health_checks() to make the replacement and ensure the comparison semantics
remain the same).
| @pytest.fixture(autouse=True) | ||
| def _wire_health_module_to_test_engine(engine: Engine): | ||
| """Point session_factory's global manager at the test engine for the | ||
| duration of each test, then restore. Also stubs the rpc_router_ref | ||
| truthy check inside _check_consensus_health.""" | ||
| from backend.database_handler import session_factory | ||
| from backend.protocol_rpc import health as health_module | ||
|
|
||
| prev_mgr = session_factory._db_manager | ||
| prev_router = health_module._rpc_router_ref | ||
|
|
||
| session_factory._db_manager = _StubManager(engine=engine) | ||
| health_module._rpc_router_ref = object() # truthy | ||
|
|
||
| yield | ||
|
|
||
| session_factory._db_manager = prev_mgr | ||
| health_module._rpc_router_ref = prev_router |
There was a problem hiding this comment.
Pin the health thresholds in this autouse fixture.
All of these assertions currently depend on the default values for HEALTH_HEAD_STUCK_AFTER_MINUTES, TRANSACTION_TIMEOUT_MINUTES, and HEALTH_DEGRADED_AT_STUCK_HEADS, but _check_consensus_health() reads them from os.environ on every call. If CI or another test changes one of those vars, this whole suite becomes order-dependent.
Suggested fix
`@pytest.fixture`(autouse=True)
-def _wire_health_module_to_test_engine(engine: Engine):
+def _wire_health_module_to_test_engine(
+ engine: Engine, monkeypatch: pytest.MonkeyPatch
+):
"""Point session_factory's global manager at the test engine for the
duration of each test, then restore. Also stubs the rpc_router_ref
truthy check inside _check_consensus_health."""
from backend.database_handler import session_factory
from backend.protocol_rpc import health as health_module
+ monkeypatch.setenv("HEALTH_HEAD_STUCK_AFTER_MINUTES", "15")
+ monkeypatch.setenv("TRANSACTION_TIMEOUT_MINUTES", "30")
+ monkeypatch.setenv("HEALTH_DEGRADED_AT_STUCK_HEADS", "3")
+
prev_mgr = session_factory._db_manager
prev_router = health_module._rpc_router_ref🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@tests/db-sqlalchemy/test_health_orphan_detection.py` around lines 42 - 59,
The autouse fixture _wire_health_module_to_test_engine should pin the
environment keys HEALTH_HEAD_STUCK_AFTER_MINUTES, TRANSACTION_TIMEOUT_MINUTES,
and HEALTH_DEGRADED_AT_STUCK_HEADS so _check_consensus_health (which reads
os.environ each call) is deterministic; modify the fixture to save
os.environ.get(...) for those three keys, set them to the canonical test values
you expect before yield (using os.environ[...] = "..."), and restore the
previous values (or delete if previously absent) after the yield while keeping
the existing session_factory/_rpc_router_ref setup and teardown.
|
🎉 This PR is included in version 0.119.2 🎉 The release is available on GitHub release Your semantic-release bot 📦🚀 |
Summary
Rewrites
_check_consensus_healthso the public Studio Health dashboard alert (orphaned_transactions) means "contract head-of-queue is stuck" instead of "lots of pending txs". Previously the alert fired DEGRADED on perfectly-healthy instances whenever a heavy user queued many txs against one slow contract.Why
Today's Rally Prod alert:
"There are 58 pending transactions"on a system that was actively draining a backlog. Two bugs in the previous detector caused this:active_workersfilter was wrong. Selected distinctworker_idfrom txs wherecreated_at > NOW() - 1 hour. Bursty workloads with no new txs in the last hour → 0 active workers reported, even though workers were actively processing 6h-old txs.total_orphaned_transactionssemantics. Previous: "all blocked txs are orphaned IF zero active workers." Combined with bug Initial Prototype #1, this declared every queued tx orphaned during normal heavy-user drain.What changed
Per the user spec — "a long queue is not a problem if the head is moving; the head being stuck IS a problem regardless of queue depth" —
_check_consensus_healthnow computes:Stuck-head contracts: per contract, oldest in-flight tx is "stuck" when (a) created >
HEAD_STUCK_AFTER_MINUTESago AND (b) no tx for that contract has an unexpired worker claim (blocked_at > NOW() - claim_window). Field nametotal_orphaned_transactionspreserved for dashboard backwards-compat; semantics changed to count CONTRACTS in this state.active_workers: distinctworker_idwith an unexpired claim, regardless of tx age.Status threshold: aligned with the dashboard's alert gate (default 3) so
consensus.statusand the top-levelissuestag agree.Codex-driven refinements during review
This PR went through Codex review (
/codex) which surfaced two genuine bugs in my first pass:blocked_atis NOT a heartbeat. Workers set it once at claim, clear on completion. My initialRECENT_ACTIVITY_WINDOW_MINUTES = 5would have falsely flagged any consensus round longer than 5 min as "no active work." Fixed by aligning the window withTRANSACTION_TIMEOUT_MINUTES(worker claim timeout, default 30).Status flip threshold mismatch —
status="degraded"at ≥1 stuck contract but the alert gate is ≥3. Fixed via a singleDEGRADED_AT_STUCK_HEADSconstant.Plus: included
LEADER_TIMEOUT/VALIDATORS_TIMEOUTin the in-flight status set (finalization/appeal claim paths process them); deterministichash ASCtie-break in the head CTE; status filter on theNOT EXISTSsubquery so finalized/canceled rows don't briefly mask a stuck head.Test plan
tests/db-sqlalchemy/test_health_orphan_detection.py— 9 tests covering: long queue with active head reads 0; stuck head reads 1 (status stays healthy below threshold); recent head doesn't trip; workers on 6h-old txs count as active; multi-contract scenarios; long-running consensus within claim window (Codex's high-priority finding); expired claim counts as stuck; timeout statuses are in-flight; status flips degraded only at threshold of 3.tests/db-sqlalchemy/suite: 143 passed, 1 xfailed (pre-existing).Follow-ups (not blocking)
/health/consensus(separate endpoint) still uses the oldcreated_at > 1hworker filter and the old per-tx orphan-hash logic. The detail endpoint should share the helper from this PR. Out of scope here; tracked as a separate task.Summary by CodeRabbit
Release Notes
Bug Fixes
Tests