Skip to content

fix(health): detect head-of-queue stalls, not just queue depth - #1621

Merged
MuncleUscles merged 1 commit into
mainfrom
fix/health-orphan-detection-head-of-queue
May 6, 2026
Merged

MuncleUscles merged 1 commit into
mainfrom
fix/health-orphan-detection-head-of-queue

Conversation

@MuncleUscles

@MuncleUscles MuncleUscles commented May 6, 2026 •

Copy link
Copy Markdown
Member

Summary

Rewrites _check_consensus_health so the public Studio Health dashboard alert (orphaned_transactions) means "contract head-of-queue is stuck" instead of "lots of pending txs". Previously the alert fired DEGRADED on perfectly-healthy instances whenever a heavy user queued many txs against one slow contract.

Why

Today's Rally Prod alert: "There are 58 pending transactions" on a system that was actively draining a backlog. Two bugs in the previous detector caused this:

  1. active_workers filter was wrong. Selected distinct worker_id from txs where created_at > NOW() - 1 hour. Bursty workloads with no new txs in the last hour → 0 active workers reported, even though workers were actively processing 6h-old txs.

  2. total_orphaned_transactions semantics. Previous: "all blocked txs are orphaned IF zero active workers." Combined with bug Initial Prototype #1, this declared every queued tx orphaned during normal heavy-user drain.

What changed

Per the user spec — "a long queue is not a problem if the head is moving; the head being stuck IS a problem regardless of queue depth" — _check_consensus_health now computes:

  • Stuck-head contracts: per contract, oldest in-flight tx is "stuck" when (a) created > HEAD_STUCK_AFTER_MINUTES ago AND (b) no tx for that contract has an unexpired worker claim (blocked_at > NOW() - claim_window). Field name total_orphaned_transactions preserved for dashboard backwards-compat; semantics changed to count CONTRACTS in this state.

  • active_workers: distinct worker_id with an unexpired claim, regardless of tx age.

  • Status threshold: aligned with the dashboard's alert gate (default 3) so consensus.status and the top-level issues tag agree.

Codex-driven refinements during review

This PR went through Codex review (/codex) which surfaced two genuine bugs in my first pass:

  1. blocked_at is NOT a heartbeat. Workers set it once at claim, clear on completion. My initial RECENT_ACTIVITY_WINDOW_MINUTES = 5 would have falsely flagged any consensus round longer than 5 min as "no active work." Fixed by aligning the window with TRANSACTION_TIMEOUT_MINUTES (worker claim timeout, default 30).

  2. Status flip threshold mismatch — status="degraded" at ≥1 stuck contract but the alert gate is ≥3. Fixed via a single DEGRADED_AT_STUCK_HEADS constant.

Plus: included LEADER_TIMEOUT / VALIDATORS_TIMEOUT in the in-flight status set (finalization/appeal claim paths process them); deterministic hash ASC tie-break in the head CTE; status filter on the NOT EXISTS subquery so finalized/canceled rows don't briefly mask a stuck head.

Test plan

  • tests/db-sqlalchemy/test_health_orphan_detection.py — 9 tests covering: long queue with active head reads 0; stuck head reads 1 (status stays healthy below threshold); recent head doesn't trip; workers on 6h-old txs count as active; multi-contract scenarios; long-running consensus within claim window (Codex's high-priority finding); expired claim counts as stuck; timeout statuses are in-flight; status flips degraded only at threshold of 3.
  • Full tests/db-sqlalchemy/ suite: 143 passed, 1 xfailed (pre-existing).

Follow-ups (not blocking)

  • /health/consensus (separate endpoint) still uses the old created_at > 1h worker filter and the old per-tx orphan-hash logic. The detail endpoint should share the helper from this PR. Out of scope here; tracked as a separate task.
  • Dashboard copy ("N orphaned transactions") is now misleading because the unit changed from txs to contracts. Worth coordinating with the dashboard team to relabel as "N stuck contracts".

Summary by CodeRabbit

Release Notes

  • Bug Fixes

    • Enhanced consensus health monitoring with richer transaction metrics for improved visibility into system processing status.
    • Improved detection of orphaned and stuck transactions for more reliable system monitoring and alerting.
  • Tests

    • Added comprehensive regression test suite validating transaction detection logic across multiple scenarios and edge cases.

The "orphaned_transactions" metric exposed by /health was producing
false-positive degraded alerts on instances with bursty traffic and
slow per-contract serialization. The dashboard would alert "There are
N pending transactions" even when consensus was healthy, just slowly
draining a backlog.

Two bugs in the previous _check_consensus_health:

1. active_workers was derived from "txs created in the last 1h with
   worker_id set". Workers processing OLD txs (a 6h-old head with a
   freshly-claimed worker) were invisible. With bursty traffic and no
   new tx in the last hour, the query returned zero, falsely
   declaring all in-flight work orphaned.

2. total_orphaned_transactions semantics mixed two unrelated things —
   "queue depth" and "no workers alive". A long queue with a moving
   head is not a problem; the head being stuck is the problem
   regardless of queue depth.

New logic, matching the spec from the discussion:

- A contract's head is the oldest non-final tx for that contract.
- The head is "stuck" when:
    a) it was created > HEAD_STUCK_AFTER_MINUTES ago, AND
    b) no tx for that contract has a fresh blocked_at within the
       RECENT_ACTIVITY_WINDOW_MINUTES window (no work in flight).
- total_orphaned_transactions now counts the number of CONTRACTS in
  this state. Field name preserved for backwards-compat with the
  external dashboard, but semantics changed: long queues with active
  heads now read 0.
- active_workers counts distinct worker_ids holding a fresh blocked_at,
  regardless of how old the underlying tx is.

Tests cover: long-queue-with-active-head reads 0, stuck head reads 1,
recent head doesn't trip, workers on 6h-old txs count as active,
multi-contract scenarios only flag the genuinely-stuck contract.

This is the metric driving the public Studio Health dashboard. The
existing /health/consensus endpoint still exposes per-contract detail
unchanged.
@coderabbitai

coderabbitai Bot commented May 6, 2026 •

Copy link
Copy Markdown
Contributor
📝 Walkthrough

Walkthrough

The consensus health check is rewritten to provide per-contract diagnostics. A new nested SQL query collects per-contract metrics including processing/pending transaction counts, time-bucket tallies, and oldest transaction ages. Results aggregate across contracts to determine overall health status and orphaned transaction counts. A new test suite validates the per-contract logic across multiple scenarios including stuck heads, expired claims, and threshold boundary conditions.

Changes

Consensus Health Per-Contract Rewrite

Layer / File(s) Summary
Core Logic Rewrite
backend/protocol_rpc/health.py
_check_consensus_health is rewritten to execute a nested SQL query that collects active workers and per-contract metrics (processing count, pending count, created-time buckets, oldest blocked/processing ages, worker transactions). In-function constants define stuck-head and activity-window thresholds. Results aggregate across contracts to compute overall status (healthy/degraded) and total orphaned transaction count, returned via background thread.
Comprehensive Test Suite
tests/db-sqlalchemy/test_health_orphan_detection.py
New test module validates the per-contract orphan detection logic across eight scenarios: active heads vs. long queues, stuck heads, recent heads, active workers processing old transactions, multiple contracts, long-running consensus within claim windows, expired claims, timeout status handling, and status threshold gating. Test utilities wire the health module to a test engine and enable direct SQL transaction insertion for synthetic test data.

Sequence Diagram

sequenceDiagram
    participant API as Health Check<br/>Endpoint
    participant HC as _check_consensus<br/>_health()
    participant DB as Database
    participant Agg as Aggregation<br/>Logic

    API->>HC: Request consensus health
    HC->>DB: Query active workers<br/>from recent transactions
    DB-->>HC: Worker list
    HC->>DB: Query per-contract metrics<br/>(nested: processing, pending,<br/>time-buckets, oldest ages)
    DB-->>HC: Per-contract rows
    HC->>Agg: Build contracts array<br/>with metrics
    Agg-->>HC: Contracts with<br/>orphaned counts
    HC->>Agg: Aggregate totals &<br/>compute status
    Agg-->>HC: status + totals
    HC-->>API: Return health response<br/>(status, total_processing,<br/>total_orphaned, contracts[])
Loading

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~25 minutes

Possibly related PRs

Suggested labels

run-tests

Suggested reviewers

  • cristiam86

Poem

🐰 Per-contract wisdom flows from nested SQL streams,
Where stuck heads reveal orphaned transaction dreams.
Each contract now tells its own health-blessed tale,
With windows and thresholds that never shall fail! ✨

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Title check ✅ Passed The title clearly describes the main change: shifting the health alert from queue depth to head-of-queue stalls, which is the core objective of this PR.
Description check ✅ Passed The description is comprehensive, covering all required template sections: Why (two bugs fixed), What (new behavior and constants), test plan, and follow-ups. It exceeds template expectations with detailed rationale and Codex-driven refinements.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch fix/health-orphan-detection-head-of-queue

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 3

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@backend/protocol_rpc/health.py`:
- Around line 528-539: The stuck-head detector currently omits PENDING because
IN_FLIGHT_STATUSES_SQL excludes it, so oldest non-final txes in PENDING don't
count toward total_orphaned_transactions; fix by introducing a separate status
constant (e.g. STUCK_HEAD_STATUSES_SQL or HEAD_NON_FINAL_STATUSES) that includes
'PENDING' while keeping IN_FLIGHT_STATUSES_SQL as-is if you want processing
counts narrower, then update the head-CTE filter, the stuck-head query (where
total_orphaned_transactions is computed), and any references at the spots noted
(including uses around lines referenced and functions/queries that reference
IN_FLIGHT_STATUSES_SQL) to use the new STUCK_HEAD_STATUSES_SQL so the detector
treats PENDING as non-final head-of-queue.
- Around line 528-530: _top-level health downgrade uses a hard-coded ">= 3"
while you introduced DEGRADED_AT_STUCK_HEADS, causing services.consensus.status
(from _check_consensus_health) to disagree with the top-level /health status and
issues; update _run_health_checks() to use the DEGRADED_AT_STUCK_HEADS constant
instead of the literal 3 so the same threshold drives both the consensus check
and the top-level status/issues logic (look for DEGRADED_AT_STUCK_HEADS,
_check_consensus_health(), and _run_health_checks() to make the replacement and
ensure the comparison semantics remain the same).

In `@tests/db-sqlalchemy/test_health_orphan_detection.py`:
- Around line 42-59: The autouse fixture _wire_health_module_to_test_engine
should pin the environment keys HEALTH_HEAD_STUCK_AFTER_MINUTES,
TRANSACTION_TIMEOUT_MINUTES, and HEALTH_DEGRADED_AT_STUCK_HEADS so
_check_consensus_health (which reads os.environ each call) is deterministic;
modify the fixture to save os.environ.get(...) for those three keys, set them to
the canonical test values you expect before yield (using os.environ[...] =
"..."), and restore the previous values (or delete if previously absent) after
the yield while keeping the existing session_factory/_rpc_router_ref setup and
teardown.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 8ebf73f3-5242-40d3-80c8-0f59bc059133

📥 Commits

Reviewing files that changed from the base of the PR and between 4809f63 and 16261c8.

📒 Files selected for processing (2)
  • backend/protocol_rpc/health.py
  • tests/db-sqlalchemy/test_health_orphan_detection.py

Comment on lines +528 to +539
# Match the dashboard alert threshold so consensus.status and the
# top-level "issues" tag agree on what counts as degraded.
DEGRADED_AT_STUCK_HEADS = int(os.environ.get("HEALTH_DEGRADED_AT_STUCK_HEADS", "3"))

# Statuses considered "in flight". Includes the *_TIMEOUT variants
# because finalization and appeal claim paths process them. Must
# match the head-CTE filter and the in-flight-count query for
# consistency.
IN_FLIGHT_STATUSES_SQL = (
"'ACTIVATED','PROPOSING','COMMITTING','REVEALING',"
"'ACCEPTED','UNDETERMINED','LEADER_TIMEOUT','VALIDATORS_TIMEOUT'"
)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major | ⚡ Quick win

PENDING heads are currently invisible to the stuck-head detector.

The docstring here defines the head as the oldest non-final tx, but the shared status set excludes PENDING. That means a contract whose oldest tx has been sitting in PENDING past the stuck threshold with no claim will never contribute to total_orphaned_transactions, which misses one real head-of-queue stall mode. If you want total_processing_transactions to stay narrower than the head detector, this probably needs separate status sets instead of one shared constant.

Also applies to: 579-600, 615-620

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@backend/protocol_rpc/health.py` around lines 528 - 539, The stuck-head
detector currently omits PENDING because IN_FLIGHT_STATUSES_SQL excludes it, so
oldest non-final txes in PENDING don't count toward total_orphaned_transactions;
fix by introducing a separate status constant (e.g. STUCK_HEAD_STATUSES_SQL or
HEAD_NON_FINAL_STATUSES) that includes 'PENDING' while keeping
IN_FLIGHT_STATUSES_SQL as-is if you want processing counts narrower, then update
the head-CTE filter, the stuck-head query (where total_orphaned_transactions is
computed), and any references at the spots noted (including uses around lines
referenced and functions/queries that reference IN_FLIGHT_STATUSES_SQL) to use
the new STUCK_HEAD_STATUSES_SQL so the detector treats PENDING as non-final
head-of-queue.

Comment on lines +528 to +530
# Match the dashboard alert threshold so consensus.status and the
# top-level "issues" tag agree on what counts as degraded.
DEGRADED_AT_STUCK_HEADS = int(os.environ.get("HEALTH_DEGRADED_AT_STUCK_HEADS", "3"))

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major | ⚡ Quick win

DEGRADED_AT_STUCK_HEADS doesn’t fully align /health yet.

This new threshold only affects _check_consensus_health(). _run_health_checks() still hard-codes >= 3 when it downgrades the top-level /health payload, so any env override will make services.consensus.status disagree with top-level status/issues.

Suggested fix
-        elif consensus_health.get("total_orphaned_transactions", 0) >= 3:
+        elif consensus_status == "degraded":
             if overall_status == "healthy":
                 overall_status = "degraded"
             issues.append("orphaned_transactions")
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@backend/protocol_rpc/health.py` around lines 528 - 530, _top-level health
downgrade uses a hard-coded ">= 3" while you introduced DEGRADED_AT_STUCK_HEADS,
causing services.consensus.status (from _check_consensus_health) to disagree
with the top-level /health status and issues; update _run_health_checks() to use
the DEGRADED_AT_STUCK_HEADS constant instead of the literal 3 so the same
threshold drives both the consensus check and the top-level status/issues logic
(look for DEGRADED_AT_STUCK_HEADS, _check_consensus_health(), and
_run_health_checks() to make the replacement and ensure the comparison semantics
remain the same).

Comment on lines +42 to +59
@pytest.fixture(autouse=True)
def _wire_health_module_to_test_engine(engine: Engine):
"""Point session_factory's global manager at the test engine for the
duration of each test, then restore. Also stubs the rpc_router_ref
truthy check inside _check_consensus_health."""
from backend.database_handler import session_factory
from backend.protocol_rpc import health as health_module

prev_mgr = session_factory._db_manager
prev_router = health_module._rpc_router_ref

session_factory._db_manager = _StubManager(engine=engine)
health_module._rpc_router_ref = object() # truthy

yield

session_factory._db_manager = prev_mgr
health_module._rpc_router_ref = prev_router

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major | ⚡ Quick win

Pin the health thresholds in this autouse fixture.

All of these assertions currently depend on the default values for HEALTH_HEAD_STUCK_AFTER_MINUTES, TRANSACTION_TIMEOUT_MINUTES, and HEALTH_DEGRADED_AT_STUCK_HEADS, but _check_consensus_health() reads them from os.environ on every call. If CI or another test changes one of those vars, this whole suite becomes order-dependent.

Suggested fix
 `@pytest.fixture`(autouse=True)
-def _wire_health_module_to_test_engine(engine: Engine):
+def _wire_health_module_to_test_engine(
+    engine: Engine, monkeypatch: pytest.MonkeyPatch
+):
     """Point session_factory's global manager at the test engine for the
     duration of each test, then restore. Also stubs the rpc_router_ref
     truthy check inside _check_consensus_health."""
     from backend.database_handler import session_factory
     from backend.protocol_rpc import health as health_module

+    monkeypatch.setenv("HEALTH_HEAD_STUCK_AFTER_MINUTES", "15")
+    monkeypatch.setenv("TRANSACTION_TIMEOUT_MINUTES", "30")
+    monkeypatch.setenv("HEALTH_DEGRADED_AT_STUCK_HEADS", "3")
+
     prev_mgr = session_factory._db_manager
     prev_router = health_module._rpc_router_ref
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@tests/db-sqlalchemy/test_health_orphan_detection.py` around lines 42 - 59,
The autouse fixture _wire_health_module_to_test_engine should pin the
environment keys HEALTH_HEAD_STUCK_AFTER_MINUTES, TRANSACTION_TIMEOUT_MINUTES,
and HEALTH_DEGRADED_AT_STUCK_HEADS so _check_consensus_health (which reads
os.environ each call) is deterministic; modify the fixture to save
os.environ.get(...) for those three keys, set them to the canonical test values
you expect before yield (using os.environ[...] = "..."), and restore the
previous values (or delete if previously absent) after the yield while keeping
the existing session_factory/_rpc_router_ref setup and teardown.

@MuncleUscles
MuncleUscles merged commit bdfd4d0 into main May 6, 2026
26 of 28 checks passed
@MuncleUscles
MuncleUscles deleted the fix/health-orphan-detection-head-of-queue branch May 6, 2026 23:03
@github-actions

github-actions Bot commented May 6, 2026

Copy link
Copy Markdown
Contributor

🎉 This PR is included in version 0.119.2 🎉

The release is available on GitHub release

Your semantic-release bot 📦🚀

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant