Skip to content

fix: avoid resetting active consensus progress - #1640

Merged
MuncleUscles merged 1 commit into
mainfrom
fix/recovery-progress-heartbeat
May 26, 2026
Merged

MuncleUscles merged 1 commit into
mainfrom
fix/recovery-progress-heartbeat

Conversation

@MuncleUscles

@MuncleUscles MuncleUscles commented May 26, 2026 •

Copy link
Copy Markdown
Member

Summary

  • prevent recovery from resetting an in-flight consensus tx solely because its original blocked_at claim lease expired
  • require both blocked_at and latest consensus_history.current_monitoring progress to be stale before recovery reset/cancel
  • stamp max recovery exhaustion with max_recovery_exhausted_at and expose max_recovery_cycles_exhausted as a degraded health notice
  • add DB regression tests for recent-progress recovery suppression and max-recovery exhaustion notice

Incident context

The prod tx 0xedc723838cee384ebfd0823fb048204731fc2509d8cbb4eb3cf56413cf9fa6bf reset almost exactly every 30 minutes after claim. It had recent PROPOSING/COMMITTING/REVEALING progress inside those windows, so blocked_at was acting as a stale lease rather than an active-progress heartbeat.

Verification

  • git diff --check
  • PYTHONPATH=. .venv/bin/python -m py_compile backend/consensus/worker.py backend/protocol_rpc/health.py tests/db-sqlalchemy/test_finalization_starvation.py tests/db-sqlalchemy/test_health_orphan_detection.py
  • read-only prod SQL syntax validation for the new recovery and max-exhaustion health queries

DB-backed pytest was attempted but local POSTGRES_URL is not configured and Docker is not running on this remote machine.

Summary by CodeRabbit

  • Improvements

    • Strengthened stuck-transaction detection by requiring recent per-transaction progress in addition to timing checks to avoid unnecessary resets.
  • Health Monitoring

    • Added a health metric and degradation flag for transactions that exhausted recovery cycles, improving visibility into recovery failures.
  • Tests

    • Added regression tests to ensure recovery logic respects recent progress and that exhaustion conditions are reported in health checks.

Review Change Stack

@coderabbitai

coderabbitai Bot commented May 26, 2026 •

Copy link
Copy Markdown
Contributor

Caution

Review failed

Pull request was closed or merged during review

📝 Walkthrough

Walkthrough

Worker recovery logic now requires stale consensus progress (in addition to stale blocked leases) before escalating or resetting stuck transactions, with exhaustion events timestamped. Health checks detect recent exhaustion and report max_recovery_exhausted_count as a degradation signal within a configurable notice window.

Changes

Recovery exhaustion tracking and health detection

Layer / File(s) Summary
Worker recovery escalation and exhaustion tracking
backend/consensus/worker.py
reset_transaction adds max_recovery_exhausted_at epoch timestamp to error payload. recover_stuck_transactions escalation and reset paths now verify both blocked_at staleness and numeric progress staleness from consensus_history->current_monitoring, replacing prior blocked_at-only conditions.
Health check for recovery exhaustion detection
backend/protocol_rpc/health.py
Introduces MAX_RECOVERY_EXHAUSTED_NOTICE_WINDOW_MINUTES config, SQL query to count canceled transactions with max_recovery_cycles_exceeded error within notice window (parsing timestamp from max_recovery_exhausted_at or falling back to created_at), and extends consensus degradation status to include max_recovery_exhausted_count > 0.
Worker recovery behavior validation
tests/db-sqlalchemy/test_finalization_starvation.py
Regression test ensures recover_stuck_transactions does not reset transactions when consensus_history shows recent progress, preserving transaction state despite stale lease; updates assertions for exhaustion payload fields.
Health check exhaustion validation
tests/db-sqlalchemy/test_health_orphan_detection.py
Test helper _insert_tx extended to support consensus_data parameter and JSON bind. Regression test verifies health checks detect recovery exhaustion and report degraded status.

Sequence Diagram(s)

sequenceDiagram
  participant Worker as ConsensusWorker
  participant DB as transactions table
  participant Progress as consensus_history.current_monitoring
  Worker->>DB: Query blocked/assigned transactions (blocked_at, worker_id)
  DB-->>Worker: Candidate rows
  Worker->>Progress: Extract numeric progress.value timestamps
  alt blocked_at stale AND max(progress) stale
    Worker->>DB: UPDATE -> escalate to CANCELED or reset to PENDING, set consensus_data.error and max_recovery_exhausted_at
  else blocked_at stale but progress recent
    Worker->>DB: Do not change transaction (leave worker/blocked_at)
  end
Loading
flowchart LR
  HealthCheck["_check_consensus_health()"]
  Query["SQL: count canceled txs where consensus_data->error='max_recovery_cycles_exceeded' and derived_ts within window"]
  Result["max_recovery_exhausted_count"]
  Status["consensus status (ok/degraded)"]
  HealthCheck --> Query
  Query --> Result
  Result -->|count > 0| Status
  Status -->|degraded| Output["services['consensus']"]
Loading

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~50 minutes

Possibly related PRs

Suggested labels

run-tests

Poem

🐰 I hop through logs where the stalled votes lay,
I stamp the exhaustion, record the day.
Lease and progress both must age in the sun,
Health rings its bell when recovery's done.
Hooray — gentle fixes, another race run.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Description check ⚠️ Warning The PR description provides a clear summary, incident context, and verification steps, but does not follow the provided template structure with required sections like 'What', 'Why', 'Testing done', 'Decisions made', and 'Checks'. Reorganize the description to match the template: add 'What', 'Why', 'Testing done' sections, include the verification checklist, and add 'User facing release notes' section.
✅ Passed checks (4 passed)
Check name Status Explanation
Title check ✅ Passed The title accurately summarizes the main change: preventing recovery from resetting active consensus progress by requiring both blocked_at and consensus_history progress to be stale.
Docstring Coverage ✅ Passed Docstring coverage is 92.31% which is sufficient. The required threshold is 80.00%.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch fix/recovery-progress-heartbeat

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🧹 Nitpick comments (1)
tests/db-sqlalchemy/test_health_orphan_detection.py (1)

637-670: ⚡ Quick win

Exercise the notice window with an older control row.

This only proves the positive path. If the SQL window filter regresses and starts counting all historical exhausted transactions, this test still passes because there's no out-of-window fixture to exclude. Add one older exhausted CANCELED row and keep the assertion at 1.

🧪 Minimal test hardening
     with Session_() as s:
         _insert_tx(
             s,
             tx_hash="0x" + "ab" * 32,
             to_address="0x" + "ab" * 20,
             status="CANCELED",
             nonce=0,
             created_at=now - timedelta(minutes=5),
             recovery_count=3,
             consensus_data={
                 "error": "max_recovery_cycles_exceeded",
                 "recovery_count": 3,
                 "max_recovery_exhausted_at": int(now.timestamp()),
             },
         )
+        _insert_tx(
+            s,
+            tx_hash="0x" + "ac" * 32,
+            to_address="0x" + "ac" * 20,
+            status="CANCELED",
+            nonce=1,
+            created_at=now - timedelta(hours=2),
+            recovery_count=3,
+            consensus_data={
+                "error": "max_recovery_cycles_exceeded",
+                "recovery_count": 3,
+                "max_recovery_exhausted_at": int(
+                    (now - timedelta(hours=2)).timestamp()
+                ),
+            },
+        )
         s.commit()
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@tests/db-sqlalchemy/test_health_orphan_detection.py` around lines 637 - 670,
Add an older exhausted CANCELED control row outside the configured notice window
to ensure the window filter is honored: in the
test_recent_max_recovery_exhaustion_flags_notice test, after inserting the
recent CANCELED row via _insert_tx, insert a second CANCELED row with
recovery_count and consensus_data indicating max_recovery_cycles_exceeded but
with created_at older than HEALTH_MAX_RECOVERY_EXHAUSTED_NOTICE_WINDOW_MINUTES
(e.g. now - timedelta(minutes=61)); then call
health_module._check_consensus_health and keep the assertion that
result["max_recovery_exhausted_count"] == 1 and status == "degraded".
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@backend/consensus/worker.py`:
- Around line 667-669: The DB update branches now include a new payload field
'max_recovery_exhausted_at', which breaks tests that assert exact equality of
the payload; update the assertions in
tests/db-sqlalchemy/test_finalization_starvation.py (around the current lines
asserting the payload at 482-489) to either include 'max_recovery_exhausted_at'
with the expected integer timestamp or change the assertion to check a
superset/subset (e.g., assert that required keys exist and values match) so the
test tolerates the extra key; also scan the worker.py update sites that set
'max_recovery_exhausted_at' (the SQL update blocks around the earlier change and
the similar block at 886-889) and ensure any other tests or consumers are
updated similarly.

---

Nitpick comments:
In `@tests/db-sqlalchemy/test_health_orphan_detection.py`:
- Around line 637-670: Add an older exhausted CANCELED control row outside the
configured notice window to ensure the window filter is honored: in the
test_recent_max_recovery_exhaustion_flags_notice test, after inserting the
recent CANCELED row via _insert_tx, insert a second CANCELED row with
recovery_count and consensus_data indicating max_recovery_cycles_exceeded but
with created_at older than HEALTH_MAX_RECOVERY_EXHAUSTED_NOTICE_WINDOW_MINUTES
(e.g. now - timedelta(minutes=61)); then call
health_module._check_consensus_health and keep the assertion that
result["max_recovery_exhausted_count"] == 1 and status == "degraded".
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: be361d90-f38c-4430-8f3f-7eb81ef49d5a

📥 Commits

Reviewing files that changed from the base of the PR and between e0757e1 and f67297e.

📒 Files selected for processing (4)
  • backend/consensus/worker.py
  • backend/protocol_rpc/health.py
  • tests/db-sqlalchemy/test_finalization_starvation.py
  • tests/db-sqlalchemy/test_health_orphan_detection.py

Comment thread backend/consensus/worker.py Resolved
@MuncleUscles
MuncleUscles force-pushed the fix/recovery-progress-heartbeat branch from f67297e to 89c01f5 Compare May 26, 2026 13:43
@MuncleUscles
MuncleUscles merged commit d239078 into main May 26, 2026
10 of 11 checks passed
@MuncleUscles
MuncleUscles deleted the fix/recovery-progress-heartbeat branch May 26, 2026 13:48
@github-actions

Copy link
Copy Markdown
Contributor

🎉 This PR is included in version 0.120.14 🎉

The release is available on GitHub release

Your semantic-release bot 📦🚀

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant