Repository navigation
fix: avoid resetting active consensus progress - #1640
Conversation
|
Caution Review failedPull request was closed or merged during review 📝 WalkthroughWalkthroughWorker recovery logic now requires stale consensus progress (in addition to stale blocked leases) before escalating or resetting stuck transactions, with exhaustion events timestamped. Health checks detect recent exhaustion and report ChangesRecovery exhaustion tracking and health detection
Sequence Diagram(s)sequenceDiagram
participant Worker as ConsensusWorker
participant DB as transactions table
participant Progress as consensus_history.current_monitoring
Worker->>DB: Query blocked/assigned transactions (blocked_at, worker_id)
DB-->>Worker: Candidate rows
Worker->>Progress: Extract numeric progress.value timestamps
alt blocked_at stale AND max(progress) stale
Worker->>DB: UPDATE -> escalate to CANCELED or reset to PENDING, set consensus_data.error and max_recovery_exhausted_at
else blocked_at stale but progress recent
Worker->>DB: Do not change transaction (leave worker/blocked_at)
end
flowchart LR
HealthCheck["_check_consensus_health()"]
Query["SQL: count canceled txs where consensus_data->error='max_recovery_cycles_exceeded' and derived_ts within window"]
Result["max_recovery_exhausted_count"]
Status["consensus status (ok/degraded)"]
HealthCheck --> Query
Query --> Result
Result -->|count > 0| Status
Status -->|degraded| Output["services['consensus']"]
Estimated code review effort🎯 4 (Complex) | ⏱️ ~50 minutes Possibly related PRs
Suggested labels
Poem
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (1)
tests/db-sqlalchemy/test_health_orphan_detection.py (1)
637-670: ⚡ Quick winExercise the notice window with an older control row.
This only proves the positive path. If the SQL window filter regresses and starts counting all historical exhausted transactions, this test still passes because there's no out-of-window fixture to exclude. Add one older exhausted
CANCELEDrow and keep the assertion at1.🧪 Minimal test hardening
with Session_() as s: _insert_tx( s, tx_hash="0x" + "ab" * 32, to_address="0x" + "ab" * 20, status="CANCELED", nonce=0, created_at=now - timedelta(minutes=5), recovery_count=3, consensus_data={ "error": "max_recovery_cycles_exceeded", "recovery_count": 3, "max_recovery_exhausted_at": int(now.timestamp()), }, ) + _insert_tx( + s, + tx_hash="0x" + "ac" * 32, + to_address="0x" + "ac" * 20, + status="CANCELED", + nonce=1, + created_at=now - timedelta(hours=2), + recovery_count=3, + consensus_data={ + "error": "max_recovery_cycles_exceeded", + "recovery_count": 3, + "max_recovery_exhausted_at": int( + (now - timedelta(hours=2)).timestamp() + ), + }, + ) s.commit()🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tests/db-sqlalchemy/test_health_orphan_detection.py` around lines 637 - 670, Add an older exhausted CANCELED control row outside the configured notice window to ensure the window filter is honored: in the test_recent_max_recovery_exhaustion_flags_notice test, after inserting the recent CANCELED row via _insert_tx, insert a second CANCELED row with recovery_count and consensus_data indicating max_recovery_cycles_exceeded but with created_at older than HEALTH_MAX_RECOVERY_EXHAUSTED_NOTICE_WINDOW_MINUTES (e.g. now - timedelta(minutes=61)); then call health_module._check_consensus_health and keep the assertion that result["max_recovery_exhausted_count"] == 1 and status == "degraded".
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@backend/consensus/worker.py`:
- Around line 667-669: The DB update branches now include a new payload field
'max_recovery_exhausted_at', which breaks tests that assert exact equality of
the payload; update the assertions in
tests/db-sqlalchemy/test_finalization_starvation.py (around the current lines
asserting the payload at 482-489) to either include 'max_recovery_exhausted_at'
with the expected integer timestamp or change the assertion to check a
superset/subset (e.g., assert that required keys exist and values match) so the
test tolerates the extra key; also scan the worker.py update sites that set
'max_recovery_exhausted_at' (the SQL update blocks around the earlier change and
the similar block at 886-889) and ensure any other tests or consumers are
updated similarly.
---
Nitpick comments:
In `@tests/db-sqlalchemy/test_health_orphan_detection.py`:
- Around line 637-670: Add an older exhausted CANCELED control row outside the
configured notice window to ensure the window filter is honored: in the
test_recent_max_recovery_exhaustion_flags_notice test, after inserting the
recent CANCELED row via _insert_tx, insert a second CANCELED row with
recovery_count and consensus_data indicating max_recovery_cycles_exceeded but
with created_at older than HEALTH_MAX_RECOVERY_EXHAUSTED_NOTICE_WINDOW_MINUTES
(e.g. now - timedelta(minutes=61)); then call
health_module._check_consensus_health and keep the assertion that
result["max_recovery_exhausted_count"] == 1 and status == "degraded".
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: be361d90-f38c-4430-8f3f-7eb81ef49d5a
📒 Files selected for processing (4)
backend/consensus/worker.pybackend/protocol_rpc/health.pytests/db-sqlalchemy/test_finalization_starvation.pytests/db-sqlalchemy/test_health_orphan_detection.py
f67297e to
89c01f5
Compare
|
🎉 This PR is included in version 0.120.14 🎉 The release is available on GitHub release Your semantic-release bot 📦🚀 |
Summary
Incident context
The prod tx 0xedc723838cee384ebfd0823fb048204731fc2509d8cbb4eb3cf56413cf9fa6bf reset almost exactly every 30 minutes after claim. It had recent PROPOSING/COMMITTING/REVEALING progress inside those windows, so blocked_at was acting as a stale lease rather than an active-progress heartbeat.
Verification
DB-backed pytest was attempted but local POSTGRES_URL is not configured and Docker is not running on this remote machine.
Summary by CodeRabbit
Improvements
Health Monitoring
Tests