Repository navigation
fix: cap recover_stuck_transactions cycles to prevent infinite re-stick - #1601
Conversation
The recovery loop has been repeatedly resetting the same poisoned tx back to PENDING every 20 minutes. The worker immediately re-claims it, tx re-enters COMMITTING, validators hang (e.g. all land on a bad provider), tx stuck again. The claim query's per-contract lock then blocks every newer tx for that address — yesterday we saw this produce a 1,500+ PENDING backlog on one contract that only cleared after manual cancel. Adds a persistent `recovery_count` column and a two-step recovery flow: 1. Escalate: txs with recovery_count >= MAX_RECOVERY_CYCLES (3, env-tunable via RECOVERY_MAX_CYCLES) that are still stuck → set status=CANCELED with a reason in consensus_data. The per-contract queue can now advance. 2. Reset: txs under the limit get the existing reset + a counter bump. Why this wasn't caught before: the recovery reset chain was designed for transient errors where the root cause self-resolves (provider restart, network blip). With deterministic crashes or persistently unavailable providers, the same tx keeps cycling forever. Refs: yesterday's manual cancel of 0x1AE5Eb9a chain head (recurred today), and today's cancel of 0xe8760c16.
|
Warning Rate limit exceeded
Your organization is not enrolled in usage-based pricing. Contact your admin to enable usage-based pricing to continue reviews beyond the rate limit, or try again in 49 minutes and 27 seconds. ⌛ How to resolve this issue?After the wait time has elapsed, a review can be triggered using the We recommend that you space out your commits to avoid hitting the rate limit. 🚦 How do rate limits work?CodeRabbit enforces hourly rate limits for each developer per organization. Our paid plans have higher rate limits than the trial, open-source and free plans. In all cases, we re-allow further reviews after a brief timeout. Please see our FAQ for further information. ℹ️ Review info⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (1)
📝 WalkthroughWalkthroughAdds recovery attempt tracking to the consensus worker with a configurable maximum (3 by default). Introduces Changes
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~20 minutes Poem
🚥 Pre-merge checks | ✅ 2 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (2 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
Actionable comments posted: 2
🧹 Nitpick comments (2)
backend/consensus/worker.py (2)
144-146: Add validation forRECOVERY_MAX_CYCLESenv var.
int(os.environ.get(...))will raiseValueErrorat worker startup on any non-integer value (crashing the pod), and a value of0or negative would immediately escalate every candidate transaction toCANCELEDon the first recovery sweep. Consider mirroring_parse_max_parallel_txs(Lines 151–180): try/except with a warning and floor the minimum at 1.🛠️ Proposed fix
- self._max_recovery_cycles = int( - os.environ.get("RECOVERY_MAX_CYCLES", str(self.MAX_RECOVERY_CYCLES)) - ) + self._max_recovery_cycles = self._parse_positive_int_env( + "RECOVERY_MAX_CYCLES", self.MAX_RECOVERY_CYCLES + )🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@backend/consensus/worker.py` around lines 144 - 146, Validate RECOVERY_MAX_CYCLES when initializing self._max_recovery_cycles: wrap the int(os.environ.get(...)) parse in a try/except like _parse_max_parallel_txs, log a warning via the same logger on parse failure and fall back to self.MAX_RECOVERY_CYCLES, and after parsing enforce a minimum of 1 (i.e., if parsed_value < 1 then set to 1) to avoid zero/negative values causing immediate cancellation; update the assignment to set self._max_recovery_cycles to the validated/fallback value and ensure any exception is swallowed after logging so the worker doesn't crash on startup.
787-830: Track the deferredrecovery_countreset on FINALIZE.Because the counter is never reset on successful finalization (acknowledged as deferred in the PR description), a tx that survives 2 recovery cycles and then finalizes will carry
recovery_count=2for the rest of its life. If it later gets re-stuck for any unrelated reason it will be escalated to CANCELED after a single reset cycle, which is surprising operationally. Consider either:
- Resetting
recovery_count = 0in the FINALIZE path before shipping this, or- Filing a follow-up issue and leaving a
# TODO(...)comment here so the asymmetry is discoverable from the call site.Want me to open the follow-up issue for the FINALIZE-side reset?
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@backend/consensus/worker.py` around lines 787 - 830, The recovery_count is never reset when a transaction successfully FINALIZEs, causing re-stuck transactions to escalate unexpectedly; fix by resetting recovery_count to 0 in the finalization code path (the FINALIZE/commit handler that marks transactions FINALIZED) or, if you prefer to defer, add a clear TODO comment here referencing a follow-up issue and file that issue; specifically, update the FINALIZE handler to set recovery_count = 0 for the transaction row (or add a TODO with the issue number next to the recovery logic around recovery_count and the UPDATE query shown) so the asymmetry is discoverable and resolved later.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@backend/consensus/worker.py`:
- Around line 746-784: The escalation path updates rows in bulk but omits the
refund and WS-dispatch steps other CANCELED flows perform; after you run the
set-based update and obtain escalated (RETURNING hash, recovery_count), iterate
each row, reload the Transactions ORM object (e.g., via self.get_session() or
session.query(Transactions).filter_by(hash=row.hash).one()), call
AccountsManager(s).refund_tx_value(tx.hash, tx.from_address) when tx.value > 0
and tx.from_address is set, commit the refund session, then call
ConsensusAlgorithm.dispatch_transaction_status_update(TransactionsProcessor(s),
tx.hash, TransactionStatus.CANCELED, self.msg_handler) for each tx so
subscribers see the status change, and keep the existing logger.error per row;
ensure you import/use AccountsManager, Transactions, TransactionsProcessor and
dispatch API referenced above.
- Around line 758-766: The escalation path currently sets status = 'CANCELED'
but leaves consensus_data and consensus_history intact, risking state leakage;
update the escalation/update logic that marks transactions CANCELED (the
SQL/handler that sets status = 'CANCELED' for stuck rows in
PROPOSING/COMMITTING/REVEALING) to also set consensus_data = NULL and
consensus_history = NULL (mirroring the recovery/reset path that sets status =
'PENDING' and clears those fields), ensuring canceled transactions do not carry
partial consensus state into future work.
---
Nitpick comments:
In `@backend/consensus/worker.py`:
- Around line 144-146: Validate RECOVERY_MAX_CYCLES when initializing
self._max_recovery_cycles: wrap the int(os.environ.get(...)) parse in a
try/except like _parse_max_parallel_txs, log a warning via the same logger on
parse failure and fall back to self.MAX_RECOVERY_CYCLES, and after parsing
enforce a minimum of 1 (i.e., if parsed_value < 1 then set to 1) to avoid
zero/negative values causing immediate cancellation; update the assignment to
set self._max_recovery_cycles to the validated/fallback value and ensure any
exception is swallowed after logging so the worker doesn't crash on startup.
- Around line 787-830: The recovery_count is never reset when a transaction
successfully FINALIZEs, causing re-stuck transactions to escalate unexpectedly;
fix by resetting recovery_count to 0 in the finalization code path (the
FINALIZE/commit handler that marks transactions FINALIZED) or, if you prefer to
defer, add a clear TODO comment here referencing a follow-up issue and file that
issue; specifically, update the FINALIZE handler to set recovery_count = 0 for
the transaction row (or add a TODO with the issue number next to the recovery
logic around recovery_count and the UPDATE query shown) so the asymmetry is
discoverable and resolved later.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: e2a648c9-94d1-40db-a9a3-4ac3d5dcc389
📒 Files selected for processing (3)
backend/consensus/worker.pybackend/database_handler/migration/versions/f2c3d4e5a6b7_add_recovery_count_to_transactions.pybackend/database_handler/models.py
| escalate_query = text( | ||
| """ | ||
| UPDATE transactions | ||
| SET status = 'CANCELED', | ||
| worker_id = NULL, | ||
| blocked_at = NULL, | ||
| consensus_data = COALESCE(consensus_data, '{}'::jsonb) | ||
| || jsonb_build_object( | ||
| 'error', 'max_recovery_cycles_exceeded', | ||
| 'recovery_count', recovery_count | ||
| ) | ||
| WHERE recovery_count >= :max_cycles | ||
| AND status NOT IN ('FINALIZED', 'CANCELED') | ||
| AND ( | ||
| (blocked_at IS NOT NULL | ||
| AND blocked_at < NOW() - CAST(:timeout AS INTERVAL)) | ||
| OR | ||
| (blocked_at IS NULL | ||
| AND status IN ('PROPOSING', 'COMMITTING', 'REVEALING') | ||
| AND created_at < NOW() - CAST(:orphan_timeout AS INTERVAL)) | ||
| ) | ||
| RETURNING hash, recovery_count; | ||
| """ | ||
| ) | ||
| escalated = session.execute( | ||
| escalate_query, | ||
| { | ||
| "max_cycles": self._max_recovery_cycles, | ||
| "timeout": f"{self.transaction_timeout_minutes} minutes", | ||
| "orphan_timeout": "5 minutes", | ||
| }, | ||
| ).fetchall() | ||
| if escalated: | ||
| session.commit() | ||
| for row in escalated: | ||
| logger.error( | ||
| f"[Worker {self.worker_id}] Transaction {row.hash} canceled " | ||
| f"after {row.recovery_count} recovery cycles (stuck repeatedly)" | ||
| ) |
There was a problem hiding this comment.
Escalation path is missing two behaviors that every other CANCELED path performs: value refund and status-change dispatch.
Compare with _handle_no_validators_retry (Lines 1039–1063) and _handle_generic_error_retry (Lines 1098–1123):
- No refund of
tx.value. If an escalated tx is payable (value > 0,from_addressset) and was never activated, the sender's funds are now permanently lost — the tx is marked CANCELED butAccountsManager.refund_tx_value(...)is never called. This is a silent financial bug for any payable stuck tx that exceeds the recovery cap. - No
ConsensusAlgorithm.dispatch_transaction_status_update(...)call. Frontends/subscribers relying on WebSocket status updates will not observe the CANCELED transition for escalated txs — only the row changes in the DB.
Because the set-based SQL update escalates multiple rows at once, you'll need to iterate the RETURNING rows (you already have them as escalated) and, per row, load the tx via ORM, refund if needed, and dispatch the WS event. Something along the lines of:
🛠️ Sketch of the fix
if escalated:
session.commit()
from backend.database_handler.accounts_manager import AccountsManager
for row in escalated:
with self.get_session() as s:
tx = s.query(Transactions).filter_by(hash=row.hash).one()
if tx.value and tx.value > 0 and tx.from_address:
AccountsManager(s).refund_tx_value(tx.hash, tx.from_address)
s.commit()
await ConsensusAlgorithm.dispatch_transaction_status_update(
TransactionsProcessor(s),
tx.hash,
TransactionStatus.CANCELED,
self.msg_handler,
)
logger.error(
f"[Worker {self.worker_id}] Transaction {row.hash} canceled "
f"after {row.recovery_count} recovery cycles (stuck repeatedly)"
)🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@backend/consensus/worker.py` around lines 746 - 784, The escalation path
updates rows in bulk but omits the refund and WS-dispatch steps other CANCELED
flows perform; after you run the set-based update and obtain escalated
(RETURNING hash, recovery_count), iterate each row, reload the Transactions ORM
object (e.g., via self.get_session() or
session.query(Transactions).filter_by(hash=row.hash).one()), call
AccountsManager(s).refund_tx_value(tx.hash, tx.from_address) when tx.value > 0
and tx.from_address is set, commit the refund session, then call
ConsensusAlgorithm.dispatch_transaction_status_update(TransactionsProcessor(s),
tx.hash, TransactionStatus.CANCELED, self.msg_handler) for each tx so
subscribers see the status change, and keep the existing logger.error per row;
ensure you import/use AccountsManager, Transactions, TransactionsProcessor and
dispatch API referenced above.
| AND status NOT IN ('FINALIZED', 'CANCELED') | ||
| AND ( | ||
| (blocked_at IS NOT NULL | ||
| AND blocked_at < NOW() - CAST(:timeout AS INTERVAL)) | ||
| OR | ||
| (blocked_at IS NULL | ||
| AND status IN ('PROPOSING', 'COMMITTING', 'REVEALING') | ||
| AND created_at < NOW() - CAST(:orphan_timeout AS INTERVAL)) | ||
| ) |
There was a problem hiding this comment.
🧩 Analysis chain
🏁 Script executed:
# First, let's examine the code in backend/consensus/worker.py around the flagged lines
wc -l backend/consensus/worker.pyRepository: genlayerlabs/genlayer-studio
Length of output: 104
🏁 Script executed:
# Read the relevant section (lines 740-790 to get full context)
sed -n '740,790p' backend/consensus/worker.py | cat -nRepository: genlayerlabs/genlayer-studio
Length of output: 2553
🏁 Script executed:
# Now let's look at the restore_stuck_transactions function mentioned in the learning
find . -name "server.py" -path "*/protocol_rpc/*" -type fRepository: genlayerlabs/genlayer-studio
Length of output: 54
🏁 Script executed:
# Search for transaction cancellation logic to understand what cleanup happens
rg -n "CANCEL" backend/consensus/worker.py | head -20Repository: genlayerlabs/genlayer-studio
Length of output: 671
🏁 Script executed:
# Look for state-related operations during transaction cancellation
rg -n "status.*CANCEL\|DELETE.*contract\|rollback\|cleanup" backend/consensus/worker.py | head -30Repository: genlayerlabs/genlayer-studio
Length of output: 54
🏁 Script executed:
# Search for what happens during COMMITTING and REVEALING phases
rg -n "COMMITTING\|REVEALING" backend/consensus/worker.py -B 2 -A 5 | head -60Repository: genlayerlabs/genlayer-studio
Length of output: 54
🏁 Script executed:
# Find how contract state snapshots/updates are handled
rg -n "contract.*state\|state.*snapshot\|state.*rollback\|apply.*state" backend/consensus/worker.py -i | head -20Repository: genlayerlabs/genlayer-studio
Length of output: 54
🏁 Script executed:
# Look for the restore_stuck_transactions function in server.py
cat -n backend/protocol_rpc/server.py | grep -A 30 "def restore_stuck_transactions"Repository: genlayerlabs/genlayer-studio
Length of output: 134
🏁 Script executed:
# Check if there are tests for transaction cancellation during consensus phases
find . -path "*/test*" -name "*.py" -type f | xargs rg -l "cancel.*transaction\|CANCELED.*COMMITTING\|CANCELED.*REVEALING" 2>/dev/null | head -5Repository: genlayerlabs/genlayer-studio
Length of output: 54
🏁 Script executed:
# Search for contract state handling and what persists after transaction operations
rg -n "to_address\|contract_address\|state_root\|state_data" backend/consensus/worker.py | head -25Repository: genlayerlabs/genlayer-studio
Length of output: 54
🏁 Script executed:
# Find server.py with restore_stuck_transactions
find . -name "server.py" -type f | head -10Repository: genlayerlabs/genlayer-studio
Length of output: 54
🏁 Script executed:
# Search for restore_stuck_transactions across entire codebase
rg -l "restore_stuck_transactions" --type py | head -5Repository: genlayerlabs/genlayer-studio
Length of output: 54
🏁 Script executed:
# Look at the transaction model to understand status definitions
find . -path "*/models*" -name "*.py" -type f | xargs rg -l "TransactionStatus\|transaction.*status" 2>/dev/null | head -5Repository: genlayerlabs/genlayer-studio
Length of output: 54
🏁 Script executed:
# Search for where PROPOSING/COMMITTING/REVEALING are defined or used
rg -r "PROPOSING|COMMITTING|REVEALING" --type py | head -20Repository: genlayerlabs/genlayer-studio
Length of output: 468
🏁 Script executed:
# Look at the full transaction escalation function to understand the context better
sed -n '730,850p' backend/consensus/worker.py | cat -nRepository: genlayerlabs/genlayer-studio
Length of output: 5884
🏁 Script executed:
# Look for how transactions interact with contract state during execution
rg -n "execute\|consensus\|contract" backend/consensus/worker.py | head -30Repository: genlayerlabs/genlayer-studio
Length of output: 54
🏁 Script executed:
# Check if there are any comments or documentation about state safety during cancellation
grep -n -A 3 -B 3 "PROPOSING\|COMMITTING\|REVEALING" backend/consensus/worker.pyRepository: genlayerlabs/genlayer-studio
Length of output: 840
🏁 Script executed:
# Search for any references to the learning document's concerns across the codebase
rg -n "state.*corrupt\|partial.*commit\|state.*isolation" --type py | head -10Repository: genlayerlabs/genlayer-studio
Length of output: 54
🏁 Script executed:
# Check git history or comments about this design decision
sed -n '700,850p' backend/consensus/worker.py | cat -nRepository: genlayerlabs/genlayer-studio
Length of output: 7542
Asymmetric cleanup between escalation and recovery paths creates state corruption risk.
When a transaction is escalated to CANCELED (lines 758-766), the consensus_data field is appended with error metadata but not cleared, unlike the recovery path (lines 93-94) which explicitly sets consensus_data = NULL when resetting to PENDING. If PROPOSING/COMMITTING/REVEALING phases write partial state to consensus_data, the next transaction on the same to_address could inherit corrupted state.
Either:
- Clear
consensus_dataandconsensus_historyduring escalation (parallel to recovery), or - Keep stuck transactions stuck as documented in prior guidance rather than canceling them, and surface via alerting for investigation.
The asymmetry suggests this risk was recognized during recovery-path design but overlooked during escalation-path design.
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@backend/consensus/worker.py` around lines 758 - 766, The escalation path
currently sets status = 'CANCELED' but leaves consensus_data and
consensus_history intact, risking state leakage; update the escalation/update
logic that marks transactions CANCELED (the SQL/handler that sets status =
'CANCELED' for stuck rows in PROPOSING/COMMITTING/REVEALING) to also set
consensus_data = NULL and consensus_history = NULL (mirroring the recovery/reset
path that sets status = 'PENDING' and clears those fields), ensuring canceled
transactions do not carry partial consensus state into future work.
PG 17.8 handles ADD COLUMN with constant DEFAULT as metadata-only (no table rewrite), but still needs a brief ACCESS EXCLUSIVE lock. On a hot table like `transactions` with constant consensus-worker writes, a slow unrelated query could queue the ALTER, and every new write would queue behind the ALTER — stalling consensus. Cap the wait to 5s so the migration fails fast and retries rather than blocking the whole transaction pipeline.
|
🎉 This PR is included in version 0.117.4 🎉 The release is available on GitHub release Your semantic-release bot 📦🚀 |
Summary
Adds a persistent recovery-cycle cap. After N=3 resets on the same tx, it's escalated to CANCELED so the per-contract queue can advance past a poisoned head. This stops the 20-min reset→reclaim→stick loop that has produced 1,500+ tx backlogs twice in 2 days (manual cancellations were required both times).
Why
Before this PR, `recover_stuck_transactions` resets any stuck tx back to PENDING every 20 min. The claim query's per-contract lock means ONE poisoned tx blocks all newer txs for that address — forever. Root causes vary (GenVM WASM trap, persistent provider outage, deterministic validator failure) but the pattern is the same.
Fix
Verified
Deferred
Reset `recovery_count` to 0 on successful FINALIZE: a tx that gets recovered once and then succeeds shouldn't count against its quota. Not important right now because the counter only matters when it hits the cap — transient recoveries don't reach 3.
Refs: yesterday's manual cancel of 0x1AE5Eb9a chain head (recurred today and was canceled again), and today's cancel of 0xe8760c16.
Summary by CodeRabbit
Release Notes