Skip to content

fix: cap GenVM hard-crash retries at 3, finalize with error receipt - #1595

Merged
MuncleUscles merged 3 commits into
mainfrom
fix/leader-crash-retry-cap
Apr 16, 2026
Merged

MuncleUscles merged 3 commits into
mainfrom
fix/leader-crash-retry-cap

Conversation

@MuncleUscles

@MuncleUscles MuncleUscles commented Apr 16, 2026 •

Copy link
Copy Markdown
Member

Summary

  • After 3 consecutive GenVMInternalError with code=None + causes=[] (non-classifiable WASM host-side traps) on the same tx, stop retrying.
  • Synthesize a leader receipt with execution_result=ERROR, error_code=INTERNAL_ERROR, carrying the crash fingerprint detail.
  • Tx flows through normal finalization → FINALIZED with error outcome visible in explorer.
  • One poisoned tx (0x451098…) generated ~6,500 identical Sentry events over 4 weeks. This fix caps that at 3.

Scope

  • Only GenVMInternalError with code=None + causes=[] + is_fatal=False — the deterministic "hard crash" class.
  • LLM errors (NO_PROVIDER, rate limits), fatal errors, and validator-side errors are unchanged.
  • Counter is in-memory per worker pod (matches existing _generic_error_retries pattern).
  • Configurable via LEADER_CRASH_MAX_RETRIES env var (default 3).

Test plan

  • 7 new unit tests pass (under-cap, at-cap, receipt shape, per-tx isolation, truncation, classification).
  • 76 existing related tests still pass.
  • CI green.
  • After deploy: Sentry GENLAYER-STUDIO-11X rate should drop to max 3 events per poisoned tx.

Refs: Sentry GENLAYER-STUDIO-11X, genvm#288

Summary by CodeRabbit

  • Bug Fixes
    • Improved consensus resilience: leader-failure retries capped (3 attempts) and transactions finalized with an error receipt when cap reached; normal fatal errors still stop processing.
  • Tests
    • Added unit tests covering leader-crash retry behavior, counter isolation, status dispatch, and truncation of long error details.
  • Integration
    • Updated account balance assertions to parse hex API responses.
  • Tests (contracts)
    • Adjusted test contract withdrawal parameter typing.

When GenVM produces a non-classifiable internal error (WASM host-side trap
with code=None, causes=[]), the consensus worker previously reset the
transaction for retry indefinitely. One poisoned tx generated ~6,500
identical Sentry events over 4 weeks.

After 3 consecutive hard-crash leader attempts on the same tx, synthesize
a leader receipt with execution_result=ERROR and error_code=INTERNAL_ERROR
carrying the crash fingerprint detail. The tx then flows through the normal
finalization path to FINALIZED with an error outcome visible in the explorer.

Scope:
- Only affects GenVMInternalError with code=None + causes=[] + is_fatal=False
- LLM errors (NO_PROVIDER, rate limits) and fatal errors are unchanged
- Counter is in-memory per worker pod (matches existing retry counter pattern)
- Configurable via LEADER_CRASH_MAX_RETRIES env var (default 3)

Refs: GENLAYER-STUDIO-11X, genlayerlabs/genvm#288
@coderabbitai

coderabbitai Bot commented Apr 16, 2026 •

Copy link
Copy Markdown
Contributor

Warning

Rate limit exceeded

@MuncleUscles has exceeded the limit for the number of commits that can be reviewed per hour. Please wait 20 minutes and 9 seconds before requesting another review.

Your organization is not enrolled in usage-based pricing. Contact your admin to enable usage-based pricing to continue reviews beyond the rate limit, or try again in 20 minutes and 9 seconds.

⌛ How to resolve this issue?

After the wait time has elapsed, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

We recommend that you space out your commits to avoid hitting the rate limit.

🚦 How do rate limits work?

CodeRabbit enforces hourly rate limits for each developer per organization.

Our paid plans have higher rate limits than the trial, open-source and free plans. In all cases, we re-allow further reviews after a brief timeout.

Please see our FAQ for further information.

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 25ba95cc-3173-4109-ba9e-95dc9a8c3194

📥 Commits

Reviewing files that changed from the base of the PR and between c6cd08a and 857bc21.

📒 Files selected for processing (1)
  • backend/consensus/base.py
📝 Walkthrough

Walkthrough

Added per-transaction leader-crash retry tracking to ConsensusWorker. Hard-crash GenVMInternalError instances are retried up to a cap; on cap hit the worker finalizes the transaction with an ACCEPTED status and a synthetic ERROR leader receipt instead of resetting for reprocessing.

Changes

Cohort / File(s) Summary
Leader-crash retry logic
backend/consensus/worker.py
Added MAX_LEADER_CRASH_RETRIES constant and in-memory _leader_crash_retries counter, new method _handle_leader_crash_retry(...), and updated _transaction_context and process_transaction to classify hard-crash GenVMInternalError, apply retry capping, synthesize an ERROR leader receipt on cap, skip reset when finalized, and clean up counters on success.
Unit tests: leader-crash
tests/unit/test_worker_leader_crash_retry.py
New unit test module covering hard-crash detection, per-tx retry isolation, cap enforcement, synthetic ERROR receipt contents (including truncation behavior), and cases that should bypass retry logic (structured internal errors, fatal errors).
Integration tests: balance assertions
tests/integration/accounts/test_accounts.py
Adjusted balance assertions to parse hex-encoded eth_getBalance results via int(..., 16) before comparisons.
Integration contract signature
tests/integration/icontracts/contracts/payable_escrow.py
Changed withdraw parameter type from to: Address to to: str and updated call to gl.get_contract_at(Address(to)) accordingly.

Sequence Diagram(s)

sequenceDiagram
    participant Worker as ConsensusWorker
    participant GenVM as GenVM
    participant DB as Database (tx_row)
    participant Dispatcher as ConsensusAlgorithm

    rect rgba(200,230,255,0.5)
    Worker->>GenVM: execute transaction
    GenVM-->>Worker: raises GenVMInternalError (hard-crash)
    end

    Worker->>Worker: classify error (code=None, causes=[], is_fatal=False)
    alt retry count < max
        Worker->>Worker: increment _leader_crash_retries[tx_hash]
        Worker->>DB: reset_transaction / allow reprocessing (existing flow)
    else retry count >= max
        Worker->>DB: write tx_row.consensus_data (ACCEPTED + synthetic ERROR receipt)
        Worker->>Dispatcher: dispatch_transaction_status_update(tx_hash, ACCEPTED)
        Worker->>Worker: cleanup _leader_crash_retries[tx_hash]
    end
Loading

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~25 minutes

Possibly related PRs

Suggested labels

run-tests, Studio

Suggested reviewers

  • kstroobants
  • cristiam86

Poem

🐰 I hopped the log where errors creep,

Tried thrice to wake a leader from sleep.
When retries end and bells are still,
I scribble "ERROR" on the hill.
Then off I go — another task to keep.

🚥 Pre-merge checks | ✅ 2 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 54.17% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (2 passed)
Check name Status Explanation
Title check ✅ Passed The title clearly and specifically describes the main change: capping GenVM hard-crash retries and finalizing with an error receipt.
Description check ✅ Passed The description covers all key template sections: what was changed, why it matters, testing approach, and implementation details. All required template elements are present and well-documented.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch fix/leader-crash-retry-cap

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick comments (1)
tests/unit/test_worker_leader_crash_retry.py (1)

61-85: Consider moving assertion inside the patch context for clarity.

The assertion mock_dispatch.assert_not_called() at line 85 is outside the with patch(...) block. While Python keeps mock_dispatch in scope, placing the assertion inside the context manager would make the test's intent clearer and avoid confusion about whether the mock is still active.

📝 Suggested restructure
     with patch(
         "backend.consensus.base.ConsensusAlgorithm.dispatch_transaction_status_update",
         new_callable=AsyncMock,
     ) as mock_dispatch:
         assert (
             await worker._handle_leader_crash_retry(
                 tx_hash, "transaction", _hard_crash_error()
             )
         ) is False
         assert (
             await worker._handle_leader_crash_retry(
                 tx_hash, "transaction", _hard_crash_error()
             )
         ) is False
+        mock_dispatch.assert_not_called()

     assert worker._leader_crash_retries[tx_hash]["count"] == 2
-    mock_dispatch.assert_not_called()
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@tests/unit/test_worker_leader_crash_retry.py` around lines 61 - 85, Move the
assertion that the mock wasn't called into the patch context to make intent
explicit: inside test_under_cap_returns_false(), keep the with patch(...) as
mock_dispatch block around both await calls and then call
mock_dispatch.assert_not_called() before exiting that block (referencing the
test function test_under_cap_returns_false, the mock variable mock_dispatch, and
the method _handle_leader_crash_retry).
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Nitpick comments:
In `@tests/unit/test_worker_leader_crash_retry.py`:
- Around line 61-85: Move the assertion that the mock wasn't called into the
patch context to make intent explicit: inside test_under_cap_returns_false(),
keep the with patch(...) as mock_dispatch block around both await calls and then
call mock_dispatch.assert_not_called() before exiting that block (referencing
the test function test_under_cap_returns_false, the mock variable mock_dispatch,
and the method _handle_leader_crash_retry).

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: acd596f5-ef27-40df-ac26-447a48c7f2ad

📥 Commits

Reviewing files that changed from the base of the PR and between cd554a6 and 29f90b6.

📒 Files selected for processing (2)
  • backend/consensus/worker.py
  • tests/unit/test_worker_leader_crash_retry.py

…ress type

- test_accounts.py: eth_getBalance now returns hex (0x7d0), use int(result, 16)
- payable_escrow.py: withdraw param typed as str, wrap with Address() for emit_transfer

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@tests/integration/icontracts/contracts/payable_escrow.py`:
- Around line 24-31: The withdraw method's parameter annotation is incorrect for
actual call sites: change the signature of withdraw (function withdraw) so the
parameter 'to' accepts both Address and string types (e.g., annotate as
Union[str, Address] or Address | str depending on project typing style) to match
callers that pass Address instances; keep the runtime behavior that wraps with
Address(to) before calling gl.get_contract_at(Address(to)). Ensure you import or
reference the Address type in the module to satisfy the annotation.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: b29f7d0c-c908-4465-b660-a8218343ebd5

📥 Commits

Reviewing files that changed from the base of the PR and between 29f90b6 and c6cd08a.

📒 Files selected for processing (2)
  • tests/integration/accounts/test_accounts.py
  • tests/integration/icontracts/contracts/payable_escrow.py

Comment on lines +24 to +31
def withdraw(self, to: str) -> None:
if gl.message.sender_address != self.depositor:
raise gl.vm.UserError("not depositor")
amount = self.deposited
if amount == u256(0):
raise gl.vm.UserError("nothing to withdraw")
self.deposited = u256(0)
gl.get_contract_at(to).emit_transfer(value=amount)
gl.get_contract_at(Address(to)).emit_transfer(value=amount)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor

🧩 Analysis chain

🌐 Web query:

In py-genlayer contracts, for @gl.public.writemethods, does changing a parameter annotation fromAddresstostrchange ABI argument validation/encoding behavior? Also, isAddress(Address(...)) guaranteed to be valid?

💡 Result:

No, changing a parameter annotation from Address to str in @gl.public.write methods does not change ABI argument validation/encoding behavior. GenLayer uses a custom ABI based on JSON-like calldata encoding where arguments are passed as raw Python objects (strings, numbers, etc.) via "args" array and "kwargs" map, serialized via genlayer.py.calldata.encode. Parameter annotations determine contract schema generation and static type checking/introspection via get_schema and genlayer.py.get_schema, but runtime decoding delivers raw values (str for both cases) without type-specific ABI encoding differences. Address/str both encode/decode as strings. Address(Address(...)) is not guaranteed to be valid if the inner Address(...) fails construction (e.g., invalid length/format), as Address constructor validates input size/format but skips checksum and can raise on invalid str/Buffer. Nested construction propagates errors if inner fails; if inner succeeds, outer receives valid Address object and succeeds.

Citations:


🏁 Script executed:

#!/bin/bash
# Verify local usage consistency for withdraw argument types/call sites.
# Expected: call sites should consistently pass the type intended by the ABI.
rg -nP --type=py -C3 'def\s+withdraw\s*\(\s*self,\s*to:\s*'
rg -nP --type=py -C3 '\.withdraw\s*\('

Repository: genlayerlabs/genlayer-studio

Length of output: 2500


Parameter type annotation inconsistency: to: str but call sites may pass Address objects.

At line 24, the parameter is annotated as str, but direct test callers (tests/direct/test_example_contracts_direct.py:117, 127) pass Address objects (direct_bob, direct_alice). The wrapping Address(to) at line 31 handles both string and object inputs, but the annotation should match actual call patterns. Consider either changing to to: Address with direct string wrapping only, or document that both types are accepted at runtime.

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@tests/integration/icontracts/contracts/payable_escrow.py` around lines 24 -
31, The withdraw method's parameter annotation is incorrect for actual call
sites: change the signature of withdraw (function withdraw) so the parameter
'to' accepts both Address and string types (e.g., annotate as Union[str,
Address] or Address | str depending on project typing style) to match callers
that pass Address instances; keep the runtime behavior that wraps with
Address(to) before calling gl.get_contract_at(Address(to)). Ensure you import or
reference the Address type in the module to satisfy the annotation.

…nalization

TransactionContext.__init__() eagerly calls contract_snapshot_factory for all
non-SEND txs. For txs targeting contracts with empty state (data={}), this
throws ContractNotFoundError before FinalizingState.handle() can check
should_finalize_contract. The guard at line 2382 is correct but unreachable.

Catch the error and set contract_snapshot=None. The finalization code already
handles None correctly for error-result txs (skips state update).

Fixes: GENLAYER-STUDIO-EG (2267 Sentry events since Feb 4)
@MuncleUscles
MuncleUscles merged commit ff14a1a into main Apr 16, 2026
11 checks passed
@MuncleUscles
MuncleUscles deleted the fix/leader-crash-retry-cap branch April 16, 2026 14:44
@github-actions

Copy link
Copy Markdown
Contributor

🎉 This PR is included in version 0.117.2 🎉

The release is available on GitHub release

Your semantic-release bot 📦🚀

MuncleUscles added a commit that referenced this pull request Apr 16, 2026
…it on SEND

`execute_transfer` (consensus/base.py) was crediting the recipient via
unconditional `update_account_balance`. For SEND transactions created by
`sim_fundAccount`, the endpoint already credits the recipient via
`credit_tx_value_once` (PR #1582 fix). The consensus worker then picks up
the PENDING SEND and credits again in `execute_transfer` — giving 2x the
funded amount.

This bug has existed since Sept 2024 but was hidden by the pre-#1595
`test_accounts_funding` assertion comparing strings ("0x7d0" == 1000,
which always fails inconclusively). The hex-parse fix in #1595 finally
makes the test compare real values and surfaced the bug (assert 2000 == 1000).

Fix: swap update_account_balance for credit_tx_value_once in
execute_transfer. This is a no-op for faucet txs (value_credited already
true) and a one-time credit for regular user-to-user SEND.

Fixes integration test_accounts_{funding,burn,transfers} regressions.
MuncleUscles added a commit that referenced this pull request Apr 16, 2026
…it on SEND

`execute_transfer` (consensus/base.py) was crediting the recipient via
unconditional `update_account_balance`. For SEND transactions created by
`sim_fundAccount`, the endpoint already credits the recipient via
`credit_tx_value_once` (PR #1582 fix). The consensus worker then picks up
the PENDING SEND and credits again in `execute_transfer` — giving 2x the
funded amount.

This bug has existed since Sept 2024 but was hidden by the pre-#1595
`test_accounts_funding` assertion comparing strings ("0x7d0" == 1000,
which always fails inconclusively). The hex-parse fix in #1595 finally
makes the test compare real values and surfaced the bug (assert 2000 == 1000).

Fix: swap update_account_balance for credit_tx_value_once in
execute_transfer. This is a no-op for faucet txs (value_credited already
true) and a one-time credit for regular user-to-user SEND.

Fixes integration test_accounts_{funding,burn,transfers} regressions.
MuncleUscles added a commit that referenced this pull request Apr 16, 2026
…it on SEND

`execute_transfer` (consensus/base.py) was crediting the recipient via
unconditional `update_account_balance`. For SEND transactions created by
`sim_fundAccount`, the endpoint already credits the recipient via
`credit_tx_value_once` (PR #1582 fix). The consensus worker then picks up
the PENDING SEND and credits again in `execute_transfer` — giving 2x the
funded amount.

This bug has existed since Sept 2024 but was hidden by the pre-#1595
`test_accounts_funding` assertion comparing strings ("0x7d0" == 1000,
which always fails inconclusively). The hex-parse fix in #1595 finally
makes the test compare real values and surfaced the bug (assert 2000 == 1000).

Fix: swap update_account_balance for credit_tx_value_once in
execute_transfer. This is a no-op for faucet txs (value_credited already
true) and a one-time credit for regular user-to-user SEND.

Fixes integration test_accounts_{funding,burn,transfers} regressions.
MuncleUscles added a commit that referenced this pull request Apr 16, 2026
…it on SEND (#1600)

`execute_transfer` (consensus/base.py) was crediting the recipient via
unconditional `update_account_balance`. For SEND transactions created by
`sim_fundAccount`, the endpoint already credits the recipient via
`credit_tx_value_once` (PR #1582 fix). The consensus worker then picks up
the PENDING SEND and credits again in `execute_transfer` — giving 2x the
funded amount.

This bug has existed since Sept 2024 but was hidden by the pre-#1595
`test_accounts_funding` assertion comparing strings ("0x7d0" == 1000,
which always fails inconclusively). The hex-parse fix in #1595 finally
makes the test compare real values and surfaced the bug (assert 2000 == 1000).

Fix: swap update_account_balance for credit_tx_value_once in
execute_transfer. This is a no-op for faucet txs (value_credited already
true) and a one-time credit for regular user-to-user SEND.

Fixes integration test_accounts_{funding,burn,transfers} regressions.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant