Repository navigation
fix: cap GenVM hard-crash retries at 3, finalize with error receipt - #1595
Conversation
When GenVM produces a non-classifiable internal error (WASM host-side trap with code=None, causes=[]), the consensus worker previously reset the transaction for retry indefinitely. One poisoned tx generated ~6,500 identical Sentry events over 4 weeks. After 3 consecutive hard-crash leader attempts on the same tx, synthesize a leader receipt with execution_result=ERROR and error_code=INTERNAL_ERROR carrying the crash fingerprint detail. The tx then flows through the normal finalization path to FINALIZED with an error outcome visible in the explorer. Scope: - Only affects GenVMInternalError with code=None + causes=[] + is_fatal=False - LLM errors (NO_PROVIDER, rate limits) and fatal errors are unchanged - Counter is in-memory per worker pod (matches existing retry counter pattern) - Configurable via LEADER_CRASH_MAX_RETRIES env var (default 3) Refs: GENLAYER-STUDIO-11X, genlayerlabs/genvm#288
|
Warning Rate limit exceeded
Your organization is not enrolled in usage-based pricing. Contact your admin to enable usage-based pricing to continue reviews beyond the rate limit, or try again in 20 minutes and 9 seconds. ⌛ How to resolve this issue?After the wait time has elapsed, a review can be triggered using the We recommend that you space out your commits to avoid hitting the rate limit. 🚦 How do rate limits work?CodeRabbit enforces hourly rate limits for each developer per organization. Our paid plans have higher rate limits than the trial, open-source and free plans. In all cases, we re-allow further reviews after a brief timeout. Please see our FAQ for further information. 📝 WalkthroughWalkthroughAdded per-transaction leader-crash retry tracking to ConsensusWorker. Hard-crash GenVMInternalError instances are retried up to a cap; on cap hit the worker finalizes the transaction with an ACCEPTED status and a synthetic ERROR leader receipt instead of resetting for reprocessing. Changes
Sequence Diagram(s)sequenceDiagram
participant Worker as ConsensusWorker
participant GenVM as GenVM
participant DB as Database (tx_row)
participant Dispatcher as ConsensusAlgorithm
rect rgba(200,230,255,0.5)
Worker->>GenVM: execute transaction
GenVM-->>Worker: raises GenVMInternalError (hard-crash)
end
Worker->>Worker: classify error (code=None, causes=[], is_fatal=False)
alt retry count < max
Worker->>Worker: increment _leader_crash_retries[tx_hash]
Worker->>DB: reset_transaction / allow reprocessing (existing flow)
else retry count >= max
Worker->>DB: write tx_row.consensus_data (ACCEPTED + synthetic ERROR receipt)
Worker->>Dispatcher: dispatch_transaction_status_update(tx_hash, ACCEPTED)
Worker->>Worker: cleanup _leader_crash_retries[tx_hash]
end
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~25 minutes Possibly related PRs
Suggested labels
Suggested reviewers
Poem
🚥 Pre-merge checks | ✅ 2 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (2 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
🧹 Nitpick comments (1)
tests/unit/test_worker_leader_crash_retry.py (1)
61-85: Consider moving assertion inside the patch context for clarity.The assertion
mock_dispatch.assert_not_called()at line 85 is outside thewith patch(...)block. While Python keepsmock_dispatchin scope, placing the assertion inside the context manager would make the test's intent clearer and avoid confusion about whether the mock is still active.📝 Suggested restructure
with patch( "backend.consensus.base.ConsensusAlgorithm.dispatch_transaction_status_update", new_callable=AsyncMock, ) as mock_dispatch: assert ( await worker._handle_leader_crash_retry( tx_hash, "transaction", _hard_crash_error() ) ) is False assert ( await worker._handle_leader_crash_retry( tx_hash, "transaction", _hard_crash_error() ) ) is False + mock_dispatch.assert_not_called() assert worker._leader_crash_retries[tx_hash]["count"] == 2 - mock_dispatch.assert_not_called()🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@tests/unit/test_worker_leader_crash_retry.py` around lines 61 - 85, Move the assertion that the mock wasn't called into the patch context to make intent explicit: inside test_under_cap_returns_false(), keep the with patch(...) as mock_dispatch block around both await calls and then call mock_dispatch.assert_not_called() before exiting that block (referencing the test function test_under_cap_returns_false, the mock variable mock_dispatch, and the method _handle_leader_crash_retry).
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Nitpick comments:
In `@tests/unit/test_worker_leader_crash_retry.py`:
- Around line 61-85: Move the assertion that the mock wasn't called into the
patch context to make intent explicit: inside test_under_cap_returns_false(),
keep the with patch(...) as mock_dispatch block around both await calls and then
call mock_dispatch.assert_not_called() before exiting that block (referencing
the test function test_under_cap_returns_false, the mock variable mock_dispatch,
and the method _handle_leader_crash_retry).
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: acd596f5-ef27-40df-ac26-447a48c7f2ad
📒 Files selected for processing (2)
backend/consensus/worker.pytests/unit/test_worker_leader_crash_retry.py
…ress type - test_accounts.py: eth_getBalance now returns hex (0x7d0), use int(result, 16) - payable_escrow.py: withdraw param typed as str, wrap with Address() for emit_transfer
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@tests/integration/icontracts/contracts/payable_escrow.py`:
- Around line 24-31: The withdraw method's parameter annotation is incorrect for
actual call sites: change the signature of withdraw (function withdraw) so the
parameter 'to' accepts both Address and string types (e.g., annotate as
Union[str, Address] or Address | str depending on project typing style) to match
callers that pass Address instances; keep the runtime behavior that wraps with
Address(to) before calling gl.get_contract_at(Address(to)). Ensure you import or
reference the Address type in the module to satisfy the annotation.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: b29f7d0c-c908-4465-b660-a8218343ebd5
📒 Files selected for processing (2)
tests/integration/accounts/test_accounts.pytests/integration/icontracts/contracts/payable_escrow.py
| def withdraw(self, to: str) -> None: | ||
| if gl.message.sender_address != self.depositor: | ||
| raise gl.vm.UserError("not depositor") | ||
| amount = self.deposited | ||
| if amount == u256(0): | ||
| raise gl.vm.UserError("nothing to withdraw") | ||
| self.deposited = u256(0) | ||
| gl.get_contract_at(to).emit_transfer(value=amount) | ||
| gl.get_contract_at(Address(to)).emit_transfer(value=amount) |
There was a problem hiding this comment.
🧩 Analysis chain
🌐 Web query:
In py-genlayer contracts, for @gl.public.writemethods, does changing a parameter annotation fromAddresstostrchange ABI argument validation/encoding behavior? Also, isAddress(Address(...)) guaranteed to be valid?
💡 Result:
No, changing a parameter annotation from Address to str in @gl.public.write methods does not change ABI argument validation/encoding behavior. GenLayer uses a custom ABI based on JSON-like calldata encoding where arguments are passed as raw Python objects (strings, numbers, etc.) via "args" array and "kwargs" map, serialized via genlayer.py.calldata.encode. Parameter annotations determine contract schema generation and static type checking/introspection via get_schema and genlayer.py.get_schema, but runtime decoding delivers raw values (str for both cases) without type-specific ABI encoding differences. Address/str both encode/decode as strings. Address(Address(...)) is not guaranteed to be valid if the inner Address(...) fails construction (e.g., invalid length/format), as Address constructor validates input size/format but skips checksum and can raise on invalid str/Buffer. Nested construction propagates errors if inner fails; if inner succeeds, outer receives valid Address object and succeeds.
Citations:
- 1: https://docs.genlayer.com/developers/intelligent-contracts/types/address
- 2: https://sdk.genlayer.com/v0.2.9/_modules/genlayer/gl/genvm_contracts.html
- 3: https://sdk.genlayer.com/v0.1.4/api/genlayer.html
- 4: https://sdk.genlayer.com/main/api/genlayer.html
- 5: https://sdk.genlayer.com/v0.2.11/spec/04-contract-interface/02-abi.html
- 6: https://docs.genlayer.com/api-references/genlayer-cli/contracts/call
🏁 Script executed:
#!/bin/bash
# Verify local usage consistency for withdraw argument types/call sites.
# Expected: call sites should consistently pass the type intended by the ABI.
rg -nP --type=py -C3 'def\s+withdraw\s*\(\s*self,\s*to:\s*'
rg -nP --type=py -C3 '\.withdraw\s*\('Repository: genlayerlabs/genlayer-studio
Length of output: 2500
Parameter type annotation inconsistency: to: str but call sites may pass Address objects.
At line 24, the parameter is annotated as str, but direct test callers (tests/direct/test_example_contracts_direct.py:117, 127) pass Address objects (direct_bob, direct_alice). The wrapping Address(to) at line 31 handles both string and object inputs, but the annotation should match actual call patterns. Consider either changing to to: Address with direct string wrapping only, or document that both types are accepted at runtime.
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@tests/integration/icontracts/contracts/payable_escrow.py` around lines 24 -
31, The withdraw method's parameter annotation is incorrect for actual call
sites: change the signature of withdraw (function withdraw) so the parameter
'to' accepts both Address and string types (e.g., annotate as Union[str,
Address] or Address | str depending on project typing style) to match callers
that pass Address instances; keep the runtime behavior that wraps with
Address(to) before calling gl.get_contract_at(Address(to)). Ensure you import or
reference the Address type in the module to satisfy the annotation.
…nalization
TransactionContext.__init__() eagerly calls contract_snapshot_factory for all
non-SEND txs. For txs targeting contracts with empty state (data={}), this
throws ContractNotFoundError before FinalizingState.handle() can check
should_finalize_contract. The guard at line 2382 is correct but unreachable.
Catch the error and set contract_snapshot=None. The finalization code already
handles None correctly for error-result txs (skips state update).
Fixes: GENLAYER-STUDIO-EG (2267 Sentry events since Feb 4)
|
🎉 This PR is included in version 0.117.2 🎉 The release is available on GitHub release Your semantic-release bot 📦🚀 |
…it on SEND `execute_transfer` (consensus/base.py) was crediting the recipient via unconditional `update_account_balance`. For SEND transactions created by `sim_fundAccount`, the endpoint already credits the recipient via `credit_tx_value_once` (PR #1582 fix). The consensus worker then picks up the PENDING SEND and credits again in `execute_transfer` — giving 2x the funded amount. This bug has existed since Sept 2024 but was hidden by the pre-#1595 `test_accounts_funding` assertion comparing strings ("0x7d0" == 1000, which always fails inconclusively). The hex-parse fix in #1595 finally makes the test compare real values and surfaced the bug (assert 2000 == 1000). Fix: swap update_account_balance for credit_tx_value_once in execute_transfer. This is a no-op for faucet txs (value_credited already true) and a one-time credit for regular user-to-user SEND. Fixes integration test_accounts_{funding,burn,transfers} regressions.
…it on SEND `execute_transfer` (consensus/base.py) was crediting the recipient via unconditional `update_account_balance`. For SEND transactions created by `sim_fundAccount`, the endpoint already credits the recipient via `credit_tx_value_once` (PR #1582 fix). The consensus worker then picks up the PENDING SEND and credits again in `execute_transfer` — giving 2x the funded amount. This bug has existed since Sept 2024 but was hidden by the pre-#1595 `test_accounts_funding` assertion comparing strings ("0x7d0" == 1000, which always fails inconclusively). The hex-parse fix in #1595 finally makes the test compare real values and surfaced the bug (assert 2000 == 1000). Fix: swap update_account_balance for credit_tx_value_once in execute_transfer. This is a no-op for faucet txs (value_credited already true) and a one-time credit for regular user-to-user SEND. Fixes integration test_accounts_{funding,burn,transfers} regressions.
…it on SEND `execute_transfer` (consensus/base.py) was crediting the recipient via unconditional `update_account_balance`. For SEND transactions created by `sim_fundAccount`, the endpoint already credits the recipient via `credit_tx_value_once` (PR #1582 fix). The consensus worker then picks up the PENDING SEND and credits again in `execute_transfer` — giving 2x the funded amount. This bug has existed since Sept 2024 but was hidden by the pre-#1595 `test_accounts_funding` assertion comparing strings ("0x7d0" == 1000, which always fails inconclusively). The hex-parse fix in #1595 finally makes the test compare real values and surfaced the bug (assert 2000 == 1000). Fix: swap update_account_balance for credit_tx_value_once in execute_transfer. This is a no-op for faucet txs (value_credited already true) and a one-time credit for regular user-to-user SEND. Fixes integration test_accounts_{funding,burn,transfers} regressions.
…it on SEND (#1600) `execute_transfer` (consensus/base.py) was crediting the recipient via unconditional `update_account_balance`. For SEND transactions created by `sim_fundAccount`, the endpoint already credits the recipient via `credit_tx_value_once` (PR #1582 fix). The consensus worker then picks up the PENDING SEND and credits again in `execute_transfer` — giving 2x the funded amount. This bug has existed since Sept 2024 but was hidden by the pre-#1595 `test_accounts_funding` assertion comparing strings ("0x7d0" == 1000, which always fails inconclusively). The hex-parse fix in #1595 finally makes the test compare real values and surfaced the bug (assert 2000 == 1000). Fix: swap update_account_balance for credit_tx_value_once in execute_transfer. This is a no-op for faucet txs (value_credited already true) and a one-time credit for regular user-to-user SEND. Fixes integration test_accounts_{funding,burn,transfers} regressions.
Summary
GenVMInternalErrorwithcode=None + causes=[](non-classifiable WASM host-side traps) on the same tx, stop retrying.execution_result=ERROR,error_code=INTERNAL_ERROR, carrying the crash fingerprint detail.0x451098…) generated ~6,500 identical Sentry events over 4 weeks. This fix caps that at 3.Scope
GenVMInternalErrorwithcode=None + causes=[] + is_fatal=False— the deterministic "hard crash" class.NO_PROVIDER, rate limits), fatal errors, and validator-side errors are unchanged._generic_error_retriespattern).LEADER_CRASH_MAX_RETRIESenv var (default 3).Test plan
GENLAYER-STUDIO-11Xrate should drop to max 3 events per poisoned tx.Refs: Sentry
GENLAYER-STUDIO-11X, genvm#288Summary by CodeRabbit