Repository navigation
fix: handle LLM provider failures in validator voting and leader recovery - #1424
Conversation
…very LLM provider failures (401, 429, 503) arrive as USER_ERROR from GenVM, but _set_vote only checked VM_ERROR for timeout — causing validators to vote DETERMINISTIC_VIOLATION (which is discarded in consensus) instead of TIMEOUT. Leaders also returned a normal receipt instead of triggering worker recovery. - Validator: LLM error codes in _set_vote now vote TIMEOUT with ERROR log - Leader: LLM error codes in _run_genvm now raise GenVMInternalError (resets tx to PENDING, kills worker for K8s restart) - Add LLM_ERROR_CODES frozenset to error_codes.py
📝 WalkthroughWalkthroughLeader/validator GenVM execution and consensus flows were updated to treat GenVM fatal errors as replacement-triggering exceptions, introduce retry/replacement pools for idle leaders/validators, add LLM error code exports, change vote semantics to include IDLE, and add GenVM execution failure tracking with health endpoints and callback wiring. Changes
Sequence Diagram(s)sequenceDiagram
rect rgba(200,200,255,0.5)
participant LeaderNode as Leader Node
participant GenVM as GenVM Host
participant ReplacementPool as Replacement Pool
participant Validators as Validators
end
LeaderNode->>GenVM: run_genvm_host(request)
alt genvm returns success
GenVM-->>LeaderNode: success receipt
LeaderNode->>Validators: reveal/commit
else genvm returns non-fatal error
GenVM-->>LeaderNode: error (non-fatal)
LeaderNode->>LeaderNode: emit receipt (ERROR)
else genvm returns fatal error
GenVM-->>LeaderNode: raw_error.fatal = True
LeaderNode->>LeaderNode: raise GenVMInternalError(is_fatal=True)
LeaderNode->>ReplacementPool: select next leader
alt replacement available
ReplacementPool-->>LeaderNode: new_leader
LeaderNode->>GenVM: run_genvm_host(request) (retry)
else no replacements left
ReplacementPool-->>LeaderNode: none
LeaderNode->>LeaderNode: raise GenVMInternalError(ALL_LEADERS_IDLE)
end
end
Estimated code review effort🎯 4 (Complex) | ⏱️ ~45 minutes Possibly related PRs
Suggested labels
Suggested reviewers
Poem
🚥 Pre-merge checks | ✅ 2 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (2 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing touches
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
Actionable comments posted: 3
🤖 Fix all issues with AI agents
In `@backend/node/base.py`:
- Line 535: The long conditional at the if statement using receipt.genvm_result
and receipt.genvm_result.get("error_code") in LLM_ERROR_CODES exceeds the
project's line-length and fails Black; reformat the expression into a
Black-friendly multiline form (e.g., wrap the condition in parentheses and split
after logical operators) and then run `black` to apply the canonical formatting
so the file passes CI, keeping the same logic involving receipt.genvm_result and
LLM_ERROR_CODES.
In `@backend/node/genvm/error_codes.py`:
- Around line 94-102: CI flagged Black formatting for the LLM_ERROR_CODES
constant; reformat the frozenset definition so it matches Black's style (e.g.,
put each GenVMErrorCode entry on its own line with trailing commas and proper
indentation) or simply run your project's Black/pre-commit formatter to auto-fix
the block containing LLM_ERROR_CODES and the GenVMErrorCode entries
(GenVMErrorCode.LLM_RATE_LIMITED, GenVMErrorCode.LLM_NO_PROVIDER,
GenVMErrorCode.LLM_PROVIDER_ERROR, GenVMErrorCode.LLM_INVALID_API_KEY,
GenVMErrorCode.LLM_TIMEOUT).
In `@tests/unit/test_leader_llm_recovery.py`:
- Around line 38-59: The leader recovery path is reading
raw_error.get("is_fatal", True) but GenVM uses "fatal"; update the logic in
backend.node.base (in the _run_genvm / GenVMInternalError construction) to
derive is_fatal = raw_error.get("is_fatal") if present else
raw_error.get("fatal", True) (or equivalent precedence), so GenVM's "fatal":
False is respected; also add the suggested pytest that constructs an
ExecutionResult with raw_error["fatal"]=False and asserts the raised
GenVMInternalError.is_fatal is False to prevent regression.
🧹 Nitpick comments (2)
tests/unit/test_set_vote.py (1)
13-36: Consider extracting shared test helpers to avoid duplication withtest_leader_llm_recovery.py.
_make_validatorand_make_nodeare duplicated across both new test files (with slight differences in the validator address). These could be consolidated into a shared fixture intests/common/or aconftest.py, but it's fine to defer.tests/unit/test_leader_llm_recovery.py (1)
76-88: Unused lambda parameteraddr.Static analysis flags
addras unused on line 82. Use_to signal it's intentionally ignored.♻️ Proposed fix
- contract_snapshot_factory=lambda addr: _make_snapshot(), + contract_snapshot_factory=lambda _addr: _make_snapshot(),
CodeRabbit caught that Lua produces {"fatal": true} but we read
"is_fatal" which always missed → defaulted to True. Also adds test
for fatal=False propagation.
…ARNING floor The genvm_log filter had a hardcoded minimum of WARNING that silently discarded GenVM metrics (logged at INFO level) regardless of the configured LOG_LEVEL. This made it impossible to see GenVM's built-in performance metrics (LLM/web module time, host time, compilation time).
- Add Vote.IDLE for infrastructure failures (counted as DISAGREE) - Leader retry: on fatal GenVMInternalError, replace leader from remaining_validators (up to 3 attempts) - Validator retry: on fatal raw_error, replace from shared pool with asyncio.Lock (up to 3 attempts per slot) - All-idle check: if every validator voted IDLE, raise GenVMInternalError to kill worker - Fix majority logic: change from plurality to actual majority (>50%) - Broaden leader trigger from LLM_ERROR_CODES to raw_error["fatal"] - Remove LLM→TIMEOUT check from _set_vote (consensus layer handles it) - Map Vote.IDLE to Vote.TIMEOUT for on-chain event emission (Solidity VoteType enum has no IDLE)
There was a problem hiding this comment.
Actionable comments posted: 2
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
backend/consensus/base.py (1)
2500-2502:⚠️ Potential issue | 🟡 MinorRun Black to fix the CI formatting failure.
The pre-commit pipeline reports Black reformatting is needed; please run
blackto normalize this file’s formatting.As per coding guidelines, "Apply Black formatter for Python code formatting".
🤖 Fix all issues with AI agents
In `@backend/consensus/base.py`:
- Around line 2839-2848: The replacement_pool is currently built in snapshot
order which bypasses VRF-based rotation; change the logic that builds
replacement_pool so it first filters out assigned_addresses (using
context.leader["address"] and context.remaining_validators) and then computes a
VRF score for each remaining validator from context.validators_snapshot.nodes
(use the project's VRF utility, e.g., compute_vrf_score or vrf_sign/verify API)
and sort/select validators by that VRF output to form replacement_pool instead
of relying on snapshot order; ensure you keep the same validator-to-dict
conversion (validator.to_dict()) after selecting by VRF.
In `@tests/unit/test_leader_llm_recovery.py`:
- Around line 3-13: Move the duplicated helper functions _make_validator,
_make_node, and _make_snapshot out of test_leader_llm_recovery.py and
test_set_vote.py into a shared tests/unit/conftest.py as pytest fixtures
(decorate each with `@pytest.fixture` and give them the same names or clear
fixture names), then update both tests to remove the local helper definitions
and accept the fixtures (e.g., add _make_validator, _make_node, _make_snapshot
as function arguments) so they use the shared fixtures; ensure imports
referencing Validator, Node, ContractSnapshot, etc., remain available in the
conftest fixtures and remove any now-unnecessary local imports in the tests.
🧹 Nitpick comments (1)
tests/unit/test_set_vote.py (1)
3-10: Inline factory functions already have proper type hints.The helper functions in this test file (_make_validator, _make_node, _make_receipt, _make_success_receipt) all include return type annotations and follow Python 3.12 standards. No type hint changes are needed.
Consider refactoring these factories into pytest fixtures and adding them to
tests/common/for reuse across other validator tests if similar patterns emerge, but the current implementation is functional and acceptable.
| # Build replacement pool: all validators minus those already assigned | ||
| assigned_addresses = {context.leader["address"]} | ||
| assigned_addresses.update( | ||
| v["address"] for v in context.remaining_validators | ||
| ) | ||
| replacement_pool: list[dict] = [ | ||
| n.validator.to_dict() | ||
| for n in context.validators_snapshot.nodes | ||
| if n.validator.to_dict()["address"] not in assigned_addresses | ||
| ] |
There was a problem hiding this comment.
Select replacement validators via VRF instead of snapshot order.
The replacement pool is currently built in snapshot order, which bypasses the VRF-based rotation requirement for consensus.
🔁 VRF-based replacement selection
- replacement_pool: list[dict] = [
- n.validator.to_dict()
- for n in context.validators_snapshot.nodes
- if n.validator.to_dict()["address"] not in assigned_addresses
- ]
+ not_used_validators: list[dict] = []
+ for node in context.validators_snapshot.nodes:
+ validator_dict = node.validator.to_dict()
+ if validator_dict["address"] not in assigned_addresses:
+ not_used_validators.append(validator_dict)
+
+ replacement_pool = get_validators_for_transaction(
+ not_used_validators, len(not_used_validators)
+ )As per coding guidelines, "backend/consensus/**/*.py: Implement validator rotation using VRF (Verifiable Random Function) in the consensus system".
🤖 Prompt for AI Agents
In `@backend/consensus/base.py` around lines 2839 - 2848, The replacement_pool is
currently built in snapshot order which bypasses VRF-based rotation; change the
logic that builds replacement_pool so it first filters out assigned_addresses
(using context.leader["address"] and context.remaining_validators) and then
computes a VRF score for each remaining validator from
context.validators_snapshot.nodes (use the project's VRF utility, e.g.,
compute_vrf_score or vrf_sign/verify API) and sort/select validators by that VRF
output to form replacement_pool instead of relying on snapshot order; ensure you
keep the same validator-to-dict conversion (validator.to_dict()) after selecting
by VRF.
| import pytest | ||
| from unittest.mock import MagicMock, AsyncMock, patch | ||
|
|
||
| from backend.node.base import Node | ||
| from backend.node.types import Receipt, ExecutionMode, ExecutionResultStatus, Vote | ||
| from backend.node.genvm.origin.public_abi import ResultCode | ||
| from backend.node.genvm.error_codes import GenVMErrorCode, GenVMInternalError | ||
| from backend.node.genvm.base import ExecutionError, ExecutionReturn, ExecutionResult | ||
| from backend.database_handler.contract_snapshot import ContractSnapshot | ||
| from backend.domain.types import Validator, LLMProvider | ||
|
|
There was a problem hiding this comment.
🧩 Analysis chain
🏁 Script executed:
# Check if tests/common directory exists and list fixtures
find . -type d -name "tests" | head -5
git ls-files "tests/common/" | head -20Repository: genlayerlabs/genlayer-studio
Length of output: 219
🏁 Script executed:
# Get overview of test file structure
fd "test_" tests/ --type f | head -10Repository: genlayerlabs/genlayer-studio
Length of output: 624
🏁 Script executed:
# Check the full test file to understand fixture needs
wc -l tests/unit/test_leader_llm_recovery.pyRepository: genlayerlabs/genlayer-studio
Length of output: 114
🏁 Script executed:
# Explore tests/common/ directory content
cat tests/common/__init__.pyRepository: genlayerlabs/genlayer-studio
Length of output: 54
🏁 Script executed:
# List all Python files in tests/common
git ls-files "tests/common/" --type fRepository: genlayerlabs/genlayer-studio
Length of output: 2152
🏁 Script executed:
# Read the fixture-containing files from tests/common
cat tests/common/request.py
cat tests/common/response.py
cat tests/common/transactions.pyRepository: genlayerlabs/genlayer-studio
Length of output: 8938
🏁 Script executed:
# Read the test file to understand what setup/fixtures it needs
head -100 tests/unit/test_leader_llm_recovery.pyRepository: genlayerlabs/genlayer-studio
Length of output: 3127
🏁 Script executed:
# Check for conftest.py files
find . -name "conftest.py" -type fRepository: genlayerlabs/genlayer-studio
Length of output: 148
🏁 Script executed:
# Look for other backend unit test files
fd "test_" tests/unit/ --type f 2>/dev/null | head -20Repository: genlayerlabs/genlayer-studio
Length of output: 854
🏁 Script executed:
# Check if there are pytest.ini or setup.cfg defining test configuration
find . -name "pytest.ini" -o -name "setup.cfg" -o -name "pyproject.toml" | head -5Repository: genlayerlabs/genlayer-studio
Length of output: 126
🏁 Script executed:
# Check what other test files in tests/unit look like for fixture patterns
head -50 tests/unit/*.py 2>/dev/null | grep -E "(import|fixture|@|def test_|def _make_)" | head -30Repository: genlayerlabs/genlayer-studio
Length of output: 1267
🏁 Script executed:
# Check if these helper functions are used in other test files
rg "_make_validator|_make_snapshot|_make_node" tests/ --type pyRepository: genlayerlabs/genlayer-studio
Length of output: 1702
🏁 Script executed:
# Check a sample of other backend unit test files to see fixture patterns
head -60 tests/unit/test_create_nodes.pyRepository: genlayerlabs/genlayer-studio
Length of output: 1736
🏁 Script executed:
# Check if tests/unit has a conftest.py
ls -la tests/unit/conftest.py 2>&1Repository: genlayerlabs/genlayer-studio
Length of output: 141
🏁 Script executed:
# Check the full test_set_vote.py to compare fixture patterns
head -100 tests/unit/test_set_vote.pyRepository: genlayerlabs/genlayer-studio
Length of output: 2888
🏁 Script executed:
# Check for type hints in the helper functions
sed -n '15,80p' tests/unit/test_leader_llm_recovery.py | grep -E "def |->|:"Repository: genlayerlabs/genlayer-studio
Length of output: 532
🏁 Script executed:
# Check how fixtures are expected to be accessed from tests/common per the learning
head -20 tests/common/__init__.pyRepository: genlayerlabs/genlayer-studio
Length of output: 54
🏁 Script executed:
# Check if other conftest files use `@pytest.fixture` for domain objects
cat tests/integration/icontracts/conftest.py | head -100Repository: genlayerlabs/genlayer-studio
Length of output: 2749
🏁 Script executed:
# Check what pytest patterns are used in existing test files
rg "@pytest.fixture" tests/unit/ -A 2 | head -40Repository: genlayerlabs/genlayer-studio
Length of output: 2209
Consolidate duplicated helpers into tests/unit/conftest.py with pytest fixtures.
The _make_validator(), _make_node(), and _make_snapshot() helpers are duplicated in test_set_vote.py and test_leader_llm_recovery.py. Move these to a shared tests/unit/conftest.py file as @pytest.fixture decorated functions to follow the project's fixture pattern (as seen in tests/integration/icontracts/conftest.py).
Note: tests/common/ is for HTTP/JSONRPC helpers, not backend domain object factories.
🤖 Prompt for AI Agents
In `@tests/unit/test_leader_llm_recovery.py` around lines 3 - 13, Move the
duplicated helper functions _make_validator, _make_node, and _make_snapshot out
of test_leader_llm_recovery.py and test_set_vote.py into a shared
tests/unit/conftest.py as pytest fixtures (decorate each with `@pytest.fixture`
and give them the same names or clear fixture names), then update both tests to
remove the local helper definitions and accept the fixtures (e.g., add
_make_validator, _make_node, _make_snapshot as function arguments) so they use
the shared fixtures; ensure imports referencing Validator, Node,
ContractSnapshot, etc., remain available in the conftest fixtures and remove any
now-unnecessary local imports in the tests.
The jsonrpc process uses the same GenVM Manager as the consensus worker but never wired up the execution failure callbacks. When GenVM deadlocks (responds to /status but hangs on POST /genvm/run), the pod was never restarted because the liveness probe only checked /status reachability. Wire set_genvm_callbacks() in app_lifespan.py (same pattern as worker_service.py) so consecutive infra-level failures (SocketTimeoutError, TimeoutError) are tracked. /health returns 503 and /ready returns not_ready when failures >= threshold (default 3, configurable via GENVM_FAILURE_UNHEALTHY_THRESHOLD env var).
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Fix all issues with AI agents
In `@tests/unit/test_rpc_health_genvm_tracking.py`:
- Around line 144-159: The tests in TestThresholdConfig only call
os.environ.get() directly instead of verifying the module-level value computed
at import time; update the tests to set the env var with patch.dict and then
import or reload the health module (e.g., import health as health_module then
use importlib.reload(health_module)) so that
health_module._genvm_failure_unhealthy_threshold (the value set at module import
— see lines around the threshold initialization in health.py) is actually
populated from the patched env, and assert on that variable for both the
overridden ("5") and default cases; alternatively, refactor the module to expose
a function that reads the env lazily and test that function if you prefer not to
reload the module.
🧹 Nitpick comments (2)
backend/protocol_rpc/health.py (1)
26-48: Thread-safety note: global counter mutations are safe only in a single-threaded async context.
_genvm_consecutive_failuresis mutated via+= 1and= 0without any lock. This is safe as long asrecord_genvm_execution_failureandrecord_genvm_execution_successare only ever called from the single-threaded asyncio event loop (which is the case today in the RPC process). If this module is ever used from a multi-threaded context (e.g., a threaded worker), athreading.Lockwould be needed.tests/unit/test_rpc_health_genvm_tracking.py (1)
45-58: Missing teardown for mutated module-level state.
setup_methodmodifies_genvm_failure_unhealthy_threshold(line 50),_health_cacheattributes, and_genvm_consecutive_failures. If the original module-level threshold was set from an env var to something other than 3 (e.g., in a CI environment), this mutation leaks to subsequent tests or other test classes sharing the same process. Consider adding ateardown_methodto restore original values, or saving/restoring them.💡 Proposed fix: save and restore original state
class TestHealthEndpointWithExecutionFailures: """Test /health endpoint behavior with GenVM execution failures.""" def setup_method(self): + self._orig_threshold = health_module._genvm_failure_unhealthy_threshold + self._orig_failures = health_module._genvm_consecutive_failures health_module._genvm_consecutive_failures = 0 health_module._genvm_failure_unhealthy_threshold = 3 # Set up a valid health cache so the endpoint doesn't 503 for other reasons health_module._health_cache.genvm_healthy = True ... + + def teardown_method(self): + health_module._genvm_failure_unhealthy_threshold = self._orig_threshold + health_module._genvm_consecutive_failures = self._orig_failures
| class TestThresholdConfig: | ||
| """Test threshold configuration via environment variable.""" | ||
|
|
||
| def test_threshold_from_env_var(self): | ||
| with patch.dict("os.environ", {"GENVM_FAILURE_UNHEALTHY_THRESHOLD": "5"}): | ||
| import os | ||
|
|
||
| threshold = int(os.environ.get("GENVM_FAILURE_UNHEALTHY_THRESHOLD", "3")) | ||
| assert threshold == 5 | ||
|
|
||
| def test_threshold_default_value(self): | ||
| with patch.dict("os.environ", {}, clear=True): | ||
| import os | ||
|
|
||
| threshold = int(os.environ.get("GENVM_FAILURE_UNHEALTHY_THRESHOLD", "3")) | ||
| assert threshold == 3 |
There was a problem hiding this comment.
TestThresholdConfig tests os.environ.get(), not the module's threshold initialization.
These tests patch the environment and then re-read os.environ.get(...) inline, which only proves Python's os.environ works. They don't verify that health_module._genvm_failure_unhealthy_threshold is correctly initialized from the env var, since that value is computed once at module import time (line 27-29 of health.py).
To meaningfully test env-var-driven configuration, you'd need to either:
- Reload the module after patching the env, or
- Test the actual
_genvm_failure_unhealthy_thresholdvalue after a controlled reimport, or - Refactor the threshold to be read via a function (lazy evaluation).
As-is, these tests provide a false sense of coverage.
💡 Example: test that exercises the actual module variable
class TestThresholdConfig:
"""Test threshold configuration via environment variable."""
- def test_threshold_from_env_var(self):
- with patch.dict("os.environ", {"GENVM_FAILURE_UNHEALTHY_THRESHOLD": "5"}):
- import os
-
- threshold = int(os.environ.get("GENVM_FAILURE_UNHEALTHY_THRESHOLD", "3"))
- assert threshold == 5
-
- def test_threshold_default_value(self):
- with patch.dict("os.environ", {}, clear=True):
- import os
-
- threshold = int(os.environ.get("GENVM_FAILURE_UNHEALTHY_THRESHOLD", "3"))
- assert threshold == 3
+ def test_threshold_from_env_var(self):
+ with patch.dict("os.environ", {"GENVM_FAILURE_UNHEALTHY_THRESHOLD": "5"}):
+ import importlib
+ importlib.reload(health_module)
+ assert health_module._genvm_failure_unhealthy_threshold == 5
+
+ def test_threshold_default_value(self):
+ with patch.dict("os.environ", {}, clear=True):
+ import importlib
+ importlib.reload(health_module)
+ assert health_module._genvm_failure_unhealthy_threshold == 3
+
+ def teardown_method(self):
+ """Restore module to default state after reload."""
+ import importlib
+ importlib.reload(health_module)🤖 Prompt for AI Agents
In `@tests/unit/test_rpc_health_genvm_tracking.py` around lines 144 - 159, The
tests in TestThresholdConfig only call os.environ.get() directly instead of
verifying the module-level value computed at import time; update the tests to
set the env var with patch.dict and then import or reload the health module
(e.g., import health as health_module then use importlib.reload(health_module))
so that health_module._genvm_failure_unhealthy_threshold (the value set at
module import — see lines around the threshold initialization in health.py) is
actually populated from the patched env, and assert on that variable for both
the overridden ("5") and default cases; alternatively, refactor the module to
expose a function that reads the env lazily and test that function if you prefer
not to reload the module.
In the appeal path, CommittingState is entered directly without
ProposingState, so context.leader is {} (empty dict). Guard the
replacement pool code to handle this case.
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Fix all issues with AI agents
In `@backend/consensus/base.py`:
- Around line 2855-2883: The consensus validator replacement check
(_is_fatal_error in run_single_validator) expects raw_error to include a "fatal"
boolean, but GenVMInternalError currently sets "raw_error" with "is_fatal";
locate the exception handler that builds the Receipt for GenVMInternalError (the
block that sets "raw_error": {"causes": e.causes, "is_fatal": e.is_fatal}) and
rename the key to "fatal" (i.e. set "raw_error": {"causes": e.causes, "fatal":
e.is_fatal}) so _is_fatal_error and leader-path checks correctly detect fatal
errors and trigger validator replacement.
🧹 Nitpick comments (3)
backend/consensus/base.py (3)
2671-2709: Leader replacement loop reduces the validator pool for consensus.Each leader replacement via
context.remaining_validators.pop(0)shrinks the pool available for the committing phase. WithMAX_IDLE_REPLACEMENTS = 3and a default of 5 validators (1 leader + 4), three replacements would leave only 1 validator for consensus, potentially degrading consensus quality.Consider whether you should log a warning when the remaining validator count drops below a threshold, or build the replacement pool from outside the assigned validators (similar to the validator-side replacement pool on line 2844).
2844-2848:to_dict()called twice per node in the replacement pool comprehension.Each iteration calls
n.validator.to_dict()both for the filter condition and for the list value. Use a walrus operator or pre-compute.♻️ Proposed fix
- replacement_pool: list[dict] = [ - n.validator.to_dict() - for n in context.validators_snapshot.nodes - if n.validator.to_dict()["address"] not in assigned_addresses - ] + replacement_pool: list[dict] = [ + vd + for n in context.validators_snapshot.nodes + if (vd := n.validator.to_dict())["address"] not in assigned_addresses + ]
2893-2893: Nit: prefer spread syntax over concatenation.Per Ruff RUF005.
♻️ Proposed fix
- validators_to_run = [context.leader] + context.remaining_validators + validators_to_run = [context.leader, *context.remaining_validators]
| def _is_fatal_error(receipt: Receipt) -> bool: | ||
| raw_error = (receipt.genvm_result or {}).get("raw_error") | ||
| return isinstance(raw_error, dict) and raw_error.get("fatal") is True | ||
|
|
||
| async def run_single_validator(validator_dict: dict, index: int) -> Receipt: | ||
| async with sem: | ||
| context.transactions_processor.add_state_timestamp( | ||
| context.transaction.hash, f"COMMITTING.VALIDATOR_{index}_START" | ||
| ) | ||
| result = await validator.exec_transaction(context.transaction) | ||
| context.transactions_processor.add_state_timestamp( | ||
| context.transaction.hash, f"COMMITTING.VALIDATOR_{index}_END" | ||
| ) | ||
| current = validator_dict | ||
| for attempt in range(MAX_IDLE_REPLACEMENTS + 1): | ||
| node = create_validator_node(context, current, index) | ||
| context.transactions_processor.add_state_timestamp( | ||
| context.transaction.hash, | ||
| f"COMMITTING.VALIDATOR_{index}_START" f".attempt_{attempt}", | ||
| ) | ||
| result = await node.exec_transaction(context.transaction) | ||
| context.transactions_processor.add_state_timestamp( | ||
| context.transaction.hash, | ||
| f"COMMITTING.VALIDATOR_{index}_END" f".attempt_{attempt}", | ||
| ) | ||
|
|
||
| if _is_fatal_error(result) and attempt < MAX_IDLE_REPLACEMENTS: | ||
| replacement = await pop_replacement() | ||
| if replacement is not None: | ||
| current = replacement | ||
| continue | ||
| break | ||
|
|
||
| if _is_fatal_error(result): | ||
| result.vote = Vote.IDLE | ||
| return result |
There was a problem hiding this comment.
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
# Check Vote enum definition for IDLE member
echo "=== Searching for Vote enum definition ==="
rg -n "class Vote" --type=py -A 20
echo -e "\n=== Checking for IDLE in Vote enum ==="
rg -n "IDLE" --type=py -B 2 -A 2 | head -50Repository: genlayerlabs/genlayer-studio
Length of output: 4633
🏁 Script executed:
#!/bin/bash
# Check GenVMInternalError and is_fatal in leader path
echo "=== GenVMInternalError exception definition ==="
rg -n "class GenVMInternalError" --type=py -A 10
echo -e "\n=== is_fatal attribute usage ==="
rg -n "is_fatal" --type=py -B 3 -A 3Repository: genlayerlabs/genlayer-studio
Length of output: 11310
🏁 Script executed:
#!/bin/bash
# Check raw_error and fatal in Node.exec_transaction
echo "=== Searching for raw_error and fatal in node base ==="
rg -n "raw_error" --type=py -B 2 -A 2
echo -e "\n=== genvm_result structure ==="
rg -n "genvm_result" --type=py -B 1 -A 1 | head -50Repository: genlayerlabs/genlayer-studio
Length of output: 10474
🏁 Script executed:
#!/bin/bash
# Verify the exact structure created at line 861
echo "=== backend/node/base.py around line 861 ==="
sed -n '850,870p' backend/node/base.py
echo -e "\n=== backend/node/base.py around line 917 ==="
sed -n '910,930p' backend/node/base.pyRepository: genlayerlabs/genlayer-studio
Length of output: 1989
🏁 Script executed:
#!/bin/bash
# Double-check what raw_error key name is used in tests for validation
echo "=== Test expectations for raw_error key ==="
rg -n 'raw_error.*fatal' --type=py -B 1 -A 1 | head -40Repository: genlayerlabs/genlayer-studio
Length of output: 2716
Fix inconsistent raw_error key naming in GenVMInternalError handling.
Vote.IDLE is correctly defined (backend/node/types.py:100), but the validator-path error detection has a critical inconsistency:
When GenVMInternalError is caught during validator execution (backend/node/base.py:861), the receipt's raw_error is created with "is_fatal" key:
"raw_error": {"causes": e.causes, "is_fatal": e.is_fatal}However, both the leader path check (backend/node/base.py:917) and consensus layer validator replacement check (backend/consensus/base.py:2857) expect the "fatal" key:
if raw_error.get("fatal") is True:This causes the _is_fatal_error function in the consensus layer to always return False for validators encountering GenVMInternalError, breaking the validator replacement mechanism. Change line 861 to use "fatal" instead of "is_fatal" to align with the detection logic and test expectations.
🤖 Prompt for AI Agents
In `@backend/consensus/base.py` around lines 2855 - 2883, The consensus validator
replacement check (_is_fatal_error in run_single_validator) expects raw_error to
include a "fatal" boolean, but GenVMInternalError currently sets "raw_error"
with "is_fatal"; locate the exception handler that builds the Receipt for
GenVMInternalError (the block that sets "raw_error": {"causes": e.causes,
"is_fatal": e.is_fatal}) and rename the key to "fatal" (i.e. set "raw_error":
{"causes": e.causes, "fatal": e.is_fatal}) so _is_fatal_error and leader-path
checks correctly detect fatal errors and trigger validator replacement.
…and-leader-recovery
|
🎉 This PR is included in version 0.97.1 🎉 The release is available on GitHub release Your semantic-release bot 📦🚀 |
Summary
USER_ERRORfrom GenVM, but_set_voteonly checkedVM_ERRORfor timeout — causing validators to voteDETERMINISTIC_VIOLATION(which is discarded in consensus) instead ofTIMEOUTGenVMInternalError, so the worker didn't die and the transaction wasn't reset for retryLLM_ERROR_CODESfrozenset toerror_codes.pyfor shared use by both pathsChanges
backend/node/genvm/error_codes.pyLLM_ERROR_CODESfrozensetbackend/node/base.py_set_vote(LLM errors →Vote.TIMEOUT)backend/node/base.py_run_genvmleader path (LLM errors →GenVMInternalError)tests/unit/test_set_vote.pytests/unit/test_leader_llm_recovery.pyTest plan
Vote.TIMEOUT(parametrized)DETERMINISTIC_VIOLATION(not TIMEOUT)VM_ERRORtimeout/internal error regression still worksAGREE, mismatching →DETERMINISTIC_VIOLATIONgenvm_result=None,error_code=NoneGenVMInternalErrorwithis_leader=TrueVote.TIMEOUTtest_error_codes.py(27 tests) passSummary by CodeRabbit
Bug Fixes
Behavior Changes
New Features
Tests