Skip to content

fix: validator VM crash should vote DISAGREE, not AGREE - #1448

Merged
MuncleUscles merged 1 commit into
mainfrom
fix/validator-vote-vm-crash-disagree
Feb 18, 2026
Merged

MuncleUscles merged 1 commit into
mainfrom
fix/validator-vote-vm-crash-disagree

Conversation

@MuncleUscles

@MuncleUscles MuncleUscles commented Feb 18, 2026 •

Copy link
Copy Markdown
Member

Summary

  • Bug: _set_vote compared raw result bytes between leader and validator. When both crashed with the same VM error (e.g. exit_code 1), they matched and voted AGREE — even though neither validated anything.
  • Fix: VM errors (exit_code, OOM, etc.) now always vote DISAGREE. Only RETURN and USER_ERROR with matching deterministic properties (execution_result, contract_state, pending_transactions) vote AGREE.
  • Tests: 5 new test cases covering the critical behavior changes, all existing tests pass.

Vote determination logic (new)

GenVM result Vote
VM_ERROR + timeout / internal error TIMEOUT
nondet_disagree set DISAGREE
VM_ERROR + exit_code / OOM / other DISAGREE (was AGREE)
RETURN / USER_ERROR + state mismatch DETERMINISTIC_VIOLATION
RETURN / USER_ERROR + state matches AGREE

Production incident

On Rally testnet, a contract's leader crashed with exit_code 1 (LLM returned malformed response). Validators also crashed (couldn't unpack leader's error). All voted AGREE because raw error bytes matched. Transaction finalized as "agreed" with no actual validation.

Test plan

  • All 13 test_set_vote.py tests pass (8 existing + 5 new)
  • All 303 unit tests pass
  • All 52 DB/SQLAlchemy tests pass
  • CI pipeline

Summary by CodeRabbit

Release Notes

  • Bug Fixes
    • Enhanced validator consensus voting with improved handling of VM errors, timeouts, and exception cases
    • Better detection of non-deterministic scenarios during validator consensus validation
    • Refined deterministic violation detection across validator execution results and contract states
    • Improved prioritization of validation failures for more accurate consensus decisions

The _set_vote logic was comparing raw result bytes between leader and
validator receipts. When both crashed with the same VM error (e.g.
exit_code, OOM), they'd match byte-for-byte and vote AGREE — even
though neither actually validated anything.

New logic:
- RETURN/USER_ERROR → check deterministic properties (state, pending_txs,
  execution_result) for DETERMINISTIC_VIOLATION, otherwise AGREE
- VM_ERROR + timeout/internal → TIMEOUT
- VM_ERROR + anything else (exit_code, OOM) → DISAGREE
- nondet_disagree signal from GenVM → DISAGREE

Result bytes are no longer compared for vote determination. Only
execution outcome category and contract state matter.
@coderabbitai

coderabbitai Bot commented Feb 18, 2026 •

Copy link
Copy Markdown
Contributor
📝 Walkthrough

Walkthrough

The changes refine the _set_vote method in Node's voting mechanism to distinguish between different VM error types (timeouts vs. other errors), add handling for nondeterministic disagreements, and shift deterministic violation detection from strict conjunction to disjunctive conditions. Corresponding test cases validate these new behaviors.

Changes

Cohort / File(s) Summary
Voting Logic Enhancement
backend/node/base.py
Enhanced _set_vote method with granular VM error handling: differentiates timeout/GenVM internal errors (Vote.TIMEOUT) from other VM_ERROR cases (Vote.DISAGREE), adds nondeterministic disagreement detection, and refines deterministic violation checks using disjunctive conditions on execution_result, contract_state, and pending_transactions fields.
Test Suite Updates
tests/unit/test_set_vote.py
Added comprehensive test cases covering VM_ERROR non-timeout scenarios expecting DISAGREE, nondeterministic disagreement precedence, and USER_ERROR handling with state matching for AGREE decisions.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~25 minutes

Poem

🐰 Hark! The votes now discern with care,
Timeouts caught, VM errors laid bare,
When nondet whispers disagreement's call,
And state divergence breaks the wall,
Yet matching fields let concord ring true—
Logic refined, the votes renew! ✨

🚥 Pre-merge checks | ✅ 2 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 66.67% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (2 passed)
Check name Status Explanation
Title check ✅ Passed The title accurately and specifically describes the main bug fix: VM errors should result in DISAGREE votes instead of AGREE, directly reflecting the core change in the changeset.
Description check ✅ Passed The description is comprehensive and well-structured, covering the bug, fix, voting logic, production incident, and test results. However, some template sections are incomplete: 'Decisions made', 'Reviewing tips', and 'User facing release notes' are missing or minimal.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
  • 📝 Generate docstrings
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Post copyable unit tests in a comment
  • Commit unit tests in branch fix/validator-vote-vm-crash-disagree

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick comments (2)
backend/node/base.py (1)

700-702: Defensive check: receipt.result must be non-empty.

receipt.result[0] will raise IndexError if result is an empty bytes object. While current callers always produce a result with at least one byte (via encode_result_to_bytes), a defensive guard would make this more robust against future changes.

🛡️ Optional defensive check
 def _set_vote(self, receipt: Receipt) -> Receipt:
+    if not receipt.result:
+        receipt.vote = Vote.DISAGREE
+        return receipt
     result_code = receipt.result[0]
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@backend/node/base.py` around lines 700 - 702, The _set_vote method accesses
receipt.result[0] without verifying receipt.result is non-empty; add a defensive
check at the start of _set_vote (method name: _set_vote, type: Receipt,
property: Receipt.result) that verifies receipt.result is not empty (e.g., if
not receipt.result: raise ValueError(...) or return an explicit error), and
include a clear error message referencing the Receipt (or its id) so callers can
diagnose unexpected empty results; ensure the rest of the function uses the
validated result[0] after this guard.
tests/unit/test_set_vote.py (1)

301-319: Add explicit test for pending_transactions divergence in the AGREE path.

The _set_vote logic compares pending_transactions as part of the deterministic violation check (line 727 of backend/node/base.py). However, the _make_receipt helper does not support setting pending_transactions, so all current tests implicitly rely on the default empty tuple (). This means the comparison branch is never explicitly exercised. Add a test where pending_transactions differs between leader and validator to ensure it triggers Vote.DETERMINISTIC_VIOLATION.

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@tests/unit/test_set_vote.py` around lines 301 - 319, Add a new unit test
variant that exercises the deterministic-violation branch by creating a leader
receipt and a validator receipt via _make_receipt where you explicitly set
differing pending_transactions values (e.g., leader uses default or () and
validator uses a non-empty tuple) and then call
node._set_vote(validator_receipt) to assert the returned vote is
Vote.DETERMINISTIC_VIOLATION; update or extend
test_user_error_matching_state_votes_agree (or create a new test function) so
_make_receipt is called with a pending_transactions argument to force the
divergence and validate the deterministic-violation path in the _set_vote logic.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Nitpick comments:
In `@backend/node/base.py`:
- Around line 700-702: The _set_vote method accesses receipt.result[0] without
verifying receipt.result is non-empty; add a defensive check at the start of
_set_vote (method name: _set_vote, type: Receipt, property: Receipt.result) that
verifies receipt.result is not empty (e.g., if not receipt.result: raise
ValueError(...) or return an explicit error), and include a clear error message
referencing the Receipt (or its id) so callers can diagnose unexpected empty
results; ensure the rest of the function uses the validated result[0] after this
guard.

In `@tests/unit/test_set_vote.py`:
- Around line 301-319: Add a new unit test variant that exercises the
deterministic-violation branch by creating a leader receipt and a validator
receipt via _make_receipt where you explicitly set differing
pending_transactions values (e.g., leader uses default or () and validator uses
a non-empty tuple) and then call node._set_vote(validator_receipt) to assert the
returned vote is Vote.DETERMINISTIC_VIOLATION; update or extend
test_user_error_matching_state_votes_agree (or create a new test function) so
_make_receipt is called with a pending_transactions argument to force the
divergence and validate the deterministic-violation path in the _set_vote logic.

@MuncleUscles
MuncleUscles merged commit 03483f1 into main Feb 18, 2026
25 checks passed
@github-actions

Copy link
Copy Markdown
Contributor

🎉 This PR is included in version 0.101.1 🎉

The release is available on GitHub release

Your semantic-release bot 📦🚀

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants