Skip to content

feat: forward LLM error details to transaction receipts - #1441

Merged
cristiam86 merged 8 commits into
mainfrom
feat/include-llm-error-in-receipts
Feb 13, 2026
Merged

cristiam86 merged 8 commits into
mainfrom
feat/include-llm-error-in-receipts

Conversation

@cristiam86

@cristiam86 cristiam86 commented Feb 12, 2026 •

Copy link
Copy Markdown
Contributor

What

  • Enhanced llm.lua to capture structured error info (HTTP status, model, provider, error message) from LLM provider failures and pass it through lib.rs.user_error() ctx
  • Added parse_ctx_from_module_error_string() parser in error_codes.py to extract primary_error / fallback_error from the Rust debug format string that GenVM produces for INTERNAL_ERROR
  • Added ctx field to GenVMInternalError exception so error context flows through the Python exception chain
  • Added _extract_llm_error_result() to Receipt to surface LLM errors as llm_error_result in the transaction receipt dict
  • Fixed raw_error construction in node/base.py to include ctx and use correct "fatal" key

Why

  • When an LLM provider fails (invalid API key, rate limiting, etc.), the error details were lost during the GenVM → Python error propagation. The result_data arrives as a Rust debug format string (not a Python dict), so the structured ctx containing error details was silently discarded.
  • This made it impossible for users and operators to understand why a validator failed — the receipt only showed a generic "NO_PROVIDER_FOR_PROMPT" error code without the HTTP status, provider name, or error message.

Testing done

  • Added 7 unit tests for parse_ctx_from_module_error_string covering: field extraction, nested Maps, escaped quotes, Nil values, integer status conversion
  • All 295 backend unit tests pass
  • All 31 DB/SQLAlchemy integration tests pass
  • All 103 frontend unit tests pass

Decisions made

  • Used character-by-character brace matching (not regex) for extracting nested Map({...}) content from Rust debug strings, since nested structures break regex-based approaches
  • Only extract status, model, provider, error_message fields — these are the actionable fields users need. The body field (which may contain large nested error payloads) is deliberately excluded from llm_error_result
  • The llm_error_result field is None when no LLM errors occur, keeping receipts clean for successful transactions

Checks

  • I have tested this code
  • I have reviewed my own PR
  • I have created an issue for this PR
  • I have set a descriptive PR title compliant with conventional commits

Reviewing tips

  • Start with backend/node/llm.lua to see how error info is now captured at the source
  • Then follow the propagation: error_codes.py (parser) → genvm/base.py (provide_result) → node/base.py (receipt construction) → types.py (receipt serialization)
  • The Rust debug format parsing in error_codes.py is the core new logic — the test cases show the format clearly

User facing release notes

  • Transaction receipts now include detailed LLM error information (llm_error_result) when a validator's LLM provider fails, showing the HTTP status code, provider name, model, and error message

🤖 Generated with Claude Code

Summary by CodeRabbit

Release Notes

  • New Features

    • Enhanced error reporting with detailed context information, including status codes and error details for improved diagnostics.
  • Improvements

    • Improved system reliability through increased health check retry attempts.
    • Optimized transaction processing performance with eager data loading.
    • Enhanced error handling and logging for better visibility into system failures.

cristiam86 and others added 5 commits February 11, 2026 12:38
Add eager loading with selectinload for the triggered_transactions
relationship in TransactionsProcessor methods that fetch multiple
transactions and call _parse_transaction_data. This eliminates the
N+1 query problem detected by Sentry (GENLAYER-STUDIO-NA).

Methods updated:
- get_activated_transactions_older_than
- get_transactions_for_address
- get_transactions_for_block (conditionally, only when include_full_tx)
- get_newer_transactions
- transactions_in_process_by_contract

Fixes GENLAYER-STUDIO-NA

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
@coderabbitai

coderabbitai Bot commented Feb 12, 2026 •

Copy link
Copy Markdown
Contributor
📝 Walkthrough

Walkthrough

The PR introduces enhanced error context propagation through GenVM execution and consensus layers, adds structured error reporting with contextual metadata, optimizes database query performance via eager loading, refines error handling with improved diagnostic logging across multiple execution paths, and increases healthcheck resilience in Docker configuration.

Changes

Cohort / File(s) Summary
Database Query Optimization
backend/database_handler/transactions_processor.py
Added selectinload eager-loading for triggered_transactions across multiple query methods (get_activated_transactions_older_than, get_transactions_for_address, get_transactions_for_block, get_newer_transactions, transactions_in_process_by_contract) to reduce lazy-loading overhead; no signature changes.
GenVM Error Context Infrastructure
backend/node/genvm/error_codes.py
Added parse_ctx_from_module_error_string function with parsing helpers to extract error context (status, model, provider, error_message) from ModuleError strings; updated GenVMInternalError constructor to accept optional ctx parameter and include it in repr; added internal parsing patterns and utilities.
GenVM Error Handling
backend/node/genvm/base.py, backend/node/base.py
Modified INTERNAL_ERROR handling to distinguish between dict and string result_data, extracting context via parse_ctx_from_module_error_string; updated GenVMInternalError invocation to pass ctx; simplified _run_genvm to collapse error handling into single re-raise path with is_leader flag.
LLM Provider Error Reporting
backend/node/llm.lua
Enhanced try_provider to return structured error_info objects with status, error_message, body, model, and provider fields; updated just_in_backend to capture primary and fallback error details and propagate context via lib.rs.user_error; added diagnostic logging for provider availability and execution results.
Consensus Error Handling & Transaction Ordering
backend/consensus/base.py, backend/consensus/worker.py
Added internal error handling receipts for GenVMInternalError in leader replacement and validator execution paths; reordered transaction status updates to occur after triggered transaction insertion or event emission to prevent race conditions; added logging with context (ctx field) for error diagnostics.
Test Coverage
tests/unit/test_error_codes.py
Added comprehensive test suite for parse_ctx_from_module_error_string covering missing primary_error, field extraction (status, model, provider, error_message), nested map handling, fallback_error presence, escaped quotes, and type validation.
Infrastructure Configuration
docker-compose.yml
Increased jsonrpc healthcheck retries from 3 to 5 to improve service resilience without altering check interval or timeout.

Sequence Diagram

sequenceDiagram
    participant GenVM as GenVM Execution
    participant GenVMBase as genvm/base.py
    participant ErrorCodes as error_codes.py
    participant Consensus as Consensus Layer
    participant Logger as Logging

    GenVM->>GenVM: Execute (ModuleError occurs)
    GenVM->>GenVMBase: Raise error with result_data
    GenVMBase->>GenVMBase: Check if result_data is dict
    alt result_data is dict
        GenVMBase->>GenVMBase: Extract ctx, error_code, causes, is_fatal
    else result_data is string
        GenVMBase->>ErrorCodes: parse_ctx_from_module_error_string(error_str)
        ErrorCodes->>ErrorCodes: Parse ModuleError format
        ErrorCodes->>ErrorCodes: Extract primary_error, fallback_error
        ErrorCodes->>ErrorCodes: Extract fields (status, model, provider, error_message)
        ErrorCodes->>GenVMBase: Return ctx dict
        GenVMBase->>GenVMBase: Derive ctx from string parsing
    end
    GenVMBase->>GenVMBase: Create GenVMInternalError(ctx=ctx, ...)
    GenVMBase->>Consensus: Raise GenVMInternalError with ctx
    Consensus->>Consensus: Build internal error receipt
    Consensus->>Logger: Log GenVMInternalError
    Logger->>Logger: Include ctx in log output
    Consensus->>Consensus: Update transaction status (with propagated context)
Loading

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~50 minutes

Possibly related PRs

Suggested labels

run-tests

Suggested reviewers

  • kstroobants

Poem

🐰 A context-aware error hops through the GenVM,
Each layer extracts wisdom from the ModuleError stem,
Consensus now knows the model, provider, and why,
While databases eager-load without a sigh,
Healthchecks retry with patience, and logs shine bright—
This refactor makes error-handling just right! ✨

🚥 Pre-merge checks | ✅ 2 | ❌ 1
❌ Failed checks (1 warning)
Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 58.33% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (2 passed)
Check name Status Explanation
Title check ✅ Passed The title 'feat: forward LLM error details to transaction receipts' clearly and specifically describes the main change: capturing and propagating LLM provider error information into transaction receipts.
Description check ✅ Passed The PR description comprehensively covers all template sections: What (specific changes made), Why (rationale), Testing done (with test counts), Decisions made (parsing approach and field selection), and Checking tips for reviewers.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing touches
  • 📝 Generate docstrings
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Post copyable unit tests in a comment
  • Commit unit tests in branch feat/include-llm-error-in-receipts

No actionable comments were generated in the recent review. 🎉


Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🤖 Fix all issues with AI agents
In `@backend/node/genvm/base.py`:
- Around line 200-201: Remove the unused alias import "from loguru import logger
as _ilog" (symbol _ilog) in base.py; either delete that import line or replace
it with the intended logger usage (e.g., use the existing logger variable
instead of introducing _ilog) so there are no unused imports left.

In `@backend/node/llm.lua`:
- Around line 218-233: The code may index a nil as_user_error.ctx when
evaluating llm.overloaded_statuses[as_user_error.ctx.status]; to fix, capture
the context safely (e.g. local ctx = as_user_error.ctx) and use a guarded status
lookup (local status = ctx and ctx.status) before referencing it, update
error_info.status with that safe status, and only index llm.overloaded_statuses
when status is non-nil; ensure the lib.log calls use the safe ctx/status
variables and keep existing branches (as_user_error present vs nil) intact.

Comment on lines +200 to +201
from loguru import logger as _ilog

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor

Unused import: _ilog is imported but never referenced.

from loguru import logger as _ilog on Line 200 appears to be a debug leftover — it's not used anywhere in this block.

🧹 Proposed fix
-            from loguru import logger as _ilog
-
             error_ctx = None
🤖 Prompt for AI Agents
In `@backend/node/genvm/base.py` around lines 200 - 201, Remove the unused alias
import "from loguru import logger as _ilog" (symbol _ilog) in base.py; either
delete that import line or replace it with the intended logger usage (e.g., use
the existing logger variable instead of introducing _ilog) so there are no
unused imports left.

Comment thread backend/node/llm.lua
Comment on lines +218 to +233
local as_user_error = lib.rs.as_user_error(result)
if as_user_error ~= nil then
error_info.status = as_user_error.ctx and as_user_error.ctx.status or nil
error_info.error_message = tostring(result)
if as_user_error.ctx and as_user_error.ctx.body then
error_info.body = as_user_error.ctx.body
end
if llm.overloaded_statuses[as_user_error.ctx.status] then
lib.log{level = "warning", message = "service is overloaded", error = as_user_error, request = request}
else
lib.log{level = "warning", message = "provider failed", error = as_user_error, request = request}
end
else
error_info.error_message = tostring(result)
lib.log{level = "warning", message = "provider failed with unexpected error", error = tostring(result), request = request}
end

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

Potential nil dereference on as_user_error.ctx at Line 225.

Lines 220 and 222 guard against nil ctx (as_user_error.ctx and ...), but Line 225 accesses as_user_error.ctx.status without a nil check. If as_user_error.ctx is nil, this will throw a Lua "attempt to index a nil value" error, breaking the error info construction.

🐛 Proposed fix — add nil guard
-		if llm.overloaded_statuses[as_user_error.ctx.status] then
+		if as_user_error.ctx and llm.overloaded_statuses[as_user_error.ctx.status] then
🤖 Prompt for AI Agents
In `@backend/node/llm.lua` around lines 218 - 233, The code may index a nil
as_user_error.ctx when evaluating
llm.overloaded_statuses[as_user_error.ctx.status]; to fix, capture the context
safely (e.g. local ctx = as_user_error.ctx) and use a guarded status lookup
(local status = ctx and ctx.status) before referencing it, update
error_info.status with that safe status, and only index llm.overloaded_statuses
when status is non-nil; ensure the lib.log calls use the safe ctx/status
variables and keep existing branches (as_user_error present vs nil) intact.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)
backend/node/genvm/base.py (1)

199-238: ⚠️ Potential issue | 🔴 Critical

NameError when result_data is a dict: error_str is only assigned in the else branch.

When res.result_data is a dict (Line 205 branch), execution skips Line 212 where error_str is assigned, but Line 238 unconditionally references error_str[:1000]. This will crash with a NameError instead of properly raising GenVMInternalError.

🐛 Proposed fix
         elif res.result_kind == ResultCode.INTERNAL_ERROR:
-            from loguru import logger as _ilog
-
             error_ctx = None
+            error_str = str(res.result_data)

             # Try to extract structured data if result_data is a dict
             if isinstance(res.result_data, dict):

This ensures error_str is always defined before it's used at Line 238. It also removes the unused _ilog import (see below).

@MuncleUscles

Copy link
Copy Markdown
Member

I'm not sure those errors belong in the receipt, I think logs is the right place

  • Receipt - for developers/users of the network
  • Logs - for manager of the studio

@cristiam86

Copy link
Copy Markdown
Contributor Author

I'm not sure those errors belong in the receipt, I think logs is the right place

  • Receipt - for developers/users of the network

  • Logs - for manager of the studio

You are right, the intention here is to be able to access them fast in the studio explorer

- Removed error from receipt
- Log error with context
- Fixed race condition on triggered transactions
@cristiam86

Copy link
Copy Markdown
Contributor Author

I'm not sure those errors belong in the receipt, I think logs is the right place

  • Receipt - for developers/users of the network
  • Logs - for manager of the studio

Removed from the first level of the receipt and also fixed a race condition in the triggered transactions integration tests due to the query optimization that you did recently

@cristiam86
cristiam86 merged commit 20a31b9 into main Feb 13, 2026
10 checks passed
@github-actions

Copy link
Copy Markdown
Contributor

🎉 This PR is included in version 0.100.0 🎉

The release is available on GitHub release

Your semantic-release bot 📦🚀

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants