Skip to content

feat: worker generic error retry + DB test coverage + error detail logging - #1439

Merged
MuncleUscles merged 10 commits into
mainfrom
feat/worker-error-retry-and-db-tests-v2
Feb 11, 2026
Merged

MuncleUscles merged 10 commits into
mainfrom
feat/worker-error-retry-and-db-tests-v2

Conversation

@MuncleUscles

@MuncleUscles MuncleUscles commented Feb 11, 2026 •

Copy link
Copy Markdown
Member

Summary

  • Worker error surfacing: Transactions hitting generic exceptions in _transaction_context now get exponential backoff (10s base) and cancel after 3 retries instead of silently retrying forever. Health endpoint exposes generic_error_retries count.
  • GenVM error detail logging: GenVMInternalError now carries a detail field with the raw error string (truncated 1000 chars) so worker logs show the actual upstream HTTP status/response from OpenRouter instead of just LLM_EXECUTION_ERROR.
  • DB test coverage: 25 new integration tests for TransactionsProcessor raw SQL methods.

Test plan

  • DB integration tests pass (docker compose -f tests/db-sqlalchemy/docker-compose.yml --project-directory . run --build --rm tests)
  • Worker service starts and processes transactions normally
  • Health endpoint includes generic_error_retries field
  • Transactions that fail 3 times are CANCELED with diagnostic consensus_data
  • Worker logs show detail=ModuleError { causes: [...], ctx: { status: 429, ... } } on GenVM errors

Summary by CodeRabbit

  • New Features

    • Bounded concurrency for GenVM calls to prevent unbounded parallel operations.
  • Bug Fixes

    • Transactions encountering transient/internal VM errors now auto-retry with exponential backoff and cancel after max attempts.
  • Chores

    • Error reports include richer contextual detail.
    • Health endpoints now surface generic-error retry counts and refined worker metrics (occupied/runnable/needed).
    • Increased startup healthcheck grace period.
  • Tests

    • Expanded test coverage for transaction processing and consensus behaviors; added shorter test fixture alias.

…coverage

Worker: transactions that fail with generic exceptions now get exponential
backoff (10s base) and cancel after 3 retries instead of silently retrying
forever. Health endpoint exposes retry count for observability.

Tests: 25 new DB integration tests covering update_transaction_status,
consensus_history, set_transaction_result, timestamps, appeals,
state_timestamp, get_transactions, contract_snapshot, and rotation_count.
Add `detail` field to GenVMInternalError carrying the raw error string
(truncated to 1000 chars). Worker logs now include the full Rust
ModuleError with ctx (HTTP status, response body from OpenRouter),
so we can see *why* LLM calls fail instead of just "LLM_EXECUTION_ERROR".
@coderabbitai

coderabbitai Bot commented Feb 11, 2026 •

Copy link
Copy Markdown
Contributor
📝 Walkthrough

Walkthrough

Added a generic-error retry mechanism with exponential backoff and max-retry limits to the consensus worker; GenVM INTERNAL_ERROR now carries an optional detail string; worker health now reports active generic-error retries; JSON-RPC GenVM calls are concurrency-limited; tests and docker-compose adjustments included.

Changes

Cohort / File(s) Summary
GenVM error shape & propagation
backend/node/genvm/error_codes.py, backend/node/genvm/base.py
Add optional `detail: str
Consensus worker: generic-error retry
backend/consensus/worker.py
Add MAX_GENERIC_ERROR_RETRIES, per-tx _generic_error_retries and _generic_error_base_backoff, implement _handle_generic_error_retry(...), integrate generic-error backoff into _is_in_backoff, clear retry state on success, and extend GenVM error logging.
Health endpoint (worker service)
backend/consensus/worker_service.py
health_check now includes generic_error_retries (counts entries in worker._generic_error_retries when present).
JSON-RPC GenVM concurrency limiter
backend/protocol_rpc/endpoints.py
Introduce module-level semaphore (_genvm_semaphore) bounded by GENVM_MAX_CONCURRENT and wrap GenVM call paths to limit concurrent GenVM socket usage.
Tests & fixtures
tests/db-sqlalchemy/conftest.py, tests/db-sqlalchemy/transactions_processor_test.py, tests/integration/rpc/test_jsonrpc_endpoint.py
Add tp fixture alias; expand transactions processor tests with helpers/mocks and new imports; adjust JSON-RPC readiness test to set GenVM health flag.
Metrics & health
backend/protocol_rpc/health.py
Replace single active-workers gauge with per-request Prometheus registry exposing genlayer_occupied_contracts, genlayer_runnable_contracts, and recomputed genlayer_needed_workers.
Deployment config
docker-compose.yml
Increase start_period for two healthchecks from 40s to 180s.

Sequence Diagram

sequenceDiagram
    participant Client as Client
    participant TxWorker as ConsensusWorker
    participant GenVM as GenVM Executor
    participant State as State/DB
    participant WS as WebSocket Notifier

    Client->>TxWorker: Submit transaction
    TxWorker->>GenVM: Execute transaction
    GenVM-->>TxWorker: GenVMInternalError (with detail)

    TxWorker->>TxWorker: _handle_generic_error_retry(tx_hash, error)
    alt retries < MAX_GENERIC_ERROR_RETRIES
        TxWorker->>TxWorker: increment retry count & set base backoff
        TxWorker->>TxWorker: schedule exponential backoff
        TxWorker-->>Client: defer / retry later
    else retries >= MAX_GENERIC_ERROR_RETRIES
        TxWorker->>State: persist tx status = CANCELED with error detail
        TxWorker->>WS: notify cancellation
        TxWorker->>TxWorker: remove retry tracking
    end

    Note over TxWorker: On next processing attempt\n_is_in_backoff checks generic-error backoff
Loading

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~45 minutes

Possibly related PRs

Suggested labels

Studio

Suggested reviewers

  • kstroobants
  • cristiam86

Poem

🐰 I nibbled logs beneath the tree,
retried a bug—backoff times three,
details tucked in error notes,
canceled when the last bell tolled,
I hop, I watch, persistent—then free. 🥕

🚥 Pre-merge checks | ✅ 2 | ❌ 1
❌ Failed checks (1 warning)
Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 32.56% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (2 passed)
Check name Status Explanation
Title check ✅ Passed The PR title accurately summarizes the three main changes: worker generic error retry capability, database test coverage expansion, and error detail logging enhancement.
Description check ✅ Passed The PR description provides a clear summary, concrete test plan, and explains the what/why of changes, though it lacks sections for decisions made and reviewing tips from the template.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing touches
  • 📝 Generate docstrings
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Post copyable unit tests in a comment
  • Commit unit tests in branch feat/worker-error-retry-and-db-tests-v2

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

…by_hash compat

_prepare_basic_transaction_data expects leader_result[0]["node_config"]["address"]
to exist when consensus_results is present. Pass _MockReceipt with proper
node_config instead of None.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Fix all issues with AI agents
In `@tests/db-sqlalchemy/transactions_processor_test.py`:
- Around line 529-535: In test_get_highest_timestamp_ignores_null, the local
variable tx2 is assigned by calling _make_tx(tp) but never used; to silence
static analysis and indicate the value is intentionally unused, replace the tx2
assignment with an underscore (e.g., _ = _make_tx(tp)) or remove the assignment
entirely so the call to _make_tx(tp) is still executed; update the test body
around the _make_tx call in test_get_highest_timestamp_ignores_null accordingly.
🧹 Nitpick comments (5)
backend/node/genvm/error_codes.py (1)

57-62: __repr__ does not include the new detail field.

All other instance attributes (error_code, causes, is_fatal, is_leader) are surfaced in __repr__, but the newly added detail is omitted. This means repr(exc) won't show the diagnostic context that detail was specifically added to carry. Consider adding it (perhaps truncated) for debugging consistency.

Proposed fix
     def __repr__(self) -> str:
         return (
             f"GenVMInternalError(message={self.args[0]!r}, "
             f"error_code={self.error_code!r}, causes={self.causes!r}, "
-            f"is_fatal={self.is_fatal!r}, is_leader={self.is_leader!r})"
+            f"is_fatal={self.is_fatal!r}, is_leader={self.is_leader!r}, "
+            f"detail={self.detail!r})"
         )
tests/db-sqlalchemy/transactions_processor_test.py (2)

24-43: Missing type hints on _make_tx helper.

Per coding guidelines, all Python code should include type hints. The function parameters and return type are untyped.

Proposed fix
-def _make_tx(tp, **overrides):
+def _make_tx(tp: TransactionsProcessor, **overrides: object) -> str:
     """Create a transaction with sensible defaults and unique hash."""

As per coding guidelines: "Include type hints in all Python code".


424-430: _MockReceipt lacks type hints and strip_contract_state is intentionally unused.

The unused parameter mirrors the real Receipt.to_dict interface, which is fine. Adding a type hint would satisfy the coding guidelines.

Proposed fix
 class _MockReceipt:
-    def to_dict(self, strip_contract_state=False):
+    def to_dict(self, strip_contract_state: bool = False) -> dict:
         return {

As per coding guidelines: "Include type hints in all Python code".

backend/consensus/worker.py (2)

993-1001: Unbounded str(error) stored in consensus_data — consider truncating.

str(error) can be arbitrarily long (e.g., full stack traces or large HTTP response bodies). This is persisted to the DB's JSONB column and returned through APIs. The detail field on GenVMInternalError already truncates to 1000 chars; applying similar bounds here would be consistent.

Proposed fix
                 tx.status = TransactionStatus.CANCELED
                 tx.consensus_data = {
                     "error": "max_generic_retries_exceeded",
-                    "last_error": str(error),
+                    "last_error": str(error)[:1000],
                     "retries": retry_info["count"],
                 }

43-43: MAX_GENERIC_ERROR_RETRIES is a class constant while the analogous no-validators limit is env-configurable.

This is fine for now, but if operators need to tune retry behavior without redeploying, consider making it env-configurable like NO_VALIDATORS_MAX_RETRIES.

Comment on lines +529 to +535
def test_get_highest_timestamp_ignores_null(self, tp):
tx1 = _make_tx(tp)
tx2 = _make_tx(tp)
tp.set_transaction_timestamp_awaiting_finalization(tx1, 9999)
tp.session.commit()
# tx2 has no timestamp set
assert tp.get_highest_timestamp() == 9999

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor

Unused variable tx2 — assign to _ or remove the name.

Static analysis flags tx2 as assigned but never used. The intent is to create a second transaction without a timestamp, but the returned hash isn't needed. Use _ to signal this is intentional.

Proposed fix
     def test_get_highest_timestamp_ignores_null(self, tp):
         tx1 = _make_tx(tp)
-        tx2 = _make_tx(tp)
+        _make_tx(tp)  # second tx with no timestamp set
         tp.set_transaction_timestamp_awaiting_finalization(tx1, 9999)
         tp.session.commit()
         # tx2 has no timestamp set
         assert tp.get_highest_timestamp() == 9999
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
def test_get_highest_timestamp_ignores_null(self, tp):
tx1 = _make_tx(tp)
tx2 = _make_tx(tp)
tp.set_transaction_timestamp_awaiting_finalization(tx1, 9999)
tp.session.commit()
# tx2 has no timestamp set
assert tp.get_highest_timestamp() == 9999
def test_get_highest_timestamp_ignores_null(self, tp):
tx1 = _make_tx(tp)
_make_tx(tp) # second tx with no timestamp set
tp.set_transaction_timestamp_awaiting_finalization(tx1, 9999)
tp.session.commit()
# tx2 has no timestamp set
assert tp.get_highest_timestamp() == 9999
🧰 Tools
🪛 Ruff (0.15.0)

[error] 531-531: Local variable tx2 is assigned to but never used

Remove assignment to unused variable tx2

(F841)

🤖 Prompt for AI Agents
In `@tests/db-sqlalchemy/transactions_processor_test.py` around lines 529 - 535,
In test_get_highest_timestamp_ignores_null, the local variable tx2 is assigned
by calling _make_tx(tp) but never used; to silence static analysis and indicate
the value is intentionally unused, replace the tx2 assignment with an underscore
(e.g., _ = _make_tx(tp)) or remove the assignment entirely so the call to
_make_tx(tp) is still executed; update the test body around the _make_tx call in
test_get_highest_timestamp_ignores_null accordingly.

…ilation

GenVM precompiles 6+ WASM modules on first start (~18s each = ~108s total).
The previous 40s start_period caused healthcheck failures in CI before the
Flask app could even start.
Commit 621a568 changed HealthCache.genvm_healthy default from True to
False. The readiness integration test has no GenVM manager, so the check
short-circuits to 503 before evaluating the RPC router. Set the cache
flag explicitly so the test exercises the intended code path.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)
tests/db-sqlalchemy/transactions_processor_test.py (1)

1-1: ⚠️ Potential issue | 🟡 Minor

Rename to test_transactions_processor.py to follow pytest auto-discovery conventions.

The file uses the *_test.py pattern, but pytest's default discovery only recognizes test_*.py and Test*.py patterns. No custom pytest configuration was found in the repository. The coding guidelines require test_<feature>.py naming for pytest auto-discovery. Rename to test_transactions_processor.py.

🧹 Nitpick comments (2)
tests/db-sqlalchemy/transactions_processor_test.py (2)

21-43: Global _tx_counter can leak state across test runs in the same process.

If pytest runs multiple test files in the same process (the default), _tx_counter accumulates across modules and never resets. This isn't a correctness bug today since hashes remain unique, but it makes test isolation fragile. Consider using a pytest fixture or itertools.count() instance scoped to conftest instead.


463-478: session fixture parameter is unused in these test methods.

Both test_update_consensus_history_empty_initial and test_update_consensus_history_appends_rounds accept session but only use tp (which already owns a session). Safe to remove the extra parameter to silence static analysis.

Proposed fix
-    def test_update_consensus_history_empty_initial(self, tp, session):
+    def test_update_consensus_history_empty_initial(self, tp):
         tx_hash = _make_tx(tp)
-    def test_update_consensus_history_appends_rounds(self, tp, session):
+    def test_update_consensus_history_appends_rounds(self, tp):
         tx_hash = _make_tx(tp)

gen_call had no concurrency limit, allowing unbounded concurrent GenVM
executions that cause uvloop file descriptor registry collisions
(RuntimeError: File descriptor N is used by transport). Workers already
use asyncio.Semaphore(8) in consensus/base.py; this adds the same
limit to the jsonrpc gen_call path. Configurable via GENVM_MAX_CONCURRENT.
Replace worker_id-based genlayer_needed_workers metric with per-contract
schedulable work measurement. Old metric counted distinct worker_ids from
last hour (supply-based, double-counted during rolling upgrades). New
metric counts distinct contracts with occupied (in-flight) or runnable
(pending, no in-flight) transactions — directly measuring max useful
parallelism since transactions per contract are processed sequentially.

New gauges: genlayer_occupied_contracts, genlayer_runnable_contracts

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Fix all issues with AI agents
In `@backend/protocol_rpc/health.py`:
- Around line 1151-1169: The metrics SQL in the per_contract CTE currently
filters statuses to ('PENDING','PROPOSING','COMMITTING','UNDETERMINED') which
diverges from the consensus health check’s broader processing set; either expand
the status list in the CTE to match the consensus health-counting states
(include 'ACTIVATED','REVEALING','ACCEPTED') so occupied/runnable calculations
(SELECT ... AS occupied, ... AS runnable) align with the health/debug endpoints,
or add a clear comment above the query explaining why the narrower set is
intentionally used for HPA scaling decisions; also replace the stray
logging.exception() call with the module logger.exception() used elsewhere to
keep logging consistent.
🧹 Nitpick comments (2)
backend/protocol_rpc/health.py (2)

1154-1161: to_address can be NULL — deploy transactions will be grouped into a single bucket.

Other queries in this file (e.g., Line 526) filter WHERE to_address IS NOT NULL. Without that filter, all deploy transactions (NULL to_address) collapse into one group, which is mostly harmless but may slightly skew the occupied/runnable counts.

Consider adding AND to_address IS NOT NULL to the CTE's WHERE clause for consistency.

Proposed fix
                         FROM transactions
-                        WHERE status IN ('PENDING', 'PROPOSING', 'COMMITTING', 'UNDETERMINED')
+                        WHERE status IN ('PENDING', 'PROPOSING', 'COMMITTING', 'UNDETERMINED')
+                          AND to_address IS NOT NULL
                         GROUP BY to_address

1204-1205: Use logger.exception instead of logging.exception.

Line 1205 uses the module-level logging.exception(...) while the rest of the file consistently uses the logger instance defined on Line 18. This bypasses the module's logger name, making log filtering/routing harder.

Proposed fix
-        logging.exception("Metrics endpoint failed")
+        logger.exception("Metrics endpoint failed")

Comment on lines +1151 to +1169
row = conn.execute(
text(
"""
WITH per_contract AS (
SELECT
to_address,
BOOL_OR(status = 'PENDING') AS has_pending,
BOOL_OR(status IN ('PROPOSING', 'COMMITTING', 'UNDETERMINED')) AS has_inflight
FROM transactions
WHERE status IN ('PENDING', 'PROPOSING', 'COMMITTING', 'UNDETERMINED')
GROUP BY to_address
)
SELECT
COALESCE(COUNT(*) FILTER (WHERE has_inflight), 0) AS occupied,
COALESCE(COUNT(*) FILTER (WHERE has_pending AND NOT has_inflight), 0) AS runnable
FROM per_contract
"""
)
)
).fetchone()

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
# First, let's check the file size and read the relevant sections
wc -l backend/protocol_rpc/health.py

Repository: genlayerlabs/genlayer-studio

Length of output: 107


🏁 Script executed:

#!/bin/bash
# Check the status usage around line 523 (the consensus health check)
sed -n '515,535p' backend/protocol_rpc/health.py

Repository: genlayerlabs/genlayer-studio

Length of output: 972


🏁 Script executed:

#!/bin/bash
# Check the status usage in the metrics query (lines 1144-1176)
sed -n '1144,1176p' backend/protocol_rpc/health.py

Repository: genlayerlabs/genlayer-studio

Length of output: 1702


🏁 Script executed:

#!/bin/bash
# Check the logging inconsistency around line 1205
sed -n '1200,1210p' backend/protocol_rpc/health.py

Repository: genlayerlabs/genlayer-studio

Length of output: 534


🏁 Script executed:

#!/bin/bash
# Search for all transaction status checks in the file to understand the full context
rg -n "status\s*(?:IN|=|!=)" backend/protocol_rpc/health.py -A2 -B2

Repository: genlayerlabs/genlayer-studio

Length of output: 8095


Status set in metrics query differs from health checks; clarify intent.

The metrics query (lines 1160, 1158) uses ('PENDING', 'PROPOSING', 'COMMITTING', 'UNDETERMINED') to measure occupied and runnable contracts, but the consensus health check (line 523) and debug endpoint (line 1024) count ('ACTIVATED', 'PROPOSING', 'COMMITTING', 'REVEALING', 'ACCEPTED', 'UNDETERMINED') as processing transactions. If transactions spend non-trivial time in ACTIVATED, REVEALING, or ACCEPTED states, the metrics endpoint will undercount occupied contracts and could cause HPA to scale down prematurely.

Consider aligning the status set with the broader definition used in consensus health checks, or add a comment explaining why the narrower set is intentional for HPA scaling decisions.

Additionally, line 1205 uses logging.exception() instead of the logger.exception() pattern used elsewhere in the file (e.g., line 887).

🤖 Prompt for AI Agents
In `@backend/protocol_rpc/health.py` around lines 1151 - 1169, The metrics SQL in
the per_contract CTE currently filters statuses to
('PENDING','PROPOSING','COMMITTING','UNDETERMINED') which diverges from the
consensus health check’s broader processing set; either expand the status list
in the CTE to match the consensus health-counting states (include
'ACTIVATED','REVEALING','ACCEPTED') so occupied/runnable calculations (SELECT
... AS occupied, ... AS runnable) align with the health/debug endpoints, or add
a clear comment above the query explaining why the narrower set is intentionally
used for HPA scaling decisions; also replace the stray logging.exception() call
with the module logger.exception() used elsewhere to keep logging consistent.

@MuncleUscles
MuncleUscles merged commit e5b5cf0 into main Feb 11, 2026
23 checks passed
@github-actions

Copy link
Copy Markdown
Contributor

🎉 This PR is included in version 0.99.0 🎉

The release is available on GitHub release

Your semantic-release bot 📦🚀

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant