Skip to content

fix: genvm invalid contract error - #1363

Merged
cristiam86 merged 9 commits into
mainfrom
debug-genvm-invalid-contract-error
Dec 18, 2025
Merged

cristiam86 merged 9 commits into
mainfrom
debug-genvm-invalid-contract-error

Conversation

@cristiam86

@cristiam86 cristiam86 commented Dec 18, 2025 •

Copy link
Copy Markdown
Contributor

Summary

Improves GenVM/consensus-worker reliability and health reporting by increasing timeouts, surfacing blocked_at state, and making the worker health check fail when a transaction is stuck too long.

What changed

  • Bump GenVM version: GENVM_TAG updated from v0.2.4 → v0.2.7 in docker-compose.yml.
  • Reduce flaky GenVM calls: Node-side request timeout increased from 5s → 30s in backend/node/base.py.
  • Tighten consensus transaction timeout: default transaction_timeout_minutes reduced from 30 → 20 in backend/consensus/worker.py.
  • Expose blocked transaction state: blocked_at is now selected and returned in worker transaction queries in backend/consensus/worker.py.
  • More robust health check (backend/consensus/worker_service.py):
    • Handles blocked_at as either ISO string or datetime
    • Normalizes timezone-aware values to naive UTC for comparisons
    • Returns 500 if a transaction has been blocked longer than the configured threshold (currently elapsed > 600s)
    • Logs parsing errors instead of silently swallowing them
    • Captures the pkill return code (for observability/debugging)

Why

  • Prevents false negatives and intermittent failures due to overly aggressive timeouts.
  • Makes “stuck transaction” conditions visible and actionable via health checks (better orchestration/restarts).
  • Keeps GenVM runtime aligned with the newer tagged version.

Testing / verification

  • Manual: start stack and confirm worker /health reports blocked_at when present and returns 500 once the blocked duration exceeds the threshold.
  • Spot-check: confirm Node GenVM calls no longer fail under normal latency spikes with the 30s timeout.

Summary by CodeRabbit

  • Bug Fixes

    • Improved health monitoring to detect/report blocked work items, with robust parsing and a 10-minute unhealthiness threshold.
  • Chores

    • Updated GenVM container to v0.2.7.
    • Shortened default transaction timeout to 20 minutes.
    • Increased contract schema retrieval timeout for stability.
    • Added development tooling configuration enabling three editor plugins by default.

✏️ Tip: You can customize this high-level summary in your review settings.

@coderabbitai

coderabbitai Bot commented Dec 18, 2025 •

Copy link
Copy Markdown
Contributor

Walkthrough

Adds a Claude plugin config, reduces ConsensusWorker transaction timeout to 20 minutes, propagates blocked_at in claim queries/returns, tightens health-check logic to mark items blocked >10 minutes as unhealthy, increases host schema timeout to 30s, and bumps GenVM image tag to v0.2.7.

Changes

Cohort / File(s) Summary
Claude Plugin Configuration
\.claude/settings.json
New file enabling three Claude plugins: code-review@claude-plugins-official, feature-dev@claude-plugins-official, and hookify@claude-plugins-official.
Consensus Worker: claims & timeout
backend/consensus/worker.py
Default transaction_timeout_minutes changed from 30 → 20; SQL SELECTs updated to include blocked_at; claim_next_transaction, claim_next_appeal, and claim_next_finalization now return blocked_at in their result dicts.
Consensus Worker Service: health check
backend/consensus/worker_service.py
Health endpoint now accepts blocked_at as ISO string or datetime, converts to naive UTC, computes blocked duration, returns unhealthy (500) when >10 minutes, and logs parsing errors without failing health evaluation; removed PKILL side-effect in favor of capturing return code.
Node host timeout
backend/node/base.py
Increased get_contract_schema host execution timeout from 5s → 30s.
Container image update
docker-compose.yml
Updated GENVM_TAG for jsonrpc service from v0.2.4 → v0.2.7.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~22 minutes

  • Attention areas:
    • backend/consensus/worker_service.py: verify robust handling of blocked_at types, timezone normalization, and correct threshold use (10 minutes).
    • backend/consensus/worker.py: confirm blocked_at is selected in all DB paths and consumers handle optional/null values.
    • backend/node/base.py and docker-compose.yml: validate new timeouts and GenVM tag operate as expected in integration.

Possibly related PRs

Suggested reviewers

  • MuncleUscles

Poem

🐰 Hopping through commits with a twitchy nose,

Timeouts trimmed where the river flows,
Blocked-at timestamps now wake the bell,
Health checks listen and ring the knell,
Docker hops forward — new GenVM grows.

Pre-merge checks and finishing touches

❌ Failed checks (1 inconclusive)
Check name Status Explanation Resolution
Title check ❓ Inconclusive The PR title 'fix: genvm invalid contract error' is only partially related to the main changes; the core work involves timeouts, health checks, and blocked_at state tracking, not exclusively GenVM invalid contract errors. Consider a more comprehensive title that reflects the primary focus: 'fix: improve consensus worker reliability with timeout and health check enhancements' or similar.
✅ Passed checks (2 passed)
Check name Status Explanation
Description check ✅ Passed The PR description provides comprehensive coverage of changes, rationale, and testing approach, though testing is described as manual rather than automated, which aligns with the template's expectations for the Testing done section.
Docstring Coverage ✅ Passed Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.
✨ Finishing touches
  • 📝 Generate docstrings
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Post copyable unit tests in a comment
  • Commit unit tests in branch debug-genvm-invalid-contract-error

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🧹 Nitpick comments (2)
backend/consensus/worker_service.py (1)

64-64: Captured return code is unused.

_pkill_rc captures the return code but it's never used. The underscore prefix suggests intentional non-use, but per the PR summary, this was added "for observability." Consider logging the return code if observability was the intent.

-    _pkill_rc = os.system("pkill -9 -f 'genvm (llm|web)' 2>/dev/null || true")
-    logger.info("GenVM cleanup complete")
+    pkill_rc = os.system("pkill -9 -f 'genvm (llm|web)' 2>/dev/null || true")
+    logger.info(f"GenVM cleanup complete (exit code: {pkill_rc})")
backend/consensus/worker.py (1)

45-45: Mismatch between parameter default and environment variable default.

The transaction_timeout_minutes parameter defaults to 20 minutes, but the TRANSACTION_TIMEOUT_MINUTES environment variable in docker-compose.yml defaults to 30 minutes. When running via docker-compose, the environment variable takes precedence, but these inconsistent code-level defaults can cause confusion. Align the parameter default to 20 to match the code intent, or change it to 30 for consistency with docker-compose configuration.

📜 Review details

Configuration used: defaults

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between afe5895 and dcb8f79.

📒 Files selected for processing (5)
  • .claude/settings.json (1 hunks)
  • backend/consensus/worker.py (7 hunks)
  • backend/consensus/worker_service.py (4 hunks)
  • backend/node/base.py (1 hunks)
  • docker-compose.yml (1 hunks)
🧰 Additional context used
📓 Path-based instructions (3)
**/*.py

📄 CodeRabbit inference engine (CLAUDE.md)

Use Black formatter for Python code

Target Python 3.12, use 4-space indentation, and rely on Black via pre-commit for formatting consistency

Files:

  • backend/consensus/worker.py
  • backend/node/base.py
  • backend/consensus/worker_service.py
backend/**/*.py

📄 CodeRabbit inference engine (CLAUDE.md)

backend/**/*.py: Include type hints in Python code
Use SQLAlchemy ORM with Alembic migrations for database operations

Align backend filenames with their behavior (e.g., validators/llm_validator.py) and mirror that pattern in tests

Files:

  • backend/consensus/worker.py
  • backend/node/base.py
  • backend/consensus/worker_service.py
docker-compose*.{yml,yaml}

📄 CodeRabbit inference engine (CLAUDE.md)

Use Docker Compose for orchestrating microservices (genlayer up/down commands)

Files:

  • docker-compose.yml
🧠 Learnings (1)
📚 Learning: 2025-12-04T13:40:37.336Z
Learnt from: CR
Repo: genlayerlabs/genlayer-studio PR: 0
File: CLAUDE.md:0-0
Timestamp: 2025-12-04T13:40:37.336Z
Learning: Applies to docker-compose*.{yml,yaml} : Use Docker Compose for orchestrating microservices (genlayer up/down commands)

Applied to files:

  • docker-compose.yml
🧬 Code graph analysis (1)
backend/consensus/worker_service.py (2)
backend/node/genvm/origin/logger.py (1)
  • error (23-24)
backend/protocol_rpc/message_handler/fastapi_handler.py (1)
  • error (156-165)
🪛 Ruff (0.14.8)
backend/consensus/worker_service.py

64-64: Starting a process with a shell: seems safe, but may be changed in the future; consider rewriting without shell

(S605)


64-64: Starting a process with a partial executable path

(S607)


371-371: Do not catch blind exception: Exception

(BLE001)

⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (1)
  • GitHub Check: backend-unit-tests
🔇 Additional comments (8)
backend/node/base.py (1)

695-695: LGTM! Timeout increase for schema retrieval is appropriate.

The 6x increase from 5s to 30s provides adequate headroom for GenVM schema retrieval during latency spikes, aligning with the PR's reliability improvements.

.claude/settings.json (1)

1-7: Configuration file for Claude plugins looks reasonable.

This enables official Claude plugins for code review, feature development, and hookify. While unrelated to the GenVM debugging objective of this PR, the configuration is valid.

backend/consensus/worker.py (3)

161-161: LGTM! blocked_at field properly added to finalization claim.

The field is correctly selected in the SQL RETURNING clause and mapped to the returned dictionary.


267-268: LGTM! blocked_at field properly added to appeal claim.

Consistent with the finalization claim changes.


370-370: LGTM! blocked_at field properly added to transaction claim.

All three claim methods now consistently return the blocked_at field for health check monitoring.

backend/consensus/worker_service.py (2)

344-356: LGTM! Robust blocked_at parsing handles both string and datetime types.

The parsing logic correctly handles ISO strings (with Z suffix) and datetime objects, normalizing to naive UTC for consistent comparison. This aligns with the worker.py changes that now return blocked_at from SQL queries.


371-374: Exception handling is appropriate for defensive parsing.

Logging the error without failing the health check is reasonable here—a parsing failure shouldn't mark the pod as unhealthy. The bare Exception catch is acceptable given the context.

docker-compose.yml (1)

93-93: GenVM v0.2.7 is a documented, official release and correctly configured.

The version bump from v0.2.4 to v0.2.7 is properly implemented in both docker-compose.yml (line 93) and Dockerfile.backend, which download the binaries from the official genlayerlabs/genvm GitHub releases. The environment variable is correctly passed through and used for multi-platform binary extraction.

Comment thread backend/consensus/worker_service.py Resolved

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🧹 Nitpick comments (1)
backend/consensus/worker_service.py (1)

377-380: Consider more specific exception handling.

While catching Exception is defensive here, consider narrowing to specific exceptions like ValueError (for ISO parsing) and TypeError (for type issues) to avoid masking unexpected errors.

Also, the pass statement at line 380 is redundant since there's no code after the except block.

-            except Exception as e:
+            except (ValueError, TypeError, AttributeError) as e:
                 # Log the error but don't fail the health check
                 logger.error(f"Error parsing blocked_at timestamp: {e}")
-                pass
📜 Review details

Configuration used: defaults

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between dcb8f79 and 16892d2.

📒 Files selected for processing (1)
  • backend/consensus/worker_service.py (4 hunks)
🧰 Additional context used
📓 Path-based instructions (2)
**/*.py

📄 CodeRabbit inference engine (CLAUDE.md)

Use Black formatter for Python code

Target Python 3.12, use 4-space indentation, and rely on Black via pre-commit for formatting consistency

Files:

  • backend/consensus/worker_service.py
backend/**/*.py

📄 CodeRabbit inference engine (CLAUDE.md)

backend/**/*.py: Include type hints in Python code
Use SQLAlchemy ORM with Alembic migrations for database operations

Align backend filenames with their behavior (e.g., validators/llm_validator.py) and mirror that pattern in tests

Files:

  • backend/consensus/worker_service.py
🧬 Code graph analysis (1)
backend/consensus/worker_service.py (2)
backend/node/genvm/origin/logger.py (1)
  • error (23-24)
backend/protocol_rpc/message_handler/fastapi_handler.py (1)
  • error (156-165)
🪛 Ruff (0.14.8)
backend/consensus/worker_service.py

64-64: Starting a process with a shell: seems safe, but may be changed in the future; consider rewriting without shell

(S605)


64-64: Starting a process with a partial executable path

(S607)


377-377: Do not catch blind exception: Exception

(BLE001)

⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (1)
  • GitHub Check: backend-unit-tests
🔇 Additional comments (1)
backend/consensus/worker_service.py (1)

64-64: Good observability improvement.

Capturing the return code is a helpful addition for debugging, even if not currently used. The leading underscore convention appropriately signals the variable is intentionally unused.

Note: The static analysis warning about shell commands is a false positive here since the command string is fixed with no user input.

Comment on lines +344 to +368
blocked_at = tx["blocked_at"]

# Handle both datetime objects and ISO strings
if isinstance(blocked_at, str):
blocked_at = datetime.fromisoformat(
blocked_at.replace("Z", "+00:00")
)

# Convert to naive UTC datetime for comparison
if blocked_at.tzinfo is not None:
blocked_at = blocked_at.replace(tzinfo=None)

elapsed = datetime.utcnow() - blocked_at

# Check if blocked for more than 10 minutes - pod is unhealthy
if elapsed.total_seconds() > 600: # 10 minutes = 600 seconds
return JSONResponse(
status_code=500,
content={
"status": "unhealthy",
"worker_id": worker.worker_id,
"error": "Transaction blocked for more than 10 minutes",
"blocked_duration_seconds": elapsed.total_seconds(),
},
)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor

Replace datetime.utcnow() with datetime.now(timezone.utc) for Python 3.12 compliance.

datetime.utcnow() is deprecated in Python 3.12. Replace it with datetime.now(timezone.utc) at line 356. Import timezone from the datetime module.

Additionally, the current code converts parsed blocked_at to naive datetime. To align with Python 3.12 best practices, ensure blocked_at remains timezone-aware (UTC) throughout the function, then perform the subtraction with an aware datetime.now(timezone.utc). This eliminates the manual timezone stripping.

Add return type hints to health_check() per coding guidelines for backend Python code.

🤖 Prompt for AI Agents
In backend/consensus/worker_service.py around lines 344 to 368, the code uses
datetime.utcnow() and strips tzinfo from parsed blocked_at; update to keep
blocked_at timezone-aware (parse ISO strings into UTC-aware datetimes and do not
replace tzinfo=None) and replace datetime.utcnow() with
datetime.now(timezone.utc) so the subtraction is between two aware UTC
datetimes; add "from datetime import timezone" to imports and adjust parsing to
ensure blocked_at has tzinfo=timezone.utc when it's a string; also add an
appropriate return type hint to the health_check() function signature per
backend guidelines.

@sonarqubecloud

Copy link
Copy Markdown

Quality Gate Failed Quality Gate failed

Failed conditions
0.0% Coverage on New Code (required ≥ 80%)
C Reliability Rating on New Code (required ≥ A)

See analysis details on SonarQube Cloud

Catch issues before they fail your Quality Gate with our IDE extension SonarQube for IDE

@cristiam86 cristiam86 changed the title Debug genvm invalid contract error fix: genvm invalid contract error Dec 18, 2025
@cristiam86
cristiam86 merged commit b121467 into main Dec 18, 2025
26 of 29 checks passed
@github-actions

Copy link
Copy Markdown
Contributor

🎉 This PR is included in version 0.83.6 🎉

The release is available on GitHub release

Your semantic-release bot 📦🚀

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant