Skip to content

perf: background health checker for fast /health endpoint - #1372

Merged
cristiam86 merged 5 commits into
mainfrom
perf/background-health-checker
Jan 13, 2026
Merged

cristiam86 merged 5 commits into
mainfrom
perf/background-health-checker

Conversation

@MuncleUscles

@MuncleUscles MuncleUscles commented Jan 7, 2026 •

Copy link
Copy Markdown
Member

Summary

  • Move expensive health checks (DB, consensus, memory, Redis, GenVM) to a background task that runs periodically
  • /health endpoint now returns cached results (~1ms response time instead of 14+ seconds)
  • Background task runs every 10s by default (configurable via HEALTH_CHECK_INTERVAL_SECONDS env var)
  • GenVM failures still return 503 to trigger container restart (critical for K8s health)
  • Added /ping endpoint for ultra-lightweight load balancer checks
  • Kept detailed /health/* endpoints for debugging (run checks synchronously)

Problem

The /health endpoint was taking 14+ seconds intermittently due to:

  1. Complex SQL aggregation queries for consensus status
  2. Database pool stat collection
  3. Memory/psutil checks
  4. Redis ping
  5. GenVM HTTP probe

When Cloudflare/LB health checks coincided with user requests, this caused cascading delays.

Solution

Background task runs health checks periodically and caches results. The /health endpoint just returns the cached data, making it consistently fast (~1ms).

Test plan

  • DB-sqlalchemy tests pass (24/24)
  • CI should run full test suite
  • Verify /health responds quickly after deployment
  • Verify /health still reflects actual service health (not stale)

Summary by CodeRabbit

  • New Features

    • Continuous background health monitoring with cached results for faster responses.
    • Automatic start/stop of the health checker during app startup/shutdown.
    • Added lightweight /ping and readiness endpoints for load balancers and orchestration.
    • Expanded per-service health endpoints with detailed reporting; overall health now returns 503 if core service is unhealthy.
  • Bug Fixes

    • Normalized contract address input during admin upgrade flow to align validation and signer checks.

✏️ Tip: You can customize this high-level summary in your review settings.

- Move expensive health checks (DB, consensus, memory, Redis, GenVM) to background task
- Background task runs every 10s (configurable via HEALTH_CHECK_INTERVAL_SECONDS)
- /health endpoint now returns cached results (~1ms response vs 14s+ before)
- GenVM failures still return 503 to trigger container restart
- Add /ping endpoint for ultra-lightweight load balancer checks
- Keep detailed /health/* endpoints for debugging (run checks synchronously)

This fixes intermittent RPC slowness caused by /health taking 14+ seconds
when load balancer health checks coincide with request processing.
@coderabbitai

coderabbitai Bot commented Jan 7, 2026 •

Copy link
Copy Markdown
Contributor

Caution

Review failed

The pull request is closed.

📝 Walkthrough

Walkthrough

Starts a background health checker at app startup that periodically probes services (DB, consensus, memory, Redis, GenVM), caches results in a HealthCache, exposes cached /health and per-service endpoints plus /ping, and stops the checker on app shutdown.

Changes

Cohort / File(s) Summary
App Lifecycle Integration
backend/protocol_rpc/app_lifespan.py
Import start_background_health_checker and stop_background_health_checker; start the background checker after RPC router creation on startup and stop it during shutdown.
Background Health System
backend/protocol_rpc/health.py
Add HealthCache dataclass, background loop and task control (start_background_health_checker, stop_background_health_checker, get_health_check_interval); implement _run_health_checks and _background_health_loop, per-service _check_* helpers, cached /health (503 if GenVM unhealthy), /ping, and detailed /health/* endpoints; improved error handling and logging.
Endpoint Normalization
backend/protocol_rpc/endpoints.py
Normalize contract_address to checksum format in admin_upgrade_contract_code before authorization/nonce/signature checks.

Sequence Diagram(s)

sequenceDiagram
    participant App as Application
    participant Lifespan as App Lifespan
    participant Router as RPC Router
    participant BG as Background Checker
    participant Services as Services (DB, Consensus, Memory, Redis, GenVM)
    participant Cache as HealthCache
    participant Endpoint as /health Endpoint
    participant Client as Client

    App->>Lifespan: startup
    activate Lifespan
    Lifespan->>Router: create RPC router
    Lifespan->>BG: start_background_health_checker(rpc_router)
    activate BG
    Lifespan-->>App: startup complete
    deactivate Lifespan

    loop every N seconds
        BG->>Services: run _check_* probes
        Services-->>BG: probe results
        BG->>Cache: update status, timestamps, durations, issues
    end

    Client->>Endpoint: GET /health
    Endpoint->>Cache: read latest health
    Cache-->>Endpoint: return cached payload
    alt GenVM unhealthy
        Endpoint-->>Client: 503 + details
    else
        Endpoint-->>Client: 200 + cached health
    end

    App->>Lifespan: shutdown
    activate Lifespan
    Lifespan->>BG: stop_background_health_checker()
    deactivate BG
    Lifespan-->>App: shutdown complete
Loading

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~20 minutes

Poem

🐇 I nibble logs in silent night,
I ping the nodes till morning light.
I store their hum in tidy cache,
From startup bell to shutdown's hush.
A tiny rabbit, keeping watch.

🚥 Pre-merge checks | ✅ 3
✅ Passed checks (3 passed)
Check name Status Explanation
Title check ✅ Passed The title clearly and specifically describes the main change: implementing a background health checker to improve /health endpoint performance.
Description check ✅ Passed The description covers the key sections including summary, problem, solution, and test plan, but is missing several template sections like 'What', 'Why', 'Decisions made', and 'Checks'.
Docstring Coverage ✅ Passed Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing touches
  • 📝 Generate docstrings

📜 Recent review details

Configuration used: defaults

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between eb605d7 and 9750f47.

📒 Files selected for processing (1)
  • backend/protocol_rpc/health.py

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

Comment thread backend/protocol_rpc/health.py Fixed
Comment thread backend/protocol_rpc/health.py Fixed

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 4

🤖 Fix all issues with AI agents
In @backend/protocol_rpc/health.py:
- Around line 410-423: The health endpoint currently returns raw internal error
details via the "error" field from _health_cache, exposing sensitive internals;
remove or replace that field in the returned dict so external consumers only get
a sanitized status (e.g., error: "internal_error" or None) and keep full
exception details logged server-side instead; update the function building this
response to stop including _health_cache.error and ensure any logging of
_health_cache.error uses the server logger (not the HTTP response) and retains
get_health_check_interval(), _health_cache.status, _health_cache.issues, and
_health_cache.services as before.
- Around line 386-387: The endpoint decorator for health_check is causing
FastAPI to try to generate a Pydantic model from the return type Union[dict,
JSONResponse]; update the route decorator on @health_router.get("/health") to
disable response model generation by adding response_model=None, and optionally
adjust health_check's return type annotation (e.g., to Any or
starlette.responses.Response) to avoid Union-based type hints that trigger
Pydantic processing.
- Around line 392-402: The HTTP response in the genvm health check currently
returns internal error text via _health_cache.genvm_error in the JSONResponse
(see the block that returns status_code=503); remove or replace the "detail"
field so it does not expose internal errors to clients — for example, omit
"detail" or set it to a non-sensitive string like "contact support" or "internal
error" — and rely on the existing internal logging in _check_genvm_health to
retain full error details for operators. Ensure the change is applied where
JSONResponse is constructed for genvm health and do not alter the internal
logging that records _health_cache.genvm_error.
- Around line 337-350: The Redis client created in _check_redis_health via
redis.from_url is not closed and can leak connections; fix it by ensuring the
client is closed after the ping (use a try/finally or context manager).
Specifically, create the client with redis.from_url, call redis_client.ping(),
and in a finally block call redis_client.close() (or use the client's context
manager if available) so the connection pool is released even on exceptions.
🧹 Nitpick comments (6)
backend/protocol_rpc/health.py (5)

159-173: Use asyncio.get_running_loop() instead of deprecated asyncio.get_event_loop().

asyncio.get_event_loop() is deprecated in Python 3.10+ and may raise a DeprecationWarning. Since this function is called from an async context (app startup), use asyncio.get_running_loop() instead.

♻️ Proposed fix
 def start_background_health_checker(
     rpc_router: Optional[FastAPIRPCRouter] = None,
 ) -> None:
     """Start the background health checker task. Call from app startup."""
     global _background_task, _rpc_router_ref

     _rpc_router_ref = rpc_router

     if _background_task is not None:
         logger.warning("Background health checker already running")
         return

-    loop = asyncio.get_event_loop()
-    _background_task = loop.create_task(_background_health_loop())
+    _background_task = asyncio.create_task(_background_health_loop())
     logger.info("Background health checker started")

176-183: Consider awaiting task cancellation for graceful cleanup.

When cancelling the background task, the cancellation is not awaited, which may cause the task to be left in an incomplete state. Consider making this function async and awaiting the task.

♻️ Suggested improvement
-def stop_background_health_checker() -> None:
+async def stop_background_health_checker() -> None:
     """Stop the background health checker task. Call from app shutdown."""
     global _background_task

     if _background_task is not None:
         _background_task.cancel()
+        try:
+            await _background_task
+        except asyncio.CancelledError:
+            pass
         _background_task = None
         logger.info("Background health checker stopped")

This would require updating app_lifespan.py to await stop_background_health_checker().


233-238: Use logger.exception for better error context.

Per static analysis hint TRY400, using logger.exception instead of logger.error automatically includes the stack trace, which is helpful for debugging database connectivity failures.

♻️ Proposed fix
         except Exception as e:
-            logger.error(f"Database connectivity test failed: {e}")
+            logger.exception("Database connectivity test failed")

152-156: Redundant exception object in logging call.

Per static analysis hint TRY401, the exception object e is redundant when using logger.exception since it automatically captures the exception context. Additionally, asyncio.sleep reads the interval once at startup and won't pick up runtime changes.

♻️ Proposed fix
         except Exception as e:
-            logger.exception(f"Background health check error: {e}")
+            logger.exception("Background health check error")

-        await asyncio.sleep(interval)
+        await asyncio.sleep(get_health_check_interval())

The second change allows the interval to be updated at runtime via environment variable reload.


250-317: Async function blocks event loop with synchronous database calls.

The _check_consensus_health() function is declared async but uses synchronous db_manager.engine.connect() calls which block the event loop. This pattern is repeated across multiple health check functions in this file. While this background task runs infrequently (every 10 seconds) with minimal query overhead, consider refactoring to use loop.run_in_executor() to avoid blocking other tasks, or migrate to async SQLAlchemy if the codebase adopts it project-wide.

backend/protocol_rpc/app_lifespan.py (1)

408-410: Shutdown integration is correct, but order could be improved.

The health checker is stopped early in the shutdown sequence, which is appropriate. However, if the suggested change to make stop_background_health_checker async is applied, this would need to be await stop_background_health_checker().

Note: The log message on line 410 is slightly redundant since stop_background_health_checker already logs the same message internally.

♻️ Optional: Remove redundant log
         # Stop background health checker
         stop_background_health_checker()
-        logger.info("[SHUTDOWN] Background health checker stopped")

The function already logs "Background health checker stopped" internally.

📜 Review details

Configuration used: defaults

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between cf19db3 and 453c320.

📒 Files selected for processing (2)
  • backend/protocol_rpc/app_lifespan.py
  • backend/protocol_rpc/health.py
🧰 Additional context used
📓 Path-based instructions (2)
**/*.py

📄 CodeRabbit inference engine (AGENTS.md)

Target Python 3.12, use 4-space indentation, and rely on Black via pre-commit for formatting consistency

**/*.py: Apply Black formatter for Python code formatting
Include type hints in all Python code

Files:

  • backend/protocol_rpc/app_lifespan.py
  • backend/protocol_rpc/health.py
backend/**/*.py

📄 CodeRabbit inference engine (AGENTS.md)

Align backend filenames with their behavior (e.g., validators/llm_validator.py) and mirror that pattern in tests

Files:

  • backend/protocol_rpc/app_lifespan.py
  • backend/protocol_rpc/health.py
🧠 Learnings (1)
📚 Learning: 2025-12-28T17:12:28.860Z
Learnt from: CR
Repo: genlayerlabs/genlayer-studio PR: 0
File: CLAUDE.md:0-0
Timestamp: 2025-12-28T17:12:28.860Z
Learning: Applies to backend/node/base/leader_election.py : Backend consensus should use configurable validator sets from backend/node/base/leader_election.py

Applied to files:

  • backend/protocol_rpc/app_lifespan.py
🧬 Code graph analysis (2)
backend/protocol_rpc/app_lifespan.py (1)
backend/protocol_rpc/health.py (2)
  • start_background_health_checker (159-173)
  • stop_background_health_checker (176-183)
backend/protocol_rpc/health.py (2)
backend/consensus/monitoring.py (2)
  • get_monitor (197-199)
  • get_status (130-173)
backend/database_handler/models.py (1)
  • Transactions (68-147)
🪛 GitHub Actions: Unit Tests
backend/protocol_rpc/health.py

[error] 386-386: FastAPIError: Invalid args for response field! Hint: check that typing.Union[dict, starlette.responses.JSONResponse] is a valid Pydantic field type. If you are using a return type annotation that is not a valid Pydantic field (e.g. Union[Response, dict, None]) you can disable generating the response model from the type annotation with the path operation decorator parameter response_model=None.

🪛 GitHub Check: CodeQL
backend/protocol_rpc/health.py

[warning] 396-401: Information exposure through an exception
Stack trace information flows to this location and may be exposed to an external user.


[warning] 410-423: Information exposure through an exception
Stack trace information flows to this location and may be exposed to an external user.

🪛 Ruff (0.14.10)
backend/protocol_rpc/health.py

154-154: Redundant exception object included in logging.exception call

(TRY401)


210-211: try-except-pass detected, consider logging the exception

(S110)


210-210: Do not catch blind exception: Exception

(BLE001)


216-217: try-except-pass detected, consider logging the exception

(S110)


216-216: Do not catch blind exception: Exception

(BLE001)


222-223: try-except-pass detected, consider logging the exception

(S110)


222-222: Do not catch blind exception: Exception

(BLE001)


225-225: Do not catch blind exception: Exception

(BLE001)


237-237: Do not catch blind exception: Exception

(BLE001)


238-238: Use logging.exception instead of logging.error

Replace with exception

(TRY400)


246-246: Do not catch blind exception: Exception

(BLE001)


333-333: Do not catch blind exception: Exception

(BLE001)


348-348: Consider moving this statement to an else block

(TRY300)


349-349: Do not catch blind exception: Exception

(BLE001)


369-369: Do not catch blind exception: Exception

(BLE001)


490-491: try-except-pass detected, consider logging the exception

(S110)


490-490: Do not catch blind exception: Exception

(BLE001)


496-497: try-except-pass detected, consider logging the exception

(S110)


496-496: Do not catch blind exception: Exception

(BLE001)


502-503: try-except-pass detected, consider logging the exception

(S110)


502-502: Do not catch blind exception: Exception

(BLE001)


508-509: try-except-pass detected, consider logging the exception

(S110)


508-508: Do not catch blind exception: Exception

(BLE001)


522-523: try-except-pass detected, consider logging the exception

(S110)


522-522: Do not catch blind exception: Exception

(BLE001)


525-525: Do not catch blind exception: Exception

(BLE001)


539-539: Use logging.exception instead of logging.error

Replace with exception

(TRY400)

⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (4)
  • GitHub Check: Analyze (javascript)
  • GitHub Check: Analyze (python)
  • GitHub Check: test
  • GitHub Check: load-test
🔇 Additional comments (5)
backend/protocol_rpc/health.py (3)

27-44: HealthCache dataclass and global state look reasonable.

The cache structure is well-designed for storing health check results. One minor note: issues: list should specify a type hint like list[str] for better type safety, but this is optional.


380-383: The /ping endpoint is a good addition.

This ultra-lightweight endpoint for load balancers is a best practice for health checks that don't need to verify backend dependencies.


52-139: Background health check logic is well-structured.

The _run_health_checks function properly updates the cache with all service statuses and handles exceptions gracefully. The overall structure of collecting health from multiple services and aggregating the status is sound.

backend/protocol_rpc/app_lifespan.py (2)

37-40: Import additions look correct.

The imports are properly added for the new background health checker lifecycle functions.


313-315: Background health checker startup is correctly integrated.

Starting the background health checker after RPC router creation ensures the router reference is available for consensus health checks. Good placement in the startup sequence.

Comment thread backend/protocol_rpc/health.py Resolved
Comment thread backend/protocol_rpc/health.py Resolved

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 3

🤖 Fix all issues with AI agents
In @backend/protocol_rpc/health.py:
- Around line 386-402: The health_check handler returns internal details via
_health_cache.genvm_error; remove or sanitize that sensitive field from the HTTP
response in the health_check function so external consumers only see a generic
error (e.g., set "detail" to a non-sensitive string like "internal_error" or
omit it entirely) while keeping the full error logged inside
_check_genvm_health; update the JSONResponse construction in health_check to
stop including _health_cache.genvm_error and instead return only safe fields
("status", "error", "timestamp") or a generic detail.
- Around line 337-350: The _check_redis_health function creates a Redis client
via redis.from_url but never closes it, leaking connection pools; modify
_check_redis_health to ensure the created redis_client is explicitly closed
(call redis_client.close()) in a finally block (or use the client's context
manager if available) so that regardless of success or exception the connection
pool is released; reference the redis_client variable and the
_check_redis_health function when making this change.
🧹 Nitpick comments (3)
backend/protocol_rpc/health.py (3)

142-157: Remove redundant exception from logging.exception call.

Line 154 includes the exception object in the message string, but logging.exception already logs the exception details automatically.

♻️ Proposed fix
-            logger.exception(f"Background health check error: {e}")
+            logger.exception("Background health check error")

228-238: Use logger.exception for better error context.

Line 238 logs a database connectivity failure using logger.error. Since this occurs within an exception handler, logger.exception would automatically include the stack trace.

♻️ Proposed fix
-            logger.error(f"Database connectivity test failed: {e}")
+            logger.exception("Database connectivity test failed")

529-539: Use logger.exception for better error context.

Line 539 logs a database connectivity failure using logger.error. Since this occurs within an exception handler, logger.exception would automatically include the stack trace.

♻️ Proposed fix
         except Exception as e:
-            logger.error(f"Database connectivity test failed: {e}")
+            logger.exception("Database connectivity test failed")
📜 Review details

Configuration used: defaults

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 453c320 and 69ab110.

📒 Files selected for processing (1)
  • backend/protocol_rpc/health.py
🧰 Additional context used
📓 Path-based instructions (2)
**/*.py

📄 CodeRabbit inference engine (AGENTS.md)

Target Python 3.12, use 4-space indentation, and rely on Black via pre-commit for formatting consistency

**/*.py: Apply Black formatter for Python code formatting
Include type hints in all Python code

Files:

  • backend/protocol_rpc/health.py
backend/**/*.py

📄 CodeRabbit inference engine (AGENTS.md)

Align backend filenames with their behavior (e.g., validators/llm_validator.py) and mirror that pattern in tests

Files:

  • backend/protocol_rpc/health.py
🧬 Code graph analysis (1)
backend/protocol_rpc/health.py (3)
backend/consensus/monitoring.py (2)
  • get_monitor (197-199)
  • get_status (130-173)
backend/database_handler/session_factory.py (1)
  • get_database_manager (56-59)
backend/protocol_rpc/message_handler/redis_worker_handler.py (1)
  • health_check (251-276)
🪛 GitHub Check: CodeQL
backend/protocol_rpc/health.py

[warning] 396-401: Information exposure through an exception
Stack trace information flows to this location and may be exposed to an external user.


[warning] 410-423: Information exposure through an exception
Stack trace information flows to this location and may be exposed to an external user.

🪛 Ruff (0.14.10)
backend/protocol_rpc/health.py

154-154: Redundant exception object included in logging.exception call

(TRY401)


210-211: try-except-pass detected, consider logging the exception

(S110)


210-210: Do not catch blind exception: Exception

(BLE001)


216-217: try-except-pass detected, consider logging the exception

(S110)


216-216: Do not catch blind exception: Exception

(BLE001)


222-223: try-except-pass detected, consider logging the exception

(S110)


222-222: Do not catch blind exception: Exception

(BLE001)


225-225: Do not catch blind exception: Exception

(BLE001)


237-237: Do not catch blind exception: Exception

(BLE001)


238-238: Use logging.exception instead of logging.error

Replace with exception

(TRY400)


246-246: Do not catch blind exception: Exception

(BLE001)


333-333: Do not catch blind exception: Exception

(BLE001)


348-348: Consider moving this statement to an else block

(TRY300)


349-349: Do not catch blind exception: Exception

(BLE001)


369-369: Do not catch blind exception: Exception

(BLE001)


490-491: try-except-pass detected, consider logging the exception

(S110)


490-490: Do not catch blind exception: Exception

(BLE001)


496-497: try-except-pass detected, consider logging the exception

(S110)


496-496: Do not catch blind exception: Exception

(BLE001)


502-503: try-except-pass detected, consider logging the exception

(S110)


502-502: Do not catch blind exception: Exception

(BLE001)


508-509: try-except-pass detected, consider logging the exception

(S110)


508-508: Do not catch blind exception: Exception

(BLE001)


522-523: try-except-pass detected, consider logging the exception

(S110)


522-522: Do not catch blind exception: Exception

(BLE001)


525-525: Do not catch blind exception: Exception

(BLE001)


539-539: Use logging.exception instead of logging.error

Replace with exception

(TRY400)

⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (1)
  • GitHub Check: backend-unit-tests
🔇 Additional comments (2)
backend/protocol_rpc/health.py (2)

52-140: LGTM! Robust orchestration of health checks.

The function properly orchestrates all health checks, aggregates status, and handles failures gracefully by updating the cache with error information. Exception handling ensures the background task remains resilient.


380-383: LGTM! Perfect lightweight endpoint for load balancer checks.

This ultra-lightweight endpoint is ideal for load balancer health probes without expensive checks.

Comment thread backend/protocol_rpc/health.py Resolved
Comment thread backend/protocol_rpc/health.py Resolved
Comment thread backend/protocol_rpc/health.py Resolved
- Close Redis client after health check to prevent connection leaks
- Remove internal error details from HTTP responses (CodeQL info exposure)

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🤖 Fix all issues with AI agents
In @backend/protocol_rpc/health.py:
- Around line 372-375: The except block that currently logs "GenVM health probe
failed unexpectedly: {e}" and returns True, None should not mark the service
healthy; update the handler to log the full exception (use logger.exception or
include stack/exception details) and return False, None so unexpected errors
cause an unhealthy result; modify the except Exception block (the one calling
logger.warning(f"GenVM health probe failed unexpectedly: {e}")) accordingly.
- Around line 171-172: The current synchronous start_background_health_checker
uses asyncio.get_event_loop().create_task(...) which is less explicit; change it
to call asyncio.get_running_loop().create_task(_background_health_loop()) so it
uses the running loop from the async rpc_app_lifespan context, or alternatively
make start_background_health_checker async and await it from rpc_app_lifespan;
update the assignment to _background_task accordingly and keep reference to
_background_health_loop() and rpc_app_lifespan so the correct symbols are
modified.
🧹 Nitpick comments (4)
backend/protocol_rpc/health.py (4)

154-154: Remove redundant exception object from log message.

logger.exception() automatically includes the exception traceback and details, so explicitly including {e} in the message is redundant.

♻️ Suggested fix
-        logger.exception(f"Background health check error: {e}")
+        logger.exception("Background health check error")

237-238: Use logging.exception for better error diagnostics.

When logging from an exception handler, prefer logging.exception() over logging.error() to automatically include the full traceback.

♻️ Suggested fix
         except Exception as e:
-            logger.error(f"Database connectivity test failed: {e}")
+            logger.exception("Database connectivity test failed")

352-353: Log Redis health check failures for observability.

Silent exception swallowing makes debugging connectivity issues difficult. Consider logging the exception before returning "unhealthy".

♻️ Suggested fix
     except Exception:
+        logger.warning("Redis health check failed", exc_info=True)
         return "unhealthy"

542-542: Use logging.exception for better error diagnostics.

When logging from an exception handler, prefer logging.exception() over logging.error() to automatically include the full traceback.

♻️ Suggested fix
         except Exception as e:
-            logger.error(f"Database connectivity test failed: {e}")
+            logger.exception("Database connectivity test failed")
📜 Review details

Configuration used: defaults

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 69ab110 and d556f90.

📒 Files selected for processing (1)
  • backend/protocol_rpc/health.py
🧰 Additional context used
📓 Path-based instructions (2)
**/*.py

📄 CodeRabbit inference engine (AGENTS.md)

Target Python 3.12, use 4-space indentation, and rely on Black via pre-commit for formatting consistency

**/*.py: Apply Black formatter for Python code formatting
Include type hints in all Python code

Files:

  • backend/protocol_rpc/health.py
backend/**/*.py

📄 CodeRabbit inference engine (AGENTS.md)

Align backend filenames with their behavior (e.g., validators/llm_validator.py) and mirror that pattern in tests

Files:

  • backend/protocol_rpc/health.py
🪛 Ruff (0.14.10)
backend/protocol_rpc/health.py

154-154: Redundant exception object included in logging.exception call

(TRY401)


210-211: try-except-pass detected, consider logging the exception

(S110)


210-210: Do not catch blind exception: Exception

(BLE001)


216-217: try-except-pass detected, consider logging the exception

(S110)


216-216: Do not catch blind exception: Exception

(BLE001)


222-223: try-except-pass detected, consider logging the exception

(S110)


222-222: Do not catch blind exception: Exception

(BLE001)


225-225: Do not catch blind exception: Exception

(BLE001)


237-237: Do not catch blind exception: Exception

(BLE001)


238-238: Use logging.exception instead of logging.error

Replace with exception

(TRY400)


246-246: Do not catch blind exception: Exception

(BLE001)


333-333: Do not catch blind exception: Exception

(BLE001)


352-352: Do not catch blind exception: Exception

(BLE001)


372-372: Do not catch blind exception: Exception

(BLE001)


493-494: try-except-pass detected, consider logging the exception

(S110)


493-493: Do not catch blind exception: Exception

(BLE001)


499-500: try-except-pass detected, consider logging the exception

(S110)


499-499: Do not catch blind exception: Exception

(BLE001)


505-506: try-except-pass detected, consider logging the exception

(S110)


505-505: Do not catch blind exception: Exception

(BLE001)


511-512: try-except-pass detected, consider logging the exception

(S110)


511-511: Do not catch blind exception: Exception

(BLE001)


525-526: try-except-pass detected, consider logging the exception

(S110)


525-525: Do not catch blind exception: Exception

(BLE001)


528-528: Do not catch blind exception: Exception

(BLE001)


542-542: Use logging.exception instead of logging.error

Replace with exception

(TRY400)

⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (1)
  • GitHub Check: backend-unit-tests

Comment on lines +171 to +172
loop = asyncio.get_event_loop()
_background_task = loop.create_task(_background_health_loop())

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor

🧩 Analysis chain

🏁 Script executed:

rg -nP -A5 -B5 'start_background_health_checker' --type=py

Repository: genlayerlabs/genlayer-studio

Length of output: 2578


🏁 Script executed:

fd -t f 'app_lifespan' -e py

Repository: genlayerlabs/genlayer-studio

Length of output: 108


🏁 Script executed:

fd -t f 'health\.py' backend --type f

Repository: genlayerlabs/genlayer-studio

Length of output: 102


🏁 Script executed:

sed -n '150,185p' backend/protocol_rpc/health.py

Repository: genlayerlabs/genlayer-studio

Length of output: 1176


🏁 Script executed:

sed -n '300,330p' backend/protocol_rpc/app_lifespan.py

Repository: genlayerlabs/genlayer-studio

Length of output: 1291


🏁 Script executed:

sed -n '1,50p' backend/protocol_rpc/app_lifespan.py

Repository: genlayerlabs/genlayer-studio

Length of output: 1896


🏁 Script executed:

sed -n '200,350p' backend/protocol_rpc/app_lifespan.py

Repository: genlayerlabs/genlayer-studio

Length of output: 5990


🏁 Script executed:

rg -nP '@asynccontextmanager|async def|def ' backend/protocol_rpc/app_lifespan.py | head -30

Repository: genlayerlabs/genlayer-studio

Length of output: 766


Use asyncio.get_running_loop().create_task() for clarity.

This function is called from an async context (rpc_app_lifespan), so asyncio.get_running_loop().create_task() is more explicit and correct than asyncio.get_event_loop().create_task(). Alternatively, make start_background_health_checker async and await it from the lifespan.

Note: asyncio.create_task() cannot be used directly from a synchronous function.

🤖 Prompt for AI Agents
In @backend/protocol_rpc/health.py around lines 171 - 172, The current
synchronous start_background_health_checker uses
asyncio.get_event_loop().create_task(...) which is less explicit; change it to
call asyncio.get_running_loop().create_task(_background_health_loop()) so it
uses the running loop from the async rpc_app_lifespan context, or alternatively
make start_background_health_checker async and await it from rpc_app_lifespan;
update the assignment to _background_task accordingly and keep reference to
_background_health_loop() and rpc_app_lifespan so the correct symbols are
modified.

Comment on lines +372 to +375
except Exception as e:
# Don't fail on unexpected errors
logger.warning(f"GenVM health probe failed unexpectedly: {e}")
return True, None

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

Returning True on unexpected GenVM errors masks failures.

Line 375 returns True (healthy) when an unexpected exception occurs during the GenVM health check. This causes the system to report GenVM as healthy even when the check itself failed, which defeats the purpose of health checking.

🐛 Proposed fix
     except Exception as e:
-        # Don't fail on unexpected errors
-        logger.warning(f"GenVM health probe failed unexpectedly: {e}")
-        return True, None
+        # Log unexpected errors and mark as unhealthy
+        logger.exception("GenVM health probe failed unexpectedly")
+        return False, f"unexpected_error: {type(e).__name__}"
🧰 Tools
🪛 Ruff (0.14.10)

372-372: Do not catch blind exception: Exception

(BLE001)

🤖 Prompt for AI Agents
In @backend/protocol_rpc/health.py around lines 372 - 375, The except block that
currently logs "GenVM health probe failed unexpectedly: {e}" and returns True,
None should not mark the service healthy; update the handler to log the full
exception (use logger.exception or include stack/exception details) and return
False, None so unexpected errors cause an unhealthy result; modify the except
Exception block (the one calling logger.warning(f"GenVM health probe failed
unexpectedly: {e}")) accordingly.

MuncleUscles and others added 2 commits January 7, 2026 23:25
Address case mismatch between frontend and backend caused signature
verification to fail. Frontend calls getContractNonce which normalizes
to checksum, but upgrade verification used raw address. Now both use
checksum for consistent tx_count and message hash.
@cristiam86
cristiam86 merged commit 9783db6 into main Jan 13, 2026
9 of 10 checks passed
@github-actions

Copy link
Copy Markdown
Contributor

🎉 This PR is included in version 0.89.0 🎉

The release is available on GitHub release

Your semantic-release bot 📦🚀

@sonarqubecloud

Copy link
Copy Markdown

Quality Gate Failed Quality Gate failed

Failed conditions
13.7% Coverage on New Code (required ≥ 80%)
10.2% Duplication on New Code (required ≤ 3%)

See analysis details on SonarQube Cloud

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants