Repository navigation
perf: background health checker for fast /health endpoint - #1372
Conversation
- Move expensive health checks (DB, consensus, memory, Redis, GenVM) to background task - Background task runs every 10s (configurable via HEALTH_CHECK_INTERVAL_SECONDS) - /health endpoint now returns cached results (~1ms response vs 14s+ before) - GenVM failures still return 503 to trigger container restart - Add /ping endpoint for ultra-lightweight load balancer checks - Keep detailed /health/* endpoints for debugging (run checks synchronously) This fixes intermittent RPC slowness caused by /health taking 14+ seconds when load balancer health checks coincide with request processing.
|
Caution Review failedThe pull request is closed. 📝 WalkthroughWalkthroughStarts a background health checker at app startup that periodically probes services (DB, consensus, memory, Redis, GenVM), caches results in a HealthCache, exposes cached /health and per-service endpoints plus /ping, and stops the checker on app shutdown. Changes
Sequence Diagram(s)sequenceDiagram
participant App as Application
participant Lifespan as App Lifespan
participant Router as RPC Router
participant BG as Background Checker
participant Services as Services (DB, Consensus, Memory, Redis, GenVM)
participant Cache as HealthCache
participant Endpoint as /health Endpoint
participant Client as Client
App->>Lifespan: startup
activate Lifespan
Lifespan->>Router: create RPC router
Lifespan->>BG: start_background_health_checker(rpc_router)
activate BG
Lifespan-->>App: startup complete
deactivate Lifespan
loop every N seconds
BG->>Services: run _check_* probes
Services-->>BG: probe results
BG->>Cache: update status, timestamps, durations, issues
end
Client->>Endpoint: GET /health
Endpoint->>Cache: read latest health
Cache-->>Endpoint: return cached payload
alt GenVM unhealthy
Endpoint-->>Client: 503 + details
else
Endpoint-->>Client: 200 + cached health
end
App->>Lifespan: shutdown
activate Lifespan
Lifespan->>BG: stop_background_health_checker()
deactivate BG
Lifespan-->>App: shutdown complete
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~20 minutes Poem
🚥 Pre-merge checks | ✅ 3✅ Passed checks (3 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing touches
📜 Recent review detailsConfiguration used: defaults Review profile: CHILL Plan: Pro 📒 Files selected for processing (1)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
Actionable comments posted: 4
🤖 Fix all issues with AI agents
In @backend/protocol_rpc/health.py:
- Around line 410-423: The health endpoint currently returns raw internal error
details via the "error" field from _health_cache, exposing sensitive internals;
remove or replace that field in the returned dict so external consumers only get
a sanitized status (e.g., error: "internal_error" or None) and keep full
exception details logged server-side instead; update the function building this
response to stop including _health_cache.error and ensure any logging of
_health_cache.error uses the server logger (not the HTTP response) and retains
get_health_check_interval(), _health_cache.status, _health_cache.issues, and
_health_cache.services as before.
- Around line 386-387: The endpoint decorator for health_check is causing
FastAPI to try to generate a Pydantic model from the return type Union[dict,
JSONResponse]; update the route decorator on @health_router.get("/health") to
disable response model generation by adding response_model=None, and optionally
adjust health_check's return type annotation (e.g., to Any or
starlette.responses.Response) to avoid Union-based type hints that trigger
Pydantic processing.
- Around line 392-402: The HTTP response in the genvm health check currently
returns internal error text via _health_cache.genvm_error in the JSONResponse
(see the block that returns status_code=503); remove or replace the "detail"
field so it does not expose internal errors to clients — for example, omit
"detail" or set it to a non-sensitive string like "contact support" or "internal
error" — and rely on the existing internal logging in _check_genvm_health to
retain full error details for operators. Ensure the change is applied where
JSONResponse is constructed for genvm health and do not alter the internal
logging that records _health_cache.genvm_error.
- Around line 337-350: The Redis client created in _check_redis_health via
redis.from_url is not closed and can leak connections; fix it by ensuring the
client is closed after the ping (use a try/finally or context manager).
Specifically, create the client with redis.from_url, call redis_client.ping(),
and in a finally block call redis_client.close() (or use the client's context
manager if available) so the connection pool is released even on exceptions.
🧹 Nitpick comments (6)
backend/protocol_rpc/health.py (5)
159-173: Useasyncio.get_running_loop()instead of deprecatedasyncio.get_event_loop().
asyncio.get_event_loop()is deprecated in Python 3.10+ and may raise aDeprecationWarning. Since this function is called from an async context (app startup), useasyncio.get_running_loop()instead.♻️ Proposed fix
def start_background_health_checker( rpc_router: Optional[FastAPIRPCRouter] = None, ) -> None: """Start the background health checker task. Call from app startup.""" global _background_task, _rpc_router_ref _rpc_router_ref = rpc_router if _background_task is not None: logger.warning("Background health checker already running") return - loop = asyncio.get_event_loop() - _background_task = loop.create_task(_background_health_loop()) + _background_task = asyncio.create_task(_background_health_loop()) logger.info("Background health checker started")
176-183: Consider awaiting task cancellation for graceful cleanup.When cancelling the background task, the cancellation is not awaited, which may cause the task to be left in an incomplete state. Consider making this function async and awaiting the task.
♻️ Suggested improvement
-def stop_background_health_checker() -> None: +async def stop_background_health_checker() -> None: """Stop the background health checker task. Call from app shutdown.""" global _background_task if _background_task is not None: _background_task.cancel() + try: + await _background_task + except asyncio.CancelledError: + pass _background_task = None logger.info("Background health checker stopped")This would require updating
app_lifespan.pytoawait stop_background_health_checker().
233-238: Uselogger.exceptionfor better error context.Per static analysis hint TRY400, using
logger.exceptioninstead oflogger.errorautomatically includes the stack trace, which is helpful for debugging database connectivity failures.♻️ Proposed fix
except Exception as e: - logger.error(f"Database connectivity test failed: {e}") + logger.exception("Database connectivity test failed")
152-156: Redundant exception object in logging call.Per static analysis hint TRY401, the exception object
eis redundant when usinglogger.exceptionsince it automatically captures the exception context. Additionally,asyncio.sleepreads the interval once at startup and won't pick up runtime changes.♻️ Proposed fix
except Exception as e: - logger.exception(f"Background health check error: {e}") + logger.exception("Background health check error") - await asyncio.sleep(interval) + await asyncio.sleep(get_health_check_interval())The second change allows the interval to be updated at runtime via environment variable reload.
250-317: Async function blocks event loop with synchronous database calls.The
_check_consensus_health()function is declaredasyncbut uses synchronousdb_manager.engine.connect()calls which block the event loop. This pattern is repeated across multiple health check functions in this file. While this background task runs infrequently (every 10 seconds) with minimal query overhead, consider refactoring to useloop.run_in_executor()to avoid blocking other tasks, or migrate to async SQLAlchemy if the codebase adopts it project-wide.backend/protocol_rpc/app_lifespan.py (1)
408-410: Shutdown integration is correct, but order could be improved.The health checker is stopped early in the shutdown sequence, which is appropriate. However, if the suggested change to make
stop_background_health_checkerasync is applied, this would need to beawait stop_background_health_checker().Note: The log message on line 410 is slightly redundant since
stop_background_health_checkeralready logs the same message internally.♻️ Optional: Remove redundant log
# Stop background health checker stop_background_health_checker() - logger.info("[SHUTDOWN] Background health checker stopped")The function already logs "Background health checker stopped" internally.
📜 Review details
Configuration used: defaults
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (2)
backend/protocol_rpc/app_lifespan.pybackend/protocol_rpc/health.py
🧰 Additional context used
📓 Path-based instructions (2)
**/*.py
📄 CodeRabbit inference engine (AGENTS.md)
Target Python 3.12, use 4-space indentation, and rely on Black via pre-commit for formatting consistency
**/*.py: Apply Black formatter for Python code formatting
Include type hints in all Python code
Files:
backend/protocol_rpc/app_lifespan.pybackend/protocol_rpc/health.py
backend/**/*.py
📄 CodeRabbit inference engine (AGENTS.md)
Align backend filenames with their behavior (e.g.,
validators/llm_validator.py) and mirror that pattern in tests
Files:
backend/protocol_rpc/app_lifespan.pybackend/protocol_rpc/health.py
🧠 Learnings (1)
📚 Learning: 2025-12-28T17:12:28.860Z
Learnt from: CR
Repo: genlayerlabs/genlayer-studio PR: 0
File: CLAUDE.md:0-0
Timestamp: 2025-12-28T17:12:28.860Z
Learning: Applies to backend/node/base/leader_election.py : Backend consensus should use configurable validator sets from backend/node/base/leader_election.py
Applied to files:
backend/protocol_rpc/app_lifespan.py
🧬 Code graph analysis (2)
backend/protocol_rpc/app_lifespan.py (1)
backend/protocol_rpc/health.py (2)
start_background_health_checker(159-173)stop_background_health_checker(176-183)
backend/protocol_rpc/health.py (2)
backend/consensus/monitoring.py (2)
get_monitor(197-199)get_status(130-173)backend/database_handler/models.py (1)
Transactions(68-147)
🪛 GitHub Actions: Unit Tests
backend/protocol_rpc/health.py
[error] 386-386: FastAPIError: Invalid args for response field! Hint: check that typing.Union[dict, starlette.responses.JSONResponse] is a valid Pydantic field type. If you are using a return type annotation that is not a valid Pydantic field (e.g. Union[Response, dict, None]) you can disable generating the response model from the type annotation with the path operation decorator parameter response_model=None.
🪛 GitHub Check: CodeQL
backend/protocol_rpc/health.py
[warning] 396-401: Information exposure through an exception
Stack trace information flows to this location and may be exposed to an external user.
[warning] 410-423: Information exposure through an exception
Stack trace information flows to this location and may be exposed to an external user.
🪛 Ruff (0.14.10)
backend/protocol_rpc/health.py
154-154: Redundant exception object included in logging.exception call
(TRY401)
210-211: try-except-pass detected, consider logging the exception
(S110)
210-210: Do not catch blind exception: Exception
(BLE001)
216-217: try-except-pass detected, consider logging the exception
(S110)
216-216: Do not catch blind exception: Exception
(BLE001)
222-223: try-except-pass detected, consider logging the exception
(S110)
222-222: Do not catch blind exception: Exception
(BLE001)
225-225: Do not catch blind exception: Exception
(BLE001)
237-237: Do not catch blind exception: Exception
(BLE001)
238-238: Use logging.exception instead of logging.error
Replace with exception
(TRY400)
246-246: Do not catch blind exception: Exception
(BLE001)
333-333: Do not catch blind exception: Exception
(BLE001)
348-348: Consider moving this statement to an else block
(TRY300)
349-349: Do not catch blind exception: Exception
(BLE001)
369-369: Do not catch blind exception: Exception
(BLE001)
490-491: try-except-pass detected, consider logging the exception
(S110)
490-490: Do not catch blind exception: Exception
(BLE001)
496-497: try-except-pass detected, consider logging the exception
(S110)
496-496: Do not catch blind exception: Exception
(BLE001)
502-503: try-except-pass detected, consider logging the exception
(S110)
502-502: Do not catch blind exception: Exception
(BLE001)
508-509: try-except-pass detected, consider logging the exception
(S110)
508-508: Do not catch blind exception: Exception
(BLE001)
522-523: try-except-pass detected, consider logging the exception
(S110)
522-522: Do not catch blind exception: Exception
(BLE001)
525-525: Do not catch blind exception: Exception
(BLE001)
539-539: Use logging.exception instead of logging.error
Replace with exception
(TRY400)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (4)
- GitHub Check: Analyze (javascript)
- GitHub Check: Analyze (python)
- GitHub Check: test
- GitHub Check: load-test
🔇 Additional comments (5)
backend/protocol_rpc/health.py (3)
27-44: HealthCache dataclass and global state look reasonable.The cache structure is well-designed for storing health check results. One minor note:
issues: listshould specify a type hint likelist[str]for better type safety, but this is optional.
380-383: The/pingendpoint is a good addition.This ultra-lightweight endpoint for load balancers is a best practice for health checks that don't need to verify backend dependencies.
52-139: Background health check logic is well-structured.The
_run_health_checksfunction properly updates the cache with all service statuses and handles exceptions gracefully. The overall structure of collecting health from multiple services and aggregating the status is sound.backend/protocol_rpc/app_lifespan.py (2)
37-40: Import additions look correct.The imports are properly added for the new background health checker lifecycle functions.
313-315: Background health checker startup is correctly integrated.Starting the background health checker after RPC router creation ensures the router reference is available for consensus health checks. Good placement in the startup sequence.
There was a problem hiding this comment.
Actionable comments posted: 3
🤖 Fix all issues with AI agents
In @backend/protocol_rpc/health.py:
- Around line 386-402: The health_check handler returns internal details via
_health_cache.genvm_error; remove or sanitize that sensitive field from the HTTP
response in the health_check function so external consumers only see a generic
error (e.g., set "detail" to a non-sensitive string like "internal_error" or
omit it entirely) while keeping the full error logged inside
_check_genvm_health; update the JSONResponse construction in health_check to
stop including _health_cache.genvm_error and instead return only safe fields
("status", "error", "timestamp") or a generic detail.
- Around line 337-350: The _check_redis_health function creates a Redis client
via redis.from_url but never closes it, leaking connection pools; modify
_check_redis_health to ensure the created redis_client is explicitly closed
(call redis_client.close()) in a finally block (or use the client's context
manager if available) so that regardless of success or exception the connection
pool is released; reference the redis_client variable and the
_check_redis_health function when making this change.
🧹 Nitpick comments (3)
backend/protocol_rpc/health.py (3)
142-157: Remove redundant exception from logging.exception call.Line 154 includes the exception object in the message string, but
logging.exceptionalready logs the exception details automatically.♻️ Proposed fix
- logger.exception(f"Background health check error: {e}") + logger.exception("Background health check error")
228-238: Use logger.exception for better error context.Line 238 logs a database connectivity failure using
logger.error. Since this occurs within an exception handler,logger.exceptionwould automatically include the stack trace.♻️ Proposed fix
- logger.error(f"Database connectivity test failed: {e}") + logger.exception("Database connectivity test failed")
529-539: Use logger.exception for better error context.Line 539 logs a database connectivity failure using
logger.error. Since this occurs within an exception handler,logger.exceptionwould automatically include the stack trace.♻️ Proposed fix
except Exception as e: - logger.error(f"Database connectivity test failed: {e}") + logger.exception("Database connectivity test failed")
📜 Review details
Configuration used: defaults
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (1)
backend/protocol_rpc/health.py
🧰 Additional context used
📓 Path-based instructions (2)
**/*.py
📄 CodeRabbit inference engine (AGENTS.md)
Target Python 3.12, use 4-space indentation, and rely on Black via pre-commit for formatting consistency
**/*.py: Apply Black formatter for Python code formatting
Include type hints in all Python code
Files:
backend/protocol_rpc/health.py
backend/**/*.py
📄 CodeRabbit inference engine (AGENTS.md)
Align backend filenames with their behavior (e.g.,
validators/llm_validator.py) and mirror that pattern in tests
Files:
backend/protocol_rpc/health.py
🧬 Code graph analysis (1)
backend/protocol_rpc/health.py (3)
backend/consensus/monitoring.py (2)
get_monitor(197-199)get_status(130-173)backend/database_handler/session_factory.py (1)
get_database_manager(56-59)backend/protocol_rpc/message_handler/redis_worker_handler.py (1)
health_check(251-276)
🪛 GitHub Check: CodeQL
backend/protocol_rpc/health.py
[warning] 396-401: Information exposure through an exception
Stack trace information flows to this location and may be exposed to an external user.
[warning] 410-423: Information exposure through an exception
Stack trace information flows to this location and may be exposed to an external user.
🪛 Ruff (0.14.10)
backend/protocol_rpc/health.py
154-154: Redundant exception object included in logging.exception call
(TRY401)
210-211: try-except-pass detected, consider logging the exception
(S110)
210-210: Do not catch blind exception: Exception
(BLE001)
216-217: try-except-pass detected, consider logging the exception
(S110)
216-216: Do not catch blind exception: Exception
(BLE001)
222-223: try-except-pass detected, consider logging the exception
(S110)
222-222: Do not catch blind exception: Exception
(BLE001)
225-225: Do not catch blind exception: Exception
(BLE001)
237-237: Do not catch blind exception: Exception
(BLE001)
238-238: Use logging.exception instead of logging.error
Replace with exception
(TRY400)
246-246: Do not catch blind exception: Exception
(BLE001)
333-333: Do not catch blind exception: Exception
(BLE001)
348-348: Consider moving this statement to an else block
(TRY300)
349-349: Do not catch blind exception: Exception
(BLE001)
369-369: Do not catch blind exception: Exception
(BLE001)
490-491: try-except-pass detected, consider logging the exception
(S110)
490-490: Do not catch blind exception: Exception
(BLE001)
496-497: try-except-pass detected, consider logging the exception
(S110)
496-496: Do not catch blind exception: Exception
(BLE001)
502-503: try-except-pass detected, consider logging the exception
(S110)
502-502: Do not catch blind exception: Exception
(BLE001)
508-509: try-except-pass detected, consider logging the exception
(S110)
508-508: Do not catch blind exception: Exception
(BLE001)
522-523: try-except-pass detected, consider logging the exception
(S110)
522-522: Do not catch blind exception: Exception
(BLE001)
525-525: Do not catch blind exception: Exception
(BLE001)
539-539: Use logging.exception instead of logging.error
Replace with exception
(TRY400)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (1)
- GitHub Check: backend-unit-tests
🔇 Additional comments (2)
backend/protocol_rpc/health.py (2)
52-140: LGTM! Robust orchestration of health checks.The function properly orchestrates all health checks, aggregates status, and handles failures gracefully by updating the cache with error information. Exception handling ensures the background task remains resilient.
380-383: LGTM! Perfect lightweight endpoint for load balancer checks.This ultra-lightweight endpoint is ideal for load balancer health probes without expensive checks.
- Close Redis client after health check to prevent connection leaks - Remove internal error details from HTTP responses (CodeQL info exposure)
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Fix all issues with AI agents
In @backend/protocol_rpc/health.py:
- Around line 372-375: The except block that currently logs "GenVM health probe
failed unexpectedly: {e}" and returns True, None should not mark the service
healthy; update the handler to log the full exception (use logger.exception or
include stack/exception details) and return False, None so unexpected errors
cause an unhealthy result; modify the except Exception block (the one calling
logger.warning(f"GenVM health probe failed unexpectedly: {e}")) accordingly.
- Around line 171-172: The current synchronous start_background_health_checker
uses asyncio.get_event_loop().create_task(...) which is less explicit; change it
to call asyncio.get_running_loop().create_task(_background_health_loop()) so it
uses the running loop from the async rpc_app_lifespan context, or alternatively
make start_background_health_checker async and await it from rpc_app_lifespan;
update the assignment to _background_task accordingly and keep reference to
_background_health_loop() and rpc_app_lifespan so the correct symbols are
modified.
🧹 Nitpick comments (4)
backend/protocol_rpc/health.py (4)
154-154: Remove redundant exception object from log message.
logger.exception()automatically includes the exception traceback and details, so explicitly including{e}in the message is redundant.♻️ Suggested fix
- logger.exception(f"Background health check error: {e}") + logger.exception("Background health check error")
237-238: Uselogging.exceptionfor better error diagnostics.When logging from an exception handler, prefer
logging.exception()overlogging.error()to automatically include the full traceback.♻️ Suggested fix
except Exception as e: - logger.error(f"Database connectivity test failed: {e}") + logger.exception("Database connectivity test failed")
352-353: Log Redis health check failures for observability.Silent exception swallowing makes debugging connectivity issues difficult. Consider logging the exception before returning "unhealthy".
♻️ Suggested fix
except Exception: + logger.warning("Redis health check failed", exc_info=True) return "unhealthy"
542-542: Uselogging.exceptionfor better error diagnostics.When logging from an exception handler, prefer
logging.exception()overlogging.error()to automatically include the full traceback.♻️ Suggested fix
except Exception as e: - logger.error(f"Database connectivity test failed: {e}") + logger.exception("Database connectivity test failed")
📜 Review details
Configuration used: defaults
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (1)
backend/protocol_rpc/health.py
🧰 Additional context used
📓 Path-based instructions (2)
**/*.py
📄 CodeRabbit inference engine (AGENTS.md)
Target Python 3.12, use 4-space indentation, and rely on Black via pre-commit for formatting consistency
**/*.py: Apply Black formatter for Python code formatting
Include type hints in all Python code
Files:
backend/protocol_rpc/health.py
backend/**/*.py
📄 CodeRabbit inference engine (AGENTS.md)
Align backend filenames with their behavior (e.g.,
validators/llm_validator.py) and mirror that pattern in tests
Files:
backend/protocol_rpc/health.py
🪛 Ruff (0.14.10)
backend/protocol_rpc/health.py
154-154: Redundant exception object included in logging.exception call
(TRY401)
210-211: try-except-pass detected, consider logging the exception
(S110)
210-210: Do not catch blind exception: Exception
(BLE001)
216-217: try-except-pass detected, consider logging the exception
(S110)
216-216: Do not catch blind exception: Exception
(BLE001)
222-223: try-except-pass detected, consider logging the exception
(S110)
222-222: Do not catch blind exception: Exception
(BLE001)
225-225: Do not catch blind exception: Exception
(BLE001)
237-237: Do not catch blind exception: Exception
(BLE001)
238-238: Use logging.exception instead of logging.error
Replace with exception
(TRY400)
246-246: Do not catch blind exception: Exception
(BLE001)
333-333: Do not catch blind exception: Exception
(BLE001)
352-352: Do not catch blind exception: Exception
(BLE001)
372-372: Do not catch blind exception: Exception
(BLE001)
493-494: try-except-pass detected, consider logging the exception
(S110)
493-493: Do not catch blind exception: Exception
(BLE001)
499-500: try-except-pass detected, consider logging the exception
(S110)
499-499: Do not catch blind exception: Exception
(BLE001)
505-506: try-except-pass detected, consider logging the exception
(S110)
505-505: Do not catch blind exception: Exception
(BLE001)
511-512: try-except-pass detected, consider logging the exception
(S110)
511-511: Do not catch blind exception: Exception
(BLE001)
525-526: try-except-pass detected, consider logging the exception
(S110)
525-525: Do not catch blind exception: Exception
(BLE001)
528-528: Do not catch blind exception: Exception
(BLE001)
542-542: Use logging.exception instead of logging.error
Replace with exception
(TRY400)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (1)
- GitHub Check: backend-unit-tests
| loop = asyncio.get_event_loop() | ||
| _background_task = loop.create_task(_background_health_loop()) |
There was a problem hiding this comment.
🧩 Analysis chain
🏁 Script executed:
rg -nP -A5 -B5 'start_background_health_checker' --type=pyRepository: genlayerlabs/genlayer-studio
Length of output: 2578
🏁 Script executed:
fd -t f 'app_lifespan' -e pyRepository: genlayerlabs/genlayer-studio
Length of output: 108
🏁 Script executed:
fd -t f 'health\.py' backend --type fRepository: genlayerlabs/genlayer-studio
Length of output: 102
🏁 Script executed:
sed -n '150,185p' backend/protocol_rpc/health.pyRepository: genlayerlabs/genlayer-studio
Length of output: 1176
🏁 Script executed:
sed -n '300,330p' backend/protocol_rpc/app_lifespan.pyRepository: genlayerlabs/genlayer-studio
Length of output: 1291
🏁 Script executed:
sed -n '1,50p' backend/protocol_rpc/app_lifespan.pyRepository: genlayerlabs/genlayer-studio
Length of output: 1896
🏁 Script executed:
sed -n '200,350p' backend/protocol_rpc/app_lifespan.pyRepository: genlayerlabs/genlayer-studio
Length of output: 5990
🏁 Script executed:
rg -nP '@asynccontextmanager|async def|def ' backend/protocol_rpc/app_lifespan.py | head -30Repository: genlayerlabs/genlayer-studio
Length of output: 766
Use asyncio.get_running_loop().create_task() for clarity.
This function is called from an async context (rpc_app_lifespan), so asyncio.get_running_loop().create_task() is more explicit and correct than asyncio.get_event_loop().create_task(). Alternatively, make start_background_health_checker async and await it from the lifespan.
Note: asyncio.create_task() cannot be used directly from a synchronous function.
🤖 Prompt for AI Agents
In @backend/protocol_rpc/health.py around lines 171 - 172, The current
synchronous start_background_health_checker uses
asyncio.get_event_loop().create_task(...) which is less explicit; change it to
call asyncio.get_running_loop().create_task(_background_health_loop()) so it
uses the running loop from the async rpc_app_lifespan context, or alternatively
make start_background_health_checker async and await it from rpc_app_lifespan;
update the assignment to _background_task accordingly and keep reference to
_background_health_loop() and rpc_app_lifespan so the correct symbols are
modified.
| except Exception as e: | ||
| # Don't fail on unexpected errors | ||
| logger.warning(f"GenVM health probe failed unexpectedly: {e}") | ||
| return True, None |
There was a problem hiding this comment.
Returning True on unexpected GenVM errors masks failures.
Line 375 returns True (healthy) when an unexpected exception occurs during the GenVM health check. This causes the system to report GenVM as healthy even when the check itself failed, which defeats the purpose of health checking.
🐛 Proposed fix
except Exception as e:
- # Don't fail on unexpected errors
- logger.warning(f"GenVM health probe failed unexpectedly: {e}")
- return True, None
+ # Log unexpected errors and mark as unhealthy
+ logger.exception("GenVM health probe failed unexpectedly")
+ return False, f"unexpected_error: {type(e).__name__}"🧰 Tools
🪛 Ruff (0.14.10)
372-372: Do not catch blind exception: Exception
(BLE001)
🤖 Prompt for AI Agents
In @backend/protocol_rpc/health.py around lines 372 - 375, The except block that
currently logs "GenVM health probe failed unexpectedly: {e}" and returns True,
None should not mark the service healthy; update the handler to log the full
exception (use logger.exception or include stack/exception details) and return
False, None so unexpected errors cause an unhealthy result; modify the except
Exception block (the one calling logger.warning(f"GenVM health probe failed
unexpectedly: {e}")) accordingly.
Address case mismatch between frontend and backend caused signature verification to fail. Frontend calls getContractNonce which normalizes to checksum, but upgrade verification used raw address. Now both use checksum for consistent tx_count and message hash.
|
🎉 This PR is included in version 0.89.0 🎉 The release is available on GitHub release Your semantic-release bot 📦🚀 |
|


Summary
/healthendpoint now returns cached results (~1ms response time instead of 14+ seconds)HEALTH_CHECK_INTERVAL_SECONDSenv var)/pingendpoint for ultra-lightweight load balancer checks/health/*endpoints for debugging (run checks synchronously)Problem
The
/healthendpoint was taking 14+ seconds intermittently due to:When Cloudflare/LB health checks coincided with user requests, this caused cascading delays.
Solution
Background task runs health checks periodically and caches results. The
/healthendpoint just returns the cached data, making it consistently fast (~1ms).Test plan
/healthresponds quickly after deployment/healthstill reflects actual service health (not stale)Summary by CodeRabbit
New Features
Bug Fixes
✏️ Tip: You can customize this high-level summary in your review settings.