Skip to content

fix: initialize genvm health before alert checks - #1637

Merged
MuncleUscles merged 2 commits into
mainfrom
fix-health-genvm-readiness-order
May 24, 2026
Merged

MuncleUscles merged 2 commits into
mainfrom
fix-health-genvm-readiness-order

Conversation

@MuncleUscles

@MuncleUscles MuncleUscles commented May 24, 2026 •

Copy link
Copy Markdown
Member

Summary

  • probe and cache local GenVM health before heavier DB/alert checks in the background health loop
  • keep readiness-critical GenVM state from staying uninitialized when alert queries are slow
  • add a unit test covering the ordering

Verification

  • python3 -m py_compile backend/protocol_rpc/health.py tests/unit/test_rpc_health_genvm_tracking.py
  • pytest tests/unit/test_rpc_health_genvm_tracking.py -q (local env missing python-dotenv; full validation via CI)

Summary by CodeRabbit

  • Refactor

    • Improved GenVM health tracking and permit/capacity reporting for more accurate service status.
  • New Features

    • Provider availability checks now enforce a configurable timeout, causing slow providers to be marked unavailable sooner.
  • Tests

    • Added tests to verify health-check ordering, GenVM state propagation, and provider timeout behavior.

Review Change Stack

@coderabbitai

coderabbitai Bot commented May 24, 2026 •

Copy link
Copy Markdown
Contributor
📝 Walkthrough

Walkthrough

Extracted GenVM cache updates and permit parsing into _update_genvm_health_cache(...), moved GenVM manager health check to run first inside _run_health_checks(), added tests asserting GenVM-first ordering and verifying cache fields, and added a timeout around the GenVM LLM availability probe.

Changes

GenVM Health Check Refactoring

Layer / File(s) Summary
GenVM health cache helper function
backend/protocol_rpc/health.py
New _update_genvm_health_cache(...) helper centralizes GenVM cache updates to both _health_cache and services["genvm"] details, including best-effort parsing of permit/capacity metrics (max, available, and active execution counts) from the GenVM /status JSON with error handling.
Reordered health check with GenVM prioritization
backend/protocol_rpc/health.py
_run_health_checks() now performs the GenVM manager health check first and routes results through _update_genvm_health_cache(...); step numbering/comments were adjusted accordingly.
Test validation of GenVM ordering and cache state
tests/unit/test_rpc_health_genvm_tracking.py
TestBackgroundHealthGenVMOrdering monkeypatches internal health-check functions, runs _run_health_checks(), asserts GenVM health is checked first via call ordering, and validates the shared health cache reflects expected GenVM health and permit/execution values.
LLM provider availability timeout
backend/protocol_rpc/endpoints.py, tests/unit/test_llm_provider_availability.py
check_provider_is_available now wraps genvm_manager.try_llms(...) with asyncio.wait_for using LLM_PROVIDER_AVAILABILITY_TIMEOUT_SECONDS (default 20). Tests add a slow manager and assert timeout behavior.

Sequence Diagram

sequenceDiagram
  participant HealthLoop as _run_health_checks()
  participant GenVM as genvm_manager
  participant Cache as _update_genvm_health_cache
  participant DB as database_check
  participant Consensus as consensus_check

  HealthLoop->>GenVM: probe genvm_manager.health()
  GenVM-->>HealthLoop: health response /status JSON
  HealthLoop->>Cache: _update_genvm_health_cache(response)
  Cache-->>HealthLoop: updated _health_cache (genvm fields)
  HealthLoop->>DB: database_check() (observes genvm cache)
  HealthLoop->>Consensus: consensus_check() (observes genvm cache)
Loading

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~45 minutes

Possibly related PRs

Suggested labels

run-tests

Poem

🐰 I hopped through checks at break of dawn,
GenVM first — no steps withdrawn.
Cache refreshed with permits parsed right,
Tests give order, glowing bright.
A tiny rabbit cheers the build tonight.

🚥 Pre-merge checks | ✅ 3 | ❌ 2

❌ Failed checks (1 warning, 1 inconclusive)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 11.76% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
Description check ❓ Inconclusive The description covers the key changes (reordering GenVM health checks, preventing uninitialized state, adding tests) and verification steps, but omits most required template sections like 'What', 'Why', 'Testing done', and 'Decisions made'. Expand the description to address template sections: detail specific changes in 'What', explain the problem and value in 'Why', describe all test coverage in 'Testing done', and note any implementation decisions.
✅ Passed checks (3 passed)
Check name Status Explanation
Title check ✅ Passed The title 'fix: initialize genvm health before alert checks' directly describes the main change: reordering GenVM health initialization to occur before alert checks.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch fix-health-genvm-readiness-order

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@backend/protocol_rpc/endpoints.py`:
- Around line 319-321: The code assigns timeout_s by directly casting an env var
to float, which will raise on malformed values or accept non-positive numbers;
change this to robust parsing: fetch the string with
os.environ.get("LLM_PROVIDER_AVAILABILITY_TIMEOUT_SECONDS", "20"), try
converting to float inside a try/except, on exception or if the resulting value
is <= 0 set timeout_s to a safe default (e.g., 20.0), and optionally clamp to a
sensible max; update the logic around the timeout_s variable in endpoints.py so
all availability checks use this validated, positive timeout value.

In `@tests/unit/test_llm_provider_availability.py`:
- Around line 45-51: Add explicit type hints to the test helper signatures:
annotate SlowGenVMManager.__init__(self) -> None and give self.logger the type
MagicMock (from unittest.mock) if you want an attribute annotation; annotate
SlowGenVMManager.try_llms(self, providers: list, prompt: str) -> Coroutine[Any,
Any, List[Dict[str, Any]]] (import Any, Dict, List, Coroutine from typing) or
simpler async def try_llms(...) -> List[Dict[str, Any]] if you prefer using the
runtime return type; apply the same style to the other test helper/function near
lines 66-67 (add parameter and return types and necessary typing imports).
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: b9c4d558-f1f0-4f80-8d3f-3a0a7a17241f

📥 Commits

Reviewing files that changed from the base of the PR and between 4e72457 and 342a74f.

📒 Files selected for processing (2)
  • backend/protocol_rpc/endpoints.py
  • tests/unit/test_llm_provider_availability.py

Comment on lines +319 to +321
timeout_s = float(
os.environ.get("LLM_PROVIDER_AVAILABILITY_TIMEOUT_SECONDS", "20")
)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor | ⚡ Quick win

Harden timeout env parsing and bounds.

Line 319 directly casts the env var to float; malformed or non-positive values can make availability checks fail immediately and mark providers unavailable. Add guarded parsing with a positive fallback.

Proposed fix
-        timeout_s = float(
-            os.environ.get("LLM_PROVIDER_AVAILABILITY_TIMEOUT_SECONDS", "20")
-        )
+        raw_timeout = os.environ.get("LLM_PROVIDER_AVAILABILITY_TIMEOUT_SECONDS", "20")
+        try:
+            timeout_s = float(raw_timeout)
+            if timeout_s <= 0:
+                raise ValueError("timeout must be > 0")
+        except (TypeError, ValueError):
+            genvm_manager.logger.warning(
+                "Invalid LLM provider availability timeout; using default",
+                raw_timeout=raw_timeout,
+            )
+            timeout_s = 20.0
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@backend/protocol_rpc/endpoints.py` around lines 319 - 321, The code assigns
timeout_s by directly casting an env var to float, which will raise on malformed
values or accept non-positive numbers; change this to robust parsing: fetch the
string with os.environ.get("LLM_PROVIDER_AVAILABILITY_TIMEOUT_SECONDS", "20"),
try converting to float inside a try/except, on exception or if the resulting
value is <= 0 set timeout_s to a safe default (e.g., 20.0), and optionally clamp
to a sensible max; update the logic around the timeout_s variable in
endpoints.py so all availability checks use this validated, positive timeout
value.

Comment on lines +45 to +51
class SlowGenVMManager:
def __init__(self):
self.logger = MagicMock()

async def try_llms(self, providers, prompt):
await asyncio.sleep(1)
return [{"response": "ok"}]

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🛠️ Refactor suggestion | 🟠 Major | ⚡ Quick win

Add type hints to newly introduced test helper/test function signatures.

Lines 45-51 and Line 66 introduce new Python functions without annotations; please add parameter and return types for compliance.

Proposed fix
 import asyncio
+from typing import Any
 from unittest.mock import MagicMock
@@
 class SlowGenVMManager:
-    def __init__(self):
+    def __init__(self) -> None:
         self.logger = MagicMock()
 
-    async def try_llms(self, providers, prompt):
+    async def try_llms(
+        self, providers: list[dict[str, Any]], prompt: dict[str, Any]
+    ) -> list[dict[str, str]]:
         await asyncio.sleep(1)
         return [{"response": "ok"}]
@@
-async def test_provider_availability_timeout_returns_unavailable(monkeypatch):
+async def test_provider_availability_timeout_returns_unavailable(
+    monkeypatch: pytest.MonkeyPatch,
+) -> None:

As per coding guidelines, "Include type hints in all Python code".

Also applies to: 66-67

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@tests/unit/test_llm_provider_availability.py` around lines 45 - 51, Add
explicit type hints to the test helper signatures: annotate
SlowGenVMManager.__init__(self) -> None and give self.logger the type MagicMock
(from unittest.mock) if you want an attribute annotation; annotate
SlowGenVMManager.try_llms(self, providers: list, prompt: str) -> Coroutine[Any,
Any, List[Dict[str, Any]]] (import Any, Dict, List, Coroutine from typing) or
simpler async def try_llms(...) -> List[Dict[str, Any]] if you prefer using the
runtime return type; apply the same style to the other test helper/function near
lines 66-67 (add parameter and return types and necessary typing imports).

@MuncleUscles
MuncleUscles merged commit d6a82c7 into main May 24, 2026
25 checks passed
@MuncleUscles
MuncleUscles deleted the fix-health-genvm-readiness-order branch May 24, 2026 21:08
@github-actions

Copy link
Copy Markdown
Contributor

🎉 This PR is included in version 0.120.11 🎉

The release is available on GitHub release

Your semantic-release bot 📦🚀

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant