Skip to content

fix: avoid expensive no-progress health scan - #1638

Merged
MuncleUscles merged 1 commit into
mainfrom
fix/rally-health-no-progress-scan
May 24, 2026
Merged

MuncleUscles merged 1 commit into
mainfrom
fix/rally-health-no-progress-scan

Conversation

@MuncleUscles

@MuncleUscles MuncleUscles commented May 24, 2026 •

Copy link
Copy Markdown
Member

Summary

  • split the no-consensus-progress detector into a cheap backlog gate followed by the JSON history scan only when an old backlog exists
  • add a DB statement timeout around the expensive progress scan so the background health task cannot wedge indefinitely
  • add regression coverage that verifies the progress-history scan is skipped when there is no backlog

Verification

  • .venv/bin/python -m py_compile backend/protocol_rpc/health.py
  • .venv/bin/python -m pytest tests/unit/test_rpc_health_genvm_tracking.py -q
  • .venv/bin/python -m black --check backend/protocol_rpc/health.py tests/db-sqlalchemy/test_health_orphan_detection.py

Note: tests/db-sqlalchemy/test_health_orphan_detection.py requires POSTGRES_URL locally; CI should run it in the DB-backed environment.

Summary by CodeRabbit

  • Improvements

    • Optimized consensus health check with more efficient progress detection logic
    • Enhanced reliability with timeout protection and error recovery for health monitoring queries
    • Added error tracking for failed health check operations
  • Tests

    • Added regression test verifying optimized health check behavior during low-backlog scenarios

Review Change Stack

@coderabbitai

coderabbitai Bot commented May 24, 2026 •

Copy link
Copy Markdown
Contributor
📝 Walkthrough

Walkthrough

The PR refactors consensus "no progress" detection in _check_consensus_health() to improve efficiency and resilience. A new timeout constant gates the expensive history scan; a backlog-age check determines whether to run the scan at all. Query failures are caught and recorded without breaking the health check. The response payload now exposes scan error status, and a regression test validates that the optimization correctly avoids scanning when unnecessary.

Changes

Consensus Health No-Progress Detection Optimization

Layer / File(s) Summary
No-Progress Detection with Backlog Gating and Timeout Protection
backend/protocol_rpc/health.py
Adds HEALTH_NO_PROGRESS_QUERY_TIMEOUT_MS constant and refactors progress detection to compute a backlog-age gate, then conditionally run the expensive history scan only when it can affect the decision. The scan executes under statement timeout protection and errors are caught, allowing the health check to continue while recording scan failure.
Progress Evaluation and Health Payload Extension
backend/protocol_rpc/health.py
Uses the refactored progress detection variables to compute the final no_consensus_progress flag and extends the returned consensus health payload with no_progress_check_error to expose scan timeout/error status.
Regression Test for Progress Scan Optimization
tests/db-sqlalchemy/test_health_orphan_detection.py
Adds SQLAlchemy event hook to intercept cursor execution and verify the progress detector skips history scanning when backlog is 0, validating healthy consensus result with no progress alert and zero backlog count.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~25 minutes

Possibly related PRs

  • genlayerlabs/genlayer-studio#1621: Both PRs modify _check_consensus_health() logic and orphan/no-progress detection behavior in the same health-check function.
  • genlayerlabs/genlayer-studio#1636: Both PRs directly modify consensus "no progress" detection logic and extend test_health_orphan_detection.py with backlog and progress-check regression tests.
  • genlayerlabs/genlayer-studio#1372: Both PRs modify the consensus health computation inside _check_consensus_health() function, with this PR refactoring the no-progress detection logic within that same health-check path.

Suggested labels

run-tests

Poem

🐰 With gates that check before you scan,
Your progress queries now have a plan.
When backlog sleeps, no history needed—
Health checks faster, timeouts heeded!
Error-safe, the checks proceed again. 🌟

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 66.67% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Title check ✅ Passed The title 'fix: avoid expensive no-progress health scan' is clear, concise, and accurately describes the main optimization made in the pull request—avoiding unnecessary database scans in the health check logic.
Description check ✅ Passed The pull request description includes a comprehensive summary, specific verification steps, and implementation details that align well with the template's intent, though some template sections are not explicitly filled.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch fix/rally-health-no-progress-scan

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🧹 Nitpick comments (1)
tests/db-sqlalchemy/test_health_orphan_detection.py (1)

667-674: ⚡ Quick win

Add type hints to the new event callback (and test return) for guideline compliance.

Suggested fix
+from typing import Any
@@
-async def test_no_progress_detector_skips_history_scan_without_backlog(engine: Engine):
+async def test_no_progress_detector_skips_history_scan_without_backlog(
+    engine: Engine,
+) -> None:
@@
-    def fail_on_progress_scan(
-        conn, cursor, statement, parameters, context, executemany
-    ):
+    def fail_on_progress_scan(
+        conn: Any,
+        cursor: Any,
+        statement: str,
+        parameters: Any,
+        context: Any,
+        executemany: bool,
+    ) -> None:

As per coding guidelines, "**/*.py: Include type hints in all Python code".

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@tests/db-sqlalchemy/test_health_orphan_detection.py` around lines 667 - 674,
The new test callback lacks type annotations; add type hints to the event
callback function fail_on_progress_scan and to the test return type for
guideline compliance: annotate fail_on_progress_scan's parameters (e.g., conn,
cursor, statement, parameters, context, executemany) with appropriate types from
sqlalchemy and typing (such as Connection/Cursor or DBAPI cursor, str, Any, Any,
bool) and declare the test function
test_no_progress_detector_skips_history_scan_without_backlog -> None; ensure
imports (typing.Any or sqlalchemy types) are added if needed.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@backend/protocol_rpc/health.py`:
- Around line 915-916: The computed no_progress_check_error returned from
_check_consensus_health() is not being propagated into the cached /health
payload; update _run_health_checks() so when it builds/updates
services["consensus"] it copies the no_progress_check_error (and keep
NO_PROGRESS_WINDOW_MINUTES if needed) into that dict—i.e., locate where
_run_health_checks() consumes _check_consensus_health() results and add
"no_progress_check_error": no_progress_check_error to the services["consensus"]
response so clients of /health can see the scan timeout/error state.

---

Nitpick comments:
In `@tests/db-sqlalchemy/test_health_orphan_detection.py`:
- Around line 667-674: The new test callback lacks type annotations; add type
hints to the event callback function fail_on_progress_scan and to the test
return type for guideline compliance: annotate fail_on_progress_scan's
parameters (e.g., conn, cursor, statement, parameters, context, executemany)
with appropriate types from sqlalchemy and typing (such as Connection/Cursor or
DBAPI cursor, str, Any, Any, bool) and declare the test function
test_no_progress_detector_skips_history_scan_without_backlog -> None; ensure
imports (typing.Any or sqlalchemy types) are added if needed.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 29eae775-680c-4e60-8a84-c89b49409187

📥 Commits

Reviewing files that changed from the base of the PR and between d6a82c7 and 6ff6133.

📒 Files selected for processing (2)
  • backend/protocol_rpc/health.py
  • tests/db-sqlalchemy/test_health_orphan_detection.py

Comment on lines +915 to 916
"no_progress_check_error": no_progress_check_error,
"no_progress_window_minutes": NO_PROGRESS_WINDOW_MINUTES,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major | ⚡ Quick win

no_progress_check_error is computed but not exposed through cached /health consensus payload.

Line 915 adds the field to _check_consensus_health() output, but _run_health_checks() does not copy it into services["consensus"], so clients of /health cannot see scan timeout/error state.

Suggested fix
diff --git a/backend/protocol_rpc/health.py b/backend/protocol_rpc/health.py
@@
         services["consensus"] = {
             "processing_transactions": consensus_health.get(
                 "total_processing_transactions", 0
             ),
@@
             "seconds_since_consensus_progress": consensus_health.get(
                 "seconds_since_consensus_progress"
             ),
+            "no_progress_check_error": consensus_health.get(
+                "no_progress_check_error", False
+            ),
             "active_workers": consensus_health.get("active_workers", 0),
             "status": consensus_status,
         }
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@backend/protocol_rpc/health.py` around lines 915 - 916, The computed
no_progress_check_error returned from _check_consensus_health() is not being
propagated into the cached /health payload; update _run_health_checks() so when
it builds/updates services["consensus"] it copies the no_progress_check_error
(and keep NO_PROGRESS_WINDOW_MINUTES if needed) into that dict—i.e., locate
where _run_health_checks() consumes _check_consensus_health() results and add
"no_progress_check_error": no_progress_check_error to the services["consensus"]
response so clients of /health can see the scan timeout/error state.

@MuncleUscles
MuncleUscles merged commit 529343d into main May 24, 2026
25 checks passed
@MuncleUscles
MuncleUscles deleted the fix/rally-health-no-progress-scan branch May 24, 2026 23:09
@github-actions

Copy link
Copy Markdown
Contributor

🎉 This PR is included in version 0.120.12 🎉

The release is available on GitHub release

Your semantic-release bot 📦🚀

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant