Skip to content

fix: avoid false no-progress alerts on scan timeout - #1639

Merged
MuncleUscles merged 1 commit into
mainfrom
fix/no-progress-timeout-not-degraded
May 25, 2026
Merged

MuncleUscles merged 1 commit into
mainfrom
fix/no-progress-timeout-not-degraded

Conversation

@MuncleUscles

@MuncleUscles MuncleUscles commented May 24, 2026 •

Copy link
Copy Markdown
Member

Summary

  • do not assert no_consensus_progress when the optional JSON history scan times out
  • expose no_progress_check_error in /health consensus details
  • add DB regression coverage for scan timeout fallback

Verification

  • .venv/bin/python -m py_compile backend/protocol_rpc/health.py
  • .venv/bin/python -m black --check backend/protocol_rpc/health.py tests/db-sqlalchemy/test_health_orphan_detection.py
  • .venv/bin/python -m pytest tests/unit/test_rpc_health_genvm_tracking.py -q
  • .venv/bin/python -m pytest tests/db-sqlalchemy/test_health_orphan_detection.py -q (blocked locally: POSTGRES_URL not set)

Summary by CodeRabbit

  • Bug Fixes
    • Fixed health monitoring to correctly handle database query timeouts and errors during consensus checks. Previously, temporary database issues would incorrectly trigger consensus outage alerts and degrade system status. The system now distinguishes between actual consensus failures and transient query problems, preventing false alerts while maintaining accurate health status reporting for operators.

Review Change Stack

@coderabbitai

coderabbitai Bot commented May 24, 2026 •

Copy link
Copy Markdown
Contributor

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 9a0c47e5-bf87-4708-9fc9-4a12124292ba

📥 Commits

Reviewing files that changed from the base of the PR and between 529343d and 78881c7.

📒 Files selected for processing (2)
  • backend/protocol_rpc/health.py
  • tests/db-sqlalchemy/test_health_orphan_detection.py

📝 Walkthrough

Walkthrough

The consensus health detector is updated to expose progress-scan errors in the cached health response and to avoid flagging a consensus outage when the progress-history scan fails. A regression test validates that an error during the progress scan is recorded without triggering degraded consensus status.

Changes

Consensus Health Progress-Scan Error Handling

Layer / File(s) Summary
Progress-scan error field and consensus logic
backend/protocol_rpc/health.py, tests/db-sqlalchemy/test_health_orphan_detection.py
The cached consensus health response now exposes no_progress_check_error alongside progress and backlog metrics. The no_recent_progress boolean is updated to require that the progress-scan did not error, preventing a query timeout or failure from asserting consensus outage. A regression test forces the progress scan to fail and verifies that no_progress_check_error is recorded, the overall status remains healthy, and no_consensus_progress is not asserted.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~20 minutes

Possibly related PRs

  • genlayerlabs/genlayer-studio#1638: Both PRs update consensus "no progress" detection to handle progress-scan timeout/error via a no_progress_check_error flag and prevent asserting no_consensus_progress, with overlapping DB-backed test coverage.
  • genlayerlabs/genlayer-studio#1636: Both PRs modify consensus "no progress"/no_consensus_progress computation and extend health orphan detection tests to cover scan/query failure scenarios.
  • genlayerlabs/genlayer-studio#1372: Both PRs refine consensus health orchestration and cache logic within backend/protocol_rpc/health.py.

Suggested labels

run-tests

Poem

🐰 A scan may stumble, a query may freeze,
But we catch the error with gentle ease—
No false alarms when progress can't see,
Health stays intact, consensus runs free! ✨

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 71.43% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Title check ✅ Passed The PR title clearly and concisely describes the main change: preventing false no-progress alerts when a scan times out, which directly matches the core fix in the changeset.
Description check ✅ Passed The PR description includes a summary section covering all key changes and a verification section with testing steps, though it lacks the detailed 'What', 'Why', and 'Testing done' sections from the template.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch fix/no-progress-timeout-not-degraded

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@MuncleUscles
MuncleUscles merged commit e0757e1 into main May 25, 2026
14 checks passed
@MuncleUscles
MuncleUscles deleted the fix/no-progress-timeout-not-degraded branch May 25, 2026 00:21
@github-actions

Copy link
Copy Markdown
Contributor

🎉 This PR is included in version 0.120.13 🎉

The release is available on GitHub release

Your semantic-release bot 📦🚀

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant