Skip to content

Rafal/fix hscale deployment - #1342

Merged
cristiam86 merged 112 commits into
mainfrom
rafal/fix-hscale-deployment
Oct 29, 2025
Merged

cristiam86 merged 112 commits into
mainfrom
rafal/fix-hscale-deployment

Conversation

@rrabenda

@rrabenda rrabenda commented Oct 17, 2025 •

Copy link
Copy Markdown
Contributor

Fixes #issue-number-here

What

  • changed thing a for b
  • also did this other unrelated thing in my path

Why

  • to fix a bug
  • to add more value to the user

Testing done

  • tested the new feature
  • tested the bug fix

Decisions made

Checks

  • I have tested this code
  • I have reviewed my own PR
  • I have created an issue for this PR
  • I have set a descriptive PR title compliant with conventional commits

Reviewing tips

User facing release notes

Summary by CodeRabbit

  • Infrastructure

    • Moved services to Traefik host/TLS routing, added TLS certificate config and file-based provider, extended timeouts, resource limits, and adjusted restart policies; added DOCKER_GID hint.
  • New Features

    • Added standalone health monitoring dashboard (UI + docs) and expanded aggregated health API with per-worker, per-contract and CPU metrics.
  • Tests

    • Load test now captures deploy output, extracts contract addresses to determine success, and starts Phase 2 sooner.
  • Bug Fixes

    • Safer handling when consensus history is absent; recovery now clears consensus state; RPC endpoints raise explicit errors for missing data.
  • Chores

    • Improved worker lifecycle, signal/shutdown handling, startup logging, and runtime logging/timeout tweaks.

jmlago and others added 30 commits September 8, 2025 19:45

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

♻️ Duplicate comments (1)
docker-compose.yml (1)

150-152: Secure health endpoints with HTTPS to match main API architecture.

The review comment is correct. Health endpoints currently use unencrypted HTTP (web entrypoint, line 151) while the main jsonrpc API uses HTTPS (websecure entrypoint + tls: true, lines 146–147). This creates a security gap inconsistent with the TLS-only architecture (frontend also uses websecure on lines 64–65).

Additionally, the comment on line 149 ("no TLS required") suggests this was an intentional choice but contradicts the stated TLS-only design. Apply the suggested fix and update the comment:

-      # HTTP router for health endpoints (no TLS required)
+      # HTTPS router for health endpoints
       traefik.http.routers.jsonrpc-health.rule: Host(`${SERVER_NAME}`) && (PathPrefix(`/health`) || PathPrefix(`/ready`))
-      traefik.http.routers.jsonrpc-health.entrypoints: web
+      traefik.http.routers.jsonrpc-health.entrypoints: websecure
+      traefik.http.routers.jsonrpc-health.tls: true
       traefik.http.routers.jsonrpc-health.service: jsonrpc
🧹 Nitpick comments (2)
docker-compose.yml (2)

125-128: MINOR: Redundant memory configuration — both top-level and deploy.resources present.

Lines 125–128 define memory limits at the service top-level (mem_limit, mem_reservation, memswap_limit, oom_kill_disable), and lines 131–137 duplicate these within deploy.resources.limits and deploy.resources.reservations.

In Docker Compose:

  • Top-level memory fields apply in single-container mode.
  • deploy.resources apply in Swarm mode.
  • Both present together may cause ambiguity or override conflicts depending on runtime context.

Recommendation: Remove top-level memory fields (lines 125–128) and rely solely on deploy.resources (lines 131–137) for consistency in Swarm deployments. If single-container mode support is needed, document this dual config.

Same applies to consensus-worker (lines 268–271 + 279–285).

Also applies to: 131-137


15-20: OPTIONAL: Very long timeout values — verify these are intentional and don't mask underlying issues.

Lines 15–20 set Traefik entrypoint timeouts to 600 seconds (10 minutes) for read, write, and idle. Line 143 sets jsonrpc response timeout to 12 minutes (720s). These are unusually long and suggest slow operations (e.g., large contract state loading on x86_64 per the comment).

Verify:

  • Are these timeouts necessary for your workload, or do they indicate a performance/architecture issue worth addressing separately?
  • Consider adding monitoring alerts if requests regularly approach these limits, so you can optimize downstream.

Also applies to: 143-143

📜 Review details

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 7042f60 and 1a7ec79.

📒 Files selected for processing (1)
  • docker-compose.yml (8 hunks)
🧰 Additional context used
📓 Path-based instructions (1)
docker-compose*.yml

📄 CodeRabbit inference engine (AGENTS.md)

Use docker-compose*.yml files for composing services

Files:

  • docker-compose.yml
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (1)
  • GitHub Check: backend-unit-tests
🔇 Additional comments (1)
docker-compose.yml (1)

63-63: The review comment incorrectly identifies a routing conflict that does not exist.

Traefik's default router priority is based on rule length (longer rules receive higher priority). The jsonrpc router rule is approximately 150+ characters—much longer than the frontend's ~32-character rule. Therefore, jsonrpc naturally receives higher priority and will match its specific paths (/api, /socket.io, /ws, /monitoring) first, preventing the frontend rule from intercepting that traffic.

The current configuration works correctly without explicit priorities. While adding explicit priority labels could improve clarity, they are not a critical fix for a non-existent routing conflict.

Likely an incorrect or invalid review comment.

Comment thread docker-compose.yml
Comment on lines +275 to +278
condition: on-failure # Only restart on failure (not on clean exit)
delay: 5s
max_attempts: 3
max_attempts: 0
window: 120s

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🔴 Critical

🧩 Analysis chain

MAJOR: Contradictory intent between max_attempts: 0 value and restart behavior.

Line 277 sets max_attempts: 0 within the restart_policy block of consensus-worker deploy config. In Docker Swarm, 0 means unlimited retries. However, the surrounding config suggests finite/controlled restart behavior (condition: on-failure, delay: 5s, window: 120s).

Clarify intent:

  • If you want to limit retries (e.g., to 5), change value to max_attempts: 5.
  • If you want unlimited retries, update/add a comment clarifying that 0 = unlimited to avoid future confusion.

Previous review flagged a similar issue; verify this is intentional or apply a fix.


I'll verify this review comment by examining the docker-compose.yml file and confirming the Docker Compose semantics for the max_attempts setting.





🏁 Script executed:

#!/bin/bash
# View the docker-compose.yml file around lines 275-278
cat docker-compose.yml | head -290 | tail -30

Length of output: 1037


🌐 Web query:

Docker Compose restart policy max_attempts 0 meaning unlimited retries

💡 Result:

Short answer: The Compose/Swarm spec treats the default (when max_attempts is omitted) as "never give up" (unlimited retries). The spec/docs do not document that explicitly setting max_attempts: 0 equals unlimited — the documented way to get unlimited retries is to omit max_attempts. (compose-spec.github.io)

If you need definitive behaviour for your tool/version (some CLI/SDKs treat 0 specially), tell me which Docker/Compose/version and I’ll look it up.

Citations:






🌐 Web query:

Docker Compose max_attempts 0 behavior explicit value unlimited

💡 Result:

Short answer: max_attempts does not mean "zero retries" — 0 is treated as "no limit" (i.e., unlimited / never give up). Official Compose docs state the default is "never give up" and the Engine API notes the MaxAttempts default is 0 which is ignored (unbounded). [1][2]

Sources:

  • Compose file — restart_policy (max_attempts default: never give up). [1]
  • Docker Engine API (MaxAttempts default 0 = ignored / unbounded). [2]

🌐 Web query:

Docker Compose service restart versus deploy restart_policy conflict

💡 Result:

Short answer

  • "restart:" is the Docker Compose (container) restart setting used by docker-compose / docker compose up. "deploy.restart_policy:" is the Swarm/service restart setting used by docker stack deploy (Swarm mode). When you run a compose file with docker-compose, deploy.* is ignored; when you deploy a stack to Swarm, restart (top-level) is ignored and deploy.restart_policy is used. [1][2][3]

Details & recommendations

  • Use restart: for local/docker-compose workflows. Use deploy.restart_policy: for Swarm/service-level control (condition, delay, max_attempts, window). [2][3]
  • If you need one file for both: either keep both sections (each tool will pick the relevant one) or run docker compose --compatibility to have Compose map deploy settings to container restart behavior (note: some subfields may be ignored). [1][5]

References

  • Docker Compose "restart" / command reference. [1]
  • Compose deploy.restart_policy (Compose spec / Docker docs). [2][3]
  • Community notes / Q&A on ignored options and behavior. [4]

Sources
[1] Docker Compose CLI / restart docs.
[2] Compose deploy.restart_policy — Compose specification / Docker docs.
[3] Docker Compose file reference: restart ignored by docker stack deploy.
[4] Stack Overflow / community explanations about ignored options and --compatibility.


Clarify restart intent: max_attempts: 0 (unlimited retries) may conflict with surrounding configuration.

The review comment is valid. According to Docker Compose and Engine API specs, max_attempts: 0 is treated as "no limit" (unlimited retries). However, the surrounding config—condition: on-failure, delay: 5s, window: 120s—with its comment "Only restart on failure (not on clean exit)" suggests intentional, controlled restart behavior, which contradicts unlimited retries.

Action required: Either:

  • Change max_attempts: 0 to a finite number (e.g., 5) if retries should be limited, or
  • Add a comment (e.g., # 0 = unlimited retries) if unlimited behavior is intentional.
🤖 Prompt for AI Agents
docker-compose.yml around lines 275 to 278: the restart policy uses
max_attempts: 0 which means "unlimited retries" and conflicts with the
surrounding intent of controlled restarts; either set max_attempts to a finite
value (e.g., 5) to limit retries, or keep 0 but add an explicit comment like "#
0 = unlimited retries (intentional)" and update the existing comment to clarify
the chosen behavior.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 0

♻️ Duplicate comments (3)
docker-compose.yml (3)

149-151: ⚠️ Major: Health endpoints exposed over plain HTTP while main service uses HTTPS—security inconsistency.

The jsonrpc health router (lines 149–151) routes /health and /ready endpoints over the unencrypted web entrypoint, while the main API router (line 145) uses websecure (TLS). This creates a security inconsistency: health monitoring data is exposed in cleartext and accessible to network eavesdroppers.

Recommended fix: Secure health endpoints over HTTPS by changing the entrypoint to websecure and enabling TLS:

     # HTTP router for health endpoints (no TLS required)
     traefik.http.routers.jsonrpc-health.rule: Host(`${SERVER_NAME}`) && (PathPrefix(`/health`) || PathPrefix(`/ready`))
-    traefik.http.routers.jsonrpc-health.entrypoints: web
+    traefik.http.routers.jsonrpc-health.entrypoints: websecure
+    traefik.http.routers.jsonrpc-health.tls: true

Alternatively, if plaintext health endpoints are intentionally required (e.g., for load balancer health probes), add an explicit comment documenting the security justification and trade-off.


62-66: ⚠️ CRITICAL: Traefik router priority conflict—frontend broad rule will intercept jsonrpc traffic.

The frontend router uses an unqualified Host(${SERVER_NAME}) rule that matches all traffic to the host regardless of path. The jsonrpc service (line 144) defines a more specific Host(${SERVER_NAME}) && (PathPrefix(...)) rule. Without explicit priorities, Traefik's matching heuristic is ambiguous and the broad frontend rule can incorrectly intercept requests intended for jsonrpc paths like /api, /socket.io, /ws, /health, or /monitoring.

Action: Add explicit router priorities to both routers. Set a lower priority for frontend and a higher priority for jsonrpc so path-specific rules win:

   labels:
     traefik.enable: true
     traefik.http.routers.frontend.rule: Host(`${SERVER_NAME}`)
+    traefik.http.routers.frontend.priority: "10"
     traefik.http.routers.frontend.entrypoints: websecure
     traefik.http.routers.frontend.tls: true

And for jsonrpc (line 144):

     traefik.http.routers.jsonrpc.rule: Host(`${SERVER_NAME}`) && (PathPrefix(`/api`) || PathPrefix(`/socket.io`) || PathPrefix(`/ws`) || PathPrefix(`/monitoring`))
+    traefik.http.routers.jsonrpc.priority: "20"

87-90: ⚠️ CRITICAL: Port binding incompatible with configured replicas—will fail in Docker Swarm mode.

The jsonrpc service defines:

  • Lines 87–88: ports: - "${RPCPORT}:${RPCPORT}" (explicit host port binding)
  • Line 131: replicas: ${JSONRPC_REPLICAS:-3} (default 3 replicas in Swarm)

In Docker Swarm mode, multiple container replicas cannot simultaneously bind the same host port; this configuration will cause deployment failure when replicas > 1. The existing Traefik routing (with load balancing and service discovery via expose) already provides external access and internal communication.

Recommended action: Remove the explicit port binding or set replicas to 1. If host port access is required for local development, limit to a single replica:

-    ports:
-      - "${RPCPORT}:${RPCPORT}"
     expose:
       - "${RPCPORT}"

Or enforce single-instance deployment:

     deploy:
-      replicas: ${JSONRPC_REPLICAS:-3}
+      replicas: 1

Secondary issues:

  • Memory duplication (lines 119–129 and deploy.resources): Both top-level (mem_limit, mem_reservation, memswap_limit) and deploy.resources.limits/reservations define memory constraints. In Swarm mode, deploy.resources takes precedence and top-level settings are ignored; this creates confusion. Consolidate to one location.
  • Health endpoints over HTTP (Traefik label jsonrpc-health.entrypoints: web): Health endpoints are exposed via plain HTTP while the main API uses HTTPS. Either enforce TLS for health endpoints or explicitly document the rationale for unencrypted health traffic.
🧹 Nitpick comments (2)
docker-compose.yml (2)

126-129: ⚠️ Recommended: Duplicate memory resource configuration—clarify intent for top-level vs. deploy.resources.

The jsonrpc service defines memory limits in two places:

  • Top-level (lines 126–129): mem_limit, mem_reservation, memswap_limit, oom_kill_disable
  • Deploy (lines 132–138): deploy.resources.limits.memory, deploy.resources.reservations.memory

In Docker Compose:

  • Top-level memory settings apply to the container runtime (used by docker-compose up in standalone mode).
  • deploy.* settings apply to Swarm mode (docker stack deploy).

Since this file uses replicas (line 131), indicating Swarm mode, the deploy.resources configuration takes precedence and the top-level settings are effectively ignored. This creates confusion and dead code.

Recommended action: Remove the top-level memory settings (lines 126–129) and rely solely on deploy.resources for consistency, or add a comment explaining why both are present (e.g., if the file is meant to support both docker-compose and Swarm workflows).

     expose:
       - "${RPCPORT}"
-    mem_limit:  ${COMPOSE_RPC_MEM_LIMIT:-8g}
-    mem_reservation: ${COMPOSE_RPC_MEM_RESERVATION:-2g}
-    memswap_limit: ${COMPOSE_RPC_MEM_SWAP_LIMIT:-8g}
-    oom_kill_disable: false
     deploy:
       replicas: ${JSONRPC_REPLICAS:-3}

Also applies to: 132-138


267-270: ⚠️ Recommended: Duplicate memory resource configuration—same as jsonrpc service above.

The consensus-worker service defines memory limits in both top-level (lines 267–270) and deploy.resources (lines 278–284). In Swarm mode (indicated by replicas on line 272), the top-level settings are ignored; only deploy.resources applies.

Recommended action: Remove top-level memory settings (lines 267–270) and rely solely on deploy.resources for clarity:

     start_period: 40s
-    mem_limit:  ${COMPOSE_WORKER_MEM_LIMIT:-8g}
-    mem_reservation: ${COMPOSE_WORKER_MEM_RESERVATION:-2g}
-    memswap_limit: ${COMPOSE_WORKER_MEM_SWAP_LIMIT:-8g}
-    oom_kill_disable: false
     deploy:
       replicas: ${CONSENSUS_WORKERS:-1}

Also applies to: 278-284

📜 Review details

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 1a7ec79 and b7e6038.

📒 Files selected for processing (1)
  • docker-compose.yml (8 hunks)
🧰 Additional context used
📓 Path-based instructions (1)
docker-compose*.yml

📄 CodeRabbit inference engine (AGENTS.md)

Use docker-compose*.yml files for composing services

Files:

  • docker-compose.yml
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (1)
  • GitHub Check: backend-unit-tests
🔇 Additional comments (4)
docker-compose.yml (4)

3-3: ✓ Traefik profile and extended timeouts are well-justified.

The profiles: ["studio"] addition enables selective service startup, and the 600-second transport timeouts align with the stated goal of handling large contract state loading. These changes don't conflict with container healthchecks (which measure startup time, not HTTP request duration).

Also applies to: 15-21


153-212: ✓ Other services (webdriver, ollama, postgres, database-migration, redis, vector) have no issues flagged.

These services retain appropriate security settings (no-new-privileges=true), logging configurations, and health checks. No changes related to routing, restart policies, or resource conflicts noted.

Also applies to: 317-359


1-368: Summary: Critical and major issues unresolved in Traefik migration.

This docker-compose.yml file shows progress toward Traefik-based routing but contains multiple unresolved critical and major issues from previous reviews that remain unaddressed or have been re-introduced:

Issue Severity Lines Status
Router priority conflict (frontend broad rule) 🔴 Critical 62–66, 144 Unresolved
Port binding + replicas incompatibility 🔴 Critical 87–88, 131 Marked addressed; still present
Health endpoints on plain HTTP (security) 🟠 Major 149–151 Unresolved
Restart policy max_attempts: 0 contradiction 🟠 Major 273–277, 285 Marked addressed; still present
Memory config duplication (Swarm context) 🟠 Major 126–129, 267–270 New concern

Recommended next steps:

  1. Immediately: Address the critical router priority and port/replicas conflicts to prevent deployment failures.
  2. Before release: Fix the health endpoint security inconsistency and clarify the restart policy intent.
  3. Cleanup: Consolidate duplicate memory configurations for clarity.

Detailed feedback is provided in individual review comments above.


273-277: All technical claims in the review are verified as correct.

The file contains the exact configuration described: restart_policy with condition: on-failure, delay: 5s, max_attempts: 0, and window: 120s along with the comment "Only restart on failure (not on clean exit)", and the top-level restart: unless-stopped is present at line 285.

The contradiction is real: max_attempts: 0 has undefined/ambiguous semantics in the Docker Compose spec (sources note 0 is not explicitly documented, but default behavior is unlimited retries), while the comment and surrounding config suggest intentional controlled restart behavior.

The top-level restart: unless-stopped is confirmed dead code—Swarm's docker stack deploy ignores the top-level restart key and only applies deploy.restart_policy.

The review comment requires developer decision: clarify whether retries should be limited (set max_attempts to a specific number like 5) or unlimited (update comment to explicitly state intent). Remove the dead code at line 285 regardless.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🧹 Nitpick comments (1)
backend/node/genvm/base.py (1)

462-462: Verify the timeout value is appropriate for scaled deployments.

The timeout has been doubled from 10 to 20 minutes. While this appears intentional for the hscale deployment fix, please ensure this value has been tested and is appropriate for your use case, as 20-minute timeouts can significantly impact user experience if operations fail.

Optionally, consider making this timeout configurable via a parameter or environment variable to allow different values for different deployment scenarios without code changes.

📜 Review details

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between a8a499f and 41d855e.

📒 Files selected for processing (2)
  • backend/node/genvm/base.py (1 hunks)
  • docker-compose.yml (8 hunks)
🧰 Additional context used
📓 Path-based instructions (4)
{backend,examples,tests}/**/*.py

📄 CodeRabbit inference engine (CLAUDE.md)

Python code must be formatted with Black and include type hints

Files:

  • backend/node/genvm/base.py
backend/node/genvm/**/*.py

📄 CodeRabbit inference engine (CLAUDE.md)

GenLayer VM implementation code should be placed under backend/node/genvm/

Files:

  • backend/node/genvm/base.py
**/*.py

📄 CodeRabbit inference engine (AGENTS.md)

Target Python 3.12, use 4-space indentation, and rely on Black (via pre-commit) for formatting

Files:

  • backend/node/genvm/base.py
docker-compose*.yml

📄 CodeRabbit inference engine (AGENTS.md)

Use docker-compose*.yml files for composing services

Files:

  • docker-compose.yml
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (1)
  • GitHub Check: backend-unit-tests
🔇 Additional comments (6)
docker-compose.yml (6)

3-26: Extended timeouts and Traefik profile configuration look good.

The 1200-second (20-minute) timeout aligns with GenVM processing and the symmetrical configuration across web/websecure entrypoints ensures consistent behavior. The profiles: ["studio"] addition enables conditional deployment without side effects.


131-134: Memory configuration is well-structured.

The addition of memory limits (mem_limit), reservations (mem_reservation), swap limits (memswap_limit), and OOM controls (oom_kill_disable: false) provides good resource management and prevents runaway memory usage. The values are configurable via environment variables with sensible defaults (8GB limit, 2GB reservation).


273-276: Memory configuration is consistent and well-structured.

The consensus-worker memory limits, reservations, swap controls, and OOM settings mirror the jsonrpc service configuration, providing uniform resource management. Environment variable defaults are sensible (8GB limit, 2GB reservation).


92-93: Looking at the web search results, I need to verify a critical misunderstanding in the original review comment.


The original review comment is factually incorrect about Docker Swarm behavior.

In Docker Swarm, when you publish a service port, the swarm makes the service accessible at the target port on every node, regardless of replica count. On the swarm nodes themselves, port 8080 may not actually be bound, but the routing mesh knows how to route the traffic and prevents any port conflicts from happening.

The review conflates Docker Compose with Docker Swarm: In Docker Compose, each container trying to bind to the same port causes a conflict because Docker Compose lacks built-in load balancing for service ports. However, in Docker Swarm, the ingress network handles internal service discovery and load balancing within the swarm, and the ingress network is responsible for redirecting traffic across all containers on the port.

The port binding at line 93 combined with replicas at line 136 is valid and expected when deploying via docker stack deploy (Swarm mode). The configuration does not cause deployment failure in Swarm. If this file is used only for local development with docker-compose up, then the configuration would be problematic only if replicas > 1; otherwise it works as intended.

Likely an incorrect or invalid review comment.


67-71: INCORRECT: Traefik's default priority behavior already prevents the routing conflict described.

Traefik automatically assigns priority based on rule length—longer rules get higher priority. The jsonrpc router's rule (line 150) is significantly longer due to multiple PathPrefix conditions, giving it higher default priority than the frontend's simple Host rule (line 69). Requests to /api, /socket.io, /ws, and /monitoring will correctly match the more specific jsonrpc rule first.

While explicit priorities are a best practice for clarity and maintainability, the current configuration is functionally safe and does not have the critical routing conflict the review comment describes. No fix is required.

Likely an incorrect or invalid review comment.


145-157: Remove router priority suggestion; clarify health endpoint exposure decision.

The review comment's primary concern about router priority is incorrect. By default Traefik computes each router's priority from the length/complexity of its rule: longer rules get higher priority. In this configuration, the jsonrpc router rule (Host(${SERVER_NAME}) && (PathPrefix(...))) is more complex than the frontend router rule (Host(${SERVER_NAME})), so jsonrpc automatically receives higher priority and will not be intercepted. No explicit priority field is needed.

The second concern—health endpoints exposed over unencrypted HTTP—is valid from a security standpoint. Health endpoints at /health and /ready are routed to the same external hostname as your API. If this is intentional for operational reasons (e.g., external monitoring requires unencrypted checks), document the security trade-off. Otherwise, align health endpoint transport with your TLS-first design:

      traefik.http.routers.jsonrpc-health.rule: Host(`${SERVER_NAME}`) && (PathPrefix(`/health`) || PathPrefix(`/ready`))
-      traefik.http.routers.jsonrpc-health.entrypoints: web
+      traefik.http.routers.jsonrpc-health.entrypoints: websecure
+      traefik.http.routers.jsonrpc-health.tls: true
       traefik.http.routers.jsonrpc-health.service: jsonrpc

If unencrypted health checks are required for external monitoring, clarify the architecture (e.g., load balancer IP allowlist) and document this constraint.

Likely an incorrect or invalid review comment.

Comment thread docker-compose.yml
Comment on lines 256 to +283
restart_policy:
condition: on-failure
condition: on-failure # Only restart on failure (not on clean exit)
delay: 5s
max_attempts: 3
max_attempts: 0
window: 120s

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🔴 Critical

🧩 Analysis chain

Clarify max_attempts: 0 intent — contradicts surrounding controlled-restart configuration.

In Docker Compose/Swarm, max_attempts: 0 means unlimited retries, but the surrounding config—condition: on-failure, delay: 5s, window: 120s—with its comment "Only restart on failure" suggests intentional, controlled restart behavior. This is contradictory.

Previous reviews flagged similar issues but this line was not updated. Clarify your intent:

  • If you want limited retries (e.g., to 5), change the value: max_attempts: 5
  • If you want unlimited retries, update the comment to reflect that 0 = unlimited (or document why unlimited retries are acceptable for consensus workers)

Apply one of these diffs:

Option A: Limit retries to match comment intent

-        max_attempts: 0
+        max_attempts: 5

Option B: Clarify unlimited retries are intentional

+        # 0 = unlimited retries (intentional for consensus resilience)
         max_attempts: 0

Based on past review comments that flagged this contradiction, please resolve it now by either limiting retries or confirming unlimited retries are desired and adding a clarifying comment.



---

🏁 Script executed:

```shell
#!/bin/bash
# Inspect docker-compose.yml file around the consensus-worker restart_policy section
cat docker-compose.yml | head -n 300 | tail -n 100

Length of output: 2946


🏁 Script executed:

#!/bin/bash
# Search for consensus-worker service definition and surrounding context
rg -n "consensus-worker|max_attempts|restart_policy" docker-compose.yml -A2 -B2

Length of output: 543


🏁 Script executed:

#!/bin/bash
# Check git history for mentions of max_attempts or related changes
git log --oneline -n 20 -- docker-compose.yml

Length of output: 909


🏁 Script executed:

#!/bin/bash
# Look for any comments in docker-compose.yml that mention restart strategy
rg -n "restart|consensus-worker" docker-compose.yml -C5

Length of output: 3914


🏁 Script executed:

#!/bin/bash
# Search for PR or review comments that might reference commit c76e3c0
git log --all --oneline | grep -i "c76e3c0\|restart\|max_attempts"

Length of output: 103


🏁 Script executed:

#!/bin/bash
# Get full context of consensus-worker service (lines 235-298 as mentioned in scratchpad)
sed -n '235,298p' docker-compose.yml

Length of output: 2042



---

🌐 Web query:

```web
Docker Compose deploy restart_policy max_attempts 0 unlimited retries

💡 Result:

Short answer: max_attempts is "unlimited" by default — a value of 0 is treated as no limit (i.e., unlimited retries). See the Compose/Deploy spec and Docker engine docs which state the default is "never give up" / MaxAttempts 0 in API examples. [1][2]

Sources:

  • Compose specification — deploy.restart_policy (default = never give up / unlimited retries). [1]
  • Docker Engine API examples / RestartPolicy (shows MaxAttempts: 0 used). [2]

🌐 Web query:

Docker Swarm max_attempts 0 restart policy unlimited

💡 Result:

Short answer: In Docker Swarm a RestartPolicy MaxAttempts value of 0 is treated as "ignore the limit" (i.e., no attempt cap — effectively unlimited restarts when the policy applies). [1][2]

Sources:

  • Docker Engine API docs (RestartPolicy: "MaxAttempts — ... default value is 0, which is ignored"). [1]
  • Docker Swarm API/Go types (RestartPolicy.MaxAttempts field definition). [2]

🌐 Web query:

Docker consensus worker restart policy best practices

💡 Result:

Concise best-practices for Docker restart policy on consensus/validator workers:

  1. Prefer controlled restarts (on-failure[:max-retries]) over always/unless-stopped to avoid tight infinite restart loops; set a reasonable max retries. [1][2]
  2. Combine restart policy with healthchecks so Docker only restarts truly unhealthy containers. [2]
  3. Ensure the node handles SIGTERM/SIGINT and exits with correct codes (0 on graceful shutdown) so Docker’s policy behaves as expected. [1][2][3]
  4. Use graceful halt mechanisms built into many consensus clients (halt-height or similar) for planned shutdowns so they exit cleanly (exit code 0) and aren’t restarted. [3][4]
  5. If you need always-on behaviour at host reboot, prefer an orchestration layer (systemd, Docker Swarm, Kubernetes) or use unless-stopped only when you’re certain manual stops won’t be needed; with systemd/Kubernetes you get backoff, RestartSec, liveness/readiness probes and better cluster-aware handling. [1][2][4]
  6. Add restart backoff and restart-delay (via systemd RestartSec or orchestrator probe settings) to avoid rapid flapping and to allow dependent services (DB, peers) to recover first. [2][4]
  7. Monitor and alert on restarts (docker events / metrics); maintain a failover validator/node in another location so short restarts don’t cause prolonged downtime or slashing risk. [2][5]

References: Docker restart policy docs and examples; guidance on combining healthchecks and restart limits; validator docs recommending on-failure and graceful halt; systemd/service examples for Restart and RestartSec. [1] Docker docs — Restart policies. [2] Docker best-practice notes (healthchecks, backoff). [3] Avail validator run guide (uses --restart=on-failure). [4] Validator/systemd examples with Restart and RestartSec.


Clarify unlimited retries for consensus worker — contradicts best practices.

The max_attempts: 0 means unlimited retries in Docker Compose, which contradicts best practices for consensus workers. Per Docker documentation and validator guidelines, consensus workers should use controlled restart limits (e.g., max_attempts: 5) with backoff to prevent tight restart loops and allow dependent services to recover.

Your options:

  • Recommended: Limit retries to prevent flapping: max_attempts: 5
  • Alternative: If unlimited retries are intentional for your use case, add a clarifying comment: # 0 = unlimited retries (intentional)

Note: Commit 217eda5 "remove max amount of restart for consensus worker" appears to have introduced this intentionally, but Docker best practices recommend against unlimited retries for consensus/validator services.

🤖 Prompt for AI Agents
In docker-compose.yml around lines 279 to 283, the restart_policy sets
max_attempts: 0 (unlimited retries) which may cause tight restart loops for the
consensus worker; update this to a controlled retry policy such as max_attempts:
5 with an appropriate delay/window to add backoff, or if unlimited retries are
intentional, keep max_attempts: 0 but add a clarifying inline comment like "# 0
= unlimited retries (intentional)" to document the decision.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

♻️ Duplicate comments (2)
docker-compose.yml (2)

277-280: max_attempts: 0 contradicts controlled-restart intent — causes unlimited retry loops.

Line 279 sets max_attempts: 0 (unlimited retries), but line 277's comment states "Only restart on failure (not on clean exit)", implying controlled/limited restart behavior. This contradiction was flagged in previous reviews but remains unresolved.

In Docker Compose/Swarm, max_attempts: 0 means unlimited retries, which can cause tight restart loops. Docker and validator best practices recommend limited retry counts (e.g., 5) with backoff delay for consensus workers.

Recommended fix: Limit retries to match the stated intent:

        condition: on-failure  # Only restart on failure (not on clean exit)
         delay: 5s
-        max_attempts: 0
+        max_attempts: 5  # Limit retries to prevent flapping; backoff via delay
         window: 120s

Alternatively, if unlimited retries are intentional, add a clarifying comment:

+        # 0 = unlimited retries (intentional for consensus resilience)
         max_attempts: 0

152-154: Health endpoints exposed over unencrypted HTTP — security inconsistency confirmed.

The configuration exposes /health and /ready endpoints on the web entrypoint (plain HTTP) while the main API uses websecure with TLS enabled. This leaves health monitoring data unencrypted, contradicting the security posture of the primary API endpoints.

Current state (lines 152–154):

  • Health router uses entrypoints: web (HTTP only)
  • Main API router (line 148) uses entrypoints: websecure with TLS

Traefik best practices recommend either:

  1. Restrict health endpoints to internal-only access, or
  2. Serve them over HTTPS/TLS if external load balancers must access them

Apply the suggested fix to secure health endpoints:

      traefik.http.routers.jsonrpc-health.rule: Host(`${SERVER_NAME}`) && (PathPrefix(`/health`) || PathPrefix(`/ready`))
-     traefik.http.routers.jsonrpc-health.entrypoints: web
+     traefik.http.routers.jsonrpc-health.entrypoints: websecure
+     traefik.http.routers.jsonrpc-health.tls: true
      traefik.http.routers.jsonrpc-health.service: jsonrpc

If plaintext health endpoints are required for load balancer compatibility, document the security trade-off and restrict access via network policies or IP allowlisting.

🧹 Nitpick comments (1)
docker-compose.yml (1)

129-142: Memory configuration defined at both service and deploy levels — potential maintenance burden.

Memory limits and reservations are configured at both service level (lines 129–132: mem_limit, mem_reservation, memswap_limit) and deploy level (lines 135–141: deploy.resources.limits/reservations). While both are valid in Compose, this creates two sources of truth that could diverge.

In Docker Swarm, deploy.resources governs Swarm scheduling; service-level mem_* fields govern individual container behavior. Consider consolidating to one authoritative location or adding a comment explaining their distinct roles.

📜 Review details

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 41d855e and abc6753.

📒 Files selected for processing (1)
  • docker-compose.yml (8 hunks)
🧰 Additional context used
📓 Path-based instructions (1)
docker-compose*.yml

📄 CodeRabbit inference engine (AGENTS.md)

Use docker-compose*.yml files for composing services

Files:

  • docker-compose.yml
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (1)
  • GitHub Check: backend-unit-tests
🔇 Additional comments (3)
docker-compose.yml (3)

3-3: Extended timeouts and profiles are well-justified for GenVM deployments.

The 1200s timeouts (20 minutes) across read/write/idle and server transport are documented to match GenVM contract state loading constraints. Profile-based service enablement is appropriate.

Also applies to: 15-24


288-288: Dual restart policies support multiple deployment workflows — intentional design pattern.

The service-level restart: unless-stopped (line 288) complements the deploy.restart_policy (lines 276–280), enabling compatibility with both docker-compose up (which uses service-level restart) and docker stack deploy (which uses deploy.restart_policy). This is a reasonable approach for universal Compose file compatibility.

Note: Once the max_attempts: 0 issue at line 279 is resolved, this dual configuration will provide consistent behavior across both workflows.


65-69: Explicit router priorities are not required to prevent the routing conflict described.

Based on Traefik v3 behavior, Traefik computes a router's default priority from the rule length (longer rule → higher priority). The jsonrpc router uses a significantly longer rule (Host(...) && PathPrefix(...) with multiple conditions) compared to the frontend router's simple Host(...) rule. Therefore, a router with Host(...) + PathPrefix(...) (longer rule) will take precedence over a router with just Host(...) by default, without requiring explicit priority settings.

While explicitly setting priority for specific-over-generic rules is a best practice for clarity and future maintainability, the routing conflict you described is not actually at risk in the current configuration. The jsonrpc routes will naturally be matched first due to their more specific rule length.

Likely an incorrect or invalid review comment.

Comment thread docker-compose.yml
Comment on lines +90 to +91
ports:
- "${RPCPORT}:${RPCPORT}"

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🔴 Critical

CRITICAL: Port mapping conflicts with multi-replica deployment.

Lines 90–91 define ports: - "${RPCPORT}:${RPCPORT}", but line 134 configures replicas: ${JSONRPC_REPLICAS:-3} (default 3 replicas). In Docker Swarm, multiple container replicas cannot simultaneously bind the same host port; deployment will fail when attempting to scale up.

This was flagged and marked as addressed in previous reviews, but the port mapping remains in the code. Traefik's service routing (lines 143–154) with the load balancer already provides external access and service discovery without explicit port binding.

Fix: Remove the port mapping and rely on Traefik's internal load balancing:

-    ports:
-      - "${RPCPORT}:${RPCPORT}"
     expose:
       - "${RPCPORT}"

Also applies to: 134-134

🤖 Prompt for AI Agents
In docker-compose.yml around lines 90–91 (and note related replicas at line
134), remove the explicit host port mapping `ports: - "${RPCPORT}:${RPCPORT}"`
so replicas can scale; instead rely on Traefik labels and Docker internal
service discovery/load-balancing (or use `expose: - "${RPCPORT}"` if internal
container-only exposure is needed), and ensure no other services bind the same
host port so the Swarm deployment with `replicas: ${JSONRPC_REPLICAS:-3}` can
start successfully.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

📜 Review details

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between abc6753 and 90e24cb.

📒 Files selected for processing (1)
  • backend/protocol_rpc/rpc_endpoint_manager.py (1 hunks)
🧰 Additional context used
📓 Path-based instructions (3)
{backend,examples,tests}/**/*.py

📄 CodeRabbit inference engine (CLAUDE.md)

Python code must be formatted with Black and include type hints

Files:

  • backend/protocol_rpc/rpc_endpoint_manager.py
backend/protocol_rpc/**/*.py

📄 CodeRabbit inference engine (CLAUDE.md)

Implement JSON-RPC endpoint handlers under backend/protocol_rpc/

Files:

  • backend/protocol_rpc/rpc_endpoint_manager.py
**/*.py

📄 CodeRabbit inference engine (AGENTS.md)

Target Python 3.12, use 4-space indentation, and rely on Black (via pre-commit) for formatting

Files:

  • backend/protocol_rpc/rpc_endpoint_manager.py
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (1)
  • GitHub Check: backend-unit-tests

scope=EventScope.RPC,
message=f"RPC method completed: {request.method}",
data={"method": request.method},
data={"method": request.method, "params": request.params},

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🔴 Critical

🧩 Analysis chain

Risk: Logging potentially sensitive data without sanitization.

The success log now includes request.params directly, which may contain PII (emails, user identifiers), credentials, API keys, or other sensitive data depending on the RPC methods exposed. Logging such data without sanitization creates compliance risks (GDPR, CCPA).

Additionally, the main error handlers (lines 149-154, 168) don't include params in their logs, creating an inconsistency where success is more verbose than failure.

Consider one of these approaches:

  1. Sanitize params before logging (recommended if params may contain sensitive data):
+            # Sanitize params to remove sensitive fields
+            sanitized_params = self._sanitize_params(request.params)
             if should_log and definition.log_policy.log_success:
                 self._logger.send_message(
                     LogEvent(
                         name="endpoint_success",
                         type=EventType.SUCCESS,
                         scope=EventScope.RPC,
                         message=f"RPC method completed: {request.method}",
-                        data={"method": request.method, "params": request.params},
+                        data={"method": request.method, "params": sanitized_params},
                     )
                 )

Then add a sanitization helper:

def _sanitize_params(self, params: Any) -> Any:
    """Remove sensitive fields from params before logging."""
    if params is None:
        return None
    if isinstance(params, dict):
        # Remove known sensitive keys
        sensitive_keys = {"password", "token", "secret", "apiKey", "api_key"}
        return {k: "***" if k in sensitive_keys else v for k, v in params.items()}
    return params  # For list/primitive params, log as-is or redact entirely
  1. Add params to error logs for consistency:
                         data={
                             "method": request.method,
+                            "params": request.params,
                             "code": exc.code,
                             "error_data": getattr(exc, "data", None),
                             "traceback": stack_trace,
                         },

Run this script to identify which RPC methods might pass sensitive data:

Also verify if any existing documentation or data classification policy exists:


🏁 Script executed:

#!/bin/bash
# Find RPC method registrations and check for potentially sensitive param names
rg -n -A 10 'def (.*)\(' backend/protocol_rpc/ | rg -i 'password|token|secret|api_key|credential|auth'

Length of output: 244


🏁 Script executed:

#!/bin/bash
# Search for data handling or privacy documentation
fd -e md -e txt . | xargs rg -l -i 'pii|sensitive.data|privacy|gdpr'

Length of output: 0


🏁 Script executed:

#!/bin/bash
# View the actual error logging at lines 149-154 and 168 to verify they don't log params
sed -n '145,175p' backend/protocol_rpc/rpc_endpoint_manager.py

Length of output: 1478


🏁 Script executed:

#!/bin/bash
# Check line 121 to confirm request log structure includes params
sed -n '118,125p' backend/protocol_rpc/rpc_endpoint_manager.py

Length of output: 275


🏁 Script executed:

#!/bin/bash
# Search for RPC method handler registrations to see what methods exist and param patterns
rg -n 'def (register|handle|process).*' backend/protocol_rpc/ -A 5 | head -50

Length of output: 3873


🏁 Script executed:

#!/bin/bash
# Look for any existing sanitization, filtering, or redaction utilities
rg -n 'sanitize|redact|filter.*param|mask' backend/ -i

Length of output: 1509


🏁 Script executed:

#!/bin/bash
# Check what types of RPC endpoints/methods are registered by searching the codebase
find backend/protocol_rpc -name '*.py' -exec grep -l 'endpoint\|method' {} \; | head -10

Length of output: 798


🏁 Script executed:

#!/bin/bash
# Look at imports and overall structure of rpc_endpoint_manager.py to understand context
head -50 backend/protocol_rpc/rpc_endpoint_manager.py

Length of output: 1303


🏁 Script executed:

#!/bin/bash
# Search for actual RPC method definitions in backend - what endpoints are exposed?
rg -n '@.*\(.*method|register.*handler' backend/protocol_rpc/ -B 2 -A 3

Length of output: 928


🏁 Script executed:

#!/bin/bash
# Check if there's any data classification or handling policy file
find backend -name '*.py' -o -name '*.md' | xargs grep -l 'data.classif|handling.policy|sensitive|pii' -i 2>/dev/null | head -5

Length of output: 0


🏁 Script executed:

#!/bin/bash
# View the full context around line 136 to understand what request object is and its properties
sed -n '130,145p' backend/protocol_rpc/rpc_endpoint_manager.py

Length of output: 715


🏁 Script executed:

#!/bin/bash
# Search for any RPC method implementations that accept user/auth/sensitive data
rg -n 'def.*\(.*request.*\)' backend/protocol_rpc/ -A 2 | rg -i 'auth|user|password|credential|token|secret'

Length of output: 0


🏁 Script executed:

#!/bin/bash
# Get type information for request.params
rg -n 'class.*Request|params.*:.*Any' backend/protocol_rpc/ -B 2 -A 2

Length of output: 2570


🏁 Script executed:

#!/bin/bash
# Check logging module/format to understand what fields are logged by default
rg -n 'LogEvent\|log.*success\|log.*error' backend/protocol_rpc/rpc_endpoint_manager.py -B 1 -A 1 | head -30

Length of output: 0


🏁 Script executed:

#!/bin/bash
# Verify actual error handler code doesn't include params
sed -n '140,180p' backend/protocol_rpc/rpc_endpoint_manager.py

Length of output: 1856


🏁 Script executed:

#!/bin/bash
# Search for any validators, filters applied to params before use/logging
rg -n 'request\.params' backend/protocol_rpc/rpc_endpoint_manager.py -B 2 -A 2

Length of output: 1142


🏁 Script executed:

#!/bin/bash
# Check if there's a data schema/validation for RPC methods
find backend -name '*schema*' -o -name '*validator*' | head -10

Length of output: 514


🏁 Script executed:

#!/bin/bash
# Look for test files that might show what kind of data is passed to RPC
find backend -path '*test*' -name '*.py' | xargs grep -l 'rpc\|endpoint' | head -5

Length of output: 67


🏁 Script executed:

#!/bin/bash
# Check the complete context of logging in success path from line 121-145
sed -n '121,145p' backend/protocol_rpc/rpc_endpoint_manager.py

Length of output: 1075


🏁 Script executed:

#!/bin/bash
# Verify if error logging at lines 149-154 includes params - get exact code
sed -n '149,155p' backend/protocol_rpc/rpc_endpoint_manager.py

Length of output: 303


🏁 Script executed:

#!/bin/bash
# Check line 168 error logging
sed -n '165,172p' backend/protocol_rpc/rpc_endpoint_manager.py

Length of output: 385


🏁 Script executed:

#!/bin/bash
# Check line 193 mentioned as exception that DOES include params
sed -n '190,200p' backend/protocol_rpc/rpc_endpoint_manager.py

Length of output: 447


🏁 Script executed:

#!/bin/bash
# Search broader to understand RPC protocol and what methods might be exposed
find backend -name '*.py' | xargs grep -l 'JsonRPC\|json.rpc\|rpc.*method' -i | head -10

Length of output: 434


🏁 Script executed:

#!/bin/bash
# Look at the full file to understand data flow and where params originate
wc -l backend/protocol_rpc/rpc_endpoint_manager.py

Length of output: 48


🏁 Script executed:

#!/bin/bash
# Check if there's a README or documentation about the RPC protocol/endpoints
find . -maxdepth 3 -name 'README*' -o -name '*rpc*' -type f | grep -E '\.(md|txt|rst)$'

Length of output: 224


Sanitize RPC params before logging—privacy and compliance risk confirmed.

Verification confirms that request.params is logged without sanitization in multiple places (request log at line 121, success log at line 136, validation errors at line 242, and InvalidParams exceptions at line 193). Since params is typed as Any | None, it may contain user-supplied data including PII, credentials, or API keys. This creates a compliance risk (GDPR, CCPA).

The codebase already has precedent for sanitization: _redact_consensus_data_for_log() in backend/consensus/base.py. Apply similar sanitization to RPC params before logging them in these locations:

  • backend/protocol_rpc/rpc_endpoint_manager.py, line 121 (request log)
  • backend/protocol_rpc/rpc_endpoint_manager.py, line 136 (success log)
  • backend/protocol_rpc/rpc_endpoint_manager.py, line 193 (InvalidParams error)
  • backend/protocol_rpc/rpc_endpoint_manager.py, line 242 (validation error)

Consider implementing a _sanitize_rpc_params() method that masks or redacts sensitive keys (e.g., "password", "token", "secret", "apiKey").

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

♻️ Duplicate comments (1)
docker-compose.yml (1)

91-92: 🔴 CRITICAL: Port mapping prevents multi-replica deployment.

In Docker Swarm, multiple container replicas cannot simultaneously bind the same host port—this configuration will fail when attempting to scale. The current docker-compose.yml has this exact issue:

  • Line 91–92: ports: - "${RPCPORT}:${RPCPORT}" defines explicit host port binding
  • Line ~143: deploy: replicas: ${JSONRPC_REPLICAS:-3} defaults to 3 replicas

The default Docker Swarm ingress routing mesh makes the published port available on every node and load-balances traffic to any replica, so you can run N replicas without host port conflicts. However, the explicit ports: configuration overrides this and creates a binding conflict.

Since Traefik labels provide service discovery and load-balancing, and expose: is already present, remove the problematic host port binding:

-    ports:
-      - "${RPCPORT}:${RPCPORT}"
     expose:
       - "${RPCPORT}"

The expose directive combined with Traefik's ingress routing (labels at lines ~147–155) is the correct pattern for multi-replica services.

📜 Review details

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 90e24cb and d8be379.

📒 Files selected for processing (1)
  • docker-compose.yml (8 hunks)
🧰 Additional context used
📓 Path-based instructions (1)
docker-compose*.yml

📄 CodeRabbit inference engine (AGENTS.md)

Use docker-compose*.yml files for composing services

Files:

  • docker-compose.yml
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (1)
  • GitHub Check: backend-unit-tests
🔇 Additional comments (4)
docker-compose.yml (4)

3-3: ✅ Traefik configuration improvements look solid.

The profile restriction, extended timeouts (20 min to match GenVM), and SSL path mount are all well-motivated for contract state operations and production-ready TLS setup.

Also applies to: 15-24, 31-31


101-103: ✅ Good additions: Traefik soft dependency, memory management, and restart strategy.

  • Traefik soft dependency (required: false, line 102): Allows graceful startup if Traefik isn't ready.
  • Memory configuration (lines 130–133, 271–274): Proper tiered limits (hard, soft, swap) with OOM handling prevent runaway memory consumption.
  • Restart strategy (line 289): unless-stopped is appropriate for Swarm mode—respects manual docker stop commands (unlike always).

These are production-ready improvements.

Also applies to: 130-133, 271-274, 289-289


66-70: Frontend and jsonrpc routers route correctly without explicit priority fields.

Traefik defaults to using rule length as priority—longer rules automatically get higher priority unless explicitly overridden. The frontend rule Host(\${SERVER_NAME}`)is short, while the jsonrpc ruleHost(`${SERVER_NAME}`) && (PathPrefix(...))` is longer. This means Traefik automatically prioritizes path-specific routes (jsonrpc) over the catch-all frontend route, ensuring correct routing without explicit priority configuration.

The original review comment incorrectly assumes missing priorities create ambiguity. The implicit rule-length-based priority mechanism already solves this problem.


277-281: Rewrite is incorrect; the configuration is not contradictory.

The comment at line 278 ("Only restart on failure") correctly describes the condition: on-failure setting, which ensures restarts only occur on failures, not clean exits. The max_attempts: 0 setting (unlimited retries) is orthogonal and does not contradict this—it simply means that when failures occur, retries are unlimited.

Per Docker Compose specification: max_attempts: 0 is valid syntax meaning unlimited retries (the default behavior when omitted). The configuration is internally consistent.

If the concern is whether unlimited retries are appropriate for this consensus worker service, that is a design decision (and may warrant reconsideration for stability), but it is not a logical contradiction in the configuration as the original review claims.

Likely an incorrect or invalid review comment.

Comment thread docker-compose.yml
Comment on lines +152 to +155
# HTTP router for health endpoints (no TLS required)
traefik.http.routers.jsonrpc-health.rule: Host(`${SERVER_NAME}`) && (PathPrefix(`/health`) || PathPrefix(`/ready`))
traefik.http.routers.jsonrpc-health.entrypoints: web
traefik.http.routers.jsonrpc-health.service: jsonrpc

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🔴 Critical

❓ Verification inconclusive

⚠️ Major: Health endpoints use unencrypted HTTP while main API uses HTTPS.

The main jsonrpc router (line 148–150) uses websecure entrypoint with TLS enabled, but the health router (lines 153–155) uses the plain web entrypoint with no TLS. This exposes health monitoring data (status, availability, readiness) in cleartext, creating a security inconsistency.

Recommended: Secure health endpoints over HTTPS to match the main API:

traefik.http.routers.jsonrpc-health.entrypoints: websecure
traefik.http.routers.jsonrpc-health.tls: true

Alternatively, document and justify why unencrypted health access is acceptable for your security posture.


I'll verify this security concern by examining the docker-compose configuration and checking industry standards for health endpoint security patterns.

Now let me search for health endpoint security best practices more specifically:

Now let me examine the actual docker-compose.yml file to verify the configuration described in the review:

Based on my analysis of industry best practices, security standards, and Kubernetes documentation:

The review comment is valid and correctly identifies a security issue.

The review comment appropriately flags a significant security inconsistency: API security best practices recommend enforcing HTTPS for all APIs, even if the content seems trivial. Every web API should use TLS, which protects information by encrypting messages in transit.

The concern is substantiated by multiple authoritative sources that recommend secure REST APIs should only expose HTTPS endpoints, which ensure all API communication is encrypted using SSL/TLS. Importantly, health check probes can be configured to use HTTPS protocol with the container port set to HTTPS port 443, making the current HTTP-only configuration a choice rather than a technical necessity.

The health endpoint security issue remains unresolved and warrants immediate attention to align the health router configuration with the main API's security posture.

@sonarqubecloud

Copy link
Copy Markdown

Quality Gate Failed Quality Gate failed

Failed conditions
6.6% Coverage on New Code (required ≥ 80%)
C Reliability Rating on New Code (required ≥ A)

See analysis details on SonarQube Cloud

Catch issues before they fail your Quality Gate with our IDE extension SonarQube for IDE

@cristiam86
cristiam86 merged commit 520e446 into main Oct 29, 2025
1 check passed
@github-actions

github-actions Bot commented Nov 3, 2025

Copy link
Copy Markdown
Contributor

🎉 This PR is included in version 0.79.3 🎉

The release is available on GitHub release

Your semantic-release bot 📦🚀

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants