Skip to content

chore: horizontal scalability - #1338

Merged
cristiam86 merged 62 commits into
mainfrom
hscale
Oct 17, 2025
Merged

cristiam86 merged 62 commits into
mainfrom
hscale

Conversation

@cristiam86

@cristiam86 cristiam86 commented Oct 8, 2025 •

Copy link
Copy Markdown
Contributor

Summary by CodeRabbit

  • New Features

    • Distributed consensus workers with a runnable worker service and Redis-backed event broadcasting
    • FastAPI/ASGI RPC handler and endpoint registry for JSON-RPC requests
  • Infrastructure & Deployment

    • Expanded production config and compose updates for multi-replica deployments, resource limits, and new container images
    • Redis required for worker-to-RPC broadcasting; GenVM path/configuration added
  • Improvements

    • Orphan transaction recovery, concurrency handling, and enhanced health/logging
  • Tests & Docs

    • New integration BDD tests and comprehensive backend architecture documentation

jmlago and others added 30 commits September 8, 2025 19:45

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 3

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)
backend/consensus/base.py (1)

2005-2031: Remove unused parameter stop_pending_queue.

The parameter stop_pending_queue (line 2006) is not used in the method body. Based on static analysis and review of the implementation, this parameter appears to be legacy and should be removed.

Apply this diff to remove the unused parameter:

-    async def rollback_transactions(
-        self, context: TransactionContext, stop_pending_queue
-    ):
+    async def rollback_transactions(self, context: TransactionContext):
         """
         Rollback newer transactions.
         In the simplified system, we just need to reset future transactions to PENDING.
         """

Then update all call sites to remove the second argument:

  • Line 1188: await self.rollback_transactions(context, False) → await self.rollback_transactions(context)
  • Line 1749: await self.rollback_transactions(context, True) → await self.rollback_transactions(context)
  • Line 1852: await self.rollback_transactions(context, True) → await self.rollback_transactions(context)
  • Line 1973: await self.rollback_transactions(context, True) → await self.rollback_transactions(context)
♻️ Duplicate comments (6)
backend/node/base.py (1)

38-74: Address previously flagged issues.

The concerns raised in the previous review about type safety (line 67), validation of configured log level, and type hints remain valid and should be addressed.

backend/protocol_rpc/message_handler/redis_worker_handler.py (1)

57-71: Remove await when creating the Redis client

redis.asyncio.from_url() returns a ready Redis instance; awaiting it triggers TypeError and the worker never initializes. Instantiate without await.

-                self.redis_client = await aioredis.from_url(
+                self.redis_client = aioredis.from_url(
                     self.redis_url, encoding="utf-8", decode_responses=True
                 )
backend/consensus/worker.py (4)

85-122: Verify INTERVAL parameter binding syntax.

The SQL query uses INTERVAL :timeout syntax (lines 97, 103) for PostgreSQL interval arithmetic. Based on past review comments, this was flagged as requiring explicit casting like (:timeout)::interval to avoid SQL errors. Verify that the current syntax works correctly with SQLAlchemy's parameter binding.

If the INTERVAL binding causes issues, apply this pattern (as suggested in past reviews):

-                         OR t.blocked_at < NOW() - INTERVAL :timeout)
+                         OR t.blocked_at < NOW() - (:timeout)::interval)

And similarly for line 103.


169-207: Same INTERVAL binding issue as finalization query.

The SQL query uses INTERVAL :timeout syntax (lines 180, 187) which may require explicit casting as (:timeout)::interval based on past review feedback.


259-298: Same INTERVAL binding issue as other claim methods.

The SQL query uses INTERVAL :timeout syntax (lines 266, 272) which may require explicit casting based on past review feedback.


362-381: INTERVAL binding issue in recovery query.

The SQL query uses INTERVAL :timeout and INTERVAL :orphan_timeout syntax (lines 371, 377) which may require explicit casting as per past review feedback.

🧹 Nitpick comments (1)
backend/protocol_rpc/message_handler/redis_worker_handler.py (1)

105-112: Redundant transaction_hash in message payload

The transaction_hash is included both at the root level (line 110) and within the data dictionary via log_event.to_dict() (line 109), which already serializes transaction_hash. This creates unnecessary duplication in the message payload.

If the root-level transaction_hash is intentional for routing or convenience, consider adding a comment to clarify. Otherwise, apply this diff to remove the duplication:

             message = json.dumps(
                 {
                     "worker_id": self.worker_id,
                     "event": log_event.name,
                     "data": log_event.to_dict(),
-                    "transaction_hash": log_event.transaction_hash,
                 }
             )
📜 Review details

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 8e53027 and 90352fb.

📒 Files selected for processing (5)
  • backend/consensus/base.py (20 hunks)
  • backend/consensus/worker.py (1 hunks)
  • backend/node/base.py (6 hunks)
  • backend/protocol_rpc/message_handler/redis_worker_handler.py (1 hunks)
  • tests/integration/rpc/test_fastapi_router.py (1 hunks)
🧰 Additional context used
📓 Path-based instructions (6)
{backend,examples,tests}/**/*.py

📄 CodeRabbit inference engine (CLAUDE.md)

Python code must be formatted with Black and include type hints

Files:

  • tests/integration/rpc/test_fastapi_router.py
  • backend/consensus/worker.py
  • backend/node/base.py
  • backend/consensus/base.py
  • backend/protocol_rpc/message_handler/redis_worker_handler.py
tests/**/*.py

📄 CodeRabbit inference engine (CLAUDE.md)

Backend tests should be written and run with pytest (unit, integration, e2e)

Files:

  • tests/integration/rpc/test_fastapi_router.py
tests/{unit,integration,e2e,load}/**

📄 CodeRabbit inference engine (AGENTS.md)

Organize Pytest suites by scope under tests/{unit,integration,e2e,load}

Files:

  • tests/integration/rpc/test_fastapi_router.py
**/*.py

📄 CodeRabbit inference engine (AGENTS.md)

Target Python 3.12, use 4-space indentation, and rely on Black (via pre-commit) for formatting

Files:

  • tests/integration/rpc/test_fastapi_router.py
  • backend/consensus/worker.py
  • backend/node/base.py
  • backend/consensus/base.py
  • backend/protocol_rpc/message_handler/redis_worker_handler.py
backend/consensus/**/*.py

📄 CodeRabbit inference engine (CLAUDE.md)

Consensus logic must reside under backend/consensus/

Files:

  • backend/consensus/worker.py
  • backend/consensus/base.py
backend/protocol_rpc/**/*.py

📄 CodeRabbit inference engine (CLAUDE.md)

Implement JSON-RPC endpoint handlers under backend/protocol_rpc/

Files:

  • backend/protocol_rpc/message_handler/redis_worker_handler.py
🧬 Code graph analysis (3)
backend/consensus/worker.py (7)
backend/database_handler/models.py (2)
  • Transactions (68-145)
  • TransactionStatus (31-42)
backend/database_handler/transactions_processor.py (1)
  • TransactionsProcessor (67-1317)
backend/database_handler/chain_snapshot.py (1)
  • ChainSnapshot (15-70)
backend/domain/types.py (3)
  • Transaction (154-273)
  • LLMProvider (81-98)
  • Validator (102-144)
backend/consensus/base.py (6)
  • ConsensusAlgorithm (350-2255)
  • contract_snapshot_factory (188-220)
  • can_finalize_transaction (1570-1618)
  • process_leader_appeal (1659-1751)
  • process_leader_timeout_appeal (1753-1854)
  • process_validator_appeal (1856-2003)
backend/consensus/worker_service.py (1)
  • get_session (62-63)
backend/validators/__init__.py (3)
  • Manager (136-324)
  • temporal_snapshot (273-290)
  • snapshot (259-270)
backend/consensus/base.py (4)
backend/protocol_rpc/message_handler/base.py (2)
  • MessageHandler (29-154)
  • send_message (151-154)
backend/protocol_rpc/message_handler/types.py (3)
  • LogEvent (20-38)
  • EventType (5-10)
  • EventScope (13-16)
backend/protocol_rpc/message_handler/redis_worker_handler.py (2)
  • send_message_async (167-186)
  • send_message (149-165)
tests/unit/consensus/test_helpers.py (1)
  • send_message (725-726)
backend/protocol_rpc/message_handler/redis_worker_handler.py (5)
backend/protocol_rpc/message_handler/types.py (1)
  • LogEvent (20-38)
backend/protocol_rpc/configuration.py (1)
  • GlobalConfiguration (5-8)
backend/protocol_rpc/message_handler/base.py (1)
  • _log_message (63-92)
backend/consensus/worker_service.py (1)
  • health_check (147-157)
backend/protocol_rpc/redis_subscriber.py (1)
  • health_check (242-268)
🪛 Ruff (0.14.0)
backend/consensus/worker.py

43-43: PEP 484 prohibits implicit Optional

Convert to T | None

(RUF013)


502-502: Redundant exception object included in logging.exception call

(TRY401)


567-567: Redundant exception object included in logging.exception call

(TRY401)


656-656: Redundant exception object included in logging.exception call

(TRY401)


728-728: Redundant exception object included in logging.exception call

(TRY401)

backend/consensus/base.py

86-86: Do not catch blind exception: Exception

(BLE001)


1225-1225: Use explicit conversion flag

Replace with conversion flag

(RUF010)


1225-1225: Use explicit conversion flag

Replace with conversion flag

(RUF010)


2006-2006: Unused method argument: stop_pending_queue

(ARG002)

backend/protocol_rpc/message_handler/redis_worker_handler.py

138-138: Store a reference to the return value of asyncio.create_task

(RUF006)


217-217: Do not catch blind exception: Exception

(BLE001)

⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (1)
  • GitHub Check: backend-unit-tests
🔇 Additional comments (18)
tests/integration/rpc/test_fastapi_router.py (1)

94-94: LGTM!

The test expectation correctly reflects the protocol change where JSON-RPC error responses now include an always-present "data": {} field.

backend/node/base.py (4)

10-10: LGTM!

The logging import is appropriately added to support log level constants used in the new filtering function.


260-275: LGTM!

The updated signature and implementation correctly handle both string and datetime.datetime inputs while ensuring timezone awareness. The logic appropriately adds UTC timezone info when missing and maintains backwards compatibility.


355-373: LGTM!

The log filtering is correctly applied before emitting the LogEvent, ensuring only relevant log entries are included based on the configured log level.


379-386: LGTM!

The log filtering is correctly applied to the error data in get_contract_schema, maintaining consistency with the filtering applied in _execution_finished.

backend/protocol_rpc/message_handler/redis_worker_handler.py (7)

1-28: LGTM!

The imports, class definition, and channel constants are well-organized and appropriate for the Redis pub/sub implementation.


29-55: LGTM!

The initialization logic correctly sets up the worker ID, Redis URL from environment configuration, and properly initializes the parent handler with None for socketio since this implementation uses Redis pub/sub instead.


73-88: LGTM!

The channel routing logic correctly maps event scopes to their respective Redis channels.


149-165: LGTM!

The message sending logic correctly filters important events and delegates to the logging and publishing mechanisms. Note that improvements to _socket_emit() (flagged separately) will enhance this method's reliability.


167-186: LGTM! Prefer this pattern for async contexts

This async method provides a cleaner, more reliable pattern than send_message() by directly awaiting the Redis publish operation, ensuring completion before continuing.


188-193: LGTM!

The cleanup logic properly closes the Redis client and resets state, following best practices for resource management in async contexts.


195-220: LGTM!

The health check implementation follows a robust pattern, providing comprehensive diagnostic information while gracefully handling connection failures. The broad exception catch on line 217 is appropriate for health checks to ensure the endpoint remains available even when Redis is unreachable.

backend/consensus/base.py (5)

77-139: LGTM! Comprehensive redaction logic.

The new _redact_receipt_data helper and the updated _redact_consensus_data_for_log provide robust redaction of sensitive data including private keys, model configurations, and contract state. The conditional stdout removal (lines 137-138) when stderr is absent is a sensible approach for preserving diagnostic information.


1241-1300: LGTM! Async conversion correctly applied.

The execute_transfer method is properly converted to async, with all calls to dispatch_transaction_status_update correctly awaited (lines 1270, 1295).


2323-2323: LGTM! Consistent async/await pattern.

All state handlers correctly await the async dispatch_transaction_status_update and execute_transfer calls. The async conversion is consistently applied throughout the state machine.

Also applies to: 2482-2482, 2605-2605, 2682-2682, 2980-2980, 1994-1994, 3194-3194, 3263-3263, 3355-3355, 3387-3387


84-88: Catching Exception is acceptable for defensive logging.

The broad exception catch (line 86) in _redact_consensus_data_for_log is appropriate here since the function is used for logging and should never break the main execution flow, even if deepcopy fails unexpectedly.


1195-1238: Fallback to synchronous send_message is required. Only RedisWorkerMessageHandler provides send_message_async; the base MessageHandler, FastAPI handler, and test mocks only implement send_message, so the hasattr fallback is necessary.

backend/consensus/worker.py (1)

663-734: LGTM! Well-structured worker loop.

The main run loop implements a sensible priority system (appeals > finalizations > regular transactions), periodic recovery of stuck transactions, and proper session management with separate sessions for each processing operation. The graceful shutdown via stop() is straightforward and effective.

Comment on lines +37 to +74
def __init__(
self,
get_session: Callable[[], Session],
msg_handler: MessageHandler,
consensus_service: ConsensusService,
validators_manager: validators.Manager,
worker_id: str = None,
poll_interval: int = 5,
transaction_timeout_minutes: int = 30,
):
"""
Initialize the consensus worker.

Args:
get_session: Function to get a database session
msg_handler: Message handler for events
consensus_service: Consensus service for rollup interaction
validators_manager: Validators manager
worker_id: Unique identifier for this worker (auto-generated if not provided)
poll_interval: Seconds to wait between polls when no work available
transaction_timeout_minutes: Minutes before a stuck transaction is recovered
"""
self.worker_id = worker_id or f"worker-{uuid.uuid4().hex[:8]}"
self.get_session = get_session
self.msg_handler = msg_handler
self.consensus_service = consensus_service
self.validators_manager = validators_manager
self.poll_interval = poll_interval
self.transaction_timeout_minutes = transaction_timeout_minutes
self.running = True

# Create a ConsensusAlgorithm instance to reuse its exec_transaction method
self.consensus_algorithm = ConsensusAlgorithm(
get_session,
msg_handler,
consensus_service,
validators_manager,
)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor

Fix implicit Optional type hint.

The worker_id parameter (line 43) uses an implicit Optional by having a default value of None. PEP 484 requires explicit Optional or T | None syntax.

Apply this diff to make the type hint explicit:

     def __init__(
         self,
         get_session: Callable[[], Session],
         msg_handler: MessageHandler,
         consensus_service: ConsensusService,
         validators_manager: validators.Manager,
-        worker_id: str = None,
+        worker_id: str | None = None,
         poll_interval: int = 5,
         transaction_timeout_minutes: int = 30,
     ):
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
def __init__(
self,
get_session: Callable[[], Session],
msg_handler: MessageHandler,
consensus_service: ConsensusService,
validators_manager: validators.Manager,
worker_id: str = None,
poll_interval: int = 5,
transaction_timeout_minutes: int = 30,
):
"""
Initialize the consensus worker.
Args:
get_session: Function to get a database session
msg_handler: Message handler for events
consensus_service: Consensus service for rollup interaction
validators_manager: Validators manager
worker_id: Unique identifier for this worker (auto-generated if not provided)
poll_interval: Seconds to wait between polls when no work available
transaction_timeout_minutes: Minutes before a stuck transaction is recovered
"""
self.worker_id = worker_id or f"worker-{uuid.uuid4().hex[:8]}"
self.get_session = get_session
self.msg_handler = msg_handler
self.consensus_service = consensus_service
self.validators_manager = validators_manager
self.poll_interval = poll_interval
self.transaction_timeout_minutes = transaction_timeout_minutes
self.running = True
# Create a ConsensusAlgorithm instance to reuse its exec_transaction method
self.consensus_algorithm = ConsensusAlgorithm(
get_session,
msg_handler,
consensus_service,
validators_manager,
)
def __init__(
self,
get_session: Callable[[], Session],
msg_handler: MessageHandler,
consensus_service: ConsensusService,
validators_manager: validators.Manager,
worker_id: str | None = None,
poll_interval: int = 5,
transaction_timeout_minutes: int = 30,
):
"""
Initialize the consensus worker.
Args:
get_session: Function to get a database session
msg_handler: Message handler for events
consensus_service: Consensus service for rollup interaction
validators_manager: Validators manager
worker_id: Unique identifier for this worker (auto-generated if not provided)
poll_interval: Seconds to wait between polls when no work available
transaction_timeout_minutes: Minutes before a stuck transaction is recovered
"""
self.worker_id = worker_id or f"worker-{uuid.uuid4().hex[:8]}"
self.get_session = get_session
self.msg_handler = msg_handler
self.consensus_service = consensus_service
self.validators_manager = validators_manager
self.poll_interval = poll_interval
self.transaction_timeout_minutes = transaction_timeout_minutes
self.running = True
# Create a ConsensusAlgorithm instance to reuse its exec_transaction method
self.consensus_algorithm = ConsensusAlgorithm(
get_session,
msg_handler,
consensus_service,
validators_manager,
)
🧰 Tools
🪛 Ruff (0.14.0)

43-43: PEP 484 prohibits implicit Optional

Convert to T | None

(RUF013)

🤖 Prompt for AI Agents
In backend/consensus/worker.py around lines 37 to 74, make the implicit Optional
on the worker_id parameter explicit by changing its annotation to Optional[str]
(or str | None for Python 3.10+) and ensure the proper import is present (from
typing import Optional) at the top of the file; update the __init__ signature to
use the explicit nullable type and run a quick static type check to confirm no
other references need adjustment.

Comment on lines +401 to +507
async def process_transaction(self, transaction_data: dict, session: Session):
"""
Process a single transaction through consensus.
Reuses the exec_transaction logic from ConsensusAlgorithm.

Args:
transaction_data: Transaction data dictionary
session: Database session
"""
try:
# Convert to Transaction domain object
transaction = Transaction.from_dict(transaction_data)

# Import factories here to avoid circular imports
from backend.consensus.base import (
contract_snapshot_factory,
contract_processor_factory,
chain_snapshot_factory,
transactions_processor_factory,
accounts_manager_factory,
node_factory,
)

# Get or create validators snapshot based on sim_config
virtual_validators = []
if transaction.sim_config and transaction.sim_config.validators:
# Handle virtual validators for simulation
for validator in transaction.sim_config.validators:
from backend.node.create_nodes.providers import (
get_default_provider_for,
validate_provider,
)
from backend.domain.types import LLMProvider, Validator

provider = validator.provider
model = validator.model
config = validator.config
plugin = validator.plugin
plugin_config = validator.plugin_config

if config is None or plugin is None or plugin_config is None:
llm_provider = get_default_provider_for(provider, model)
else:
llm_provider = LLMProvider(
provider=provider,
model=model,
config=config,
plugin=plugin,
plugin_config=plugin_config,
)
validate_provider(llm_provider)

account = accounts_manager_factory(session).create_new_account()
virtual_validators.append(
Validator(
address=account.address,
private_key=account.key.to_0x_hex(),
stake=validator.stake,
llmprovider=llm_provider,
)
)

# Use appropriate validators snapshot
if virtual_validators:
async with self.validators_manager.temporal_snapshot(
virtual_validators
) as validators_snapshot:
await self.consensus_algorithm.exec_transaction(
transaction,
transactions_processor_factory(session),
chain_snapshot_factory(session),
accounts_manager_factory(session),
lambda contract_address: contract_snapshot_factory(
contract_address, session, transaction
),
contract_processor_factory(session),
node_factory,
validators_snapshot,
)
else:
async with self.validators_manager.snapshot() as validators_snapshot:
await self.consensus_algorithm.exec_transaction(
transaction,
transactions_processor_factory(session),
chain_snapshot_factory(session),
accounts_manager_factory(session),
lambda contract_address: contract_snapshot_factory(
contract_address, session, transaction
),
contract_processor_factory(session),
node_factory,
validators_snapshot,
)

session.commit()
logger.info(
f"[Worker {self.worker_id}] Successfully processed transaction {transaction.hash}"
)

except Exception as e:
logger.exception(
f"[Worker {self.worker_id}] Error processing transaction {transaction_data['hash']}: {e}"
)
session.rollback()
finally:
# Always release the transaction when done
self.release_transaction(session, transaction_data["hash"])

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor

Remove redundant exception object in logger.exception calls.

The logger.exception() calls include the exception object explicitly (lines 502, 567, 656), which is redundant since logger.exception() automatically captures and logs the current exception information.

Apply this pattern to fix the redundant exception logging:

         except Exception as e:
-            logger.exception(
-                f"[Worker {self.worker_id}] Error processing transaction {transaction_data['hash']}: {e}"
-            )
+            logger.exception(
+                f"[Worker {self.worker_id}] Error processing transaction {transaction_data['hash']}"
+            )

Apply the same fix to lines 567, 656, and 728.

Also applies to: 509-572, 574-661

🧰 Tools
🪛 Ruff (0.14.0)

502-502: Redundant exception object included in logging.exception call

(TRY401)

🤖 Prompt for AI Agents
In backend/consensus/worker.py around lines 401-507 (and similarly apply to
509-572, 574-661, and 728), remove the redundant exception object passed to
logger.exception; replace calls like logger.exception(f"[Worker
{self.worker_id}] Error processing transaction {transaction_data['hash']}: {e}")
with logger.exception(f"[Worker {self.worker_id}] Error processing transaction
{transaction_data['hash']}:") so the logger captures the exception info
automatically, ensure session.rollback() and self.release_transaction(...)
behavior remains unchanged.

Comment on lines +126 to +147
def _socket_emit(self, log_event: LogEvent):
"""
Override socket emit to publish events to Redis.

Args:
log_event: The event to emit
"""
# Try to get the current event loop
try:
loop = asyncio.get_event_loop()
if loop.is_running():
# Schedule the async send operation
asyncio.create_task(self._publish_to_redis(log_event))
else:
# If no loop is running, run it synchronously
asyncio.run(self._publish_to_redis(log_event))
except RuntimeError:
# No event loop available
logger.error(
f"No event loop available for worker {self.worker_id}, "
f"cannot publish event: {log_event.name}"
)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

Fragile event loop handling pattern

The event loop management in this method has several issues:

  1. Line 135: asyncio.get_event_loop() is deprecated in async contexts; use asyncio.get_running_loop() instead.
  2. Line 138: asyncio.create_task() without storing a reference can cause the task to be garbage collected before completion, leading to lost events.
  3. Line 141: Calling asyncio.run() creates a new event loop, which is problematic and blocks the current thread.

Since workers should operate in an async context with a running event loop, consider simplifying this method or preferring send_message_async() for worker code paths.

Apply this diff to improve the event loop handling:

     def _socket_emit(self, log_event: LogEvent):
         """
         Override socket emit to publish events to Redis.
 
         Args:
             log_event: The event to emit
         """
-        # Try to get the current event loop
         try:
-            loop = asyncio.get_event_loop()
-            if loop.is_running():
-                # Schedule the async send operation
-                asyncio.create_task(self._publish_to_redis(log_event))
-            else:
-                # If no loop is running, run it synchronously
-                asyncio.run(self._publish_to_redis(log_event))
+            loop = asyncio.get_running_loop()
+            # Store task reference to prevent garbage collection
+            task = asyncio.create_task(self._publish_to_redis(log_event))
+            # Optionally add error handler
+            task.add_done_callback(lambda t: t.exception() if not t.cancelled() else None)
         except RuntimeError:
-            # No event loop available
+            # No running event loop - worker should always have one
             logger.error(
                 f"No event loop available for worker {self.worker_id}, "
                 f"cannot publish event: {log_event.name}"
             )
🧰 Tools
🪛 Ruff (0.14.0)

138-138: Store a reference to the return value of asyncio.create_task

(RUF006)

🤖 Prompt for AI Agents
In backend/protocol_rpc/message_handler/redis_worker_handler.py around lines 126
to 147, the current event-loop handling is fragile: replace
asyncio.get_event_loop() with asyncio.get_running_loop(), stop using
asyncio.run() (which creates and blocks a new loop), and avoid calling
asyncio.create_task() without retaining a reference; instead obtain the running
loop and schedule the coroutine with loop.create_task(...) while storing the
returned Task (e.g., append to a self._pending_tasks list or other task
registry) so it isn't garbage-collected, or simply call the existing
send_message_async()/publish helper if available for worker paths; if no running
loop is present catch the appropriate error and log a clear message without
starting a new loop.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

♻️ Duplicate comments (3)
backend/consensus/worker.py (2)

43-43: Fix implicit Optional type hint.

The worker_id parameter uses an implicit Optional with a default of None, which violates PEP 484.

Apply this diff:

-        worker_id: str = None,
+        worker_id: str | None = None,

510-514: Remove redundant exception object from logger.exception call.

logger.exception() automatically captures the current exception, so explicitly including {e} is redundant.

Apply this pattern to lines 512, 577, 666, and 738:

-            logger.exception(
-                f"[Worker {self.worker_id}] Error processing transaction {transaction_data['hash']}: {e}"
-            )
+            logger.exception(
+                f"[Worker {self.worker_id}] Error processing transaction {transaction_data['hash']}"
+            )
backend/protocol_rpc/message_handler/redis_worker_handler.py (1)

126-147: Fragile event loop handling pattern.

The event loop management has several issues:

  1. asyncio.get_event_loop() is deprecated; use asyncio.get_running_loop()
  2. asyncio.create_task() without storing a reference can cause GC before completion
  3. asyncio.run() creates a new event loop, blocking the current thread

Apply this diff to improve event loop handling:

     def _socket_emit(self, log_event: LogEvent):
         """
         Override socket emit to publish events to Redis.

         Args:
             log_event: The event to emit
         """
-        # Try to get the current event loop
         try:
-            loop = asyncio.get_event_loop()
-            if loop.is_running():
-                # Schedule the async send operation
-                asyncio.create_task(self._publish_to_redis(log_event))
-            else:
-                # If no loop is running, run it synchronously
-                asyncio.run(self._publish_to_redis(log_event))
+            loop = asyncio.get_running_loop()
+            # Store task reference to prevent garbage collection
+            task = asyncio.create_task(self._publish_to_redis(log_event))
+            # Add error handler
+            task.add_done_callback(lambda t: t.exception() if not t.cancelled() else None)
         except RuntimeError:
-            # No event loop available
+            # No running event loop - worker should always have one
             logger.error(
                 f"No event loop available for worker {self.worker_id}, "
                 f"cannot publish event: {log_event.name}"
             )
🧹 Nitpick comments (1)
backend/protocol_rpc/message_handler/fastapi_handler.py (1)

41-68: Excellent fix for async endpoint support!

The dual-path decorator correctly handles both async and sync endpoints, addressing the previous critical issue. The logic is sound and the implementation preserves function metadata.

Consider simplifying the return statements by removing the intermediate result variable:

         if inspect.iscoroutinefunction(func):
             # Async wrapper for coroutine functions
             @wraps(func)
             async def async_wrapper(*args, **kwargs):
                 # Log endpoint call
                 logger.info(f"Endpoint called: {func.__name__}")
                 try:
-                    result = await func(*args, **kwargs)
-                    return result
+                    return await func(*args, **kwargs)
                 except Exception as e:
                     logger.error(f"Endpoint error in {func.__name__}: {e}")
                     raise
 
             return async_wrapper
         else:
             # Sync wrapper for regular functions
             @wraps(func)
             def sync_wrapper(*args, **kwargs):
                 # Log endpoint call
                 logger.info(f"Endpoint called: {func.__name__}")
                 try:
-                    result = func(*args, **kwargs)
-                    return result
+                    return func(*args, **kwargs)
                 except Exception as e:
                     logger.error(f"Endpoint error in {func.__name__}: {e}")
                     raise
 
             return sync_wrapper

This simplification also naturally addresses the Ruff TRY300 hints.

📜 Review details

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 90352fb and 1fde575.

📒 Files selected for processing (8)
  • backend/consensus/worker.py (1 hunks)
  • backend/consensus/worker_service.py (1 hunks)
  • backend/protocol_rpc/health.py (1 hunks)
  • backend/protocol_rpc/message_handler/fastapi_handler.py (2 hunks)
  • backend/protocol_rpc/message_handler/redis_worker_handler.py (1 hunks)
  • backend/protocol_rpc/message_handler/types.py (1 hunks)
  • backend/protocol_rpc/redis_subscriber.py (1 hunks)
  • requirements.test.txt (1 hunks)
🚧 Files skipped from review as they are similar to previous changes (1)
  • backend/protocol_rpc/health.py
🧰 Additional context used
📓 Path-based instructions (4)
{backend,examples,tests}/**/*.py

📄 CodeRabbit inference engine (CLAUDE.md)

Python code must be formatted with Black and include type hints

Files:

  • backend/protocol_rpc/message_handler/types.py
  • backend/protocol_rpc/redis_subscriber.py
  • backend/consensus/worker.py
  • backend/protocol_rpc/message_handler/redis_worker_handler.py
  • backend/consensus/worker_service.py
  • backend/protocol_rpc/message_handler/fastapi_handler.py
backend/protocol_rpc/**/*.py

📄 CodeRabbit inference engine (CLAUDE.md)

Implement JSON-RPC endpoint handlers under backend/protocol_rpc/

Files:

  • backend/protocol_rpc/message_handler/types.py
  • backend/protocol_rpc/redis_subscriber.py
  • backend/protocol_rpc/message_handler/redis_worker_handler.py
  • backend/protocol_rpc/message_handler/fastapi_handler.py
**/*.py

📄 CodeRabbit inference engine (AGENTS.md)

Target Python 3.12, use 4-space indentation, and rely on Black (via pre-commit) for formatting

Files:

  • backend/protocol_rpc/message_handler/types.py
  • backend/protocol_rpc/redis_subscriber.py
  • backend/consensus/worker.py
  • backend/protocol_rpc/message_handler/redis_worker_handler.py
  • backend/consensus/worker_service.py
  • backend/protocol_rpc/message_handler/fastapi_handler.py
backend/consensus/**/*.py

📄 CodeRabbit inference engine (CLAUDE.md)

Consensus logic must reside under backend/consensus/

Files:

  • backend/consensus/worker.py
  • backend/consensus/worker_service.py
🧬 Code graph analysis (4)
backend/protocol_rpc/redis_subscriber.py (2)
backend/protocol_rpc/broadcast.py (2)
  • Broadcast (51-99)
  • publish (79-86)
backend/protocol_rpc/message_handler/redis_worker_handler.py (2)
  • close (188-193)
  • health_check (195-220)
backend/consensus/worker.py (7)
backend/database_handler/models.py (2)
  • Transactions (68-145)
  • TransactionStatus (31-42)
backend/database_handler/transactions_processor.py (1)
  • TransactionsProcessor (67-1317)
backend/database_handler/chain_snapshot.py (1)
  • ChainSnapshot (15-70)
backend/database_handler/accounts_manager.py (2)
  • AccountsManager (12-82)
  • create_new_account (24-31)
backend/domain/types.py (2)
  • Transaction (154-273)
  • Validator (102-144)
backend/consensus/base.py (8)
  • ConsensusAlgorithm (350-2255)
  • contract_snapshot_factory (188-220)
  • contract_processor_factory (223-227)
  • chain_snapshot_factory (230-240)
  • can_finalize_transaction (1570-1618)
  • process_leader_appeal (1659-1751)
  • process_leader_timeout_appeal (1753-1854)
  • process_validator_appeal (1856-2003)
backend/validators/__init__.py (3)
  • Manager (136-324)
  • temporal_snapshot (273-290)
  • snapshot (259-270)
backend/protocol_rpc/message_handler/redis_worker_handler.py (5)
backend/protocol_rpc/message_handler/fastapi_handler.py (4)
  • MessageHandler (25-250)
  • info (155-164)
  • error (133-142)
  • _socket_emit (87-95)
backend/protocol_rpc/message_handler/types.py (2)
  • LogEvent (21-39)
  • to_dict (30-39)
backend/protocol_rpc/configuration.py (1)
  • GlobalConfiguration (5-8)
backend/protocol_rpc/message_handler/base.py (1)
  • _log_message (63-92)
backend/protocol_rpc/redis_subscriber.py (1)
  • health_check (245-271)
backend/consensus/worker_service.py (5)
backend/consensus/worker.py (3)
  • ConsensusWorker (30-744)
  • run (673-739)
  • stop (741-744)
backend/protocol_rpc/message_handler/redis_worker_handler.py (3)
  • initialize (57-71)
  • close (188-193)
  • health_check (195-220)
backend/rollup/consensus_service.py (1)
  • ConsensusService (15-288)
backend/database_handler/models.py (1)
  • Base (46-47)
backend/validators/__init__.py (1)
  • Manager (136-324)
🪛 Ruff (0.14.0)
backend/protocol_rpc/redis_subscriber.py

23-23: Mutable class attributes should be annotated with typing.ClassVar

(RUF012)


133-133: Do not catch blind exception: Exception

(BLE001)


139-139: Do not catch blind exception: Exception

(BLE001)


185-185: Do not catch blind exception: Exception

(BLE001)


231-231: Do not catch blind exception: Exception

(BLE001)


268-268: Do not catch blind exception: Exception

(BLE001)


294-294: Do not catch blind exception: Exception

(BLE001)

backend/consensus/worker.py

43-43: PEP 484 prohibits implicit Optional

Convert to T | None

(RUF013)


512-512: Redundant exception object included in logging.exception call

(TRY401)


577-577: Redundant exception object included in logging.exception call

(TRY401)


666-666: Redundant exception object included in logging.exception call

(TRY401)


738-738: Redundant exception object included in logging.exception call

(TRY401)

backend/protocol_rpc/message_handler/redis_worker_handler.py

138-138: Store a reference to the return value of asyncio.create_task

(RUF006)


217-217: Do not catch blind exception: Exception

(BLE001)

backend/consensus/worker_service.py

37-37: Unused function argument: app

(ARG001)


139-139: Do not catch blind exception: Exception

(BLE001)


201-201: Possible binding to all interfaces

(S104)

backend/protocol_rpc/message_handler/fastapi_handler.py

49-49: Consider moving this statement to an else block

(TRY300)


63-63: Consider moving this statement to an else block

(TRY300)

⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (1)
  • GitHub Check: backend-unit-tests
🔇 Additional comments (5)
backend/protocol_rpc/message_handler/fastapi_handler.py (1)

3-3: LGTM!

The inspect import is necessary for the iscoroutinefunction check in the decorator below.

backend/consensus/worker_service.py (1)

135-141: LGTM: Validators manager termination properly implemented.

The validators manager is now correctly terminated during shutdown, addressing the past review comment. The error handling ensures graceful shutdown even if termination fails.

backend/protocol_rpc/message_handler/types.py (1)

17-17: LGTM: Transaction scope added to EventScope enum.

The new TRANSACTION member enables transaction-scoped event routing. Note that the enum value is "Transaction" (with capital T), which must be used for scope comparisons.

backend/protocol_rpc/redis_subscriber.py (1)

139-147: LGTM: Reconnection logic properly implemented.

The reconnection flow now correctly clears is_running and subscription_task before attempting to restart, ensuring the subscriber can reinitialize cleanly.

requirements.test.txt (1)

9-9: genlayer-test 0.9.0 is the latest version and is compatible with pytest 8.x and Python 3.12/3.13 — no action required.

The original review comment's premise is incorrect. Pinning genlayer-test to 0.9.0 is not a downgrade; it is the current stable release on PyPI. According to official metadata, genlayer-test 0.9.0:

  • Requires Python >=3.12 (supports 3.12 and 3.13).
  • Is compatible with pytest 8.x (no conflicting pytest requirement declared).

The pin itself is sound and does not introduce incompatibilities with the other test dependencies (pytest 8.3.5, pytest-asyncio 0.26.0, pytest-xdist 3.6.1).

Likely an incorrect or invalid review comment.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 4

♻️ Duplicate comments (4)
backend/consensus/worker.py (2)

35-44: Make worker_id explicitly nullable

worker_id: str = None triggers Ruff’s RUF013 and violates our explicit-Optional guideline. Please annotate it as str | None (Python 3.12 style) so the type hint matches the default.

-        worker_id: str = None,
+        worker_id: str | None = None,

As per coding guidelines.


117-129: CTE still claims multiple finalizations and blocks them

ready_for_finalization keeps one row per contract (rn = 1), but the UPDATE fires for all of those rows. The statement returns only the first row, so every other updated transaction stays blocked_at = NOW() with worker_id = this worker, yet never gets processed. We end up freezing those records until recovery. Select a single candidate (e.g., add a candidate CTE with LIMIT 1) before running the UPDATE.

-            ready_for_finalization AS (
+            ready_for_finalization AS (
                 ...
-            )
-            UPDATE transactions
+            ),
+            candidate AS (
+                SELECT hash
+                FROM ready_for_finalization
+                WHERE rn = 1
+                ORDER BY created_at ASC
+                LIMIT 1
+            )
+            UPDATE transactions
             SET blocked_at = NOW(),
                 worker_id = :worker_id
-            FROM ready_for_finalization
-            WHERE transactions.hash = ready_for_finalization.hash
-                AND ready_for_finalization.rn = 1
+            WHERE hash = (SELECT hash FROM candidate)
backend/protocol_rpc/message_handler/redis_worker_handler.py (2)

126-147: Critical: Fragile event loop handling pattern (duplicate issue).

This method still has the event loop issues previously flagged:

  1. Line 135: asyncio.get_event_loop() is deprecated in async contexts; use asyncio.get_running_loop() instead.
  2. Line 138: asyncio.create_task() without storing a reference can cause the task to be garbage collected before completion.
  3. Line 141: Calling asyncio.run() creates a new event loop, which blocks the current thread.

Since workers operate in an async context with a running event loop, simplify this method or prefer send_message_async() for worker code paths.

Apply this diff to fix the event loop handling:

     def _socket_emit(self, log_event: LogEvent):
         """
         Override socket emit to publish events to Redis.
 
         Args:
             log_event: The event to emit
         """
-        # Try to get the current event loop
         try:
-            loop = asyncio.get_event_loop()
-            if loop.is_running():
-                # Schedule the async send operation
-                asyncio.create_task(self._publish_to_redis(log_event))
-            else:
-                # If no loop is running, run it synchronously
-                asyncio.run(self._publish_to_redis(log_event))
+            loop = asyncio.get_running_loop()
+            # Store task reference to prevent garbage collection
+            task = loop.create_task(self._publish_to_redis(log_event))
+            # Optionally add error handler
+            task.add_done_callback(lambda t: t.exception() if not t.cancelled() else None)
         except RuntimeError:
-            # No event loop available
+            # No running event loop - worker should always have one
             logger.error(
                 f"No event loop available for worker {self.worker_id}, "
                 f"cannot publish event: {log_event.name}"
             )

167-186: Critical: Incorrect scope value comparisons (duplicate issue).

Lines 182-184 have the same uppercase string comparison issue as send_message(). Events won't match these conditions.

Apply this diff to fix the scope comparisons:

         # Publish to Redis if it's an important event and await completion
         if log_event.transaction_hash or log_event.scope.value in [
-            "TRANSACTION",
-            "CONSENSUS",
+            "Transaction",
+            "Consensus",
         ]:
             await self._publish_to_redis(log_event)
🧹 Nitpick comments (3)
tests/integration/icontracts/tests/test_intelligent_oracle_factory_pattern.py (2)

108-113: Consider extracting the wait interval to a constant.

The hardcoded wait_interval=5000 value could be extracted to a module-level or shared constant for better maintainability, especially if this value needs to be adjusted across multiple integration tests.

Example:

# At the top of the file or in a shared test configuration
DEFAULT_WAIT_INTERVAL_MS = 5000

# Then in the transact call
).transact(
    wait_transaction_status=TransactionStatus.FINALIZED,
    wait_triggered_transactions=True,
    wait_triggered_transactions_status=TransactionStatus.ACCEPTED,
    wait_interval=DEFAULT_WAIT_INTERVAL_MS,
)

153-155: Verify the minimal wait configuration is intentional.

The transact() call here only includes wait_interval=5000, unlike the market creation call (lines 108-113) which includes additional wait parameters like wait_transaction_status and wait_triggered_transactions. Please confirm this difference is intentional based on the resolution transaction's behavior.

Additionally, consider extracting the hardcoded 5000 value to a shared constant for consistency with the suggestion on lines 108-113.

backend/protocol_rpc/message_handler/redis_worker_handler.py (1)

73-88: LGTM with optional refactor suggestion.

The event routing now correctly uses mixed-case enum values ("Transaction" and "Consensus").

Optionally, consider comparing enum members directly instead of string values for type safety:

-        if log_event.scope.value == "Transaction":
+        if log_event.scope == EventScope.Transaction:
             return self.TRANSACTION_CHANNEL
-        elif log_event.scope.value == "Consensus":
+        elif log_event.scope == EventScope.Consensus:
             return self.CONSENSUS_CHANNEL

You'll need to import EventScope from the types module.

📜 Review details

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 1fde575 and 395e73e.

📒 Files selected for processing (5)
  • backend/consensus/worker.py (1 hunks)
  • backend/protocol_rpc/message_handler/redis_worker_handler.py (1 hunks)
  • frontend/vite.config.ts (1 hunks)
  • tests/integration/icontracts/schemas/call_contract_function.py (3 hunks)
  • tests/integration/icontracts/tests/test_intelligent_oracle_factory_pattern.py (2 hunks)
🧰 Additional context used
📓 Path-based instructions (8)
frontend/**/*.{ts,tsx,vue}

📄 CodeRabbit inference engine (CLAUDE.md)

TypeScript code should follow ESLint + Prettier conventions

Files:

  • frontend/vite.config.ts
frontend/**

📄 CodeRabbit inference engine (CLAUDE.md)

Use Socket.io for real-time consensus updates in the frontend

Files:

  • frontend/vite.config.ts
{backend,examples,tests}/**/*.py

📄 CodeRabbit inference engine (CLAUDE.md)

Python code must be formatted with Black and include type hints

Files:

  • backend/protocol_rpc/message_handler/redis_worker_handler.py
  • tests/integration/icontracts/schemas/call_contract_function.py
  • backend/consensus/worker.py
  • tests/integration/icontracts/tests/test_intelligent_oracle_factory_pattern.py
backend/protocol_rpc/**/*.py

📄 CodeRabbit inference engine (CLAUDE.md)

Implement JSON-RPC endpoint handlers under backend/protocol_rpc/

Files:

  • backend/protocol_rpc/message_handler/redis_worker_handler.py
**/*.py

📄 CodeRabbit inference engine (AGENTS.md)

Target Python 3.12, use 4-space indentation, and rely on Black (via pre-commit) for formatting

Files:

  • backend/protocol_rpc/message_handler/redis_worker_handler.py
  • tests/integration/icontracts/schemas/call_contract_function.py
  • backend/consensus/worker.py
  • tests/integration/icontracts/tests/test_intelligent_oracle_factory_pattern.py
tests/**/*.py

📄 CodeRabbit inference engine (CLAUDE.md)

Backend tests should be written and run with pytest (unit, integration, e2e)

Files:

  • tests/integration/icontracts/schemas/call_contract_function.py
  • tests/integration/icontracts/tests/test_intelligent_oracle_factory_pattern.py
tests/{unit,integration,e2e,load}/**

📄 CodeRabbit inference engine (AGENTS.md)

Organize Pytest suites by scope under tests/{unit,integration,e2e,load}

Files:

  • tests/integration/icontracts/schemas/call_contract_function.py
  • tests/integration/icontracts/tests/test_intelligent_oracle_factory_pattern.py
backend/consensus/**/*.py

📄 CodeRabbit inference engine (CLAUDE.md)

Consensus logic must reside under backend/consensus/

Files:

  • backend/consensus/worker.py
🧬 Code graph analysis (2)
backend/protocol_rpc/message_handler/redis_worker_handler.py (4)
backend/protocol_rpc/message_handler/types.py (2)
  • LogEvent (21-39)
  • to_dict (30-39)
backend/protocol_rpc/configuration.py (1)
  • GlobalConfiguration (5-8)
backend/protocol_rpc/message_handler/base.py (1)
  • _log_message (63-92)
backend/protocol_rpc/redis_subscriber.py (1)
  • health_check (245-271)
backend/consensus/worker.py (5)
backend/database_handler/models.py (2)
  • Transactions (68-145)
  • TransactionStatus (31-42)
backend/domain/types.py (3)
  • Transaction (154-273)
  • LLMProvider (81-98)
  • Validator (102-144)
backend/consensus/base.py (8)
  • ConsensusAlgorithm (350-2255)
  • contract_snapshot_factory (188-220)
  • contract_processor_factory (223-227)
  • chain_snapshot_factory (230-240)
  • can_finalize_transaction (1570-1618)
  • process_leader_appeal (1659-1751)
  • process_leader_timeout_appeal (1753-1854)
  • process_validator_appeal (1856-2003)
backend/validators/__init__.py (3)
  • Manager (136-324)
  • temporal_snapshot (273-290)
  • snapshot (259-270)
backend/node/create_nodes/providers.py (2)
  • get_default_provider_for (99-110)
  • validate_provider (38-48)
🪛 Ruff (0.14.0)
backend/protocol_rpc/message_handler/redis_worker_handler.py

138-138: Store a reference to the return value of asyncio.create_task

(RUF006)


217-217: Do not catch blind exception: Exception

(BLE001)

backend/consensus/worker.py

41-41: PEP 484 prohibits implicit Optional

Convert to T | None

(RUF013)


385-385: Do not catch blind exception: Exception

(BLE001)


399-399: Do not catch blind exception: Exception

(BLE001)


551-551: Do not catch blind exception: Exception

(BLE001)


618-618: Do not catch blind exception: Exception

(BLE001)


707-707: Do not catch blind exception: Exception

(BLE001)


787-787: Do not catch blind exception: Exception

(BLE001)

⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (1)
  • GitHub Check: backend-unit-tests
🔇 Additional comments (10)
frontend/vite.config.ts (1)

26-26: LGTM: Domain addition follows existing pattern.

The addition of .genlayer.org to the preview server's allowed hosts is consistent with the existing domain configuration.

If this domain is being added to support the horizontal scalability infrastructure (deployment/testing environments), consider briefly mentioning this in the PR description for context.

tests/integration/icontracts/schemas/call_contract_function.py (2)

25-25: LGTM!

The addition of fallback_validator aligns with the horizontal scaling objectives of this PR and properly extends the primary model configuration.


13-13: No backward compatibility concerns identified — schema change is correct.

The genvm_result: dict replacement is not a breaking change. Analysis shows:

  • gas_used is defined in the Receipt class but is not serialized in to_dict() or deserialized in from_dict() methods (backend/node/types.py:205-244)
  • genvm_result is the new serialized field containing execution output (stdout, stderr)
  • Tests confirm genvm_result structure (test_error_llm.py, test_error_execution.py access genvm_result["stderr"])
  • The schema accurately reflects the API contract; gas_used was never exposed in responses
backend/protocol_rpc/message_handler/redis_worker_handler.py (7)

1-16: LGTM!

Module docstring clearly explains the Redis pub/sub architecture for horizontal scaling, and all imports are appropriately used.


18-27: LGTM!

Class structure and channel constants are well-defined for routing different event types.


29-55: LGTM!

Initialization properly sets up worker identification and Redis configuration with sensible defaults.


57-71: LGTM!

Redis initialization correctly uses aioredis.from_url() without awaiting (the client is ready immediately), and properly tests connectivity with ping.


90-124: LGTM!

Redis publishing logic is well-structured with proper error handling, lazy initialization, and informative logging.


188-193: LGTM!

Cleanup properly closes the Redis connection and logs the disconnection.


195-220: LGTM!

Health check implementation correctly tests Redis connectivity and gathers diagnostic information. The broad exception catching on line 217 is appropriate for health check endpoints to ensure they always return a status dict rather than raising exceptions.

Comment on lines +212 to +224
SET blocked_at = NOW(),
worker_id = :worker_id
FROM available_appeals
WHERE transactions.hash = available_appeals.hash
AND available_appeals.rn = 1
RETURNING transactions.hash, transactions.from_address, transactions.to_address,
transactions.data, transactions.value, transactions.type, transactions.nonce,
transactions.gaslimit, transactions.r, transactions.s, transactions.v,
transactions.leader_only, transactions.sim_config, transactions.contract_snapshot,
transactions.status, transactions.consensus_data, transactions.input_data,
transactions.created_at, transactions.appealed, transactions.appeal_failed,
transactions.timestamp_appeal, transactions.appeal_undetermined,
transactions.appeal_leader_timeout, transactions.appeal_validators_timeout;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🔴 Critical

Appeal claim has the same multi-row UPDATE bug

available_appeals updates every rn = 1 row per contract, but you only read the first returned row. The remaining appeals stay blocked forever. Apply the single-candidate CTE pattern here as well (candidate AS (...) LIMIT 1) so you claim exactly one appeal per call.

Comment on lines +305 to +315
SET blocked_at = NOW(),
worker_id = :worker_id
FROM available_transactions
WHERE transactions.hash = available_transactions.hash
AND available_transactions.rn = 1
RETURNING transactions.hash, transactions.from_address, transactions.to_address,
transactions.data, transactions.value, transactions.type, transactions.nonce,
transactions.gaslimit, transactions.r, transactions.s, transactions.v,
transactions.leader_only, transactions.sim_config, transactions.contract_snapshot,
transactions.status, transactions.consensus_data, transactions.input_data,
transactions.created_at;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🔴 Critical

Regular transaction claim also over-claims rows

Same pattern: available_transactions lets the UPDATE touch every contract’s oldest transaction, yet you only process the first. The rest remain blocked. Introduce a candidate CTE with LIMIT 1 and update using that single hash so you only lock one transaction per call.

🤖 Prompt for AI Agents
In backend/consensus/worker.py around lines 305 to 315, the UPDATE joins
available_transactions which lets the statement mark the oldest transaction for
every contract as blocked while you only intend to claim one; change the query
to first define a candidate CTE that selects a single transaction hash (ORDER/rn
criteria) with LIMIT 1 and then UPDATE transactions SET ... FROM candidate WHERE
transactions.hash = candidate.hash RETURNING ... so only the single chosen
transaction is locked and returned.

Comment on lines +149 to +165
def send_message(self, log_event: LogEvent, log_to_terminal: bool = True):
"""
Send a message (log and publish to Redis).

Args:
log_event: The event to log and send
log_to_terminal: Whether to log to terminal (default True)
"""
if log_to_terminal:
self._log_message(log_event)

# Publish to Redis if it's an important event
if log_event.transaction_hash or log_event.scope.value in [
"TRANSACTION",
"CONSENSUS",
]:
self._socket_emit(log_event)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🔴 Critical

Critical: Incorrect scope value comparisons (same issue as fixed in _get_channel_for_event).

Lines 161-164 check for uppercase "TRANSACTION" and "CONSENSUS", but the actual enum values are "Transaction" and "Consensus" (mixed case). This means events will never match these conditions and won't be published to Redis.

This is the same issue that was correctly fixed in _get_channel_for_event() (lines 73-88) but was not applied here.

Apply this diff to fix the scope comparisons:

         # Publish to Redis if it's an important event
         if log_event.transaction_hash or log_event.scope.value in [
-            "TRANSACTION",
-            "CONSENSUS",
+            "Transaction",
+            "Consensus",
         ]:
             self._socket_emit(log_event)
🤖 Prompt for AI Agents
In backend/protocol_rpc/message_handler/redis_worker_handler.py around lines 149
to 165, the scope string comparisons use uppercase "TRANSACTION" and "CONSENSUS"
which do not match the actual enum values "Transaction" and "Consensus",
preventing events from being published; change those comparisons to use the
correct mixed-case enum string values (or better, compare against the enum
members or log_event.scope == Scope.Transaction / Scope.Consensus) so the
condition correctly detects important events and calls
self._socket_emit(log_event).

Comment on lines +46 to +60
"result": int,
"gaslimit": int,
"nonce": int,
"leader_only": bool,
"contract_snapshot": dict,
"appeal_validators_timeout": bool,
"sim_config": dict,
"sender": str,
"recipient": str,
"tx_id": str,
"activator": str,
"last_leader": str,
"result_name": str,
"num_of_rounds": str,
"last_round": dict,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

Address redundancy, type inconsistency, and consider schema improvements.

Several concerns with the new root-level fields:

  1. Field redundancy: sender/from_address (lines 53/39) and recipient/to_address (lines 54/43) appear redundant. Clarify if both are needed or if one should be removed to avoid confusion.

  2. Type inconsistency: num_of_rounds is typed as str (line 59) but semantically represents a count and should be int.

  3. Missing Optional types: Many fields (e.g., last_leader, contract_snapshot, appeal_validators_timeout) likely aren't always present and should use Optional[...] to accurately reflect the schema.

  4. Large API expansion: Adding 15 fields at once significantly expands the API surface. Consider documenting each field's purpose and when it's populated.

Apply this diff to fix the type inconsistency:

-    "num_of_rounds": str,
+    "num_of_rounds": int,

Consider these additional improvements:

  • Remove redundant fields or document why both are needed
  • Add Optional[...] typing for conditional fields
  • Consider using Pydantic models or dataclasses for type safety and validation
  • Group related fields into nested objects (e.g., a metadata section) to improve schema organization
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
"result": int,
"gaslimit": int,
"nonce": int,
"leader_only": bool,
"contract_snapshot": dict,
"appeal_validators_timeout": bool,
"sim_config": dict,
"sender": str,
"recipient": str,
"tx_id": str,
"activator": str,
"last_leader": str,
"result_name": str,
"num_of_rounds": str,
"last_round": dict,
"result": int,
"gaslimit": int,
"nonce": int,
"leader_only": bool,
"contract_snapshot": dict,
"appeal_validators_timeout": bool,
"sim_config": dict,
"sender": str,
"recipient": str,
"tx_id": str,
"activator": str,
"last_leader": str,
"result_name": str,
"num_of_rounds": int,
"last_round": dict,
🤖 Prompt for AI Agents
In tests/integration/icontracts/schemas/call_contract_function.py around lines
46 to 60, address redundancy, typing and optionality: change num_of_rounds from
str to int, mark fields that may be absent (last_leader, contract_snapshot,
appeal_validators_timeout, last_round, sim_config, activator, tx_id,
result_name, leader_only, gaslimit, nonce, result, sender/recipient duplicates)
as Optional[...] to reflect conditional presence, and either remove duplicate
address fields (sender vs from_address, recipient vs to_address) or add clear
comments/documentation explaining why both are required; optionally refactor by
grouping related fields into a nested metadata object and/or convert the schema
to a Pydantic model or dataclass for validation and clearer typing.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

♻️ Duplicate comments (1)
docker-compose.yml (1)

249-253: ⚠️ [Duplicate] Ensure curl is installed in consensus-worker image for healthcheck.

This is the same critical issue flagged in the previous review: the healthcheck uses curl (line 249), but if the consensus-worker Dockerfile uses Ubuntu 24.04 as a base image (as indicated in the past review), curl is not present by default and the healthcheck will fail.

Ensure that docker/Dockerfile.consensus-worker installs curl via apt-get update && apt-get install -y curl && rm -rf /var/lib/apt/lists/*, or switch to an alternative tool like wget or nc if available in the image.

#!/bin/bash
# Verify curl is installed in the consensus-worker image
fd Dockerfile.consensus-worker -a | head -1 | xargs -I {} sh -c 'echo "=== Checking Dockerfile ==="; cat {}'
rg -nP -C3 '(apt|apk).*(curl|install)' docker/Dockerfile.consensus-worker || echo "No curl installation found"
🧹 Nitpick comments (1)
backend/database_handler/migration/env.py (1)

25-25: Simplify redundant None check.

DB_URL is always constructed as a string, so the if DB_URL is not None check is redundant.

Apply this diff to simplify:

-url = DB_URL if DB_URL is not None else config.get_main_option("sqlalchemy.url")
+url = DB_URL

Or, if you want to preserve the fallback behavior for future flexibility:

-url = DB_URL if DB_URL is not None else config.get_main_option("sqlalchemy.url")
+url = DB_URL or config.get_main_option("sqlalchemy.url")
📜 Review details

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 395e73e and b42523d.

📒 Files selected for processing (3)
  • .github/workflows/docker-build-and-push-all.yml (1 hunks)
  • backend/database_handler/migration/env.py (1 hunks)
  • docker-compose.yml (6 hunks)
🧰 Additional context used
📓 Path-based instructions (3)
{backend,examples,tests}/**/*.py

📄 CodeRabbit inference engine (CLAUDE.md)

Python code must be formatted with Black and include type hints

Files:

  • backend/database_handler/migration/env.py
**/*.py

📄 CodeRabbit inference engine (AGENTS.md)

Target Python 3.12, use 4-space indentation, and rely on Black (via pre-commit) for formatting

Files:

  • backend/database_handler/migration/env.py
docker-compose*.yml

📄 CodeRabbit inference engine (AGENTS.md)

Use docker-compose*.yml files for composing services

Files:

  • docker-compose.yml
🧬 Code graph analysis (1)
backend/database_handler/migration/env.py (1)
backend/rollup/web3_pool.py (1)
  • get (23-58)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (1)
  • GitHub Check: backend-unit-tests
🔇 Additional comments (8)
.github/workflows/docker-build-and-push-all.yml (1)

44-49: Consensus-worker job configuration verified and approved.

All checks passed:

  • Dockerfile docker/Dockerfile.consensus-worker exists ✓
  • Job configuration includes all required parameters and secrets ✓
  • Reusable workflow file exists and is properly referenced ✓
  • Hardhat job has been cleanly removed ✓
docker-compose.yml (7)

4-21: Traefik CLI-driven configuration is well-structured for local development.

The migration from static traefik.yaml to CLI-driven routing is clean and well-organized:

  • Multiple entrypoints correctly configured (web:80, websecure:443, api:4000)
  • API dashboard enabled with insecure mode for local dev (acceptable for local-only use)
  • Traefik dashboard accessible on port 8081 to avoid conflicts with frontend
  • Docker provider configured with exposedbydefault=false for explicit service opt-in

This setup aligns well with the horizontal scalability goal by enabling Traefik-based load balancing for the jsonrpc service.


61-139: Horizontal scaling configuration for jsonrpc is well-implemented.

Excellent improvements for scalability:

  • Replicas: Increased to 3 (configurable via ${JSONRPC_REPLICAS:-3})
  • Load balancing: Traefik labels configured with sticky sessions (cookie-based), service port mapping, and healthcheck path
  • Resource constraints: CPU (configurable 1-2 cores) and memory (configurable 2-8GB) limits and reservations properly set
  • Dependencies: Correctly depends on redis (service_healthy), traefik (service_started), and database-migration (service_completed_successfully)
  • Environment variables: Redis integration, Flask production settings, connection pooling, and logging level all properly configured

The sticky cookie on the jsonrpc router ensures session affinity across replicas, which is important for request context in this architecture.


216-274: consensus-worker service properly configured for distributed consensus processing.

The new service is well-structured for the horizontal scalability goal:

  • Build target: Explicit prod target (or base if not specified)
  • Environment: All necessary config passed (DB credentials, worker polling interval, transaction timeout, Redis URL, webdriver endpoints, logging)
  • Dependencies: Properly chained to wait for database-migration, webdriver health, redis health, and jsonrpc health
  • Deploy settings:
    • Replicas configurable via ${CONSENSUS_WORKERS:-1} (minimum 1 for consensus processing)
    • Restart policy with backoff (5s delay, max 3 attempts)
    • Resource constraints (1-2 CPU cores, 2-8GB memory) with reservations
  • Healthcheck: Configured with appropriate intervals and start period for convergence

One critical concern: The healthcheck uses curl (line 249), which requires verification that curl is installed in the consensus-worker image (see earlier comment on this issue).


299-317: Redis service properly configured with persistence and health monitoring.

The new Redis service follows best practices:

  • Image: redis:7-alpine is lightweight and includes redis-cli for healthcheck
  • Healthcheck: Uses redis-cli ping (line 306), which is available in the image—this is the correct approach vs. curl
  • Persistence: redis_data volume mount ensures data survives container restart
  • Port mapping: Standard Redis port (6379) mapped and configurable
  • Security & logging: Consistent with other services (no-new-privileges, json-file logging with rotation)
  • Restart policy: Restart on failure to ensure availability for dependent services

This is properly integrated with jsonrpc and consensus-worker services that depend on redis health.


206-210: Environment variables explicitly listed in database-migration instead of env_file.

The database-migration service now explicitly lists environment variables (DBUSER, DBPASSWORD, DBHOST, DBPORT, DBNAME) instead of using a shared env_file. This is consistent with other services and makes dependencies explicit in the compose file. No issues identified.


104-108: No action needed—curl is explicitly installed in the backend Dockerfile.

The verification confirms that curl is included in the backend image. Line 17 of docker/Dockerfile.backend explicitly installs curl via apt-get install -y, so the jsonrpc healthcheck will work as intended.

Likely an incorrect or invalid review comment.


320-320: Confirm production monitoring strategy—this is a breaking change.

The vector service profile change from "studio" to "monitoring" is confirmed at line 320 and consistently applied (no orphaned references). However, the .env.example shows production deployment uses COMPOSE_PROFILES= (line 46, empty), meaning vector will NOT run in production by default.

This is a breaking change if production previously used the "studio" profile for monitoring. Before merging:

  • Confirm whether vector/monitoring was previously enabled in production
  • If yes, create a production override file (e.g., docker-compose.prod.yml) or document the process to enable the monitoring profile in production deployments
  • If no, explicitly confirm this is the intended behavior

Comment on lines +13 to +20
# Construct database URL from individual environment variables for consistency with other services
db_user = os.environ.get("DBUSER", "postgres")
db_password = os.environ.get("DBPASSWORD", "postgres")
db_host = os.environ.get("DBHOST", "localhost")
db_port = os.environ.get("DBPORT", "5432")
db_name = os.environ.get("DBNAME", "genlayer")

DB_URL = f"postgresql://{db_user}:{db_password}@{db_host}:{db_port}/{db_name}"

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🔴 Critical

URL-encode credentials to prevent connection failures.

Special characters in DBUSER or DBPASSWORD (e.g., @, :, /, ?) will break the connection string parsing and cause runtime failures. Additionally, type hints are required per coding guidelines.

Apply this diff to add URL encoding and type hints:

+from urllib.parse import quote_plus
+
 # Construct database URL from individual environment variables for consistency with other services
-db_user = os.environ.get("DBUSER", "postgres")
-db_password = os.environ.get("DBPASSWORD", "postgres")
-db_host = os.environ.get("DBHOST", "localhost")
-db_port = os.environ.get("DBPORT", "5432")
-db_name = os.environ.get("DBNAME", "genlayer")
+db_user: str = quote_plus(os.environ.get("DBUSER", "postgres"))
+db_password: str = quote_plus(os.environ.get("DBPASSWORD", "postgres"))
+db_host: str = os.environ.get("DBHOST", "localhost")
+db_port: str = os.environ.get("DBPORT", "5432")
+db_name: str = os.environ.get("DBNAME", "genlayer")
 
-DB_URL = f"postgresql://{db_user}:{db_password}@{db_host}:{db_port}/{db_name}"
+DB_URL: str = f"postgresql://{db_user}:{db_password}@{db_host}:{db_port}/{db_name}"
🤖 Prompt for AI Agents
In backend/database_handler/migration/env.py around lines 13 to 20, the DB
connection URL is built using raw environment values which can break if DBUSER
or DBPASSWORD contain special characters; add URL-encoding and type hints:
import and use urllib.parse.quote_plus to encode db_user and db_password when
constructing DB_URL, add explicit type hints for db_user, db_password, db_host,
db_port, db_name as str and DB_URL as str, and preserve defaults from
os.environ.get.

@sonarqubecloud

Copy link
Copy Markdown

Quality Gate Failed Quality Gate failed

Failed conditions
3 Security Hotspots
4.4% Coverage on New Code (required ≥ 80%)
C Reliability Rating on New Code (required ≥ A)

See analysis details on SonarQube Cloud

Catch issues before they fail your Quality Gate with our IDE extension SonarQube for IDE

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 0

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)
backend/consensus/base.py (1)

2007-2039: Remove unused parameter stop_pending_queue.

The parameter stop_pending_queue is defined but never used in the function body. Either implement its intended functionality or remove it from the signature and all call sites.

Based on coding guidelines: As per static analysis hints.

Call sites to update if removing the parameter:

  • Line 1190: await self.rollback_transactions(context, False)
  • Line 1751: await self.rollback_transactions(context, True)
  • Line 1854: await self.rollback_transactions(context, True)
  • Line 1975: await self.rollback_transactions(context, True)
🧹 Nitpick comments (2)
backend/consensus/base.py (2)

133-138: Optional: Simplify the stderr check logic.

The condition on line 137 can be simplified for better readability.

Apply this diff:

     # Handle genvm_result stdout/stderr
     genvm_result = receipt.get("genvm_result")
     if isinstance(genvm_result, dict):
-        # Only remove stdout if stderr is not present
-        if "stderr" not in genvm_result or not genvm_result.get("stderr"):
+        # Only remove stdout if stderr is not present or empty
+        if not genvm_result.get("stderr"):
             genvm_result.pop("stdout", None)

1223-1233: Optional: Remove redundant string conversions.

The str() conversions may be redundant if the values are already strings.

Apply this diff if new_status.value and transaction_hash are already strings:

         log_event = LogEvent(
             "transaction_status_updated",
             EventType.INFO,
             EventScope.CONSENSUS,
-            f"{str(new_status.value)} {str(transaction_hash)}",
+            f"{new_status.value} {transaction_hash}",
             {
-                "hash": str(transaction_hash),
-                "new_status": str(new_status.value),
+                "hash": transaction_hash,
+                "new_status": new_status.value,
             },
             transaction_hash=transaction_hash,
         )
📜 Review details

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between b42523d and 0f9423c.

📒 Files selected for processing (1)
  • backend/consensus/base.py (20 hunks)
🧰 Additional context used
📓 Path-based instructions (3)
{backend,examples,tests}/**/*.py

📄 CodeRabbit inference engine (CLAUDE.md)

Python code must be formatted with Black and include type hints

Files:

  • backend/consensus/base.py
backend/consensus/**/*.py

📄 CodeRabbit inference engine (CLAUDE.md)

Consensus logic must reside under backend/consensus/

Files:

  • backend/consensus/base.py
**/*.py

📄 CodeRabbit inference engine (AGENTS.md)

Target Python 3.12, use 4-space indentation, and rely on Black (via pre-commit) for formatting

Files:

  • backend/consensus/base.py
🧬 Code graph analysis (1)
backend/consensus/base.py (3)
backend/protocol_rpc/message_handler/base.py (2)
  • MessageHandler (29-154)
  • send_message (151-154)
backend/protocol_rpc/message_handler/types.py (3)
  • LogEvent (21-39)
  • EventType (5-10)
  • EventScope (13-17)
backend/protocol_rpc/message_handler/redis_worker_handler.py (2)
  • send_message_async (167-186)
  • send_message (149-165)
🪛 Ruff (0.14.0)
backend/consensus/base.py

86-86: Do not catch blind exception: Exception

(BLE001)


1227-1227: Use explicit conversion flag

Replace with conversion flag

(RUF010)


1227-1227: Use explicit conversion flag

Replace with conversion flag

(RUF010)


2008-2008: Unused method argument: stop_pending_queue

(ARG002)

⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (1)
  • GitHub Check: backend-unit-tests

@cristiam86
cristiam86 merged commit 55ac6d2 into main Oct 17, 2025
1 check passed
@github-actions

Copy link
Copy Markdown
Contributor

🎉 This PR is included in version 0.79.2 🎉

The release is available on GitHub release

Your semantic-release bot 📦🚀

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants