Skip to content

ci: optimize integration test pipeline (~4-5 min faster) - #1440

Merged
MuncleUscles merged 5 commits into
mainfrom
ci/optimize-integration-tests
Feb 12, 2026
Merged

MuncleUscles merged 5 commits into
mainfrom
ci/optimize-integration-tests

Conversation

@MuncleUscles

@MuncleUscles MuncleUscles commented Feb 11, 2026 •

Copy link
Copy Markdown
Member

Summary

  • Cache GenVM WASM precompile artifacts across CI runs (keyed on GENVM_TAG from Dockerfile), saving ~2.5 min on warm runs
  • Add docker-compose.ci.yml overlay to mount host cache into jsonrpc + consensus-worker containers
  • Enable pip cache on setup-python with cache-dependency-path for all 3 requirements files (~30-45s savings)
  • Move Python setup + pip install before Docker build to overlap I/O
  • Combine 3 separate pip install commands into 1 (single dependency resolution pass)
  • Bump test parallelism from -n 4 to -n 8 (tests are I/O-bound, waiting on RPC responses)
  • Reduce health check poll interval from 5s to 2s
  • Replace 5-iteration validator CRUD test (~280s) with single smoke test (~30s). Full CRUD coverage lives in tests/db-sqlalchemy/validators_registry_test.py (<1s, no GenVM overhead)

Test plan

  • CI integration tests pass on this PR (validates all caching + parallelism changes)
  • GenVM precompile cache: first run populates, second run skips (check "already precompiled" in logs)
  • Validator smoke test passes (single create-delete cycle)
  • db-integration-test job still passes (validates CRUD coverage exists)

Summary by CodeRabbit

  • Chores
    • Optimized CI: early dependency setup, CI-mounted precompile cache, increased test parallelism, faster service readiness checks, and a CI compose override for cache mounting.
  • Bug Fixes
    • Ensure cache marker directories are created to prevent precompile failures.
  • Tests
    • Replaced heavy validator CRUD test with a lightweight smoke test; added unit tests for validator timeouts, state-copy behavior, state-proxy metrics, many direct-mode and integration GenVM/LLM/web tests.
  • Documentation / Tools
    • Added local GenVM web config and benchmarking scripts with README.
  • Refactor
    • Improved execution lifecycle metrics and shared decoding/snapshot caching to reduce repeated work.

- Cache GenVM precompile artifacts across CI runs (keyed on GENVM_TAG)
- Add docker-compose.ci.yml override to mount host cache into containers
- Enable pip cache on setup-python (cache-dependency-path for 3 req files)
- Move Python setup + pip install before Docker build (overlap I/O)
- Combine 3 pip install commands into 1 (single dependency resolution)
- Bump test parallelism from -n 4 to -n 8 (I/O-bound tests benefit)
- Reduce health check poll interval from 5s to 2s
- Add mkdir -p safety in entrypoints for empty cache on first run
The full CRUD suite ran 5 iterations of create/delete, each triggering a
GenVM LLM module restart (~14s per call, ~280s total). Replaced with a
single create-delete smoke test (~30s) that still exercises the GenVM
restart path. Exhaustive CRUD coverage lives in
tests/db-sqlalchemy/validators_registry_test.py (<1s, no GenVM overhead).
@coderabbitai

coderabbitai Bot commented Feb 11, 2026 •

Copy link
Copy Markdown
Contributor
📝 Walkthrough

Walkthrough

Adds GenVM precompile caching into CI and mounts it into compose services; moves Python setup earlier in CI; ensures cache-marker directories exist in entrypoints; introduces shared decoded-value & contract-snapshot caches, state‑proxy metrics, and validator execution timeout handling in backend; replaces a heavy validator CRUD integration test with a lighter create/delete smoke test and adds related unit tests and benchmarks.

Changes

Cohort / File(s) Summary
CI Workflow & Compose
.github/workflows/backend_integration_tests_pr.yml, docker-compose.ci.yml
Early Python setup with pip caching; extract GenVM tag and restore genvm-precompile-<tag>-amd64 cache; mount ${GENVM_CACHE_DIR:-/tmp/genvm-cache} into CI compose services; run docker-compose in no-build mode; increase integration test parallelism (4→8); adjust readiness probe timing/flags.
Docker Entrypoints
docker/entrypoint-backend.sh, docker/entrypoint-consensus-worker.sh
Ensure cache marker parent directory exists before touching marker (mkdir -p "$(dirname "$CACHE_MARKER")").
Backend: consensus / node / genvm
backend/consensus/base.py, backend/node/base.py, backend/node/genvm/base.py, backend/node/genvm/origin/base_host.py
Introduce shared_decoded_value_cache and shared_contract_snapshot_cache propagation; add per-slot decoded caches and snapshot-cache layers; expose SnapshotView.get_metrics; add validator exec timeout config and timeout-receipt path; change run_genvm_host to avoid deepcopy on first attempt and add perf_timeline / host_loop lifecycle metrics; thread new optional params through node/manager APIs (backwards compatible defaults).
Tests — integration & unit
tests/integration/test_validators.py, tests/unit/consensus/test_helpers.py, tests/unit/consensus/test_validator_exec_timeout.py, tests/unit/test_node_state_proxy_metrics.py, tests/unit/test_run_genvm_host_state_copy.py
Replace parametrized validator CRUD test with a single create/delete smoke test; extend test helper signatures to accept shared caches; add unit tests for validator exec timeout, state-proxy metrics, and run_genvm_host state-copy behavior.
Direct-mode & Integration Contracts / Tests
tests/direct/*, tests/integration/icontracts/*
Add multiple direct-mode contracts and tests (LLM/web/genvm smoke) and integration GenVMSmoke contract + integration test verifying web + LLM smoke checks.
Benchmarks & Docs
scripts/benchmarks/*, scripts/benchmarks/README.md
Add snapshot and validator-batch benchmark scripts and README for local performance comparisons (shared-cache experiments).
Misc / Config
config-overrides.example/genvm-module-web.yaml, manifests, minor test comments
Add local GenVM web dev override config and minor manifest/test comment updates.

Sequence Diagram(s)

sequenceDiagram
    participant CI as CI Workflow
    participant Py as Python Setup
    participant Buildx as Docker Buildx
    participant Cache as GenVM Cache Store
    participant Compose as Docker Compose (CI)
    participant Service as Containers (jsonrpc / consensus-worker)
    participant GenVM as GenVM Host (in-container)
    participant Tests as Integration Tests

    CI->>Py: setup Python 3.12 & install deps (pip cache)
    Py-->>CI: deps ready
    CI->>Buildx: build images (extract GENVM_TAG)
    Buildx->>Cache: restore genvm-precompile-<tag>-amd64
    Cache-->>Buildx: provide precompile cache
    CI->>Compose: run docker-compose (no-build) with GENVM_CACHE_DIR mounted
    Compose->>Service: start services with /genvm-cache mount
    Service->>GenVM: run precompilation (mkdir -p then touch marker)
    GenVM-->>Service: precompile marker present
    Service->>Compose: health probe success (curl)
    Compose->>Tests: run integration tests (parallel 8, validator smoke)
    Tests-->>CI: test results
Loading

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~45 minutes

Possibly related PRs

Suggested reviewers

  • kstroobants
  • cristiam86
  • dohernandez

Poem

🐰 I hopped into CI with a cache in my paw,
Tucked precompiles safe so builds won’t guffaw.
Metrics that tick, a smoke-test that squeaks,
Docker, GenVM, and tests — carrot-powered weeks. 🥕

🚥 Pre-merge checks | ✅ 2 | ❌ 1
❌ Failed checks (1 warning)
Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 12.37% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (2 passed)
Check name Status Explanation
Title check ✅ Passed The PR title 'ci: optimize integration test pipeline (~4-5 min faster)' clearly and concisely summarizes the main objective of the changeset: optimizing CI performance. It follows conventional commit format and accurately reflects the primary goal.
Description check ✅ Passed The PR description provides a detailed summary with specific optimization techniques, quantified savings estimates, and a test plan checklist. However, it lacks explicit sections for 'What', 'Why', 'Testing done', 'Decisions made', and 'Reviewing tips' as defined in the template, though the content addresses these topics informally.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing touches
  • 📝 Generate docstrings
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Post copyable unit tests in a comment
  • Commit unit tests in branch ci/optimize-integration-tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

requirements.txt pins requests==2.32.5 while backend/requirements.txt
pins requests==2.32.3. Combined into a single pip install, pip correctly
rejects the conflict. Revert to sequential installs (last writer wins)
to match existing behavior. The version mismatch should be fixed in a
separate PR.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Fix all issues with AI agents
In @.github/workflows/backend_integration_tests_pr.yml:
- Around line 130-141: The health-check loop's wall-clock timeout was
inadvertently reduced because the poll interval (sleep 2) changed but timeout
variable remained 60 (iterations): update the loop to restore the original ~300s
budget by setting timeout=150 (or compute timeout based on desired
seconds/interval) so the loop using counter, timeout, and sleep 2 will wait ~300
seconds again; alternatively, rename timeout to max_attempts and add a comment
clarifying it counts iterations to avoid future confusion.

Comment on lines 130 to 141
timeout=60
counter=0
while [[ "$counter" -lt "$timeout" ]]; do
if curl -X POST -H "Content-Type: application/json" -d '{"jsonrpc":"2.0","method":"ping","params":[],"id":1}' http://0.0.0.0:4000/api | grep -q "OK"; then
if curl -sS -X POST -H "Content-Type: application/json" -d '{"jsonrpc":"2.0","method":"ping","params":[],"id":1}' http://0.0.0.0:4000/api 2>/dev/null | grep -q "OK"; then
echo "RPC server is up!"
break
else
echo "Waiting for RPC server... ($counter/$timeout)"
sleep 5
sleep 2
counter=$((counter+1))
fi
done

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor

Effective health-check timeout halved from ~300s to ~120s.

The counter counts iterations (not seconds). With the poll interval changed from 5s to 2s but timeout=60 unchanged, the maximum wait dropped from 60×5=300s to 60×2=120s. If that's intentional, great — but if services occasionally need more than 2 minutes to start, this will cause spurious failures.

Consider either bumping the iteration count to preserve the original wall-clock budget, or renaming the variable to clarify it represents iterations:

Option: restore ~300s wall-clock timeout
-          timeout=60
+          timeout=150
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
timeout=60
counter=0
while [[ "$counter" -lt "$timeout" ]]; do
if curl -X POST -H "Content-Type: application/json" -d '{"jsonrpc":"2.0","method":"ping","params":[],"id":1}' http://0.0.0.0:4000/api | grep -q "OK"; then
if curl -sS -X POST -H "Content-Type: application/json" -d '{"jsonrpc":"2.0","method":"ping","params":[],"id":1}' http://0.0.0.0:4000/api 2>/dev/null | grep -q "OK"; then
echo "RPC server is up!"
break
else
echo "Waiting for RPC server... ($counter/$timeout)"
sleep 5
sleep 2
counter=$((counter+1))
fi
done
timeout=150
counter=0
while [[ "$counter" -lt "$timeout" ]]; do
if curl -sS -X POST -H "Content-Type: application/json" -d '{"jsonrpc":"2.0","method":"ping","params":[],"id":1}' http://0.0.0.0:4000/api 2>/dev/null | grep -q "OK"; then
echo "RPC server is up!"
break
else
echo "Waiting for RPC server... ($counter/$timeout)"
sleep 2
counter=$((counter+1))
fi
done
🤖 Prompt for AI Agents
In @.github/workflows/backend_integration_tests_pr.yml around lines 130 - 141,
The health-check loop's wall-clock timeout was inadvertently reduced because the
poll interval (sleep 2) changed but timeout variable remained 60 (iterations):
update the loop to restore the original ~300s budget by setting timeout=150 (or
compute timeout based on desired seconds/interval) so the loop using counter,
timeout, and sleep 2 will wait ~300 seconds again; alternatively, rename timeout
to max_attempts and add a comment clarifying it counts iterations to avoid
future confusion.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🤖 Fix all issues with AI agents
In `@backend/node/genvm/base.py`:
- Around line 437-447: The retry logic risks copying a mutated state_proxy
because the original host_args[state_proxy] can be altered during the first
attempt; fix by capturing a pristine snapshot up-front and using that for
retries instead of deepcopying the possibly-mutated proxy. Specifically, before
the first execution attempt, capture original_state_proxy =
host_args.get("state_proxy") and, if it has snapshot_factory, create and store a
pristine snapshot (via snapshot_factory/_copy_state_proxy) to use when building
fresh_args on retry; update the fresh_args construction (the block using
retry_count, host_args, key == "state_proxy", snapshot_factory, and
_copy_state_proxy) to prefer the preserved pristine snapshot for retries rather
than deepcopying the current host_args[state_proxy].
🧹 Nitpick comments (2)
backend/consensus/base.py (1)

2929-2938: Don’t swallow unexpected exceptions during cancel.

Catching broad Exception here hides failures from validator execution; it’s safer to only silence cancellation/timeouts.

Proposed diff
-                        except (
-                            asyncio.CancelledError,
-                            asyncio.TimeoutError,
-                            Exception,
-                        ):
-                            pass
+                        except (asyncio.CancelledError, asyncio.TimeoutError):
+                            pass
tests/unit/consensus/test_validator_exec_timeout.py (1)

101-106: Timing assertion may be flaky under CI load.

The hung validator's asyncio.wait timeout is 0.01 s, plus up to 0.1 s for the cancel grace period, leaving only ~0.04 s of headroom before the 0.15 s cap. Under heavy CI load or slow event-loop scheduling this could intermittently fail.

Consider bumping the threshold (e.g., < 0.5) or asserting a relative relationship (elapsed significantly less than the 0.2 s hung sleep) to avoid flakiness without weakening the test's intent.

Comment thread backend/consensus/base.py
Comment on lines +2882 to +2909
def _build_timeout_receipt(validator_dict: dict) -> Receipt:
timeout_ms = int(validator_exec_timeout_seconds * 1000)
return Receipt(
result=bytes([ResultCode.VM_ERROR]) + b"timeout",
calldata=b"",
gas_used=0,
mode=ExecutionMode.VALIDATOR,
contract_state={},
node_config=validator_dict,
eq_outputs={},
execution_result=ExecutionResultStatus.ERROR,
vote=None,
genvm_result={
"stdout": "",
"stderr": (
"Validator execution exceeded "
f"{validator_exec_timeout_seconds:.3f}s"
),
"error_code": "CONSENSUS_VALIDATOR_EXEC_TIMEOUT",
"raw_error": {
"causes": ["VALIDATOR_EXEC_TIMEOUT"],
# Mark as fatal so replacement validators are attempted.
"fatal": True,
},
},
processing_time=timeout_ms,
)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

Timeout-triggered replacements should still use VRF selection.

By marking timeouts as fatal, this path now triggers validator replacement; ensure the replacement selection is VRF-based (instead of pool-order pop) to match consensus requirements.

As per coding guidelines, Implement validator rotation using VRF (Verifiable Random Function) in the consensus system.

Comment on lines +437 to +447
# Avoid expensive state deep-copy on the first attempt. We only need
# a clean copy when retrying after a failed execution attempt.
if retry_count == 0:
fresh_args = dict(host_args)
else:
fresh_args = {}
for key, value in host_args.items():
if key == "state_proxy" and hasattr(value, "snapshot_factory"):
fresh_args[key] = await _copy_state_proxy(value)
else:
fresh_args[key] = copy.deepcopy(value)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

Retry now risks using a dirty state_proxy after a failed attempt.

The first attempt passes the original proxy; if GenVM mutates storage before an exception, subsequent retries deepcopy the already-mutated proxy from host_args, breaking the “clean retry” guarantee. Consider keeping a pristine snapshot for retries (or reconstructing via snapshot_factory) rather than copying the possibly-mutated proxy.

🤖 Prompt for AI Agents
In `@backend/node/genvm/base.py` around lines 437 - 447, The retry logic risks
copying a mutated state_proxy because the original host_args[state_proxy] can be
altered during the first attempt; fix by capturing a pristine snapshot up-front
and using that for retries instead of deepcopying the possibly-mutated proxy.
Specifically, before the first execution attempt, capture original_state_proxy =
host_args.get("state_proxy") and, if it has snapshot_factory, create and store a
pristine snapshot (via snapshot_factory/_copy_state_proxy) to use when building
fresh_args on retry; update the fresh_args construction (the block using
retry_count, host_args, key == "state_proxy", snapshot_factory, and
_copy_state_proxy) to prefer the preserved pristine snapshot for retries rather
than deepcopying the current host_args[state_proxy].

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 5

🤖 Fix all issues with AI agents
In `@backend/consensus/base.py`:
- Around line 2925-2944: The current cancellation block around exec_task created
from node.exec_transaction swallows all exceptions via a broad except Exception;
narrow this to only handle asyncio.CancelledError and asyncio.TimeoutError (the
expected outcomes) and let any other exceptions propagate (or re-raise/log them)
so real errors aren't suppressed; locate the block referencing exec_task,
validator_exec_timeout_seconds and node.exec_transaction and replace the generic
except Exception with explicit except clauses for CancelledError and
TimeoutError, handling those silently while ensuring other exceptions are not
caught.

In `@config-overrides.example/genvm-module-web.yaml`:
- Around line 13-17: The signer_url template value uses the wrong interpolation
syntax; locate the signer_url entry (key: signer_url) where its value is
currently "#{signerUrl}" and change it to use the repository's standard
"${signerUrl}" interpolation so it matches other YAML configs.

In `@scripts/benchmarks/snapshot_bench.py`:
- Around line 16-135: Add explicit type hints to the benchmark helpers: annotate
make_snapshot(addr: str, slots_per_contract: int, value_size: int) ->
SimpleNamespace; make_view(snapshot_cls: Type, primary_snapshot:
SimpleNamespace, snapshot_factory: Callable[[str], Any], shared_cache:
Optional[Mapping[Any, Any]] = None) -> Any (or the snapshot instance type);
build_slots(slots_per_contract: int) -> List[bytes]; run_scenario(scenario:
Dict[str, Any], snapshot_cls: Type, address_cls: Type, executions: int,
use_shared_cache: bool) -> Dict[str, Any]; and main() -> None; and add imports
for typing symbols used (Type, Callable, Any, Optional, Mapping, List, Dict) so
static checkers accept the annotations. Ensure default values/signatures remain
unchanged while adding the return types.

In `@scripts/benchmarks/validator_batch_bench.py`:
- Around line 97-112: The loop can raise ZeroDivisionError when cross_addresses
or slots are empty; validate inputs before the loop and guard modulo/division
uses: ensure that if cross_reads > 0 then cross_addresses is not empty (or
reduce/skip cross_reads), and ensure slots (from slots_per_contract) is
non-empty before using len(slots) in sidx calculations; update the code around
variables cross_addresses, slots, cross_reads, local_reads, cross_done and the
for-loop that computes sidx/cidx to either validate/raise a clear error earlier
or branch to skip cross reads and local reads when their corresponding
address/slot lists are empty so no modulo by zero occurs.

In `@tests/direct/storage_read_bench.py`:
- Around line 1-2: Replace the star import in this test module with explicit
imports to remove F403/F405 lint flags and clarify dependencies: import the
specific symbols used (gl, u256, DynArray) from genlayer (i.e., replace "from
genlayer import *" with an explicit import that lists gl, u256, DynArray) so the
test only exposes the needed names.
🧹 Nitpick comments (7)
tests/integration/icontracts/contracts/genvm_smoke_v1.py (1)

15-20: Double .strip() on prompt result.

ask_llm() already strips the result (line 18), and the return value of strict_eq is stripped again (line 20). One of them is redundant.

Proposed fix — strip in one place only
         def ask_llm() -> str:
             return gl.nondet.exec_prompt(
                 "Respond with exactly the two characters OK and nothing else."
-            ).strip()
+            )
 
-        self.prompt_result = gl.eq_principle.strict_eq(ask_llm).strip()
+        self.prompt_result = gl.eq_principle.strict_eq(ask_llm).strip()

Or keep it only inside ask_llm — either way, one call is sufficient.

tests/direct/test_genvm_smoke_direct.py (1)

3-3: Unused import pytest.

pytest is imported but never used in this file.

Remove unused import
-import pytest
tests/direct/contracts/error_llm_contract_direct.py (1)

22-41: Remove extraneous f prefix from the prompt string.

The f-string on line 23 has no interpolation placeholders — the {{ / }} are just producing literal braces. A plain string with { / } is clearer and avoids the Ruff F541 warning.

Proposed fix
     def test_execute_prompt(self):
-        prompt = f"""
+        prompt = """
 What is 2+2?
 
 Respond using ONLY the following format:
-{{
+{
 "answer": int,
-}}
+}
 It is mandatory that you respond only using the JSON format above,
 nothing else. Don't include any other words or characters,
 your output must be only JSON without any formatting prefix or suffix.
scripts/benchmarks/snapshot_bench.py (1)

26-31: Thread the shared contract snapshot cache into make_view.
Right now the benchmark only exercises shared_decoded_value_cache. Passing shared_contract_snapshot_cache when supported lets you measure the cross‑contract snapshot cache too (matching validator_batch_bench.py).

♻️ Suggested update
-def make_view(snapshot_cls, primary_snapshot, snapshot_factory, shared_cache):
+def make_view(
+    snapshot_cls,
+    primary_snapshot,
+    snapshot_factory,
+    shared_decode_cache,
+    shared_snapshot_cache,
+):
     sig = inspect.signature(snapshot_cls.__init__)
     kwargs = {"readonly": True}
     if "shared_decoded_value_cache" in sig.parameters:
-        kwargs["shared_decoded_value_cache"] = shared_cache
+        kwargs["shared_decoded_value_cache"] = shared_decode_cache
+    if "shared_contract_snapshot_cache" in sig.parameters:
+        kwargs["shared_contract_snapshot_cache"] = shared_snapshot_cache
     return snapshot_cls(primary_snapshot, snapshot_factory, **kwargs)

-    shared_cache = {} if use_shared_cache else None
+    shared_decode_cache = {} if use_shared_cache else None
+    shared_snapshot_cache = {} if use_shared_cache else None
     ...
-        view = make_view(snapshot_cls, primary_snapshot, snapshot_factory, shared_cache)
+        view = make_view(
+            snapshot_cls,
+            primary_snapshot,
+            snapshot_factory,
+            shared_decode_cache,
+            shared_snapshot_cache,
+        )
     ...
-        "shared_cache_size": len(shared_cache) if shared_cache is not None else 0,
+        "shared_decode_cache_size": len(shared_decode_cache)
+        if shared_decode_cache is not None
+        else 0,
+        "shared_snapshot_cache_size": len(shared_snapshot_cache)
+        if shared_snapshot_cache is not None
+        else 0,

Also applies to: 73-85, 125-128

tests/unit/consensus/test_helpers.py (1)

616-617: Unused parameters are intentional but could be clearer.

Static analysis flags shared_decoded_value_cache and shared_contract_snapshot_cache as unused (Ruff ARG001). Since this mock must match the production NodeFactory signature, this is expected. Consider prefixing with _ to signal intent and silence the warning.

Proposed fix
-    shared_decoded_value_cache: Optional[dict[str, bytes]],
-    shared_contract_snapshot_cache: Optional[dict[str, ContractSnapshot]],
+    _shared_decoded_value_cache: Optional[dict[str, bytes]],
+    _shared_contract_snapshot_cache: Optional[dict[str, ContractSnapshot]],
scripts/benchmarks/validator_batch_bench.py (2)

24-38: inspect.signature introspection is fragile — prefer explicit kwargs.

If _SnapshotView.__init__ renames a parameter, this will silently skip passing the cache without any error. Since the benchmark already imports the concrete class, pass the kwargs directly.

Proposed fix
 def make_view(
     snapshot_cls,
     primary_snapshot,
     snapshot_factory,
     *,
     shared_decode_cache,
     shared_snapshot_cache,
 ):
-    sig = inspect.signature(snapshot_cls.__init__)
-    kwargs = {"readonly": True}
-    if "shared_decoded_value_cache" in sig.parameters:
-        kwargs["shared_decoded_value_cache"] = shared_decode_cache
-    if "shared_contract_snapshot_cache" in sig.parameters:
-        kwargs["shared_contract_snapshot_cache"] = shared_snapshot_cache
-    return snapshot_cls(primary_snapshot, snapshot_factory, **kwargs)
+    return snapshot_cls(
+        primary_snapshot,
+        snapshot_factory,
+        readonly=True,
+        shared_decoded_value_cache=shared_decode_cache,
+        shared_contract_snapshot_cache=shared_snapshot_cache,
+    )

This also makes the import inspect on line 4 unnecessary.


14-21: Missing type hints on public functions.

make_snapshot, run_mode, and main lack type annotations. As per coding guidelines, "Include type hints in all Python code".

Also applies to: 41-52, 129-157

Comment thread backend/consensus/base.py
Comment on lines +2925 to +2944
exec_task = asyncio.create_task(
node.exec_transaction(context.transaction)
)
done, _ = await asyncio.wait(
{exec_task},
timeout=validator_exec_timeout_seconds,
return_when=asyncio.FIRST_COMPLETED,
)
if exec_task in done:
result = await exec_task
else:
exec_task.cancel()
try:
await asyncio.wait_for(exec_task, timeout=0.1)
except (
asyncio.CancelledError,
asyncio.TimeoutError,
Exception,
):
pass

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor

🧩 Analysis chain

🏁 Script executed:

cd /tmp && find . -name "base.py" -path "*/consensus/*" 2>/dev/null | head -20

Repository: genlayerlabs/genlayer-studio

Length of output: 54


🏁 Script executed:

git ls-files | grep -E "backend/consensus/base\.py"

Repository: genlayerlabs/genlayer-studio

Length of output: 97


🏁 Script executed:

wc -l backend/consensus/base.py

Repository: genlayerlabs/genlayer-studio

Length of output: 102


🏁 Script executed:

sed -n '2920,2950p' backend/consensus/base.py

Repository: genlayerlabs/genlayer-studio

Length of output: 1588


🏁 Script executed:

head -30 backend/consensus/base.py

Repository: genlayerlabs/genlayer-studio

Length of output: 1019


🏁 Script executed:

grep -n "contextlib" backend/consensus/base.py

Repository: genlayerlabs/genlayer-studio

Length of output: 54


🏁 Script executed:

grep -n "import" backend/consensus/base.py | head -40

Repository: genlayerlabs/genlayer-studio

Length of output: 1980


Narrow the exception handler to avoid swallowing unexpected errors.
The broad except Exception clause can hide real errors during task cancellation. The expected outcomes are cancellation or timeout; other exceptions should not be suppressed.

♻️ Suggested cleanup
+import contextlib
 import os
 import asyncio
-                        try:
-                            await asyncio.wait_for(exec_task, timeout=0.1)
-                        except (
-                            asyncio.CancelledError,
-                            asyncio.TimeoutError,
-                            Exception,
-                        ):
-                            pass
+                        with contextlib.suppress(
+                            asyncio.CancelledError,
+                            asyncio.TimeoutError,
+                        ):
+                            await asyncio.wait_for(exec_task, timeout=0.1)
🧰 Tools
🪛 Ruff (0.15.0)

[error] 2939-2944: try-except-pass detected, consider logging the exception

(S110)


[warning] 2942-2942: Do not catch blind exception: Exception

(BLE001)

🤖 Prompt for AI Agents
In `@backend/consensus/base.py` around lines 2925 - 2944, The current cancellation
block around exec_task created from node.exec_transaction swallows all
exceptions via a broad except Exception; narrow this to only handle
asyncio.CancelledError and asyncio.TimeoutError (the expected outcomes) and let
any other exceptions propagate (or re-raise/log them) so real errors aren't
suppressed; locate the block referencing exec_task,
validator_exec_timeout_seconds and node.exec_transaction and replace the generic
except Exception with explicit except clauses for CancelledError and
TimeoutError, handling those silently while ensuring other exceptions are not
caught.

Comment on lines +13 to +17
lua_script_path: ${exeDir}/../config/genvm-web-default.lua
vm_count: 6
lua_path: ${exeDir}/../lib/genvm-lua/?.lua
signer_url: "#{signerUrl}"
signer_headers: {}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor

🧩 Analysis chain

🏁 Script executed:

cat config-overrides.example/genvm-module-web.yaml

Repository: genlayerlabs/genlayer-studio

Length of output: 1276


🏁 Script executed:

#!/bin/bash
# Check both template syntaxes in YAML files across the repo
echo "=== \${...} syntax in YAML files ==="
rg -n '\$\{[^}]+\}' --type yaml | head -30
echo ""
echo "=== #{...} syntax in YAML files ==="
rg -n '#\{[^}]+\}' --type yaml | head -30

Repository: genlayerlabs/genlayer-studio

Length of output: 2226


Fix template syntax on line 16: use ${signerUrl} instead of #{signerUrl}.

The ${...} syntax is the standard interpolation pattern across all YAML configs in the repository (docker-compose.yml, vector.yaml, etc.). The #{...} syntax on line 16 appears nowhere else in the codebase and is likely a typo. Change "#{signerUrl}" to "${signerUrl}" for consistency.

🤖 Prompt for AI Agents
In `@config-overrides.example/genvm-module-web.yaml` around lines 13 - 17, The
signer_url template value uses the wrong interpolation syntax; locate the
signer_url entry (key: signer_url) where its value is currently "#{signerUrl}"
and change it to use the repository's standard "${signerUrl}" interpolation so
it matches other YAML configs.

Comment on lines +16 to +135
def make_snapshot(addr: str, slots_per_contract: int, value_size: int):
state = {}
for i in range(slots_per_contract):
slot = i.to_bytes(32, "big")
seed = i.to_bytes(4, "big")
value = (seed * ((value_size + 3) // 4))[:value_size]
state[b64(slot)] = b64(value)
return SimpleNamespace(contract_address=addr, states={"accepted": state}, balance=0)


def make_view(snapshot_cls, primary_snapshot, snapshot_factory, shared_cache):
sig = inspect.signature(snapshot_cls.__init__)
kwargs = {"readonly": True}
if "shared_decoded_value_cache" in sig.parameters:
kwargs["shared_decoded_value_cache"] = shared_cache
return snapshot_cls(primary_snapshot, snapshot_factory, **kwargs)


def build_slots(slots_per_contract: int):
return [i.to_bytes(32, "big") for i in range(slots_per_contract)]


def run_scenario(
scenario,
snapshot_cls,
address_cls,
executions: int,
use_shared_cache: bool,
):
slots = build_slots(scenario["slots_per_contract"])

primary_addr = ("0x" + "ab" * 20).lower()
primary_snapshot = make_snapshot(
primary_addr, scenario["slots_per_contract"], scenario["local_value_size"]
)

cross_addrs = []
cross_snapshots = {}
for i in range(scenario["cross_contracts"]):
addr = ("0x" + format(i + 1, "040x")).lower()
cross_addrs.append(addr)
cross_snapshots[addr] = make_snapshot(
addr, scenario["slots_per_contract"], scenario["cross_value_size"]
)

snapshot_calls = {"count": 0}

def snapshot_factory(addr_hex: str):
snapshot_calls["count"] += 1
addr_hex = addr_hex.lower()
if addr_hex == primary_addr:
return primary_snapshot
return cross_snapshots[addr_hex]

primary_address = address_cls(primary_addr)
cross_addresses = [address_cls(a) for a in cross_addrs]

shared_cache = {} if use_shared_cache else None
per_exec_ms = []
agg_metrics = defaultdict(int)

local_reads = scenario["local_reads"]
cross_reads = scenario["cross_reads"]
total_reads = local_reads + cross_reads
local_stride = scenario["local_stride"]
cross_stride = scenario["cross_stride"]

for _ in range(executions):
view = make_view(snapshot_cls, primary_snapshot, snapshot_factory, shared_cache)
t0 = time.perf_counter()

local_done = 0
cross_done = 0
for i in range(total_reads):
do_cross = (cross_done < cross_reads) and (
local_done >= local_reads or (i % 5 == 0)
)
if do_cross:
cidx = cross_done % len(cross_addresses) if cross_addresses else 0
sidx = (cross_done * cross_stride) % len(slots)
view.storage_read(
cross_addresses[cidx], slots[sidx], 0, scenario["cross_read_len"]
)
cross_done += 1
else:
sidx = (local_done * local_stride) % len(slots)
view.storage_read(
primary_address, slots[sidx], 0, scenario["local_read_len"]
)
local_done += 1

per_exec_ms.append((time.perf_counter() - t0) * 1000)

if hasattr(view, "get_metrics"):
metrics = view.get_metrics()
for k, v in metrics.items():
if isinstance(v, int):
agg_metrics[k] += v

warm = per_exec_ms[1:] if len(per_exec_ms) > 1 else per_exec_ms
return {
"scenario": scenario["name"],
"use_shared_cache": use_shared_cache,
"executions": executions,
"first_exec_ms": round(per_exec_ms[0], 2),
"warm_mean_ms": round(mean(warm), 2),
"mean_exec_ms": round(mean(per_exec_ms), 2),
"min_exec_ms": round(min(per_exec_ms), 2),
"max_exec_ms": round(max(per_exec_ms), 2),
"total_ms": round(sum(per_exec_ms), 2),
"snapshot_factory_calls": snapshot_calls["count"],
"shared_cache_size": len(shared_cache) if shared_cache is not None else 0,
"metrics": dict(agg_metrics),
}


def main():
parser = argparse.ArgumentParser()
parser.add_argument("--executions", type=int, default=7)
args = parser.parse_args()

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

🧩 Analysis chain

🏁 Script executed:

cat -n scripts/benchmarks/snapshot_bench.py | head -150

Repository: genlayerlabs/genlayer-studio

Length of output: 6103


🏁 Script executed:

cat -n scripts/benchmarks/snapshot_bench.py | sed -n '38,45p'

Repository: genlayerlabs/genlayer-studio

Length of output: 302


🏁 Script executed:

# Check if there are any other benchmark files to see how typing is typically done
fd "bench" -e py | head -10

Repository: genlayerlabs/genlayer-studio

Length of output: 187


🏁 Script executed:

cat -n scripts/benchmarks/validator_batch_bench.py | head -100

Repository: genlayerlabs/genlayer-studio

Length of output: 3785


🏁 Script executed:

# Check the rest of validator_batch_bench.py to see the main function
tail -50 scripts/benchmarks/validator_batch_bench.py

Repository: genlayerlabs/genlayer-studio

Length of output: 1720


🏁 Script executed:

# Check Python version targeting in the repository to validate typing syntax
grep -r "python_requires\|target-version" . --include="*.toml" --include="*.cfg" | head -5

Repository: genlayerlabs/genlayer-studio

Length of output: 54


🏁 Script executed:

cat pyproject.toml | grep -A 5 -B 5 "python"

Repository: genlayerlabs/genlayer-studio

Length of output: 118


🏁 Script executed:

find . -maxdepth 2 -name "*.toml" -o -name "*.cfg" -o -name "setup.py" | grep -v node_modules | head -10

Repository: genlayerlabs/genlayer-studio

Length of output: 83


🏁 Script executed:

cat ruff.toml

Repository: genlayerlabs/genlayer-studio

Length of output: 978


🏁 Script executed:

# Check for Python version specification and typing patterns in existing code
find . -name "*.py" -path "*/backend/*" -type f | head -5 | xargs grep -l "from typing import\|Union\|Optional" | head -3

Repository: genlayerlabs/genlayer-studio

Length of output: 133


🏁 Script executed:

head -30 ./backend/consensus/base.py

Repository: genlayerlabs/genlayer-studio

Length of output: 1019


🏁 Script executed:

# Check if dict[str, Any] syntax (PEP 585) is used elsewhere in the codebase to confirm Python 3.12 compatibility
grep -r "dict\[" backend/ --include="*.py" | head -3

Repository: genlayerlabs/genlayer-studio

Length of output: 290


Add missing type hints to benchmark helper functions.

The file violates the coding guideline "Include type hints in all Python code" for **/*.py files. Functions make_snapshot(), make_view(), build_slots(), run_scenario() parameters scenario and snapshot_cls and address_cls, and main() lack required type annotations.

✍️ Suggested typing pass
 import time
 from collections import defaultdict
 from statistics import mean
 from types import SimpleNamespace
+from typing import Any, Callable

-def make_snapshot(addr: str, slots_per_contract: int, value_size: int):
+def make_snapshot(
+    addr: str, slots_per_contract: int, value_size: int
+) -> SimpleNamespace:
     state = {}
     for i in range(slots_per_contract):
         slot = i.to_bytes(32, "big")
         seed = i.to_bytes(4, "big")
         value = (seed * ((value_size + 3) // 4))[:value_size]
         state[b64(slot)] = b64(value)
     return SimpleNamespace(contract_address=addr, states={"accepted": state}, balance=0)

-def make_view(snapshot_cls, primary_snapshot, snapshot_factory, shared_cache):
+def make_view(
+    snapshot_cls: type,
+    primary_snapshot: SimpleNamespace,
+    snapshot_factory: Callable[[str], SimpleNamespace],
+    shared_cache: dict[str, bytes] | None,
+) -> Any:
     sig = inspect.signature(snapshot_cls.__init__)
     kwargs = {"readonly": True}
     if "shared_decoded_value_cache" in sig.parameters:
         kwargs["shared_decoded_value_cache"] = shared_cache
     return snapshot_cls(primary_snapshot, snapshot_factory, **kwargs)

-def build_slots(slots_per_contract: int):
+def build_slots(slots_per_contract: int) -> list[bytes]:
     return [i.to_bytes(32, "big") for i in range(slots_per_contract)]

 def run_scenario(
-    scenario,
-    snapshot_cls,
-    address_cls,
+    scenario: dict[str, int | str],
+    snapshot_cls: type,
+    address_cls: type,
     executions: int,
     use_shared_cache: bool,
-) -> dict[str, Any]:
+) -> dict[str, Any]:
     slots = build_slots(scenario["slots_per_contract"])
     ...

-def main():
+def main() -> None:
     parser = argparse.ArgumentParser()
🤖 Prompt for AI Agents
In `@scripts/benchmarks/snapshot_bench.py` around lines 16 - 135, Add explicit
type hints to the benchmark helpers: annotate make_snapshot(addr: str,
slots_per_contract: int, value_size: int) -> SimpleNamespace;
make_view(snapshot_cls: Type, primary_snapshot: SimpleNamespace,
snapshot_factory: Callable[[str], Any], shared_cache: Optional[Mapping[Any,
Any]] = None) -> Any (or the snapshot instance type);
build_slots(slots_per_contract: int) -> List[bytes]; run_scenario(scenario:
Dict[str, Any], snapshot_cls: Type, address_cls: Type, executions: int,
use_shared_cache: bool) -> Dict[str, Any]; and main() -> None; and add imports
for typing symbols used (Type, Callable, Any, Optional, Mapping, List, Dict) so
static checkers accept the annotations. Ensure default values/signatures remain
unchanged while adding the return types.

Comment on lines +97 to +112
local_done = 0
cross_done = 0
total_reads = local_reads + cross_reads
for i in range(total_reads):
do_cross = (cross_done < cross_reads) and (
local_done >= local_reads or (i % 5 == 0)
)
if do_cross:
cidx = cross_done % len(cross_addresses)
sidx = (cross_done * 13) % len(slots)
view.storage_read(cross_addresses[cidx], slots[sidx], 0, 512)
cross_done += 1
else:
sidx = (local_done * 7) % len(slots)
view.storage_read(primary_address, slots[sidx], 0, 32)
local_done += 1

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor

ZeroDivisionError when cross_contracts=0 but cross_reads > 0, or slots_per_contract=0.

cross_addresses is empty if --cross-contracts 0, so len(cross_addresses) on line 105 causes a division-by-zero. Same for len(slots) on lines 106/110 if --slots 0.

Proposed fix — validate inputs
 def run_mode(
     mode,
     snapshot_cls,
     address_cls,
     *,
     executions,
     slots_per_contract,
     cross_contracts,
     local_reads,
     cross_reads,
     factory_delay_ms,
 ):
+    if slots_per_contract <= 0:
+        raise ValueError("slots_per_contract must be > 0")
+    if cross_reads > 0 and cross_contracts <= 0:
+        raise ValueError("cross_contracts must be > 0 when cross_reads > 0")
🤖 Prompt for AI Agents
In `@scripts/benchmarks/validator_batch_bench.py` around lines 97 - 112, The loop
can raise ZeroDivisionError when cross_addresses or slots are empty; validate
inputs before the loop and guard modulo/division uses: ensure that if
cross_reads > 0 then cross_addresses is not empty (or reduce/skip cross_reads),
and ensure slots (from slots_per_contract) is non-empty before using len(slots)
in sidx calculations; update the code around variables cross_addresses, slots,
cross_reads, local_reads, cross_done and the for-loop that computes sidx/cidx to
either validate/raise a clear error earlier or branch to skip cross reads and
local reads when their corresponding address/slot lists are empty so no modulo
by zero occurs.

Comment on lines +1 to +2
# { "Depends": "py-genlayer:1jb45aa8ynh2a9c9xn3b7qqh8sm5q93hwfp7jqmwsfhh8jpz09h6" }
from genlayer import *

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor

🧩 Analysis chain

🏁 Script executed:

cat -n tests/direct/storage_read_bench.py

Repository: genlayerlabs/genlayer-studio

Length of output: 808


Replace star import with explicit imports for clarity and to resolve Ruff lint flags F403/F405.

The file uses gl, u256, and DynArray from genlayer. Star imports obscure dependencies and can cause linting errors; explicit imports make the code's requirements clear.

♻️ Proposed change
-from genlayer import *
+from genlayer import gl, u256, DynArray
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
# { "Depends": "py-genlayer:1jb45aa8ynh2a9c9xn3b7qqh8sm5q93hwfp7jqmwsfhh8jpz09h6" }
from genlayer import *
# { "Depends": "py-genlayer:1jb45aa8ynh2a9c9xn3b7qqh8sm5q93hwfp7jqmwsfhh8jpz09h6" }
from genlayer import gl, u256, DynArray
🧰 Tools
🪛 Ruff (0.15.0)

[error] 2-2: from genlayer import * used; unable to detect undefined names

(F403)

🤖 Prompt for AI Agents
In `@tests/direct/storage_read_bench.py` around lines 1 - 2, Replace the star
import in this test module with explicit imports to remove F403/F405 lint flags
and clarify dependencies: import the specific symbols used (gl, u256, DynArray)
from genlayer (i.e., replace "from genlayer import *" with an explicit import
that lists gl, u256, DynArray) so the test only exposes the needed names.

@MuncleUscles
MuncleUscles merged commit 2c076e4 into main Feb 12, 2026
23 checks passed
@github-actions

Copy link
Copy Markdown
Contributor

🎉 This PR is included in version 0.99.1 🎉

The release is available on GitHub release

Your semantic-release bot 📦🚀

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant