Repository navigation
fix: add per-address rate limiting and reject-when-busy for gen_call - #1443
Conversation
…_call Prevents a single contract from exhausting the DB connection pool by hammering gen_call. Two layers: 1. Per-address sliding window rate limit (20 req/10s default) — rejects before any heavy resources are acquired 2. Non-blocking semaphore check — rejects immediately when all GenVM execution slots are occupied instead of queuing and holding DB connections Both return specific JSON-RPC error codes (-32005 rate limit, -32006 server busy) with retry_after_seconds so clients can back off. Adds /health/ratelimit endpoint for observability. Env vars: GEN_CALL_RATE_LIMIT_WINDOW, GEN_CALL_RATE_LIMIT_MAX, GENVM_MAX_CONCURRENT
📝 WalkthroughWalkthroughThe changes introduce per-address rate limiting for gen_call/sim_call endpoints using a sliding-window mechanism and add a diagnostic health endpoint to monitor rate limit state and GenVM capacity metrics across active addresses. Changes
Sequence Diagram(s)sequenceDiagram
participant Client
participant Validator as Validator
participant RateLimit as Rate Limiter
participant GenVM as GenVM
participant Log as Request Log
Client->>Validator: gen_call request
Validator->>RateLimit: _check_rate_limit(address)
RateLimit->>Log: query recent requests (sliding window)
Log-->>RateLimit: timestamps within window
alt Within limit
RateLimit-->>Validator: allow
Validator->>GenVM: acquire semaphore
alt Semaphore available
GenVM-->>Validator: proceed
Validator-->>Client: execute call
else Semaphore locked
GenVM-->>Validator: JSONRPCError(-32006)
Validator-->>Client: server busy
end
else Exceeded limit
RateLimit-->>Validator: JSONRPCError(-32005)
Validator-->>Client: rate limit exceeded
end
Validator->>Log: record request timestamp
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~20 minutes Possibly related PRs
Suggested labels
Poem
🚥 Pre-merge checks | ✅ 4✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing touches
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Fix all issues with AI agents
In `@backend/protocol_rpc/endpoints.py`:
- Around line 77-98: The _address_request_log dictionary accumulates stale
entries for addresses that never get checked again; modify _check_rate_limit to
perform amortised cleanup by pruning expired timestamps for all addresses as
well as the current one: compute cutoff = now - _RATE_LIMIT_WINDOW, iterate over
keys of _address_request_log and replace each list with only timestamps >
cutoff, and delete any address whose list becomes empty to free memory; then
proceed with the existing per-address rate check logic (using
_RATE_LIMIT_MAX/_RATE_LIMIT_WINDOW) and keep logging via _rate_limit_logger as
before.
In `@backend/protocol_rpc/health.py`:
- Line 941: Remove the unused "noqa: SLF001" and stop reading the CPython
internal attribute _genvm_semaphore._value; instead expose availability via a
public API or compute it from a tracked counter. Replace the direct access to
_genvm_semaphore._value in the health payload by calling a new/existing public
accessor (e.g., genvm_semaphore.available() or get_genvm_semaphore_available())
or compute availability as (concurrency_limit - genvm_in_use_counter) maintained
where acquire/release happen; update the code paths that change the semaphore to
increment/decrement that counter so the health endpoint can report availability
without touching internal attributes.
🧹 Nitpick comments (3)
backend/protocol_rpc/endpoints.py (1)
974-975: Rate-limit check happens after validator snapshot acquisition, not "early".By the time
_check_rate_limit(to_address)runs here,_execute_call_with_snapshothas already created the validator snapshot (line 875) which involves DB work. If the goal is to reject before heavyweight resource allocation, consider moving the check into_execute_call_with_snapshotright after extractingto_addressfromparams["to"], before the snapshot context manager.backend/protocol_rpc/health.py (2)
924-924: Redundantimport time as _time—timeis already imported at module level.Suggested fix
- import time as _time - - now = _time.monotonic() + now = time.monotonic()
929-936: Iterating_address_request_logwithout pruning stale entries from the shared dict.This endpoint reads the log but doesn't clean up fully-expired addresses (same issue as in
endpoints.py). For a diagnostic endpoint this is acceptable, but be aware that theaddressespayload can grow large over time if many unique contracts have been rate-limited.
| _address_request_log: dict[str, list[float]] = {} # {address: [timestamp, ...]} | ||
|
|
||
| _rate_limit_logger = logging.getLogger(__name__ + ".rate_limit") | ||
|
|
||
|
|
||
| def _check_rate_limit(address: str) -> None: | ||
| """Reject if address exceeds rate limit. Prunes old entries.""" | ||
| now = time.monotonic() | ||
| timestamps = _address_request_log.get(address, []) | ||
| cutoff = now - _RATE_LIMIT_WINDOW | ||
| timestamps = [t for t in timestamps if t > cutoff] | ||
| if len(timestamps) >= _RATE_LIMIT_MAX: | ||
| _rate_limit_logger.warning( | ||
| f"Rate limit exceeded for {address}: {len(timestamps)} requests in {_RATE_LIMIT_WINDOW}s window" | ||
| ) | ||
| raise JSONRPCError( | ||
| code=-32005, | ||
| message=f"Rate limit exceeded: max {_RATE_LIMIT_MAX} gen_call requests per {_RATE_LIMIT_WINDOW}s per contract address", | ||
| data={"address": address, "retry_after_seconds": _RATE_LIMIT_WINDOW}, | ||
| ) | ||
| timestamps.append(now) | ||
| _address_request_log[address] = timestamps |
There was a problem hiding this comment.
_address_request_log grows without bound for inactive addresses.
Entries are only pruned when the same address is checked again. Addresses that go idle leave stale entries in the dict forever. Over time in production (many distinct contract addresses), this is a slow memory leak.
Consider periodically evicting addresses whose timestamps have all expired — for example, by cleaning during each _check_rate_limit call (amortised), or via a background task.
♻️ Lightweight amortised cleanup sketch
+import itertools
+
+_CLEANUP_EVERY = 500 # prune stale addresses every N calls
+_check_count = 0
+
def _check_rate_limit(address: str) -> None:
"""Reject if address exceeds rate limit. Prunes old entries."""
+ global _check_count
now = time.monotonic()
+ cutoff = now - _RATE_LIMIT_WINDOW
+
+ # Amortised full sweep
+ _check_count += 1
+ if _check_count >= _CLEANUP_EVERY:
+ _check_count = 0
+ stale = [a for a, ts in _address_request_log.items() if ts and ts[-1] <= cutoff]
+ for a in stale:
+ del _address_request_log[a]
+
timestamps = _address_request_log.get(address, [])
- cutoff = now - _RATE_LIMIT_WINDOW
timestamps = [t for t in timestamps if t > cutoff]🤖 Prompt for AI Agents
In `@backend/protocol_rpc/endpoints.py` around lines 77 - 98, The
_address_request_log dictionary accumulates stale entries for addresses that
never get checked again; modify _check_rate_limit to perform amortised cleanup
by pruning expired timestamps for all addresses as well as the current one:
compute cutoff = now - _RATE_LIMIT_WINDOW, iterate over keys of
_address_request_log and replace each list with only timestamps > cutoff, and
delete any address whose list becomes empty to free memory; then proceed with
the existing per-address rate check logic (using
_RATE_LIMIT_MAX/_RATE_LIMIT_WINDOW) and keep logging via _rate_limit_logger as
before.
| "window_seconds": _RATE_LIMIT_WINDOW, | ||
| "max_per_window": _RATE_LIMIT_MAX, | ||
| "genvm_concurrency_limit": _GENVM_CONCURRENCY, | ||
| "genvm_semaphore_available": _genvm_semaphore._value, # noqa: SLF001 |
There was a problem hiding this comment.
🧩 Analysis chain
🌐 Web query:
asyncio.Semaphore._value Python 3.12 public API stable documentation
💡 Result:
asyncio.Semaphore._value is not a public, stable API in Python 3.12.
- The official Python 3.12 documentation for
asyncio.Semaphoredocuments the constructor and the methodsacquire(),locked(), andrelease()—but does not document any.value/._valueattribute as supported API. [1] - In CPython’s 3.12 implementation, the internal counter is stored in an instance attribute named
self._value, i.e., an internal (underscore-prefixed) implementation detail that can change without notice. [2]
If you need supported behavior, rely on acquire()/release()/locked() (or maintain your own counter / wrap the semaphore), rather than reading or writing ._value. [1][2]
Sources: [1] [2]
🏁 Script executed:
# Check linting configuration to see if SLF001 is enabled
find . -name "*.toml" -o -name "*.cfg" -o -name ".ruff*" | head -20Repository: genlayerlabs/genlayer-studio
Length of output: 83
🏁 Script executed:
# Examine the actual file and surrounding context
head -950 backend/protocol_rpc/health.py | tail -50Repository: genlayerlabs/genlayer-studio
Length of output: 1753
🏁 Script executed:
# Check imports at the top of the file
head -30 backend/protocol_rpc/health.pyRepository: genlayerlabs/genlayer-studio
Length of output: 1088
🏁 Script executed:
# Check ruff.toml to see if SLF001 is enabled or disabled
cat ruff.tomlRepository: genlayerlabs/genlayer-studio
Length of output: 978
Remove unused noqa directive and avoid accessing asyncio.Semaphore._value, a CPython implementation detail.
The _value attribute is not part of the public API and is not documented in Python 3.12 documentation. It could break on alternative runtimes or future CPython versions. The noqa: SLF001 directive is also unused since the SLF rule is not enabled in the project's ruff.toml.
Fix: remove unused noqa
- "genvm_semaphore_available": _genvm_semaphore._value, # noqa: SLF001
+ "genvm_semaphore_available": _genvm_semaphore._value,For a more robust solution, consider computing availability from the concurrency limit minus an in-use counter, or exposing it from the semaphore wrapper.
📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| "genvm_semaphore_available": _genvm_semaphore._value, # noqa: SLF001 | |
| "genvm_semaphore_available": _genvm_semaphore._value, |
🧰 Tools
🪛 Ruff (0.15.0)
[warning] 941-941: Unused noqa directive (non-enabled: SLF001)
Remove unused noqa directive
(RUF100)
🤖 Prompt for AI Agents
In `@backend/protocol_rpc/health.py` at line 941, Remove the unused "noqa: SLF001"
and stop reading the CPython internal attribute _genvm_semaphore._value; instead
expose availability via a public API or compute it from a tracked counter.
Replace the direct access to _genvm_semaphore._value in the health payload by
calling a new/existing public accessor (e.g., genvm_semaphore.available() or
get_genvm_semaphore_available()) or compute availability as (concurrency_limit -
genvm_in_use_counter) maintained where acquire/release happen; update the code
paths that change the semaphore to increment/decrement that counter so the
health endpoint can report availability without touching internal attributes.
|
🎉 This PR is included in version 0.100.1 🎉 The release is available on GitHub release Your semantic-release bot 📦🚀 |
Summary
Emergency fix for Studio PRD crash loop — a single contract hammering
gen_callexhausts the DB connection pool (QueuePool 150 limit), causing both jsonrpc pods to crash-loop with 502s.Two protection layers:
-32005before acquiring DB connections or GenVM slots-32006instead of queuing (queued requests hold DB connections while waiting)Both errors include
retry_after_secondsfor client backoff.Observability
/health/ratelimitendpoint shows per-address request counts, semaphore availabilityConfig (env vars)
GEN_CALL_RATE_LIMIT_WINDOW10GEN_CALL_RATE_LIMIT_MAX20GENVM_MAX_CONCURRENT8Test plan
/health/ratelimitafter deploySummary by CodeRabbit
Release Notes