[joryirving]

What

Bounds a ModelPool swap by the pool's swapBudget and stops a stale deactivate from undoing an activation issued after it, in the router-proxy activator.

Why

Fixes #1837

The activator could leave a pool dark until the proxy was restarted. When the last waiter for a member gave up, the deferred cleanup fired an asynchronous Deactivate outside the lock; a request for the same member arriving in that window started a swap and scaled the member up, and the stale Deactivate then scaled it back to zero. Activate is idempotent-by-check and never re-issued, so the swap goroutine waited on a member that could never become Ready. That goroutine had no deadline either: its only exit was the cancel fired when the last waiter for the target left, so a client retrying on 503 kept the doomed swap alive indefinitely. swapping stayed true, the resident member's fast path stayed closed, and every request on the pool burned its hold budget and 503ed, including requests for the member that was still loaded.

How

The deferred deactivate stays asynchronous but now re-takes the lock and re-checks before patching, skipping itself when the member is wanted again: a waiter has arrived, it became resident, or a swap newer than the generation the deactivate was scheduled against is activating it. A per-pool swap generation was added for that check, and it also stops a cancelled swap's goroutine from clearing the state of a newer swap that started while it was unwinding, which was the same class of bug one level up. The swap itself now runs under a deadline: the pool's swapBudget, since that is already the operator's statement of how long a cold load may take, falling back to twice the router's response-header timeout when a pool declares none so the bound is never tighter than a single caller's hold. On expiry the swap is abandoned - the target is scaled back down while the lock is still held, so a fresh swap for the same member cannot have its Activate undone, a swap error is published to the waiters, the resident's fast path reopens, and the event is logged at ERROR as model pool swap abandoned and counted in a new llmkube_modelpool_swap_failures_total{router,pool,member,reason} counter. Cancelled swaps keep their existing non-error behaviour and the IfIdle and coalescing contracts are untouched.

Testing

The wedge was reproduced live on router-proxy 0.9.27: the pool 503ed for roughly 40 minutes at exactly 2x the swapBudget latency with the target sitting at replicas 0, and it cleared only when the proxy was restarted. The fix is verified by two new unit tests over the existing fake MemberController - one drives a new activation into the exact window the deferred deactivate used to lose and asserts the member is left at one replica and the second waiter is served, the other asserts a swap whose target never becomes Ready is abandoned at the deadline with a non-hold-budget error, swapping cleared, the target deactivated and the resident's fast path immediately open again. Both fail on the unfixed code. make test, make lint and make lint-all all pass locally. This has not yet been deployed in-cluster.

Assisted-by: Claude Code (generated the implementation, tests and docs from my design; I reviewed the diff and ran make test, make lint and make lint-all locally)

Checklist

• Tests added/updated
• make test passes locally
• make lint passes locally
• Commit messages follow conventional commits
• All commits are signed off (git commit -s) per DCO
• AI assistance (if any) is disclosed above, per CONTRIBUTING.md
• Documentation updated (if user-facing change)

[codecov]

Codecov Report

❌ Patch coverage is 77.27273% with 10 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
internal/router/activator.go	77.27%	6 Missing and 4 partials ⚠️

📢 Thoughts on this report? Let us know!

[Defilan]

Approving. I re-ran both new tests with the fix pulled out and each caught it: taking the re-check out of deactivateIfUnwanted makes the stale deactivate land (Deactivate(coder) called 1 times), and swapping the bounded swap context back to WithCancel leaves the caller on its hold budget and fails the deadline test. The generation check reads correctly against the ordering you describe: every Activate sets its swap state under the mutex before its goroutine calls the controller, so re-taking the same lock closes the window. The fallback bound is 240s, twice the response-header timeout, matching the docs. Thanks, the #1837 repro made this easy to follow.