[Defilan]

What

LoadAware session affinity now remembers where each session runs, instead of hashing it onto a ring with a load bound:

balancing:
  mode: LoadAware
  sessionAffinity:
    headers: [x-session-affinity, x-openwebui-chat-id, x-opencode-session]
    idleTimeout: 30m   # optional, 1m to 24h

• A new session goes to the least-loaded pod, and the proxy remembers it there.
• A known session stays on its pod unless that pod is actually backed up. That means it has requests waiting in the engine and carries at least 4 more requests than the least-loaded pod (8, without fresh engine metrics). Or it is KV-full and queuing, and another pod has KV room and is no busier.
• A session that moves stays moved, and cannot move again for pressure for 60 s.
• A session whose pod left (not ready, quarantined) moves at once. A full KV cache alone never moves a session; new sessions still avoid KV-full pods.
• The table is per proxy replica, in memory. It stores only a 64-bit hash of the header value, expires entries after idleTimeout, and caps at 100,000 sessions per backend, past which new sessions fall back to the consistent-hash home without being remembered.
• Metrics: llmkube_router_placements_total gains a reason label (waiting, kv_full, not_ready for spill and migrated), and a new gauge llmkube_router_session_bindings{router,backend} reports table size.
• API:
  • sessionAffinity now requires mode: LoadAware. The webhook rejects RoundRobin plus affinity on create or on any spec change.
  • An existing router with that combination keeps serving: the controller compiles no session headers for it and emits a SessionAffinityIgnored Warning event.
  • maxLoadPercent is deprecated and ignored. It stays in the CRD so re-applying existing manifests still works, its default is gone, and the webhook warns when it is set.

Why

Fixes #2173. Refs #2172.

A two-node DGX B200 deployment served eight vLLM replicas of Qwen3.8-27B to 48 coding agents per replica, with a median prompt of 75k tokens. LoadAware in 0.10.7 came in at TTFT p95 5.94 s and 15.0 tok/s for the slowest 5%, against 2.11 s and 35.3 tok/s for the reference router and 3.6 s for plain consistent hashing. #2172 traced it to the ring:

• 10.7% of placements spilled while pods were lightly loaded, because the bound trips at about a third above the mean with nothing queued.
• The ring has no memory, so a spilled session went back home and missed the cache twice.
• First placement by hash left pods uneven.

Each needless move re-prefills a long conversation on a cold replica. A RoundRobin-plus-affinity run on the same pods measured the same numbers, so the cause was the ring and its bound, not the load signal. This PR replaces both.

How

Commit	Scope
5ff07d865, a613b2f03	internal/router/picker/sessions.go: the table and the placement rules, pure and clock-injected
8a63819d0, 4302d55f2	Dispatcher.place uses the table, the reason label, the bindings gauge and a 30 s sweeper. Ring.Pick and maxLoadPercent leave the proxy.
9d23f9c6e	API: idleTimeout, deprecated maxLoadPercent, the compile change and the SessionAffinityIgnored event. The ModelRouter reconciler gets an event recorder.
ca2eb93c7	Webhook: sessionAffinity requires LoadAware; deprecation warning for maxLoadPercent
e44c675c4, e843af364	Concept docs rewritten for the new behavior
8e36ee7e8, b92e00df1, 007ce1603	Final-review fixes (below) and wording
8bbb5c5d7	Docs: session_bindings counts every session used within idleTimeout, from the B200 benchmark

Design decisions worth noting:

• Per-replica state. Two proxy replicas no longer agree on a session's pod, as the ring guaranteed. With more than one replica, hash the session header to a proxy replica upstream (APISIX chash on the header does this). The docs say so. A proxy restart re-places live sessions once.
• A KV-full pod sheds only to a pod no busier than itself. Without that floor, a simulated KV-full pod moved all 48 of its sessions onto busier pods, a herd that would push them toward KV-full in turn.
• A full table does not scan on every request. The table keeps a lower bound on the oldest entry, so the expiry scan runs only when something can actually have expired. Without that, a client sending a fresh session value per request would hold the backend lock for about 0.8 ms per request at 100k entries.
• The rule lives in the webhook, not CEL. The webhook already skips unchanged specs, so existing objects keep accepting status writes on any Kubernetes version. It is also kept out of validateModelRouter, which drives the reconciler's Validated condition.
• Compatibility in both directions. The proxy decodes config leniently, so a 0.10.7 controller's config works on the new proxy with defaults (tested). Behind the new controller, a ModelRouter pinned to a 0.10.7 proxy image keeps the ring, and any custom maxLoadPercent falls back to 125. The upgrade note says to unpin.

Upgrade notes:

• Routers created under 0.10.7 carry maxLoadPercent: 125 from the old default. The warning says so, and the docs give the JSON patch that removes it.
• RoundRobin plus sessionAffinity must become mode: LoadAware (or drop the affinity block) before the router's spec can change.

Tests: unit tests for every placement rule (including no return home after a spill, the cooldown, KV-full with nothing waiting, all pods full, a single pod, a full table, and a herd), -race tests for racing first requests and the sweeper, a 0.10.7 config decode test, a test that no raw session value reaches logs or metrics, webhook tests, and envtest specs for the CRD bounds and the ignored-affinity event.

Locally:

• make test passes.
• GOTOOLCHAIN=go1.26.10 make lint-all reports 0 issues on darwin and linux.
• make lint-deadcode and make check-helm-rbac pass.
• make manifests generate chart-crds leaves no drift.

Benchmarked on the B200 deployment above with a pre-release router-proxy image of this branch (0.10.8-pr2173.1, built from 4ac813557; the rebase since adds only the docs commit), on its LoadAware test router with the controller still on 0.10.7 and the router's balancing block unchanged. Qwen at 48 agents per replica, 15-minute windows (results):

	0.10.7 LoadAware	This PR	Reference router
TTFT p95	5.94 s	2.54 s	2.11 s
Speed p50 / p5	56.9 / 15.0 tok/s	77.0 / 28.5 tok/s	80.3 / 35.3 tok/s
Output	4,839 tok/s	5,917 tok/s	6,188 tok/s
Prompt cache hit	80.6%	88.6%	89.6%
Errors	0	0	0.01%

Spills fell from 10.7% to 0.4% of placements, all reason="waiting", with no kv_full, not_ready or migrated. The remaining gap is in the slow tail and is tracked in #2174 (token-weighted load).

AI assistance

Assisted-by: Claude Opus 5.5 and Claude Sonnet 5.5 (Claude Code; roles: design and implementation plan, code and tests through per-task implementer agents, per-task and whole-branch reviews, docs, commit messages and this PR text)
Verified by the author: Supervised the build and read the diffs

Checklist

• Tests added/updated
• make test passes locally
• make lint passes locally
• Commit messages follow conventional commits
• All commits are signed off (git commit -s) per DCO
• AI assistance (if any) is disclosed above, per CONTRIBUTING.md
• Documentation updated (if user-facing change)

[codecov]

Codecov Report

❌ Patch coverage is 89.13043% with 20 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
internal/router/placement.go	70.58%	10 Missing ⚠️
api/v1alpha1/zz_generated.deepcopy.go	0.00%	4 Missing ⚠️
cmd/router-proxy/main.go	0.00%	3 Missing ⚠️
internal/router/picker/picker.go	0.00%	2 Missing ⚠️
cmd/main.go	0.00%	1 Missing ⚠️

📢 Thoughts on this report? Let us know!