[Defilan]

What

LoadAware placement now counts each pod's estimated uncached prompt tokens in flight, so new sessions, header-less requests, migrations and spill targets avoid a pod that is in the middle of a long cold prefill.

• Estimate: request body size / 4, capped at 262,144 tokens. A known session's request counts only how much its body grew since its last request; a new, moved or fallback placement counts the whole body.
• Per-pod count: added at dispatch, released at the first response body byte (when a streaming engine has finished prefill) or on any error, close or disconnect. A janitor releases anything older than responseHeaderTimeout + streamIdleTimeout.
• Placement score: request load × 16,384 + prefill tokens. The pressure rules still read load in requests; tokens choose the target those rules measure against.
• Metrics: llmkube_router_endpoint_prefill_tokens (gauge) and llmkube_router_prefill_tokens_estimated_total (counter, to compare with vllm:prompt_tokens_by_source_total{source="local_compute"}). endpoint_load keeps its request meaning; placements_total has no new reason.
• No API, CRD or controller change.

Why

Fixes #2174

On the two-node DGX B200 deployment in #2172, session bindings (#2180) reached TTFT p95 2.54 s and slowest-5% speed 28.5 tok/s against the reference router's 2.11 s and 35.3. Running requests were already even across pods (CV 0.06) while uncached prompt work and generation were less even: a request count cannot see one pod mid 140k-token cold prefill.

Design evidence, from a real vLLM /metrics dump of a pod on that deployment:

• Uncached prefill per request is bimodal: 63.6% of requests computed 5k tokens or fewer, 15.7% over 50k, and that sixth did more than half the prefill work. A per-pod cache hit rate (the issue's first idea) is the mean of two peaks and misprices both, so the estimate is per session instead.
• vLLM sends response headers before prefill and its first SSE chunk only after the first engine output (vLLM v0.32.0 serving.py:527-545; Starlette 0.46.2 responses.py:238-251), so release hangs off the first body byte, never headers.
• New sessions were 13.6% of placements in the feat(router): remember LoadAware sessions and move them only under real pressure #2180 bench (spills 0.4%), about the cold-prefill share, which is where better placement pays.

How

Commit	Scope
16fd5ef2a	Picker: Candidate.Prefill, Score, EstimateTokens, constants; PickLeastLoaded orders by Score
e133c557c	Session table: promptBytes per binding and the per-turn estimate; Place returns it
c76c573af	Dispatcher: per-endpoint prefill counter, first-byte release, release on every error path, janitor, HasLoadAware
b48e5f502	Metrics and docs
25d1b0a7e	Review fixes: drop the counter series when a pod departs (backends without a metrics profile have no scraper to do it), and correct the docs on when tokens can affect a move
4d3de3018	After rebasing onto #2197 and #2201: POST /v1/messages/count_tokens holds no prefill tokens, creates or resizes no session binding and is not counted as a placement; the stale-connection retry releases the estimate on every path

Tests:

• Picker: a pod mid 140k prefill loses a tie; the issue's shape (fewer requests plus one cold prefill loses to more warm requests); estimate cap.
• Sessions: new-session, spill and migration targets avoid the busy pod; growth-only estimates across sticky turns, a compacted conversation and a spill; fallback; tokens alone never move a session on a queue or a request gap under 4; the KV-full target case pinned explicitly.
• Dispatcher: tokens held after headers and released at the first byte (moving the release to headers makes TestDispatchHoldsPrefillUntilFirstByte fail); released on close without reading, connect error, 5xx and cancel before headers; RoundRobin backends and empty bodies count nothing; release after the endpoint departs; janitor expiry, the TTL with the idle guard off, and 1,000 concurrent dispatches ending at 0.
• Metrics: gauge written and deleted with the scraper; counter totals; counter series dropped on departure.
• Interactions: count_tokens on a new and a bound session holds 0 tokens, leaves no binding and no placement count, and the next real turn still estimates only its growth (TestDispatchCountTokensSkipsPrefillAndSession, written first and seen failing); a stale-connection retry counts its estimate once and ends at 0.

An independent whole-branch review found one Important issue (the counter series leak above) and a docs inaccuracy; both are fixed in 25d1b0a7e with tests. Deferred minors: a failed dispatch still advances the session's size record; subagents sharing a session header count their whole prompt each turn; non-streaming requests hold their tokens until the body starts.

Locally:

All on the branch rebased onto main at 9480d468f, under GOTOOLCHAIN=go1.27.3:

• make test GINKGO_SEED=4242 passes (53 packages).
• go test -race ./internal/router/... ./internal/metrics/... ./cmd/router-proxy/... passes.
• The pinned golangci-lint (v2.15.0) and the dead-code config report 0 issues on darwin and linux.
• make generate manifests chart-crds foreman-chart-crds federation-chart-crds leaves no drift.

AI assistance

Assisted-by: Claude Opus 5.5 (Claude Code; roles: design research, implementation plan, code and tests, independent whole-branch review, docs, commit messages and this PR text)

Verified by the author: Reviewed diffs and supervised build

Checklist

• Tests added/updated
• make test passes locally
• make lint passes locally
• Commit messages follow conventional commits
• All commits are signed off (git commit -s) per DCO
• AI assistance (if any) is disclosed above, per CONTRIBUTING.md
• Documentation updated (if user-facing change)

[codecov]

Codecov Report

❌ Patch coverage is 83.67347% with 16 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
internal/router/endpoint_load.go	87.23%	6 Missing ⚠️
internal/router/config.go	0.00%	4 Missing ⚠️
cmd/router-proxy/main.go	0.00%	3 Missing ⚠️
internal/router/backend.go	84.61%	2 Missing ⚠️
internal/router/placement.go	85.71%	1 Missing ⚠️

📢 Thoughts on this report? Let us know!