vLLM pods: detect a silently stalled engine (running > 0, generation frozen, /health still 200) #2020

Problem

A vLLM engine can wedge silently: requests stay "running" forever while generation stops, and nothing in LLMKube notices. The pod stays Ready, the InferenceService stays Ready, the ModelRouter keeps sending traffic to it, and every request routed there hangs.

Observed on a B200 (vLLM v0.31.0, Qwen3.8-27B-FP8, MTP speculative decoding, FlashInfer) during a burst of concurrent chat requests:

• vllm:num_requests_running = 21, vllm:num_requests_waiting = 0
• vllm:generation_tokens_total frozen for 50+ minutes
• GPU at 100% utilization at idle-level power; the EngineCore process at ~98% CPU
• GET /health returned 200 in 20 ms the whole time; no error in the logs, no Xid/ECC errors
• only deleting the pod recovered it

This is a known upstream failure family (vLLM #58422, #40350), intermittent and burst-triggered. It is rare (it did not recur in ~44 subsequent replica-bursts), but when it happens it is invisible to every probe LLMKube configures, because vLLM's /health only reports that the engine process is alive.

Proposed solution

Give vLLM pods a liveness check that detects a frozen engine, opt-in at first.

1. Detector logic: the engine is stalled when vllm:num_requests_running > 0 and neither vllm:generation_tokens_total nor vllm:prompt_tokens_total has increased for a configurable window (default 120 s; long prefills complete in seconds on current GPUs, so 120 s leaves wide margin). Idle engines (running == 0) are never "stalled".
2. Mechanism: an exec liveness probe in the vLLM container that reads http://localhost:<port>/metrics, keeps the last counters and timestamp in a small state file under /tmp, and exits non-zero when the stall condition has held for the window. The vLLM image ships Python, so the check can be a short inline python3 -c script. Alternatively, a tiny static helper. Keep it dependency-free and fast (well under the probe timeout).
3. Where: VLLMBackend.BuildProbes in internal/controller/runtime_vllm.go (today's liveness is an HTTP /health check). Keep startup and readiness unchanged. spec.probeOverrides.liveness (internal/controller/deployment_builder.go) must still win when set.
4. Opt-in API: a field such as spec.runtimeOptions.stallDetection (or under the vLLM-specific config if one exists), with enabled and windowSeconds. Default off for the first release; document it. Consider default-on later once it has run in the field.
5. Router follow-up (separate issue if large): the router-proxy's per-endpoint health (internal/router/backend.go, markUnhealthy) only reacts to errors, so a hung request never trips it. A per-request timeout on local backends would let it quarantine a wedged endpoint. Note the gap in the PR; implement it only if it stays small.

Tests

• Unit-test the detector as a pure function (feed it sequences of (running, gen, prompt, t) samples): idle never fails; steady progress never fails; running with frozen counters fails only after the window; a long prefill (prompt counter advancing, generation flat) does not fail. Falsify by dropping the prompt-counter check: the long-prefill case must fail.
• If the probe is an inline script, test the exact script the controller renders (run it against a stub /metrics server in a Go test, or extract it to a file and test it directly). A test that only checks that "a probe exists" is not enough.
• BuildProbes with stall detection on renders the exec liveness probe; off renders today's /health probe; probeOverrides.liveness still wins. Falsify each.

Gates

make test, make lint, make lint-deadcode, make manifests && make chart-crds && git status (clean), make check-helm-rbac.