[Defilan]

What

spec.vllmConfig.stallDetection (enabled + windowSeconds, default off) opts a vLLM InferenceService into a liveness probe that detects a silently wedged engine: requests stay "running" while generation stops and /health keeps returning 200. The probe execs a small stdlib-only Python check against the local /metrics endpoint; startup and readiness stay on /health, and spec.probeOverrides.liveness still wins.

Why

Fixes #2020.

A wedged engine is invisible to every probe LLMKube configures: the pod stays Ready, the InferenceService stays Ready, and the ModelRouter keeps sending traffic into requests that hang. Observed in the field on a B200 (vLLM v0.31.0, Qwen3.8-27B-FP8, MTP, FlashInfer): 21 running requests, vllm:generation_tokens_total frozen for 50+ minutes, /health 200 in 20ms throughout; only deleting the pod recovered it. The detector fails the probe when vllm:num_requests_running > 0 and neither vllm:generation_tokens_total nor vllm:prompt_tokens_total has increased for the window, so kubelet restarts the container, which clears the wedge.

How

• API: StallDetection *VLLMStallDetection on spec.vllmConfig (the existing per-runtime block, styled after SpeculativeConfig). Enabled *bool (nil/false = off, the current /health probe is unchanged), WindowSeconds *int32 bounded 10..3600 by the CRD schema, nil resolves to 120 in code. make generate/manifests/chart-crds outputs are committed and regeneration is idempotent (git status clean after a re-run).
• Wiring: a new optional capability interface ProbeCustomizer (internal/controller/runtime.go), following the repo's existing optional-interface idiom (CommandBuilder, EnvBuilder, IdleDetector). deployment_builder.go calls it after BuildProbes and before the probeOverrides block, so a user override always wins. Only VLLMBackend implements it; no other runtime or the metal-agent path changes (computeSpecHash is an allowlist that does not include this field).
• Detector: one Python script (internal/controller/runtime_vllm_stall.go, stdlib only, no bare % so fmt.Sprintf is safe) fetched over http://127.0.0.1:<port>/metrics. Metrics are parsed with the same semantics as parsePrometheusGaugeSum (drain_before_rollout.go): prefix match, labels tolerated, values summed across series. Exit codes: metrics unreachable = 1 (it replaced the /health liveness, so an unreachable engine must still restart); required metrics absent = 0 (never restart on missing telemetry); idle = 0; progress = 0; every signal flat for the window = 1. Progress is counter movement in either recorded direction (a decrease is a metrics-registry reset, handled below) or a change in vllm:kv_cache_usage_perc beyond float noise (1e-9), up or down. State lives in /tmp (ephemeral, so a restarted container starts with a clean baseline), written atomically via temp file + os.replace, timestamped in epoch seconds because each kubelet probe exec is a fresh process. Metric names confirmed against vLLM's metrics docs (v0.24-v0.31); series carry a model_name label, which the summing handles. The field doc states the runtime requirements (python3 on PATH, writable /tmp): a read-only root filesystem degrades the detector to fail-open, never to false restarts, and the parser's NaN hole is deliberate parity with parsePrometheusGaugeSum, which accepts NaN the same way.
• The probe runs the exact logic the tests run: stallProbeCommand(port, window) is the only source of the script; TestVLLMCustomizeProbes deep-equals the rendered probe.Exec.Command against it, and the detector tests execute those bytes (real python3 processes) against a stub /metrics server.

Detector tests (runtime_vllm_stall_test.go, window 1s so the suite stays fast; sub-window sleeps sit well under it, frozen-case sleeps double it):

• idle never fails, including after a quiet period longer than the window;
• steady generation progress never fails, across a span longer than the window;
• frozen counters with requests running: green while the window has not elapsed, non-zero once it has;
• a long prefill does not fail: running=1, both token counters flat, vllm:kv_cache_usage_perc climbing, across a span longer than the window (the measured prefill shape, see below);
• kv usage dropping holds the window open exactly like a climb (the gauge moves in both directions when real work happens);
• a present-but-frozen kv gauge fails exactly like an absent one: the measured 50-minute stall held vllm:kv_cache_usage_perc at 0.10657, and the optional gauge must not weaken the verdict;
• a counter reset (generation dropping from 100 to 5) writes a fresh baseline and keeps the detector armed: the next all-flat probe past the window still fails.
• unreachable metrics port fails (dead-engine coverage after replacing /health);
• a body without any vLLM metric fails open, and so does a body where requests are running but a counter is missing (never restart on missing telemetry);
• two labeled series are summed and progress on the second series is detected;
• the state file is valid JSON after every probe run (pins the atomic write);
• the rendered script carries no fmt verb error and still reads the state-file env var the tests set.

Rework after field measurement (kv-cache heartbeat). The original premise that a long prefill advances the prompt counter was wrong. Measured on vLLM v0.31.0 (B200, Qwen3.8-27B-FP8, chunked prefill, 16384 batched tokens) with a cache-busting ~200K-token prompt sampled every 1s: vllm:prompt_tokens_total, vllm:generation_tokens_total and vllm:iteration_tokens_total (count and sum) all stayed flat for the whole 13.7s prefill and jumped only at the first token, while vllm:kv_cache_usage_perc climbed steadily (0.013, 0.019, 0.027, 0.031, 0.035, 0.035, 0.039 ... 0.057). During the real 50-minute stall, vllm:kv_cache_usage_perc was frozen at 0.10657. Under the old logic a healthy prefill outlasting the window would have been restarted as a stall. Four fixes:

• the script treats any change in vllm:kv_cache_usage_perc beyond a 1e-9 epsilon, up or down, as progress and stores it in the state file; with the gauge absent the judgment stays exactly the counter-only behavior it was (the required-metrics guard is unchanged);
• a counter that decreases writes a fresh baseline and exits 0: counters are monotonic within an engine lifetime, so a decrease means the metrics registry restarted, and the detector stays armed for a stall that persists after the reset;
• TestStallProbeLongPrefillDoesNotFail now models the measured prefill (running=1, both counters flat, kv climbing across a span longer than the window), with new tests for the frozen-kv stall shape, the kv drop direction, and the counter reset;
• the field docs and code comments no longer claim a prefill advances the prompt counter; they state the measured behavior and name the kv gauge as the mid-prefill heartbeat.

Wiring tests: the table in TestVLLMCustomizeProbes covers no-config / config-without-stallDetection / enabled-nil / enabled-false (all byte-identical to today's probes) plus enabled with the default window (120) and a custom window (90), asserting Exec with python3, the port and window in the script, and startup/readiness untouched. TestConstructDeployment_VLLMStallLivenessAndOverrides renders the exec probe through the real deployment path and pins that probeOverrides.liveness beats it. Envtest specs pin the CRD schema: windowSeconds: 5 and 4000 rejected, enabled-without-window admitted.

Falsifications, each applied to the tree, the named test run, failure pasted, tree restored:

• Dropped the prompt-counter check from the script (gen > prev_gen only) -> TestStallProbeLongPrefillDoesNotFail: runtime_vllm_stall_test.go:193: probe during a long prefill = exit 1, want 0 (FAIL). This test has since been reworked to the measured prefill shape (below); at the time it pinned the prompt-counter premise the issue stated.
• Dropped the idle gate from the script (if running <= 0 made dead) -> TestStallProbeIdleNeverFails: runtime_vllm_stall_test.go:112: idle probe after a long quiet period = exit 1, want 0 (FAIL).
• Collapsed the window comparison to >= 0 -> TestStallProbeFrozenRunningFailsAfterWindow: runtime_vllm_stall_test.go:161: frozen probe inside the window = exit 1, want 0 (FAIL).
• Inverted the enabled gate in CustomizeProbes -> TestVLLMCustomizeProbes: runtime_vllm_test.go:933: liveness probe changed with stall detection off: {ProbeHandler:{Exec:...python3... (FAIL).
• Removed the ProbeCustomizer call from deployment_builder.go -> TestConstructDeployment_VLLMStallLivenessAndOverrides/stall_detection_renders_the_exec_liveness_probe: deployment_builder_test.go:1709: liveness probe = ...HTTPGet.../health..., want an exec probe (FAIL).
• Removed the probeOverrides block from deployment_builder.go -> TestConstructDeployment_VLLMStallLivenessAndOverrides/probeOverrides.liveness_still_wins (FAIL).
• Deleted the fail-open guard for missing metrics from the script (if not running_found or not gen_found or not prompt_found -> if False) -> TestStallProbeMissingMetricsFailsOpen/running_with_one_counter_absent: runtime_vllm_stall_test.go:255: probe with a running engine and a missing counter = exit 1, want 0 (fail open) (FAIL). Added after review: the original all-metrics-missing case stays green even with the guard deleted (the idle branch already exits 0), so it pinned nothing; the running-with-one-counter-absent case is the one that bites.
• Dropped the kv-cache progress check from the script (if prev_kv is not None and kv is not None and abs(kv - prev_kv) > EPS -> if False) -> TestStallProbeLongPrefillDoesNotFail: runtime_vllm_stall_test.go:208: probe during a long prefill (kv climbing, counters flat) = exit 1, want 0 (FAIL); TestStallProbeKvDropCountsAsProgress fails the same way (:229: probe with kv usage dropping = exit 1, want 0).
• Dropped the counter-reset branch from the script (if gen < prev_gen or prompt < prev_prompt -> if False) -> TestStallProbeCounterResetRestartsTheWindow: runtime_vllm_stall_test.go:272: probe right after a counter reset = exit 1, want 0 (fresh baseline) (FAIL).

Gates (pinned GOTOOLCHAIN=go1.26.9 everywhere; an unpinned Go 1.27 panics staticcheck and is not a real failure):

• GOTOOLCHAIN=go1.26.9 make test: PASS, exit 0, 47 packages ok (internal/controller 88.7% coverage).
• GOTOOLCHAIN=go1.26.9 make vet: PASS. make fmt: no diff.
• GOTOOLCHAIN=go1.26.9 make lint: 0 issues with .claude/ worktree copies excluded (temp config, nothing committed). The unscoped run fails in this working directory, but identically on a pristine upstream/main checkout: 18 issues, every one in .claude/worktrees/* (broken Foreman agent worktree copies inside the repo root, no go.mod in pr2019, so the root module absorbs them). None of the 18 exist on CI, which does not run make lint. My diff's own two findings (unparam on a test helper, G115 on a port conversion) were found and fixed.
• GOTOOLCHAIN=go1.26.9 make lint-all: PASS for GOOS=darwin and GOOS=linux, 0 issues each, same worktree exclusion.
• GOTOOLCHAIN=go1.26.9 make lint-deadcode: PASS (it initially caught llmkubeStallStateFileEnv as production code used only by tests; the env name now lives in the test file with an assertion pinning that the script still reads it).
• make generate && make manifests && make chart-crds && git status: clean, only the intended 11 files.
• make check-helm-rbac: PASS (294 rules across 2 operators; no RBAC change).

Not done here, noted per the issue:

• SGLang has the same gap. SGLangBackend.BuildProbes (internal/controller/runtime_sglang.go:214) also uses a /health-only liveness (startup/readiness use /health_generate), so the same wedge family is possible there. SGLang exposes sglang:num_running_reqs and token counters on /metrics, so the fix would mirror this one almost line for line. Left for a follow-up.
• Router follow-up. The router-proxy's per-endpoint health (internal/router/backend.go, markUnhealthy) only reacts to errors and 5xx, so a hung in-flight request never trips quarantine. A per-request timeout on local backends would let it quarantine a wedged endpoint; worth its own issue if it grows past a small change.

Assisted-by: GLM-5.3-Flash (local GLM coding agent; drafted the API field, probe customizer, detector script, tests, and docs from issue #2020; every gate and falsification listed above was executed in-session and its raw output is quoted here)

Checklist

• Tests added/updated
• make test passes locally
• make lint passes locally
• Commit messages follow conventional commits
• All commits are signed off (git commit -s) per DCO
• AI assistance (if any) is disclosed above, per CONTRIBUTING.md
• Documentation updated (if user-facing change)

[codecov]

Codecov Report

❌ Patch coverage is 54.54545% with 20 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
api/v1alpha1/zz_generated.deepcopy.go	0.00%	19 Missing and 1 partial ⚠️

📢 Thoughts on this report? Let us know!