fix(controller): render vllmConfig.speculative as --speculative-config by Defilan · Pull Request #2210 · defilantech/LLMKube

Defilan · 2026-10-24T16:39:22Z

What

spec.vllmConfig.speculative now renders the one argument vLLM accepts for speculative decoding, --speculative-config with a JSON document of method, model and num_speculative_tokens. Before this change the operator emitted --speculative-model and --num-speculative-tokens, neither of which exists in any vLLM release, so enabling the field made the container exit on an unrecognized-argument error.

A draft Model CR named in the block is now resolved, staged and mounted exactly like the target model, and vLLM receives its in-container path instead of the CR name. A new method field covers the methods that need no second model: mtp and dspark speculate from the target checkpoint, ngram and suffix from the prompt. An enabled block with neither a method nor a model is refused at admission.

Why

Fixes #2190

Every point the issue raises:

Issue point	Disposition
Renders flags vLLM does not have	Addressed. One `--speculative-config`, no speculative flag of any other spelling.
Passes the draft Model's name, not a loadable path	Addressed. The resolver returns the Model, storage mounts it, `servedModelPath` gives the path.
Shape cannot express mtp, EAGLE or DSpark	Addressed by the `method` field (free string, not an enum).
Render valid JSON, or reject at admission	Both: CEL rejects `enabled` with neither `method` nor `model`.
Keep a raw-JSON escape hatch	`spec.extraArgs`, which wins over the block in all five vLLM spellings.
Assert against the pinned version's parser, not the arg string	`testdata/vllm-speculative-config-fields-v0.25.1.txt`, extracted from v0.25.1's own `SpeculativeConfig` with its provenance in the file.

How

Commit	Change
`20facb180`	API: `method` field, docs naming the JSON keys, CEL rule. 6 files, +120/-37
`206e21a16`	Renderer: `appendSpeculativeConfig`, oracle-pinned key test. 5 files, +505/-26
`30f16909d`	Draft Model resolution, mount, `SpeculativeDraftNotStaged`. 6 files, +594/-21
`a1525b8f7`	Method doc corrected against v0.25.1, plus a guard test over both CRD copies. 6 files, +61/-22
`621792ca8`	Enabled vLLM block owns draft resolution; the `model` key comes only from `cfg.Model`. 4 files, +179/-8
`8fb4ee686`	Five extraArgs spellings exempted, shared by the renderer and the condition. 8 files, +128/-25
`8ca52dd5f`	Prefetch cause list names the vLLM draft; doc and comment fixes. 7 files

Design decisions worth a reviewer's attention:

One predicate for the draft. draftWeightsRefFor answers for both runtime blocks, so the mount, the fail-closed readiness gate, the Model watch mapping ([FEATURE] InferenceService cannot express draft-model speculative decoding #1528) and the prefetch refusal cannot disagree. That is what makes a vLLM draft wait for its own Model the way a llama.cpp draft does.
The model key comes only from cfg.Model. A method that needs no draft cannot pick up a path the resolver happened to return.
num_speculative_tokens is always rendered, defaulting to 4, because vLLM raises when a speculative model is provided without one.
Method-specific keys stay in extraArgs. vLLM's config dataclass forbids unknown keys, and its key set moves between releases (enable_adaptive_verification exists in v0.32.0 and not in the pinned v0.25.1), so typing them would lock the CRD to one image.
The rendered keys are pinned to the image, not to our reading of it. The test asserts every emitted key against the field list extracted from vLLM v0.25.1's vllm/config/speculative.py, recorded in internal/controller/testdata with the extraction command and the extra="forbid" fact that makes an unknown key fatal.

Two things to know before merging:

The CEL rule means an object already stored with enabled: true and neither model nor method (legal before this change, and crashlooping in practice) cannot be written until the block is fixed. That is the intended upgrade behavior, and it is worth a release note.
Deferred, with the source for each: whether a user-supplied --spec-tokens should also escape the draft readiness gate (policy, not settled here), the num_speculative_tokens cross-field rules for every draft checkpoint (needs a v0.25.1 container run), and a draft Model whose spec.format is not a directory format (the CRD doc now says it must be one).

AI assistance

Assisted-by: DeepSeek-V4.1-Flash (deepseek-ai/DeepSeek-V4.1-Flash, vllm runtime, self-hosted with LLMKube on 3x NVIDIA GB10; roles: planning, code, tests and commit messages, executing review: base-vs-head probes and mutation checks)
Assisted-by: qwen38-flash-next (unsloth/Qwen3.8-Flash-Next-GGUF UD-Q3_K_XL, generic (llmkube-gufo-rocm-gfx1151) runtime, self-hosted with LLMKube on AMD Strix Halo (Ryzen AI Max, Radeon iGPU); roles: read-only review)

Verified by the author: Read the diffs and supervised build

Checklist

Tests added/updated
make test passes locally
make lint passes locally
Commit messages follow conventional commits
All commits are signed off (git commit -s) per DCO
AI assistance (if any) is disclosed above, per CONTRIBUTING.md
Documentation updated (if user-facing change)

…g JSON document spec.vllmConfig.speculative does not render the argument vLLM accepts, so enabling it makes the container exit at startup (defilantech#2190). Fixing the argument builder needs a method field the CRD does not have, because the shape vLLM takes today is a method, optional draft weights, and a token count, and the methods whose drafter ships in the target checkpoint (mtp, dspark) need no second model at all. This commit is the API half: the `method` field, docs that name the JSON keys the operator renders instead of flags vLLM does not have, and an admission rule refusing an enabled block with neither a method nor a model. The argument builder changes in the following commits; the field docs describe behavior that lands there. Decisions: + Flag for an enabled block | changed: renders --speculative-config with a JSON document; --speculative-model and --num-speculative-tokens are never emitted again | vLLM v0.25.1 registers only --speculative-config (vllm/engine/arg_utils.py:1514) and neither old spelling appears in that parser + What spec.vllmConfig.speculative.model means | changed: a Model CR name in the ISVC namespace, staged and mounted like the target and rendered as its in-container path | the CRD field doc and the shipped sample both call it a Model CR + method as an enum or a free string | free string | vLLM registers a method per release (dspark and dflash arrive after v0.25.1's list), the same reason ToolCallParser and ReasoningParser are not enums + Method-specific JSON keys (draft_sample_method, prompt_lookup_max, enable_adaptive_verification) | not typed; the escape hatch stays spec.extraArgs --speculative-config | vLLM's SpeculativeConfig is a pydantic dataclass with extra="forbid", and enable_adaptive_verification exists in v0.32.0 but not in the pinned v0.25.1 + enabled: true with neither model nor method | changed: refused at admission and surfaced as SpeculativeMissingModel | vLLM raises "num_speculative_tokens was provided but without speculative model" + numSpeculativeTokens unset while a draft is set | changed: rendered as the CRD default 4 | vLLM raises "A speculative model was provided, but num_speculative_tokens was not provided" Behavior changes only for: InferenceServices with spec.runtime vllm and spec.vllmConfig.speculative.enabled true. Signed-off-by: Christopher Maher <chris@defilantech.com> Signed-off-by: Christopher Maher <chris@mahercode.io>

appendSpeculativeModel emitted --speculative-model and --num-speculative-tokens. Neither flag exists in any vLLM release, so enabling speculative decoding made vLLM's argument parser reject the whole command and the container exit at startup (defilantech#2190). The issue's reporter hit it planning DeepSeek-V4.1-Flash on a two-node DGX B200 deployment; the only route that worked was extraArgs carrying the JSON by hand. The operator now renders the one argument vLLM accepts: --speculative-config with a JSON document of method, model and num_speculative_tokens. The rendered keys are asserted against the field list of the pinned image's own SpeculativeConfig (vllm/config/speculative.py at v0.25.1, in testdata with its provenance), not against a list the test wrote, because that dataclass forbids unknown keys and a typo is a container that exits rather than a flag that is ignored (defilantech#1031 is the same failure mode). A draft Model renders as the CR name for now; the next commit resolves it to the staged path and mounts the weights. Signed-off-by: Christopher Maher <chris@defilantech.com> Signed-off-by: Christopher Maher <chris@mahercode.io>

The CRD documents spec.vllmConfig.speculative.model as a Model CR reference, but nothing resolved it: the value went to the runtime verbatim, which is why the issue notes it passes a name rather than a path the server can load. The same field also skipped the fail-closed readiness gate llama.cpp's draft gets, so a vLLM service could start before its drafter finished downloading. draftWeightsRefFor is now the single answer for both runtimes: the llama.cpp draftModelRef for the spec types that need weights, and spec.vllmConfig.speculative.model for an enabled vLLM block. Everything that keyed on the old predicate follows, which is what makes the mount, the fail-closed resolver, the Model watch mapping (defilantech#1528) and the prefetch refusal cover the new field without drifting apart. The rendered model key is the draft's staged path, chosen by the same servedModelPath call the target uses, so a multi-file draft is handed its directory. skipModelInit declines staging, so a draft under it gets no flags and a SpeculativeDraftNotStaged condition naming the cause rather than an argv pointing at a path that does not exist. An extraArgs --speculative-config is exempt: that user may mount the weights themselves. Signed-off-by: Christopher Maher <chris@defilantech.com> Signed-off-by: Christopher Maher <chris@mahercode.io>

Review round 1. The method field doc claimed dspark and dflash arrived after v0.25.1's method list. They are both in it: DFlashModelTypes and DSparkModelTypes are members of SpeculativeMethod at v0.25.1 (vllm/config/speculative.py:59-73). The doc ships into both CRD copies and `kubectl explain`, so the false claim was user-facing. It now gives the true reason the field is not an enum (vLLM adds and retires methods per release), and a test checks both CRD copies so the class of claim cannot come back. Settled: + method doc corrected rather than the field narrowed: the decision (free string, not an enum) stands; only its cited example was wrong, per vllm-project/vllm @ v0.25.1 vllm/config/speculative.py:59-73. The same wrong example is corrected in the first commit's ledger row in this PR's body. + numSpeculativeTokens doc reworded (nit applied, behavior-free): the key is always emitted, so "while a draft model is configured" described only one of the shapes; vllmSpeculativeConfig.NumSpeculativeTokens has no omitempty. + SpeculativeDraftNotStaged gains a one-line pointer to operatorStagesModel (nit applied, comment only): skipModelInit is the only way a vLLM pod stages nothing only because VLLMBackend.NeedsModelInit returns true. + two_sparks sample comment notes the custom image requirement (nit applied, comment only): the kept extraArgs JSON carries enable_adaptive_verification, which the operator's default v0.25.1 image does not know and would reject. + resolveBackend question checked, no change: resolveBackend switches on the same literal (case "vllm": RuntimeVLLM, runtime.go:127-145), and nothing sets spec.Runtime to an alias, so a service rendering the vLLM argv always has Runtime == RuntimeVLLM. + open question left open, not settled here: whether vLLM fills target_model_config for method-only mtp/dspark before validation (speculative.py:657-676) is engine runtime behavior this branch cannot prove without a v0.25.1 container; the renderer emits only keys the parser accepts, which is what the change can guarantee. + open question left open, not settled here: a draft Model whose spec.format is empty or gguf gets the primary file rather than the staged directory from servedModelPath, the same call the target uses; whether an empty-format draft is reachable depends on Model CRD validation, which this change does not touch. Signed-off-by: Christopher Maher <chris@defilantech.com> Signed-off-by: Christopher Maher <chris@mahercode.io>

Review round 2. A vllm service that carries both speculative blocks, which is what a llama.cpp service looks like after it is switched to vLLM, resolved the llama draft first. The renderer then consumed that Model's path blind, so --speculative-config carried the wrong Model's staged path while the vLLM block's own draft was never resolved, staged or mounted. Same failure class as the issue: vLLM pointed at weights that are not what the spec asked for. draftWeightsRefFor now consults the enabled vLLM block before the speculativeDecoding one, and the renderer keys the model key on cfg.Model rather than on whichever path the resolver happened to return, so a method that needs no draft cannot pick up a stray one. The llama block still resolves when no enabled vLLM block names a model, so a switched-over service that never set one keeps its old mount and gate. Settled: + both blocks on a vllm service | the vLLM block wins, and only its path reaches the argv | only the vLLM path reaches vLLM's argv; resolving the llama block there hands vLLM another Model's weights (in-repo evidence: deployment_builder.go computes draftPath from whichever Model the resolver returns, and appendSpeculativeConfig renders it unkeyed before this round) + renderer ignores draftPath when cfg.Model is empty | changed: the model key is emitted only from cfg.Model | keeps "method-only renders no model key" true regardless of what the resolver returns, which is the regression repair for the same class + SpeculativeDraftNotStaged message accuracy (nit, declined) | message left as written | it is accurate whenever the condition is reachable: a missing draft Model fails the service first in getDraftModelForInferenceService, so the condition only surfaces with the draft present and Ready; narrowing the text adds a second precondition to a message only this path can produce + extraArgs exemption asymmetry (open question, deferred to Chris) | renderer and condition exempt a user-supplied --speculative-config, the resolver, mount and prefetch refusal do not | not a behavior change this round made; the ledger does not say which is intended and changing it now would decide a policy question in code + num_speculative_tokens cross-field validation (open question) | left open | the oracle pins key names, not cross-field rules; verifying needs a v0.25.1 container, which this branch cannot run Signed-off-by: Christopher Maher <chris@defilantech.com> Signed-off-by: Christopher Maher <chris@mahercode.io>

…cape hatch Review round 3. The extraArgs exemption matched only --speculative-config, but vLLM registers the shorthand flags --spec-method, --spec-model and --spec-tokens that merge into the same JSON document (mutual-exclusion check at vllm/engine/arg_utils.py:1724, alias -sc at :1514). A service with an enabled block plus any of those in spec.extraArgs produced a duplicate key and the engine refused to start, which is the failure this branch exists to remove. One predicate, extraArgsSetsVLLMSpeculative, now covers all five spellings in both the argv and the SpeculativeDraftNotStaged exemption, so the two cannot disagree. hasMatchingExtraArgAny matches "-sc" as written, which the "--name"-only helper could not. Settled: + extraArgs exemption covers -sc and the three shorthand flags | changed: any spelling suppresses the typed block | vLLM merges them into one document and raises on a key already present (arg_utils.py:1724-1737), so the crash would be the issue's own class + resolver, mount and prefetch refusal still honor only the typed block (open question deferred to Chris) | left as is | whether a user who sets --spec-tokens themselves also escapes the draft readiness gate is a policy call the ledger does not settle; the crash paths are fixed without deciding it + target_model_config for method-only mtp/dspark | settled from the oracle, no code change | arg_utils.py:1745-1751 injects target_model_config before SpeculativeConfig is built, so the ValueError at speculative.py:661 cannot fire on the --speculative-config path; round 1's open question closes + method-without-weights for value names like eagle | left to the engine, admission rule unchanged | speculative.py:658-698 raises only for methods outside the no-model set; refusing the method/model matrix at admission would widen the ledger beyond the issue and needs its own decision + draft Model format | doc line added, no behavior change | Model.spec.format defaults to gguf and servedModelPath then hands back the primary file, so the CRD now says the draft must declare a directory format + CEL upgrade friction | recorded for the PR body, no release-note edit here | an object stored with enabled and neither model nor method was legal on base; after the CRD update any write to it is refused until the block is fixed + num_speculative_tokens doc wording (nit) | reworded | vLLM takes the draft config's value when present and requires one otherwise (speculative.py:86-88) Signed-off-by: Christopher Maher <chris@defilantech.com> Signed-off-by: Christopher Maher <chris@mahercode.io>

Review round 4. The prefetch field's CRD doc enumerates every cause of PrefetchUnsupported and this branch adds one: an enabled vllmConfig.speculative block naming a draft Model is refused exactly like the llama.cpp field, because its downloader runs on the serving node. The marker names it by field path like its neighbour, the guide's list matches, and neither line breaks mid-sentence. No Must-fix items in this round: qwen-review approved and the execution gate approved. Settled: + prefetch refusal cause list in the CRD marker | applied with the field path spec.vllmConfig.speculative.model, regenerated into both CRD copies | the marker is what kubectl explain reads, prefetch.go refuses on draftWeightsRefFor, and readers of a cause list grep for field paths + allowlist-style "precedence order" wording (nit) | applied | draftWeightsRefFor is a precedence chain, not a timing one + vLLM case pins the full path while the llama case pins the body (nit) | left as is | the vLLM path is new in this branch, so pinning it is what keeps the new message from drifting; the llama prefix predates the branch and is deliberately unpinned + enumeration not guarded by a test (nit) | not added | the marker is prose over a list of code branches; a text guard would pin wording, not behavior, and the guide and marker now agree + extraArgs exemption asymmetry, num_speculative_tokens cross-field rules, file-format draft (open questions) | deferred to the author, recorded in the PR body | each needs a decision or an engine run outside this branch Signed-off-by: Christopher Maher <chris@defilantech.com> Signed-off-by: Christopher Maher <chris@mahercode.io>

codecov · 2026-10-24T16:52:51Z

Codecov Report

❌ Patch coverage is 97.18310% with 2 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
internal/controller/runtime_vllm_args.go	94.11%	1 Missing and 1 partial ⚠️

📢 Thoughts on this report? Let us know!

…ings vLLM's parser rewrites underscores to dashes in a long flag's name and merges dotted flags such as --speculative-config.prompt_lookup_max=4 into one --speculative-config document appended after every other argument (vllm/utils/argparse_utils.py at v0.25.1). The escape hatch matched only the literal spellings, so it rendered the typed block beside them: --spec_tokens 3 failed the engine at startup on the mutual-exclusion check, and a dotted flag silently replaced the typed block with the user's partial document. Reduce each extraArgs flag to the name vLLM resolves before matching, and move appendSpeculativeConfig's doc comment back onto the function. Signed-off-by: Christopher Maher <chris@mahercode.io>

…layers The field defaults to 4, which suits a separate draft model. For mtp, a value above the checkpoint's MTP layer count runs the same layer again per extra token, and vLLM warns that acceptance may drop (vllm/config/speculative.py at v0.25.1). Say so in the field doc so an mtp user sets it, usually to 1. Signed-off-by: Christopher Maher <chris@mahercode.io>

… 0.10.11) (#981) This PR contains the following updates: | Package | Type | Update | Change | |---|---|---|---| | [ghcr.io/defilantech/charts/llmkube](https://github.com/defilantech/LLMKube) | HelmChart | patch | `0.10.7` → `0.10.11` | --- ### Release Notes <details> <summary>defilantech/LLMKube (ghcr.io/defilantech/charts/llmkube)</summary> ### [`v0.10.11`](https://github.com/defilantech/LLMKube/blob/HEAD/CHANGELOG.md#01011-2026-10-25) [Compare Source](defilantech/LLMKube@v0.10.10...v0.10.11) ##### Features - **api:** accept fp8, float16 and bfloat16 in vllmConfig.kvCacheDtype ([#2227](defilantech/LLMKube#2227)) ([31eb943](defilantech/LLMKube@31eb943)) - **api:** add spec.imagePullPolicy to InferenceService ([#2208](defilantech/LLMKube#2208)) ([ec54d28](defilantech/LLMKube@ec54d28)) - **controller:** move the SGLang defaults to v0.5.22 and report removed attention backends ([#2221](defilantech/LLMKube#2221)) ([33f9bb1](defilantech/LLMKube@33f9bb1)) - **controller:** move the vLLM default to v0.32.0 and check every flag against it ([#2223](defilantech/LLMKube#2223)) ([0144a4c](defilantech/LLMKube@0144a4c)) - **controller:** run every llama.cpp default from one pinned build, b11854 ([#2220](defilantech/LLMKube#2220)) ([052840e](defilantech/LLMKube@052840e)) - **router:** weigh LoadAware placement by estimated prefill tokens in flight ([#2193](defilantech/LLMKube#2193)) ([a33735c](defilantech/LLMKube@a33735c)) ##### Bug Fixes - **controller:** render vllmConfig.speculative as --speculative-config ([#2210](defilantech/LLMKube#2210)) ([191dcbd](defilantech/LLMKube@191dcbd)) - **controller:** report sglangConfig tool-call parsers SGLang does not register ([#2228](defilantech/LLMKube#2228)) ([90fed2e](defilantech/LLMKube@90fed2e)) ### [`v0.10.10`](https://github.com/defilantech/LLMKube/blob/HEAD/CHANGELOG.md#01010-2026-10-24) [Compare Source](defilantech/LLMKube@v0.10.9...v0.10.10) ##### Features - **controller:** add --model-cache-prep to skip the root cache prep ([#2206](defilantech/LLMKube#2206)) ([8bcb973](defilantech/LLMKube@8bcb973)) - **router:** compile dataPlane: AgentGateway to AgentgatewayModel ([#2162](defilantech/LLMKube#2162)) ([26b1ef4](defilantech/LLMKube@26b1ef4)) ##### Bug Fixes - **controller:** report a prefetch as downloading while its downloader runs ([#2207](defilantech/LLMKube#2207)) ([7699514](defilantech/LLMKube@7699514)) - **router:** let every waiter of a failed ModelPool swap see its outcome ([#2203](defilantech/LLMKube#2203)) ([b95db62](defilantech/LLMKube@b95db62)) ### [`v0.10.9`](https://github.com/defilantech/LLMKube/blob/HEAD/CHANGELOG.md#0109-2026-10-23) [Compare Source](defilantech/LLMKube@v0.10.8...v0.10.9) ##### Features - **api:** affinity and topology spread for the router proxy, rolling without surge under hard placement ([#2200](defilantech/LLMKube#2200)) ([3a61700](defilantech/LLMKube@3a61700)) - **router:** proxy the Anthropic Messages API like chat completions ([#2201](defilantech/LLMKube#2201)) ([86223aa](defilantech/LLMKube@86223aa)) ##### Bug Fixes - **router:** retry stale keep-alive connections and quarantine a pod only after two failures ([#2197](defilantech/LLMKube#2197)) ([b279053](defilantech/LLMKube@b279053)) ### [`v0.10.8`](https://github.com/defilantech/LLMKube/blob/HEAD/CHANGELOG.md#0108-2026-10-23) [Compare Source](defilantech/LLMKube@v0.10.7...v0.10.8) ##### Features - **api:** type the vLLM tool-call and reasoning parsers and admit GLM on SGLang ([#2168](defilantech/LLMKube#2168)) ([fd64c72](defilantech/LLMKube@fd64c72)) - **controller:** add InferenceService spec.hubOffline for serving pods ([#2188](defilantech/LLMKube#2188)) ([a911cb2](defilantech/LLMKube@a911cb2)) - **controller:** keep 3 old ReplicaSets for LLMKube-managed Deployments by default ([#2182](defilantech/LLMKube#2182)) ([5e1722f](defilantech/LLMKube@5e1722f)) - **router:** honour endpoint nodePort and externalTrafficPolicy, add proxy runtimeClassName ([#2183](defilantech/LLMKube#2183)) ([a8f9e57](defilantech/LLMKube@a8f9e57)) - **router:** on pools under 3 pods, move a session only when its pod is KV-full ([#2192](defilantech/LLMKube#2192)) ([f9b934c](defilantech/LLMKube@f9b934c)) - **router:** remember LoadAware sessions and move them only under real pressure ([#2180](defilantech/LLMKube#2180)) ([1e8ce5a](defilantech/LLMKube@1e8ce5a)) ##### Bug Fixes - **router:** answer a ModelPool swap past swapBudget with 503 and Retry-After ([#2181](defilantech/LLMKube#2181)) ([7c0d84e](defilantech/LLMKube@7c0d84e)) - **router:** release request bodies once sent and raise the proxy's default memory limit ([#2196](defilantech/LLMKube#2196)) ([140fd6b](defilantech/LLMKube@140fd6b)) </details> --- ### Configuration 📅 **Schedule**: (in timezone Europe/London) - Branch creation - At any time (no schedule defined) - Automerge - At any time (no schedule defined) 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] If you want to rebase/retry this PR, check this box --- This PR has been generated by [Mend Renovate CLI](https://github.com/renovatebot/renovate).  Reviewed-on: https://git.hayden.moe/hayden/phoebe/pulls/981

…0.10.15) (#2331) This PR contains the following updates: | Package | Type | Update | Change | |---|---|---|---| | [ghcr.io/defilantech/charts/llmkube](https://github.com/defilantech/LLMKube) | HelmChart | patch | `0.10.2` → `0.10.15` | --- > ⚠️ **Warning** > > Some dependencies could not be looked up. Check the [Dependency Dashboard](issues/567) for more information. --- ### Release Notes <details> <summary>defilantech/LLMKube (ghcr.io/defilantech/charts/llmkube)</summary> ### [`v0.10.15`](https://github.com/defilantech/LLMKube/blob/HEAD/CHANGELOG.md#01015-2026-10-31) [Compare Source](defilantech/LLMKube@v0.10.14...v0.10.15) ##### Features - **agent:** add --engine-env-passthrough to pass named variables to Metal engines ([6d4fba0](defilantech/LLMKube@6d4fba0)) - **agent:** give Metal engines an allowlisted environment (fixes [#2314](defilantech/LLMKube#2314)) ([6d4fba0](defilantech/LLMKube@6d4fba0)) - **foreman:** select the forge provider by name, remote host, or default ([#2292](defilantech/LLMKube#2292)) ([239411a](defilantech/LLMKube@239411a)) - **foreman:** stream in-turn generation progress ([#2313](defilantech/LLMKube#2313)) ([3d57743](defilantech/LLMKube@3d57743)) - **metal-agent:** read only marked Secrets through sourceSecretRef (fixes [#1956](defilantech/LLMKube#1956)) ([eaecad3](defilantech/LLMKube@eaecad3)) - **metal:** limit the metal-agent's Service and EndpointSlice writes by admission policy ([#2319](defilantech/LLMKube#2319)) ([a8ea5bf](defilantech/LLMKube@a8ea5bf)) - **release:** publish a Homebrew cask for the metal-agent ([#2234](defilantech/LLMKube#2234)) ([1b29497](defilantech/LLMKube@1b29497)) - **router:** mirror requests to the other backends of a shadow rule (fixes [#2327](defilantech/LLMKube#2327)) ([40266ab](defilantech/LLMKube@40266ab)) ##### Bug Fixes - **agent:** pin the exact powermetrics argument vector in sudoers ([d28c55f](defilantech/LLMKube@d28c55f)) - **controller:** treat a ModelPool resident removed from spec.members as gone ([#2310](defilantech/LLMKube#2310)) ([87464ad](defilantech/LLMKube@87464ad)) - **foreman:** bound the Metal engine port read from the agent's EndpointSlice ([#2321](defilantech/LLMKube#2321)) ([75837e3](defilantech/LLMKube@75837e3)) - **metal-agent:** re-check a refused InferenceService on a backoff (fixes [#2329](defilantech/LLMKube#2329)) ([6f65b30](defilantech/LLMKube@6f65b30)) - **metal-agent:** re-issue an ingress certificate the Mac's clock no longer accepts ([7e7fbaa](defilantech/LLMKube@7e7fbaa)) - **metal-agent:** read sourceSecretRef from the Model's namespace ([eaecad3](defilantech/LLMKube@eaecad3)) - **metal-agent:** retry a start that hit a transient source Secret read error ([5e13d5c](defilantech/LLMKube@5e13d5c)) - **metal-agent:** retry a transient remote size probe failure without refusing (fixes [#2326](defilantech/LLMKube#2326)) ([5e13d5c](defilantech/LLMKube@5e13d5c)) - **metal:** bind MetalNode Reachable to the Mac with a UID-named certificate (refs [#2265](defilantech/LLMKube#2265)) ([7e7fbaa](defilantech/LLMKube@7e7fbaa)) - **metal:** expire a MetalNode Reachable result no check has renewed (refs [#2265](defilantech/LLMKube#2265)) ([7e7fbaa](defilantech/LLMKube@7e7fbaa)) - **metal:** read MetalNode verifier finish times on the controller's clock (refs [#2265](defilantech/LLMKube#2265)) ([7e7fbaa](defilantech/LLMKube@7e7fbaa)) - **metal:** trust only the reported certificate in the MetalNode dial-back ([7e7fbaa](defilantech/LLMKube@7e7fbaa)) - preserve Foreman audit and transcript records per attempt ([#2311](defilantech/LLMKube#2311)) ([3e838f9](defilantech/LLMKube@3e838f9)) - **router:** dispatch only to backends with a rule's required capabilities (fixes [#2328](defilantech/LLMKube#2328)) ([e533192](defilantech/LLMKube@e533192)) - **router:** honor the weighted rule strategy on the Proxy data plane (fixes [#2325](defilantech/LLMKube#2325)) ([7b34a43](defilantech/LLMKube@7b34a43)) ##### Documentation - **macos:** explain why Login Items lists the agent as Defilan Technologies LLC ([#2307](defilantech/LLMKube#2307)) ([5b5e17e](defilantech/LLMKube@5b5e17e)) - move Foreman stream guide before references ([#2322](defilantech/LLMKube#2322)) ([d856cb1](defilantech/LLMKube@d856cb1)) - **upgrade:** a MetalNode is Reachable only when the Mac's metal-agent reports an ingress certificate naming its MetalNode. After upgrading the controller, every Mac shows Reachable=False with reason AgentUpgradeRequired, and its hostIP is cleared, until its metal-agent is upgraded too; upgrade the agents promptly. Nothing routes to hostIP yet, so serving is unaffected. The dial-back verifier image's curl must be 7.57 or later with a TLS library that accepts a self-signed certificate as its own trust anchor; the default curlimages/curl image does. ([7e7fbaa](defilantech/LLMKube@7e7fbaa)) - **upgrade:** Metal engines now inherit only PATH, HOME, USER, LOGNAME, TMPDIR, LANG, TZ, LC\_\*, HF_HOME and HF_HUB_CACHE from the agent's environment, plus the variables the agent sets for each runtime. If an engine relies on another variable from the agent's launchd environment (for example a GGML or MLX tuning variable, or proxy settings for llama-server's own `-hf` download), list it in `--engine-env-passthrough`. Engines adopted from a previous agent process keep their old environment until they restart; run `llmkube-metal-agent stop-engines` after the bootout to apply it at once. ([6d4fba0](defilantech/LLMKube@6d4fba0)) - **upgrade:** on the Proxy data plane, a ModelRouter rule with match.requiredCapabilities now sends requests only to the backends that advertise all of them, under every strategy. A primary-fallback rule no longer sends to, or falls back to, a backend that lacks them; when every capable backend fails, the request fails instead of landing on an incapable one. ([e533192](defilantech/LLMKube@e533192)) - **upgrade:** on the Proxy data plane, ModelRouter rules with strategy: shadow now send a copy of each request to the rule's other backends, including external ones, which bill those requests. A shadow rule with one backend now fails validation, and one that mirrors to a ModelPool member fails the compile and keeps the previous config. ([40266ab](defilantech/LLMKube@40266ab)) - **upgrade:** on the Proxy data plane, ModelRouter rules with strategy: weighted now split requests by backend weight instead of sending all of them to the first backend. weight: 0 drains a backend, matching the Gateway data plane. A weighted rule whose backends cannot take traffic now fails validation, and a weighted rule across two members of one ModelPool fails the compile and keeps the previous config. ([7b34a43](defilantech/LLMKube@7b34a43)) - **upgrade:** the Metal agent now reads a Model's spec.sourceSecretRef only when the Secret is labeled inference.llmkube.dev/model-source-credentials=true; otherwise the InferenceService is refused with SourceSecretNotAllowed. Before upgrading the agent, label each credentials Secret your Metal Models use: `kubectl label secret <name> -n <namespace> inference.llmkube.dev/model-source-credentials=true`. For one release, `--allow-unlabeled-source-secrets` keeps the old behavior with a warning; it is deprecated. A Hugging Face source whose Secret cannot be read is now refused instead of downloading anonymously. ([eaecad3](defilantech/LLMKube@eaecad3)) - **upgrade:** the powermetrics sudoers rule now names the exact command line the agent runs, with the sampling interval rendered in. Upgrading the agent does not change an installed /etc/sudoers.d/llmkube-powermetrics: on hosts that installed it, re-run `make install-powermetrics-sudo` (with `APPLE_POWER_INTERVAL_MS=<ms>` if the agent uses a non-default `--apple-power-interval`), then check with `sudo -ln | grep powermetrics` that the rule ends in `-i` followed by a number. ([d28c55f](defilantech/LLMKube@d28c55f)) ### [`v0.10.14`](https://github.com/defilantech/LLMKube/blob/HEAD/CHANGELOG.md#01014-2026-10-29) [Compare Source](defilantech/LLMKube@v0.10.13...v0.10.14) ##### Features - **cli:** send the endpoint API key from llmkube benchmark ([#2297](defilantech/LLMKube#2297)) ([df2e008](defilantech/LLMKube@df2e008)) - **router:** add spec.proxy.drainTimeout for the router-proxy shutdown ([#2304](defilantech/LLMKube#2304)) ([113e009](defilantech/LLMKube@113e009)) - **router:** opt-in client address and caller header on the audit log ([#2288](defilantech/LLMKube#2288)) ([4d1faac](defilantech/LLMKube@4d1faac)) ##### Bug Fixes - **controller:** keep a ModelPool resident the router re-activates during a reclaim ([#2290](defilantech/LLMKube#2290)) ([857daf2](defilantech/LLMKube@857daf2)) - **controller:** remove a key dropped from podLabels or podAnnotations from the pods ([#2289](defilantech/LLMKube#2289)) ([ca5df08](defilantech/LLMKube@ca5df08)) - **foreman:** route the remaining GitHub-only paths through the forge seams ([#2291](defilantech/LLMKube#2291)) ([a333270](defilantech/LLMKube@a333270)) - **metrics:** each binary serves only the metrics it records ([#2296](defilantech/LLMKube#2296)) ([de8ca21](defilantech/LLMKube@de8ca21)) ##### Documentation - **modelpool:** leave member replicas out of GitOps manifests ([#2300](defilantech/LLMKube#2300)) ([bcbc357](defilantech/LLMKube@bcbc357)) ### [`v0.10.13`](https://github.com/defilantech/LLMKube/blob/HEAD/CHANGELOG.md#01013-2026-10-28) [Compare Source](defilantech/LLMKube@v0.10.12...v0.10.13) ##### Features - **agent:** add the llmkube-metal-agent stop-engines command ([2cd2ec9](defilantech/LLMKube@2cd2ec9)) - **agent:** keep Metal engines serving across metal-agent restarts and adopt them on startup (refs [#2079](defilantech/LLMKube#2079)) ([2cd2ec9](defilantech/LLMKube@2cd2ec9)) - **release:** sign and notarize the macOS release binaries ([#2233](defilantech/LLMKube#2233)) ([f5ff962](defilantech/LLMKube@f5ff962)) - **router:** podLabels and podAnnotations for router-proxy pods ([#2285](defilantech/LLMKube#2285)) ([a736e18](defilantech/LLMKube@a736e18)) - **router:** publish per-backend modalities and context length on /v1/models ([#2249](defilantech/LLMKube#2249)) ([2887f54](defilantech/LLMKube@2887f54)) ##### Bug Fixes - apply CRDs server-side in make install and make deploy ([#2275](defilantech/LLMKube#2275)) ([edaa4f7](defilantech/LLMKube@edaa4f7)) - **chart:** leave --init-container-user off when its value is unset ([#2286](defilantech/LLMKube#2286)) ([ac541db](defilantech/LLMKube@ac541db)) - **cli:** pin only single-writer claims and make the cache pod image configurable ([#2178](defilantech/LLMKube#2178)) ([2fe250b](defilantech/LLMKube@2fe250b)) - **controller:** annotate router-proxy pods for scraping under --emit-scrape-annotations ([#2268](defilantech/LLMKube#2268)) ([07fb7f4](defilantech/LLMKube@07fb7f4)) - **controller:** chown the cache root to the init container's uid when fsGroup is off ([#2273](defilantech/LLMKube#2273)) ([c9d86ad](defilantech/LLMKube@c9d86ad)) - **controller:** create no HPA for multiNode services ([#2272](defilantech/LLMKube#2272)) ([c8d6f76](defilantech/LLMKube@c8d6f76)) - **controller:** delete FUSE stager Jobs in the foreground without the feature-gated field ([#2269](defilantech/LLMKube#2269)) ([e439f60](defilantech/LLMKube@e439f60)) - **controller:** make FUSE model-cache staging safe across stagers and serving pods ([#2276](defilantech/LLMKube#2276)) ([6c7c1fe](defilantech/LLMKube@6c7c1fe)) - **controller:** point the router-proxy Service at the port the proxy listens on ([#2280](defilantech/LLMKube#2280)) ([1c69118](defilantech/LLMKube@1c69118)) - **controller:** run the stall-probe tests on a fake clock ([#2267](defilantech/LLMKube#2267)) ([661856a](defilantech/LLMKube@661856a)) - **router:** tell client cancellations, upstream errors and timeouts apart ([#2284](defilantech/LLMKube#2284)) ([3a2ea74](defilantech/LLMKube@3a2ea74)) - use vLLM's real TurboQuant KV cache types in the sample and docs ([#2270](defilantech/LLMKube#2270)) ([3b4ffb6](defilantech/LLMKube@3b4ffb6)) ##### Documentation - **upgrade:** metal-agent engines now keep running when the agent stops and the next agent adopts them. To roll back to an older agent, run `launchctl bootout gui/$(id -u)/com.llmkube.metal-agent` and then `llmkube-metal-agent stop-engines` with the current binary before installing the older one, or every model is loaded twice. `make uninstall-metal-agent` stops the engines for you. Adopted engines keep the flags they started with; run stop-engines after the bootout to apply new defaults. ([2cd2ec9](defilantech/LLMKube@2cd2ec9)) ### [`v0.10.12`](https://github.com/defilantech/LLMKube/blob/HEAD/CHANGELOG.md#01012-2026-10-27) [Compare Source](defilantech/LLMKube@v0.10.11...v0.10.12) ##### Features - **api:** add MetalNode, verified by a controller dial-back, with the agent reporting through its Lease ([#2235](defilantech/LLMKube#2235)) ([fccc1bc](defilantech/LLMKube@fccc1bc)) - **cli:** add llmkube metal join and leave, with agent token renewal ([#2246](defilantech/LLMKube#2246)) ([b1f6602](defilantech/LLMKube@b1f6602)) - **cli:** add llmkube metal token create and metal remove ([#2236](defilantech/LLMKube#2236)) ([f54414a](defilantech/LLMKube@f54414a)) ##### Bug Fixes - **controller:** keep a ModelPool activation while its member is held at zero ([#2241](defilantech/LLMKube#2241)) ([8fc4143](defilantech/LLMKube@8fc4143)) - **controller:** map sglangConfig.enablePrefixCaching onto --disable-radix-cache ([#2259](defilantech/LLMKube#2259)) ([d499aa4](defilantech/LLMKube@d499aa4)) - **controller:** stage under the cache lock when the init image's stat has no -f ([#2260](defilantech/LLMKube#2260)) ([52241de](defilantech/LLMKube@52241de)) - **router:** list each backend on /v1/models under the id BackendNameMatch routes ([#2257](defilantech/LLMKube#2257)) ([4627cfc](defilantech/LLMKube@4627cfc)) - **router:** report moves off a quarantined pod as reason="quarantined" instead of reason="not_ready" ([faa7a5f](defilantech/LLMKube@faa7a5f)) - **router:** rewrite vLLM and SGLang backends to the name they serve ([#2258](defilantech/LLMKube#2258)) ([3d39111](defilantech/LLMKube@3d39111)) - validate multi-file cache staging on shared volumes ([#2185](defilantech/LLMKube#2185)) ([e6eab85](defilantech/LLMKube@e6eab85)) ### [`v0.10.11`](https://github.com/defilantech/LLMKube/blob/HEAD/CHANGELOG.md#01011-2026-10-25) [Compare Source](defilantech/LLMKube@v0.10.10...v0.10.11) ##### Features - **api:** accept fp8, float16 and bfloat16 in vllmConfig.kvCacheDtype ([#2227](defilantech/LLMKube#2227)) ([31eb943](defilantech/LLMKube@31eb943)) - **api:** add spec.imagePullPolicy to InferenceService ([#2208](defilantech/LLMKube#2208)) ([ec54d28](defilantech/LLMKube@ec54d28)) - **controller:** move the SGLang defaults to v0.5.22 and report removed attention backends ([#2221](defilantech/LLMKube#2221)) ([33f9bb1](defilantech/LLMKube@33f9bb1)) - **controller:** move the vLLM default to v0.32.0 and check every flag against it ([#2223](defilantech/LLMKube#2223)) ([0144a4c](defilantech/LLMKube@0144a4c)) - **controller:** run every llama.cpp default from one pinned build, b11854 ([#2220](defilantech/LLMKube#2220)) ([052840e](defilantech/LLMKube@052840e)) - **router:** weigh LoadAware placement by estimated prefill tokens in flight ([#2193](defilantech/LLMKube#2193)) ([a33735c](defilantech/LLMKube@a33735c)) ##### Bug Fixes - **controller:** render vllmConfig.speculative as --speculative-config ([#2210](defilantech/LLMKube#2210)) ([191dcbd](defilantech/LLMKube@191dcbd)) - **controller:** report sglangConfig tool-call parsers SGLang does not register ([#2228](defilantech/LLMKube#2228)) ([90fed2e](defilantech/LLMKube@90fed2e)) ### [`v0.10.10`](https://github.com/defilantech/LLMKube/blob/HEAD/CHANGELOG.md#01010-2026-10-24) [Compare Source](defilantech/LLMKube@v0.10.9...v0.10.10) ##### Features - **controller:** add --model-cache-prep to skip the root cache prep ([#2206](defilantech/LLMKube#2206)) ([8bcb973](defilantech/LLMKube@8bcb973)) - **router:** compile dataPlane: AgentGateway to AgentgatewayModel ([#2162](defilantech/LLMKube#2162)) ([26b1ef4](defilantech/LLMKube@26b1ef4)) ##### Bug Fixes - **controller:** report a prefetch as downloading while its downloader runs ([#2207](defilantech/LLMKube#2207)) ([7699514](defilantech/LLMKube@7699514)) - **router:** let every waiter of a failed ModelPool swap see its outcome ([#2203](defilantech/LLMKube#2203)) ([b95db62](defilantech/LLMKube@b95db62)) ### [`v0.10.9`](https://github.com/defilantech/LLMKube/blob/HEAD/CHANGELOG.md#0109-2026-10-23) [Compare Source](defilantech/LLMKube@v0.10.8...v0.10.9) ##### Features - **api:** affinity and topology spread for the router proxy, rolling without surge under hard placement ([#2200](defilantech/LLMKube#2200)) ([3a61700](defilantech/LLMKube@3a61700)) - **router:** proxy the Anthropic Messages API like chat completions ([#2201](defilantech/LLMKube#2201)) ([86223aa](defilantech/LLMKube@86223aa)) ##### Bug Fixes - **router:** retry stale keep-alive connections and quarantine a pod only after two failures ([#2197](defilantech/LLMKube#2197)) ([b279053](defilantech/LLMKube@b279053)) ### [`v0.10.8`](https://github.com/defilantech/LLMKube/blob/HEAD/CHANGELOG.md#0108-2026-10-23) [Compare Source](defilantech/LLMKube@v0.10.7...v0.10.8) ##### Features - **api:** type the vLLM tool-call and reasoning parsers and admit GLM on SGLang ([#2168](defilantech/LLMKube#2168)) ([fd64c72](defilantech/LLMKube@fd64c72)) - **controller:** add InferenceService spec.hubOffline for serving pods ([#2188](defilantech/LLMKube#2188)) ([a911cb2](defilantech/LLMKube@a911cb2)) - **controller:** keep 3 old ReplicaSets for LLMKube-managed Deployments by default ([#2182](defilantech/LLMKube#2182)) ([5e1722f](defilantech/LLMKube@5e1722f)) - **router:** honour endpoint nodePort and externalTrafficPolicy, add proxy runtimeClassName ([#2183](defilantech/LLMKube#2183)) ([a8f9e57](defilantech/LLMKube@a8f9e57)) - **router:** on pools under 3 pods, move a session only when its pod is KV-full ([#2192](defilantech/LLMKube#2192)) ([f9b934c](defilantech/LLMKube@f9b934c)) - **router:** remember LoadAware sessions and move them only under real pressure ([#2180](defilantech/LLMKube#2180)) ([1e8ce5a](defilantech/LLMKube@1e8ce5a)) ##### Bug Fixes - **router:** answer a ModelPool swap past swapBudget with 503 and Retry-After ([#2181](defilantech/LLMKube#2181)) ([7c0d84e](defilantech/LLMKube@7c0d84e)) - **router:** release request bodies once sent and raise the proxy's default memory limit ([#2196](defilantech/LLMKube#2196)) ([140fd6b](defilantech/LLMKube@140fd6b)) ### [`v0.10.7`](https://github.com/defilantech/LLMKube/blob/HEAD/CHANGELOG.md#0107-2026-10-23) [Compare Source](defilantech/LLMKube@v0.10.6...v0.10.7) ##### Features - **chart:** install the metal-agent RBAC from the Helm chart ([#2165](defilantech/LLMKube#2165)) ([521e502](defilantech/LLMKube@521e502)) - **controller:** roll a multi-replica GPU service one pod at a time ([#2164](defilantech/LLMKube#2164)) ([c4073cf](defilantech/LLMKube@c4073cf)) - **router:** load-aware, session-sticky placement for endpoint backends ([#2166](defilantech/LLMKube#2166)) ([74cfb0e](defilantech/LLMKube@74cfb0e)) - **router:** the router proxy tracks endpoint backends' pods itself ([#2159](defilantech/LLMKube#2159)) ([57070e1](defilantech/LLMKube@57070e1)) ### [`v0.10.6`](https://github.com/defilantech/LLMKube/blob/HEAD/CHANGELOG.md#0106-2026-10-22) [Compare Source](defilantech/LLMKube@v0.10.5...v0.10.6) ##### Features - **controller:** prefetch per cache claim, gated by the InferenceService ([#2138](defilantech/LLMKube#2138)) ([d76f398](defilantech/LLMKube@d76f398)) - **router:** per-backend Endpoint Picker for dataPlane: AgentGateway ([#2157](defilantech/LLMKube#2157)) ([c1bbe28](defilantech/LLMKube@c1bbe28)) ##### Bug Fixes - **cli:** read the model cache from its own pod in cache list ([#2154](defilantech/LLMKube#2154)) ([996abe9](defilantech/LLMKube@996abe9)) - **controller:** refuse to repoint a same-name Service ([#2149](defilantech/LLMKube#2149)) ([7965a1b](defilantech/LLMKube@7965a1b)) - **hf-listing:** read a tree size only when it is a whole number of bytes ([#2153](defilantech/LLMKube#2153)) ([6e6723d](defilantech/LLMKube@6e6723d)) - **model:** report the staged total instead of status.size "0" ([#2150](defilantech/LLMKube#2150)) ([f6710e7](defilantech/LLMKube@f6710e7)) ### [`v0.10.5`](https://github.com/defilantech/LLMKube/blob/HEAD/CHANGELOG.md#0105-2026-10-21) [Compare Source](defilantech/LLMKube@v0.10.4...v0.10.5) ##### Features - add make gate and make test-affected, a cached parallel round gate ([#2106](defilantech/LLMKube#2106)) ([e971675](defilantech/LLMKube@e971675)) - **agent:** admit Metal models against memory already committed to running engines (closes [#2080](defilantech/LLMKube#2080)) ([d2d2d43](defilantech/LLMKube@d2d2d43)) - **agent:** record each engine the metal-agent starts in the state directory (refs [#2079](defilantech/LLMKube#2079)) ([0224d51](defilantech/LLMKube@0224d51)) - **controller:** add spec.startupTimeout for a bigger startup probe budget ([#2129](defilantech/LLMKube#2129)) ([df0e9cd](defilantech/LLMKube@df0e9cd)) - **controller:** set fsGroupChangePolicy OnRootMismatch with the default fsGroup (closes [#2058](defilantech/LLMKube#2058)) ([2bd4538](defilantech/LLMKube@2bd4538)) ##### Bug Fixes - **agent:** escalate to SIGKILL when stopping an engine the agent did not spawn ([0224d51](defilantech/LLMKube@0224d51)) - **agent:** keep the pressure block when an eviction teardown completed ([#2107](defilantech/LLMKube#2107)) ([fd9ab77](defilantech/LLMKube@fd9ab77)) - **agent:** replace a hung oMLX daemon instead of starting a second one ([#2105](defilantech/LLMKube#2105)) ([a16c348](defilantech/LLMKube@a16c348)) - **agent:** restart services evicted under memory pressure once memory recovers ([#2093](defilantech/LLMKube#2093)) ([3db0ecc](defilantech/LLMKube@3db0ecc)) - answer 400 for request bodies that are not valid JSON ([#2075](defilantech/LLMKube#2075)) ([d9826ff](defilantech/LLMKube@d9826ff)) - **chart:** fail fast on shared cache without default storageclass and support existingClaim ([#2016](defilantech/LLMKube#2016)) ([#2114](defilantech/LLMKube#2114)) ([c4f7f3d](defilantech/LLMKube@c4f7f3d)) - **chart:** gate llama.cpp InferenceAdmissionStalled on decode progress ([#2065](defilantech/LLMKube#2065)) ([5b2f45d](defilantech/LLMKube@5b2f45d)) - **cli:** make cache clear and delete --purge-cache remove entries ([#2134](defilantech/LLMKube#2134)) ([2b668e2](defilantech/LLMKube@2b668e2)) - **controller:** keep downloader credentials off curl's command line ([#2098](defilantech/LLMKube#2098)) ([2207c2e](defilantech/LLMKube@2207c2e)) - **controller:** mount a memory-backed /dev/shm for multi-process vLLM, SGLang and TGI (closes [#2099](defilantech/LLMKube#2099)) ([a68c98f](defilantech/LLMKube@a68c98f)) - **controller:** refuse to overwrite or recreate a same-name Deployment ([#2096](defilantech/LLMKube#2096)) ([b90c8aa](defilantech/LLMKube@b90c8aa)) - **model:** canonicalize HF repo IDs so a trailing slash stages ([#2128](defilantech/LLMKube#2128)) ([9289ac6](defilantech/LLMKube@9289ac6)) - **modelpool:** stop swap thrash on reversals, idle-probe blindness, and abandoned loads ([#2130](defilantech/LLMKube#2130)) ([#2135](defilantech/LLMKube#2135)) ([e7dee43](defilantech/LLMKube@e7dee43)) - **model:** serialize downloaders that share a cache directory ([#2115](defilantech/LLMKube#2115)) ([13eca01](defilantech/LLMKube@13eca01)) - **sglang:** keep endpoint API keys out of process arguments ([#2122](defilantech/LLMKube#2122)) ([7fdae50](defilantech/LLMKube@7fdae50)) - size metal-agent KV cache estimate from GQA attention shape ([#2102](defilantech/LLMKube#2102)) ([a119234](defilantech/LLMKube@a119234)) ##### Documentation - cover per-request balancing in the ModelRouter page ([#2095](defilantech/LLMKube#2095)) ([9ec4766](defilantech/LLMKube@9ec4766)) - **model-cache:** correct the CephFS fsGroupPolicy claim and the PSA advice ([2bd4538](defilantech/LLMKube@2bd4538)) - state differentiation, audience, and project status up front ([#2100](defilantech/LLMKube#2100)) ([ee27e20](defilantech/LLMKube@ee27e20)) - **upgrade:** every InferenceService that uses the default pod security context rolls once on upgrade, because the generated pod template gains `fsGroupChangePolicy: OnRootMismatch` (a full model reload for large models) ([2bd4538](defilantech/LLMKube@2bd4538)) - **upgrade:** single-node vLLM, SGLang and TGI services with more than one GPU (or a tensor, pipeline or data-parallel size above 1) that do not mount /dev/shm themselves roll once on upgrade, because the generated pod template gains a memory-backed /dev/shm volume ([a68c98f](defilantech/LLMKube@a68c98f)) - **upgrade:** the metal-agent now admits a model only if it fits in the Mac's budget minus the memory committed to models already running, so a Mac whose models add up to more than its budget refuses the extra ones; a Model memoryBudget or memoryFraction above the Mac's budget is clamped to it unless the agent runs with --allow-model-memory-budget-override ([d2d2d43](defilantech/LLMKube@d2d2d43)) - **upgrade:** the metal-agent now writes a record per engine under --state-dir/engines (both relay and legacy mode) and fails an engine start when it cannot write it; make sure the state directory (default \~/Library/Application Support/llmkube/metal-agent) is writable by the agent's user ([0224d51](defilantech/LLMKube@0224d51)) ### [`v0.10.4`](https://github.com/defilantech/LLMKube/blob/HEAD/CHANGELOG.md#0104-2026-10-16) [Compare Source](defilantech/LLMKube@v0.10.3...v0.10.4) ##### Features - **agent:** cap the bytes a single Model download may write ([#2088](defilantech/LLMKube#2088)) ([3852411](defilantech/LLMKube@3852411)) - **api:** endpoint API key and externalTrafficPolicy for InferenceService ([#2043](defilantech/LLMKube#2043)) ([27a9214](defilantech/LLMKube@27a9214)) - **controller:** detect a stalled watch path with a lease canary, bounded reconciles and stall alerts ([#2072](defilantech/LLMKube#2072)) ([d00e0cf](defilantech/LLMKube@d00e0cf)) - **controller:** detect keyed pool siblings and vLLM env-var keys in ModelRouter ([#2068](defilantech/LLMKube#2068)) ([78889f0](defilantech/LLMKube@78889f0)) - **controller:** detect ModelRouter backends keyed through --api-key on the command line ([#2063](defilantech/LLMKube#2063)) ([3674acc](defilantech/LLMKube@3674acc)) - **controller:** stage HF mirror repos with spec.files and expand globs from the repo listing ([#2071](defilantech/LLMKube#2071)) ([9e5d7d3](defilantech/LLMKube@9e5d7d3)) ##### Bug Fixes - **agent:** health restarts stop the unhealthy engine first, wait for repeated failures, and back off ([#2090](defilantech/LLMKube#2090)) ([407e02e](defilantech/LLMKube@407e02e)) - **agent:** keep a Metal service running when its Model lookup fails transiently ([#2089](defilantech/LLMKube#2089)) ([e22eeab](defilantech/LLMKube@e22eeab)) - **controller:** merge DRA pod scheduling fields and reject multiNode with resourceClaims ([#2062](defilantech/LLMKube#2062)) ([9319745](defilantech/LLMKube@9319745)) - no device-plugin GPU limit when a Model uses DRA resourceClaims ([#2060](defilantech/LLMKube#2060)) ([8c3e434](defilantech/LLMKube@8c3e434)) ##### Documentation - APISIX in front of LLMKube ([#2053](defilantech/LLMKube#2053)) ([16286c6](defilantech/LLMKube@16286c6)) - **metal:** fix install commands and update the quickstart for relay mode ([#2087](defilantech/LLMKube#2087)) ([8bc7c6f](defilantech/LLMKube@8bc7c6f)) - use an extension-less link in the GPU fault guide ([#2056](defilantech/LLMKube#2056)) ([780e710](defilantech/LLMKube@780e710)) ### [`v0.10.3`](https://github.com/defilantech/LLMKube/blob/HEAD/CHANGELOG.md#0103-2026-10-13) [Compare Source](defilantech/LLMKube@v0.10.2...v0.10.3) ##### Features - **api:** accept SGLang's Blackwell attention backends; flag flash_attn ([#2033](defilantech/LLMKube#2033)) ([2385fd4](defilantech/LLMKube@2385fd4)) - **chart:** alert on GPU faults the device plugin ignores ([#2031](defilantech/LLMKube#2031)) ([e4697ef](defilantech/LLMKube@e4697ef)) - **controller:** add opt-in vLLM engine-stall liveness probe ([#2022](defilantech/LLMKube#2022)) ([429434a](defilantech/LLMKube@429434a)) - **controller:** apply the Model's HF_ENDPOINT mirror to runtime downloads ([#2039](defilantech/LLMKube#2039)) ([cbececa](defilantech/LLMKube@cbececa)) - **controller:** stall-aware liveness for SGLang pods; share the detector ([#2029](defilantech/LLMKube#2029)) ([212b014](defilantech/LLMKube@212b014)) - **router:** serve POST /v1/completions on the router-proxy ([#2019](defilantech/LLMKube#2019)) ([1724a02](defilantech/LLMKube@1724a02)) ##### Bug Fixes - **controller:** bind SGLang dual-stack so it can become Ready on IPv4 clusters ([#2028](defilantech/LLMKube#2028)) ([7d87c7e](defilantech/LLMKube@7d87c7e)) - **controller:** bound multiNode recreates and keep the failure evidence ([#1996](defilantech/LLMKube#1996)) ([3331610](defilantech/LLMKube@3331610)) - **controller:** refuse to overwrite or delete a same-name HPA ([#2010](defilantech/LLMKube#2010)) ([09affe5](defilantech/LLMKube@09affe5)) - **controller:** report a missing CA ConfigMap instead of wedging pods ([#1995](defilantech/LLMKube#1995)) ([c624e02](defilantech/LLMKube@c624e02)) - **controller:** report AutoscalingReady=True when the HPA is managed ([#2012](defilantech/LLMKube#2012)) ([b877ee5](defilantech/LLMKube@b877ee5)) - **controller:** split reconcileMultiNodeGroup under the gocyclo limit ([#2000](defilantech/LLMKube#2000)) ([023a5c4](defilantech/LLMKube@023a5c4)) - **controller:** stop the prefetch status loop while the CA ConfigMap is missing ([#2037](defilantech/LLMKube#2037)) ([b0b3cc6](defilantech/LLMKube@b0b3cc6)) - **foreman-agent:** make the repo package's git env hermetic and forward GIT_CONFIG\_\* ([#2014](defilantech/LLMKube#2014)) ([62837d9](defilantech/LLMKube@62837d9)) - **router:** compile AgentGateway mode against Gateway API Inference Extension v1 ([#1990](defilantech/LLMKube#1990)) ([ce817ba](defilantech/LLMKube@ce817ba)) - **router:** refuse to adopt same-name router-proxy objects ([#2008](defilantech/LLMKube#2008)) ([e8f190a](defilantech/LLMKube@e8f190a)) - **router:** remove router-proxy objects for routers not in Proxy mode ([#2006](defilantech/LLMKube#2006)) ([c04f824](defilantech/LLMKube@c04f824)) - **router:** stop truncating streams at the header timeout; quarantine stalled streams ([#2026](defilantech/LLMKube#2026)) ([c4335c9](defilantech/LLMKube@c4335c9)) - **router:** time only upstream silence in the stream idle guard ([#2035](defilantech/LLMKube#2035)) ([933426c](defilantech/LLMKube@933426c)) ##### Documentation - **air-gapped:** cover vLLM and SGLang safetensors checkpoints ([#1993](defilantech/LLMKube#1993)) ([4d3fa70](defilantech/LLMKube@4d3fa70)) - replace deprecated Strix Halo GTT sizing with TTM parameters ([#2003](defilantech/LLMKube#2003)) ([09d8d73](defilantech/LLMKube@09d8d73)) </details> --- ### Configuration 📅 **Schedule**: (UTC) - Branch creation - At any time (no schedule defined) - Automerge - At any time (no schedule defined) 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] If you want to rebase/retry this PR, check this box --- This PR has been generated by [Mend Renovate CLI](https://github.com/renovatebot/renovate).  Reviewed-on: https://git.erwanleboucher.dev/eleboucher/homelab/pulls/2331

Defilan added 7 commits October 24, 2026 02:51

Defilan added 2 commits October 24, 2026 11:36

Defilan mentioned this pull request Oct 24, 2026

[STORY] Bump the vLLM default image to v0.32.0 with flag parity checked against that release #2212

Closed

Defilan merged commit 191dcbd into defilantech:main Oct 24, 2026
31 checks passed

github-actions Bot mentioned this pull request Oct 24, 2026

chore: release 0.10.11 #2222

Merged

Navigation Menu

Repository navigation

fix(controller): render vllmConfig.speculative as --speculative-config - #2210

fix(controller): render vllmConfig.speculative as --speculative-config#2210
Defilan merged 9 commits into
defilantech:mainfrom
Defilan:fix/vllmconfig-speculative-renders-speculative-model-2190

Defilan commented Oct 24, 2026

Uh oh!

codecov Bot commented Oct 24, 2026 •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

Defilan commented Oct 24, 2026

What

Why

How

AI assistance

Checklist

Uh oh!

codecov Bot commented Oct 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

codecov Bot commented Oct 24, 2026 •

edited

Loading