[Defilan]

What

The metal-agent's pre-flight memory check sizes the KV cache from the model's full embedding width. This change sizes it from the GGUF attention shape when the file provides one: per layer, context x head_count_kv x (key_length x bytesK + value_length x bytesV). For Llama 3.1 8B at 131072 context with f16 K/V the estimate drops from about 64 GiB to exactly 16 GiB, so admission stops refusing models that fit. The charge can also rise: head-dim models whose heads times head_dim exceeds the embedding width now get the larger, correct number.

Why

Fixes #2085. The old formula over-counts grouped-query-attention models by head_count / head_count_kv (4x for Llama 3.1 8B), which blocked GQA models at the long contexts the catalog sets by default. The error was in the fail-closed direction (refusing, not over-admitting), but it blocked real workloads.

How

• pkg/gguf gains three accessors (HeadCountKV, KeyLength, ValueLength) over one shared scalar reader; absent, zero, and array-valued keys read as 0 (Jamba and Nemotron-H store head_count_kv per layer as an array, so 0 means unknown, never "substitute something").
• Model.status.gguf gains headCountKV, keyLength, valueLength (optional uint64); all three controller parse paths (local file, remote http(s), s3) fill them through one shared helper.
• The estimator uses the GQA formula when the shape is usable and falls back to the historical embedding-width formula otherwise. A missing key/value length is derived as embeddingSize / headCount, independently per side: that is llama.cpp's own allocation default for an absent length (its generic loader sets each head dim to n_embd/n_head and overrides each only when the GGUF carries that key), so the derived cell is what the server actually allocates. I range-read live HF headers to confirm the split this handles: Llama 3.1 8B and Qwen 2.5 7B GGUFs carry head_count_kv but no lengths, Qwen 3 and Gemma 3 carry all three.
• DeepSeek GGUFs are charged their stated head dims on both layouts, checked against llama.cpp source. llama.cpp's KV cache allocates K as n_head_kv x key_length per layer and skips V only when is_mla(), which requires both *_mla GGUF keys (src/llama-hparams.cpp:298-302, src/models/deepseek2.cpp:16-17). Older deepseek2 conversions (16 KV heads, key 192, value 128, no *_mla keys; shape verified against live GGUF headers) run the non-MLA path, so the charge is exact where the embedding-width formula under-charges 1.25x (V2-Lite at 131072 f16: 36,238,786,560 charged vs 28,991,029,248 under the old formula). Current-converter MLA-layout files (one KV head, key 576, value 512, both derived from kv_lora_rank) are over-charged about 1.9x because llama-server stores only the compressed key cache there; over-charging is the safe direction for a pre-flight check, and far closer than the old formula's 7x on the same files.
• Both directions are pinned at the admission path: a Llama-shaped model that the embedding-width estimate refused is admitted with GQA accounting, and a Gemma-shaped model (16 heads x 256 dims against a 3072 embedding, about 1.33x higher) that the old estimate admitted is refused with its attention shape known. A model admitted today can be refused after this change; that is the estimate becoming correct, not a regression.
• The element and byte arithmetic is overflow-guarded on every multiply in both branches, and each branch bounds its byte total so the float64 sum stays below 2^63 for every cache type (the f32 corner previously could round past the uint64 range).
• --ctx-size is the process-wide pool: llama-server shares it across --parallel slots rather than multiplying it, so the estimate takes no slot factor. This is pinned in the estimator doc comment and both docs.
• Accepted staleness: Models whose Status.GGUF was written by an older controller keep the embedding-width estimate until the metadata is re-parsed. The re-parse triggers are the next download, or any spec change when the Model's Available condition records an older generation than the spec; editing a local source's file on disk does nothing under the default IfNotPresent policy, while refreshPolicy: OnChange treats a size/mtime change as drift and re-downloads, which re-parses. A one-time status refresh for Models parsed before this change is deliberately not part of it; no ticket tracks that yet.
• Known over-charges, documented in both docs pages: sliding-window models (Gemma 3) are charged full-attention dims for every layer, and DeepSeek MLA-layout files are charged their stated head dims even though llama-server stores only the compressed key cache; older deepseek2 conversions with plain head dims are charged exactly what llama-server allocates.

Verification

• make test (49 packages, including parser accessor, estimator-table, producer-bytes parse-to-estimate, admission-path, and envtest coverage), make lint, GOTOOLCHAIN=go1.26.9 make lint-all (both GOOS), make test-chart, make validate-samples, and an idempotent make manifests generate + make chart-crds all pass locally. Every behavior change has a falsification-backed test: each guard was reverted during development to confirm its test fails, including the deepseek2 layout rows, the legacy byte bound, the per-side derive, and the staleness gate.

Checklist

• Tests added/updated
• make test passes locally
• make lint passes locally
• Commit messages follow conventional commits
• All commits are signed off (git commit -s) per DCO
• AI assistance (if any) is disclosed above, per CONTRIBUTING.md
• Documentation updated (if user-facing change)

Assisted-by: GLM-5.3-Flash local agent (generated the code, tests, and docs across eleven review rounds; I reviewed the diff and ran make test, make lint, make lint-all, make test-chart, and make validate-samples locally).

Known follow-ups

• llmkube status / llmkube inspect do not print the new GGUF fields yet, so the inputs behind an admission decision are visible only via kubectl get model -o yaml.
• A behavioral e2e pin of the --ctx-size shared-pool reading (start llama-server with -c 8192 --parallel 4, read the reported KV buffer) would harden the no-slot-factor claim against llama.cpp version drift.
• The s3 parse path is covered by the shared helper but has no dedicated field-level test analogue of the local and remote cases.
• Metal local-path Models managed without a cache download get no GGUF parse today, so their admission uses the size heuristic; pre-existing, worth documenting separately.

[codecov]

Codecov Report

❌ Patch coverage is 89.02439% with 9 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
pkg/agent/memory.go	80.95%	4 Missing and 4 partials ⚠️
internal/controller/model_controller.go	94.44%	1 Missing ⚠️

📢 Thoughts on this report? Let us know!