Eval bug: with -np 2 and --kv-unified, prompt processing drops 42-54% from the second long request onward (single GPU, sequential requests, no spill, no speculation)

@alankila

Update — known cause, previously recorded as not affecting prompt processing.
This is the unified-KV limitation in the CUDA/HIP flash-attention kernels: they skip
only the tails of the KQ mask (KV_max), not all--INF blocks in the interior, so
a sequence pays for cells belonging to other (or released) sequences in the shared pool.
Metal has interior block skipping; Vulkan got it in #17186. Extending it to CUDA was
raised in #14924 (comment) and
as far as I can tell has not landed.

#19523 is the same root cause for token generation on Vulkan, and its reporter
states "And pp is not affected." On HIP, prompt processing is affected, by 42-54%.
Confirmed by A/B on --kv-unified (-41.8% unified vs -3.1% split, matched prompt) and
reproduced with upstream's own llama-batched-bench -kvu -tgs: S_PP 885.55 -> 644.39
across B=1,2,4 with -kvu versus 857.79 -> 833.20 with -no-kvu on stock master.
Details: #28495 (comment)

Workaround: --no-kv-unified (also found independently by @alankila on Strix Halo).
Note -c is then divided across slots.

Name and Version

Reproduced on stock master, using the official release binary
(llama-b10825-bin-win-rocm-10.0-x64.zip), nothing local in it:

$ llama-server --version
version: 0.4.0-dev (build 10825, commit 9e0e22059)
built with Clang 20.1.8 for Windows x86_64

It was originally found on a local branch off b10665 (RDNA prefill tuning, a
chat-template fix, and the FA hunks from PR #28102). Stock master turns out to be
worse — -54% there versus -42% on the patched build — so those patches partially
mask the effect rather than causing it. Both sets of numbers are below.

Operating systems

Windows (11 Pro, 10.0.26200)

GGML backends

HIP (ROCm runtime 10.0.0 from TheRock, gfx1201)

Hardware

AMD Radeon AI PRO R9700 — a single discrete card, no second GPU, no iGPU in use
for compute:

GPU: Radeon AI PRO R9700 (RDNA4, gfx1201, PCI device ID 0x7551), 32 GiB VRAM
Windows display driver 32.0.22042.14002 (2026-04-30)
ROCm runtime 10.0.0 (TheRock Windows build for gfx120X)
Host: 32 GiB system RAM

Note for anyone repeating this on the same card: the official win-rocm release zip
ships no rocBLAS/hipBLASLt, so --list-devices reports (none) straight out of the
archive. The runtime libraries have to be supplied alongside it before the GPU is
visible at all.

Models

Qwen3.8-27B, UD-Q5_K_XL (locally requantised), -fa on, -ctk f16 -ctv f16.

Problem description & steps to reproduce

With -np 2, the first long prompt after server start is processed at full speed.
Every subsequent long prompt is ~42% slower, and it never recovers for the lifetime
of the server. Requests are strictly sequential — one at a time, never concurrent.

This is not the known concurrency/speculation family of reports (see "What this is not"
below): it needs neither concurrent requests nor speculative decoding.

Method: to avoid accidentally changing more than one thing, each arm was produced by
reading the command line of the running process and substituting a single token,
rather than by rebuilding the command line. Same binary, same runtime, same model, same
context size, same corpus, same request order in every arm.

Load: 100 607-token prompt, n_predict: 64, cache_prompt: false, ignore_eos: true,
posted to /completion three times in a row.

Results on stock master (b10825)

iter	`-np 2`	`-np 1`
1	335.7 t/s	323.5 t/s
2	151.4	322.3
3	154.5	321.9
change	-54.0%	-0.5%

TTFT for the same 100 607-token prompt goes 299.7 s -> 664.6 s -> 651.1 s at -np 2,
and stays at 311.0 / 312.2 / 312.5 s at -np 1. Memory is flat in both arms
(dedicated 31 892 MiB / shared 434 MiB at -np 2; 31 292 / 490 MiB at -np 1),
sampled during the requests, so nothing is being evicted to host memory.

Results on the patched b10665 branch (where it was found)

iter	`-np 2`	`-np 1`	`-np 2`, no speculation
1	690.6 t/s	683.1 t/s	752.0 t/s
2	398.0	660.7	438.3
3	394.4	649.7	430.6
change	-42.4%	-4.9%	-42.7%

-np 1 does not collapse; the mild -4.9% drift over three iterations is normal here.
Removing --spec-type draft-mtp --spec-draft-n-max 3 changes nothing about the
collapse (-42.7% vs -42.4%), so speculative decoding is not involved.
Generation speed is essentially unaffected — the loss is in prompt processing only.

A -np 3 arm behaves the same way (644.3 / 365.4 / 367.5 t/s, -43.3%), i.e. the loss
does not scale with slot count — it looks like a threshold at "more than one slot".
I am reporting that arm with a caveat, though: at -np 3 this model no longer fits and
the server loads with ~1 081 MiB spilled to host memory, so its absolute numbers are
contaminated and only the shape of the drop is meaningful.

It is not memory spill

Per-process GPU memory was sampled continuously during each request, not just after:

arm	dedicated	shared (peak = end)
`-np 2`	31 978 MiB	436 MiB, flat across all 3 iterations
`-np 1`	31 379 MiB	491 MiB, flat
`-np 2`, no spec	29 768 MiB	264 MiB, flat

Nothing is evicted to host memory, and the numbers do not move between the fast first
iteration and the slow later ones.

What this is not

I checked the adjacent open reports before filing; this is a different failure:

spec + vulkan: Performance Drop with MTP n-max > 1 and -np > 1 #27544 (MTP n-max > 1 with -np > 1) — needs draft-mtp and genuinely concurrent
sessions; degrades generation. Mine reproduces with speculation removed and with
strictly sequential requests, and generation is fine.
speculative: draft-dflash draft acceptance collapses under concurrent sequences (-np 16 pathological, -np 4 healthy, --spec-draft-n-max 1 recovers) #27117 (draft acceptance collapse under concurrent sequences) — scales with the
number of concurrently drafting sequences. Only one request is ever in flight here.
eval bug: draft-mtp roughly halves prompt processing on multi-GPU layer split (single GPU is fine) #27428 (draft-mtp halves prompt processing) — multi-GPU layer split; the author
reports single-GPU costs only ~3%. This is a single GPU.

First Bad Commit

Unknown — not bisected. An older measurement on this machine with the same harness and
the same -np 2 was flat (569.7 / 550.1 / 541.7 t/s), so conditions existed under which
it did not appear, but I cannot date the change and the -np 1 control locates the cause
without needing to.

Relevant log output

Command line for the -np 2 arm (API key redacted, paths shortened):

llama-server.exe -m Qwen3.8-27B-UD-Q5_K_XL_V2.gguf --mmproj mmproj-F16.gguf \
  --device ROCm0 -ngl 999 -c 163840 -fa on -np 2 --kv-unified --no-cache-idle-slots \
  --cache-type-k f16 --cache-type-v f16 -t 6 --poll 0 --jinja \
  --chat-template-file <patched-template>.jinja \
  --temp 0.7 --top-p 0.8 --top-k 20 --min-p 0 --presence-penalty 1.5 --repeat-penalty 1 \
  --cache-ram 0 --host 0.0.0.0 --port 8080 --api-key <REDACTED> \
  --no-mmproj-offload -ub 512 -b 1024 --spec-type draft-mtp --spec-draft-n-max 3

The -np 1 arm is the same line with -np 2 replaced by -np 1; the no-speculation arm
is the same line with --spec-type draft-mtp --spec-draft-n-max 3 removed.
GGML_CUDA_DISABLE_GRAPHS=1 is set in all arms.

Confirming that I probably have the same issue. Strix Halo starts well, with me getting around 320 tok/s prompt processing speed initially, with 4 parallel slots and kvu for about 600k tokens on Qwen3.8-Flash-Next, but after filling the context once, the new context should start from 0 prompt but speed that I get is usually less than half, like 90 instead of 320 tokens.

I think I have observed the same issue on CUDA, ROCm and Vulkan, so I doubt this is in any way backend specific on any of the hardware. I'm currently testing whether disabling KVU will fix this.

Reproduced on stock master (b10825), and it is worse there

I said above I would retest on master rather than leave the patched build as the only
evidence. Done — using the official release binary, nothing local in it:

llama-b10825-bin-win-rocm-10.0-x64.zip
version: 0.4.0-dev (build 10825, commit 9e0e22059)
built with Clang 20.1.8 for Windows x86_64

Same machine, same model, same corpus, same 100 607-token prompt, same
cache_prompt: false / n_predict: 64, requests strictly sequential. The command line
is unchanged from the report above except for the binary path; the -np 1 arm differs
from the -np 2 arm by that one token only.

iter	master `-np 2`	master `-np 1`
1	335.7 t/s	323.5 t/s
2	151.4	322.3
3	154.5	321.9
change	-54.0%	-0.5%

So on stock master the loss is -54%, larger than the -42% I measured on my patched
build — the local FA/RDNA prefill work appears to partially mask it, not cause it. The
-np 1 control on master is essentially perfectly flat (323.5 / 322.3 / 321.9), which
is a cleaner control than the one in the original report.

TTFT for the same prompt goes from 299.7 s to 664.6 s at -np 2, and stays at ~312 s
across all three iterations at -np 1.

Memory is again flat and rules out spill: dedicated 31 892 MiB / shared 434 MiB across
all three -np 2 iterations (sampled during the requests, not just after), and
31 292 / 490 MiB at -np 1.

Raw numbers, both arms, for completeness:

master b10825, -np 2:  335.7 / 151.4 / 154.5 t/s   TTFT 299.7 / 664.6 / 651.1 s
master b10825, -np 1:  323.5 / 322.3 / 321.9 t/s   TTFT 311.0 / 312.2 / 312.5 s

My own test results suggest that disabling KVU gets rid of the problem. I also may have understated what Strix Halo is capable of, when you start prompt over from scratch but have everything warmed up, as then you don't pay some kind of setup cost, it seems:

I slot print_timing: id 1 | task 15545 | prompt processing, n_tokens = 2048, progress = 0.03, t = 4.16 s / 492.88 tokens per second

I don't think I've ever seen nearly 500 tok/s from a Strix Halo before, so I'm willing to state from this evidence that KVU should be turned off as workaround.

@alankila

Controlled A/B: it is carried by `--kv-unified`, and the cost tracks stale cells in the pool, not request ordinality

@alankila's workaround holds up under a controlled test — thanks, that pointed
straight at it. Two experiments below, both on the patched b10665 build (not master),
same machine, same clone-the-running-command-line-and-change-one-token method as the
original report.

1. `--kv-unified` on/off, matched

Note for anyone repeating this: -c is the total with --kv-unified and is
divided across slots without it, so simply dropping the flag at -c 163840 -np 2
gives 81920 per sequence and a 100k prompt is rejected outright. To change only the one
thing, I used a ~50k prompt that fits both shapes, leaving each sequence the same usable
capacity. -np 2 in both arms, everything else identical:

iter	split cache (no `--kv-unified`)	unified cache
1	816.3 t/s	802.5 t/s
2	796.2	551.5
3	790.8	467.2
change	-3.1% (normal drift here)	-41.8%

Memory flat in both, sampled during the requests (shared peak 276 MiB split /
436 MiB unified), so spill is excluded again.

With per-sequence caches the collapse is absent. With one shared pool it appears
immediately.

2. The trigger is the size of what the previous sequence left behind

Fresh server, -np 2 --kv-unified, one short request before the long ones:

request	prompt	pp
short	512 tok	658.0 t/s
long, after the short one	100 605 tok	669.9 — full speed
long, after a long one	100 605 tok	398.0

690.6 t/s is the fresh-server figure, so 669.9 after a 512-token predecessor is inside
this machine's ~3% run-to-run drift. A 512-token predecessor costs nothing; a
100 605-token predecessor costs 41%. So it is not a binary "this is not the first
request" state — the penalty is proportional to how much of the pool the previous
sequence occupied.

What this suggests

Prompt processing appears to scale with the number of used-or-stale cells in the
shared pool rather than with the length of the sequence being prefilled.
cache_prompt: false removes the previous sequence's cells, but that does not appear to
lower the pool's occupied high-water mark, so each later prefill is built over roughly
the whole occupied pool.

That predicts a saturation effect, which is visible in the data and which I had not
understood before this run:

100k prompt in a 163840 pool — two sequences already overflow, so the residue
saturates after the first request: iter2 ≈ iter3 (398.0 / 394.4), a plateau.
50k prompt in the same pool — three fit, so residue accumulates and each iteration
is worse than the last (802.5 / 551.5 / 467.2), heading for the same ceiling.

The plateau in the original report and the progressive decline here are the same
phenomenon at different pool-fill rates.

One thing I could not determine: whether the extra time is host-side mask
construction or device-side attention. Process CPU time ran at ~1.02x wall in every arm,
fast or slow, but HIP's blocking sync busy-waits so the main thread tracks wall time
regardless. That measurement is inconclusive, not evidence either way.

Workaround

Dropping --kv-unified removes the effect here, matching @alankila's result on
Strix Halo. The cost is that -c is then divided across slots, so the same
per-sequence context needs -c multiplied by -np — which may not fit.

@ggerganov

Correction: this is the known unified-KV limitation in the CUDA/HIP FA kernels, and it reproduces in `llama-batched-bench`

I owe the thread a correction. In my previous comment I presented the mechanism as
something I had inferred from my own A/B. That was incomplete — this is already known
and named upstream, and I should have found it before filing. Prior art:

Misc. bug: llama-server with -kvu and --parallel 4 slows down tg with more inactive slots #19523 — -kvu with --parallel 4: populated-but-idle slots slow the active one,
worsening as each slot fills, persisting until restart. Closed after the Vulkan fix.
@ggerganov's answer there: "The unified KV cache is not well supported by the CUDA
backend... either disable the unified KV cache and take the memory hit, or switch to
1 slot."
The mechanism is skipping all--INF blocks in the FA kernels. Metal has it,
Vulkan got it in vulkan: skip all-negative-inf blocks in FA #17186 (regressed by vulkan: Preprocess FA mask to detect all-neg-inf and all-zero. #19281, restored by vulkan: restore -inf check in FA shaders #19582). CUDA/HIP skips only
the tails (the KV_max scan), not -INF blocks in the interior — which is exactly
what @ggerganov raised in
CUDA: skip masked KV slices for all FA kernels #14924 (comment):
"Do you plan to extend the logic to search for -INF blocks within the entire context,
not just at the tails? This would be useful in use cases where the unified KV cache
(-kvu) is enabled."

I could not find a PR implementing that for CUDA. HIP compiles the CUDA backend, so
gfx1201 inherits it.

What is still new here: #19523 is about token generation, on Vulkan, and its
reporter states explicitly "And pp is not affected." This report is about prompt
processing, on HIP, where it costs 42-54%. Same root cause, a phase and a backend that
were previously recorded as unaffected.

Reproduced with `llama-batched-bench -kvu -tgs`

Since #17103 exists precisely to measure this, here it is with no custom harness — same
machine, same model, -npp 8192 -ntg 32 -npl 1,2,4 -c 33792 -fa on, only -kvu /
-no-kvu differing:

master b10825

	B=1	B=2	B=4	change
`-kvu` S_PP	885.55	792.90	644.39	-27.2%
`-no-kvu` S_PP	857.79	847.80	833.20	-2.9%
`-kvu` S_TG	24.84	24.20	23.20	-6.6%
`-no-kvu` S_TG	24.78	24.72	24.68	-0.4%

patched b10665 build (has the #28102 FA hunks)

	B=1	B=2	B=4	change
`-kvu` S_PP	1070.25	1021.34	937.78	-12.4%
`-no-kvu` S_PP	1057.30	1050.54	1036.89	-1.9%

With -no-kvu S_PP is flat across B, as in the #17103 example where the -INF block
optimizations are active. With -kvu it falls monotonically with the number of
sequences in the pool. The -kvu S_TG column also reproduces the #19523 symptom on HIP.

Note the penalty here (-27%) is smaller than the -54% in the report body because it
scales with how much of the pool is occupied: 8 192-token prompts in a 33 792 pool versus
100 607-token prompts in a 163 840 pool. Same effect, further along the curve.

Ask

Extend the CUDA/HIP FA -INF block skipping beyond the tails, as proposed in the #14924
comment. That would fix prompt processing here and the generation case in #19523 on
CUDA/HIP at the same time. Happy to test a patch on gfx1201.

Interim workaround remains --no-kv-unified, with the caveat that -c is then divided
across slots, so preserving per-sequence context requires multiplying -c by -np —
which may not fit.

Offer to test, concretely

To make the offer at the end of my last comment more useful than a one-liner — if anyone
picks up extending the -INF block skipping to the interior of the context in the
CUDA/HIP FA kernels, I can turn a patch around the same day on the exact configuration in
this report:

gfx1201 / RDNA4 — Radeon AI PRO R9700, 32 GiB, Windows 11, ROCm runtime 10.0.0.
I build from source and already keep stock master (b10825) and a patched build side by
side, so a branch or a raw diff both work.
I can report back llama-batched-bench -kvu -tgs before/after in the table format
above, and the server-level case that actually hurts — 100 607-token prompts,
strictly sequential, in a 163 840 pool, where the loss is 54% rather than the 27% the
8k-prompt bench shows.

And a correctness check, not only speed. This is a class of change where a
speed-only test can look excellent and still be wrong: skip a block that was not entirely
-INF and the output changes quietly instead of crashing. I have a perplexity baseline on
this card that is bit-identical across runs and across two ROCm runtimes — same value and
the same per-chunk sequence, not just the same final number — so I can tell the
difference between "unchanged" and "close enough to miss". I would run that alongside the
benchmark and report both.

Happy to test partial or exploratory patches too — it does not have to be a finished PR.

I hit this on Volta (V100 32GB, sm_70, CUDA 12.9) and measured it carefully. I think the
mechanism is narrower than "unified KV is slower": with --kv-unified, the mask-skip
optimisation is never computed during token generation at all.

The gate

ggml/src/ggml-cuda/fattn-common.cuh:

if (mask && K->ne[1] % FATTN_KQ_STRIDE == 0 && (Q->ne[1] >= 1024 || Q->ne[3] > 1)) {
    // ... flash_attn_mask_to_KV_max
}

During token generation Q->ne[1] == 1, so the first clause fails. With a unified KV cache
n_stream == 1, so Q->ne[3] == 1 and the second clause fails too. KV_max is therefore
null for every TG step and each sequence walks the full padded n_kv, including the cells
belonging to every other sequence in the pool.

With a split cache Q->ne[3] == n_seq_max > 1, the gate passes and the skip is active --
which is exactly why the split configuration does not show the problem.

The other two conditions always hold in this configuration: n_kv is padded to 256 in
llama_kv_cache::get_n_kv and FATTN_KQ_STRIDE is 256.

The cost is the sum of all resident sequences, not the largest

Four identical 15.6k-token conversations, one per slot, generating one at a time
(no concurrency), token counts taken from the server's own print_timing lines:

resident	tg
1 x 15.6k	23.58 t/s
2 x 15.6k	21.82 t/s
3 x 15.6k	20.16 t/s
4 x 15.6k	18.5-18.65 t/s (each)

4 x 15.6k = 62k total, and a single 67k conversation measures 18.1 t/s on the same build.
So a 15.6k conversation with three others parked generates at the speed of a 62k one.

Unified vs split, matched token counts, clean restart per condition

qwen3 hybrid, Q6_K, -c 262144, --flash-attn on, KV q8_0, -np 4 except where noted:

measurement	unified 4slot	split 4slot	unified 4slot	split 2slot
big ctx size (tokens)	39,960	40,006	66,985	66,965
small req, empty cache	25.92	25.81	25.79	25.87
generation inside the big ctx	20.70	20.68	18.20	18.13
small req, big ctx parked	20.78	25.35	18.24	25.07
re-eval tokens on return to big	20	20	20	20

Two things worth noting: a conversation's own speed is identical in both modes (20.70 vs
20.68, 18.20 vs 18.13) -- unified costs nothing for the sequence that defines the high-water
mark. And nothing is being evicted in either mode (20 tokens re-evaluated on return), so this
is not a cache-thrashing effect.

Tail trimming alone would not fix it

I checked whether cell placement rescues this, since flash_attn_mask_to_KV_max trims from
the tail: a small conversation seeded first (so its cells sit low in the shared pool) with a
67k context parked above it still generated at 18.11 t/s. With a unified cache, find_slot
hands out cells sequentially from a moving cursor, so a sequence that grows after others were
allocated ends up with its cells split across the array and the fully-masked blocks sit in the
interior, where a tail trim cannot reach them.

Possible direction

Rather than compacting the live blocks into a list (which changes which CUDA block accumulates
which KV blocks, and therefore the floating-point grouping), a bitmap of live blocks that the
kernels skip in place keeps the iteration order and grouping identical. A fully-masked block
contributes exactly 0 to both the softmax numerator and denominator and leaves the running
maximum untouched, so skipping those blocks should be bit-for-bit identical to processing
them, which makes it straightforward to validate.

Happy to share numbers from anything else that would be useful.

Following up with prompt-processing numbers, since this affects prefill as well as token
generation and the prefill case is the more painful one in practice.

Same machine as my earlier comment (V100, sm_70, CUDA 12.9, qwen3 hybrid Q6_K, -c 262144,
-np 4, --flash-attn on, KV q8_0, --kv-unified). A 14,626-token prompt, measured from
the server's own prompt eval time line:

cache state	prompt eval
empty	637.65 / 633.70 / 632.30 t/s (three runs)
one unrelated 49k conversation resident	325.12 / 303.37 t/s

So ~46-49% of prefill throughput is lost to cells belonging to a conversation that has nothing
to do with the request. Nothing is evicted in either case; the 49k conversation is simply
parked in another slot.

The mechanism is the same one from my earlier comment, but it reaches prefill by a different
route. Generation (1 query column) uses the vec/tile kernels, where the gate

(Q->ne[1] >= 1024 || Q->ne[3] > 1)

fails outright with a unified cache, so no skip information is computed at all. Prefill is wide
enough that the first clause passes, so KV_max is computed - but flash_attn_mask_to_KV_max
only trims fully-masked blocks off the TAIL. With a unified cache the sequences share one cell
pool, so the other conversations' cells are interleaved and their fully-masked blocks sit in the
INTERIOR, where the tail trim cannot reach them.

Worth noting for anyone reproducing this on Volta: prompt processing here selects the mma kernel
(Q->ne[1]*gqa_ratio_eff > 16), and with a typical prefill tiles_efficiency_percent comes out
around 98, so use_stream_k is false and it runs one CUDA block per output tile - structurally
the same shape as the vec/tile kernels, not the stream-k split.

Measurement notes, since two of these cost me a rerun: time prefill against the server's own
timing line rather than wall clock; make sure the probe prompt is not a prompt-cache hit (check
the token count in print_timing matches what you sent); and size test contexts by measured
tokens, not by line count.

Navigation Menu

Repository navigation

Eval bug: with -np 2 and --kv-unified, prompt processing drops 42-54% from the second long request onward (single GPU, sequential requests, no spill, no speculation) #28495

Name and Version

Operating systems

GGML backends

Hardware

Models

Problem description & steps to reproduce

Results on stock master (b10825)

Results on the patched b10665 branch (where it was found)

It is not memory spill

What this is not

First Bad Commit

Relevant log output

Reproduced on stock master (b10825), and it is worse there

Controlled A/B: it is carried by `--kv-unified`, and the cost tracks stale cells in the pool, not request ordinality

1. `--kv-unified` on/off, matched

2. The trigger is the size of what the previous sequence left behind

What this suggests

Workaround

Correction: this is the known unified-KV limitation in the CUDA/HIP FA kernels, and it reproduces in `llama-batched-bench`

Reproduced with `llama-batched-bench -kvu -tgs`

Ask

Offer to test, concretely

The gate

The cost is the sum of all resident sequences, not the largest

Unified vs split, matched token counts, clean restart per condition

Tail trimming alone would not fix it

Possible direction

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Eval bug: with -np 2 and --kv-unified, prompt processing drops 42-54% from the second long request onward (single GPU, sequential requests, no spill, no speculation) #28495

Description

Name and Version

Operating systems

GGML backends

Hardware

Models

Problem description & steps to reproduce

Results on stock master (b10825)

Results on the patched b10665 branch (where it was found)

It is not memory spill

What this is not

First Bad Commit

Relevant log output

Activity

alankila commented on Sep 6, 2026

DanoPTT commented on Sep 6, 2026

Reproduced on stock master (b10825), and it is worse there

alankila commented on Sep 6, 2026

DanoPTT commented on Sep 6, 2026

Controlled A/B: it is carried by --kv-unified, and the cost tracks stale cells in the pool, not request ordinality

1. --kv-unified on/off, matched

2. The trigger is the size of what the previous sequence left behind

What this suggests

Workaround

DanoPTT commented on Sep 6, 2026

Correction: this is the known unified-KV limitation in the CUDA/HIP FA kernels, and it reproduces in llama-batched-bench

Reproduced with llama-batched-bench -kvu -tgs

Ask

DanoPTT commented on Sep 6, 2026

Offer to test, concretely

superslor commented on Sep 24, 2026

The gate

The cost is the sum of all resident sequences, not the largest

Unified vs split, matched token counts, clean restart per condition

Tail trimming alone would not fix it

Possible direction

superslor commented on Sep 25, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

Controlled A/B: it is carried by `--kv-unified`, and the cost tracks stale cells in the pool, not request ordinality

1. `--kv-unified` on/off, matched

Correction: this is the known unified-KV limitation in the CUDA/HIP FA kernels, and it reproduces in `llama-batched-bench`

Reproduced with `llama-batched-bench -kvu -tgs`