[donschen]

--kv q8_0 was rejected on HIP builds only because no ROCm machine had run it (docs/Q8_KV.md). This removes the rejection, builds poly_kv_q8_test on HIP too, and adds the AMD measurements to docs/Q8_KV.md.

Measured on an RX 9070 XT (16 GB, gfx1201), Ryzen 7 9800X3D, 30 GB RAM, Fedora, ROCm 7 from AMD's wheels, Tiel Coder 35B-A3B UD-Q4_K_XL with --kv-whole.

Kernels

The kernels are portable. poly_kv_q8_test passes on gfx1201, with 91,530 values checked.

Speed and VRAM

Each row is a fresh server:

	Experts in VRAM	256-token answer	4.5K prompt	64K prompt	Answer after 64K
fp16, 131072	4,415	110–127 tok/s	1,814–1,822 tok/s	2,065 tok/s	66 tok/s
Q8_0, 196608	4,729	106–130 tok/s	1,843–1,862 tok/s	2,075 tok/s	63 tok/s

Answers

Greedy decoding with --no-draft, seven prompts:

• The four short prompts gave identical answers.
• The three longer ones diverged after a while: a docstring, a summary of a 24K-character prompt, and a list taken from a 60K-character prompt. Each Q8_0 answer was as correct as the fp16 one.

Testing

• These numbers were taken on the 0.1.44.1 source with the AMD expert cache fix from poly, HIP: the expert cache gives VRAM back and grows again on AMD (#13) #28 also applied. The expert counts depend on that fix; the comparison between Q8_0 and fp16 does not.
• This branch was built and unit-tested together with that fix, not on its own. On its own it only removes the check in main.cpp and changes the test's CMake condition.

🤖 Generated with Claude Code

[donschen]

Overlaps with #27, which also lifts this HIP check and replaces the q8_0 readers with int8 ones. This is the light-touch version: it enables the existing q8_0 path on AMD with measurements from an RX 9070 XT. Happy to rebase onto #27 if that lands first, or rerun these tests on AMD against #27.

[VecSzn]

Merged, thanks. This one went in first since it's the small change and comes with AMD numbers. #27 has been asked to split into separate PRs. If its int8 cache comes back, a run on your RX 9070 XT against q8_0 as it is now would help a lot: same model, context and prompts as the table here.