Repository navigation
Prefill speed curve: omphalos vs llama.cpp, K4/V4 and K8/V4, with and without MTP #232
Description
The #229 pelican test says nothing about prefill: its prompt is a few dozen tokens. There is no head-to-head prefill measurement under identical conditions yet: bench/m0_llama_bench.sh covers llama.cpp alone, and #166 (NIAH) measures accuracy.
Test: prefill of the same corpus text (bench/get_corpus.sh) at 512, 2k, 8k, 16k, 32k, 64k and 100k tokens, median of 5 runs after a warm-up (PLAN.md §17). Report prefill t/s and peak VRAM.
- omphalos:
omph-run --last-logits - llama.cpp:
llama-bench -p N, HIP build,-fa on
KV settings, both engines (the maintainer's target is K4/V4):
- k4q4 (the target): omphalos
OMPH_KV_K4=1, llama.cpp-ctk q4_0 -ctv q4_0 - k8q4: omphalos default (K4 on the 8 least sensitive layers, design: per-layer K precision and the K4/V4 default (VRAM at long context) #175; and
OMPH_KV_K4_LAYERS=nonefor pure K8), llama.cpp-ctk q8_0 -ctv q4_0
MTP: every point also with the MTP block loaded. PLAN.md requires that MTP does not slow prefill; M0's trace showed llama.cpp's prefill slowing down with MTP on.
Output: bench/results/prefill-curve-<n>.csv plus a summary, tagged with the engine commit. Milestone: PLAN.md §17 benchmarks / M8 (prefill).
Activity
Nihmar commentedon Oct 3, 2026
Paused at 14:55 on 2026-10-03 (branch
test/232-prefill-curve, partial CSV committed). Done: omphalos K4/V4 with MTP loaded, all points (956 / 948 / 919 / 884 / 821 / 719 / 635 t/s at 512 / 2k / 8k / 16k / 32k / 64k / 100k), and without MTP up to 64k (953 / 947 / 922 / 889 / 827 / 725). Loading MTP costs at most 0.8 % of prefill speed. Remaining: llama.cpp K4/V4 and the four K8/V4 configurations.tools/prefill_curve.pyresumes from the CSV.Nihmar commentedon Oct 14, 2026
K8/V4 prefill curve, re-measured today (PR #486,
bench/results/speeds-k8v4-396.txt)Main 9175738, RX 9060 XT, desktop active, fresh prompt, median of 5 engine runs against
llama-bench -r 5/3. llama.cpp is the pin's RX 9060 XT overlay (f19c20f9); the upstream pinbuild lacks the
q8_0-q4_0flash-attention kernel and is slower at long context (#396).At-depth chunk: llama's pp512@d is 634 / 593 / 527 / 461 t/s at depth 0 / 4k / 16k / 32k; the
omphalos equivalent (an appending turn through the server) is ~715-770 t/s at 20k (#234).
Speculation costs the prefill: with DFlash2 loaded, omphalos' prefill of a 2073/4073-token
scenario prompt is 2162/4060 ms against 2132/4008 without (
+1.4 %, the drafter's injection+5-10 %).only, #449); llama's moves 3530-3705 to 3875-3882 ms on the 2k prompt (
Still missing from this issue's original matrix: K4/V4 (the #262 table has omphalos' k4q4
scenario cells), and an at-depth omphalos prefill curve (the
omph-rundriver measures wholeprompts; the server's
prompt_msis the comparable number).