[jadidbourbaki]

Overview

The n-gram caches in llama.cpp are currently implemented as nested std::unordered_maps. An outer map sends each n-gram to an inner map of the tokens that follow it and their counts. This is a small performance fix. I found that the inner maps were being copied unnecessarily in multiple places on every drafting step. I created this simple PR to read them by reference instead. This immediately made drafting 4.5x to 25.6x faster depending on the size of the corpus (see figure below). The latency is the average time spent drafting per drafted token.

Additional information

• Command: llama-lookup-stats -m qwen2.5-0.5b-instruct-q4_k_m.gguf -f wiki.test.raw -lcs <static cache> -c 4096 --spec-draft-n-max 8 -ngl 0
• Benchmark setup: follows lookup: complement data from context with general text statistics #5479 (WikiText-103 static caches, llama-lookup-stats, 4096-token context)
• Data: WikiText-103 test, static caches built from prefixes of WikiText-103 train, median of 3 runs
• Hardware: Apple M4 Pro, 14 cores, 48 GB, macOS 26.5.1, CPU only
• Drafting timings from llama-lookup-stats, not end-to-end generation

Corpus (MB)	master (µs / drafted token)	PR (µs / drafted token)	Speedup
0 (no static cache)	8.54	1.89	4.52x
25	45.61	4.12	11.08x
50	59.73	4.42	13.52x
100	83.46	4.64	17.99x
200	113.46	5.62	20.18x
541	165.48	6.47	25.58x

• Static cache load time at 541 MB: 5485 ms (master), 5283 ms (PR)
• Peak memory at 541 MB: 3465 MB (master), 3548 MB (PR)
• Accepted drafted tokens: within 0.15 percentage points of master on every corpus

Benchmark code and full results: https://github.com/jadidbourbaki/ngram-cache-bench

Requirements

• I have read and agree with the contributing guidelines
• AI usage disclosure: YES. Removing unnecessary copies of the n-gram caches was my idea. The LLMs (Claude Opus 5.5 and Gemma 4 31B) helped find every place that copied a map by value, change those to references, and run the benchmarks. I verified every line of the code change and the benchmark results.

[am17an]

Nice! Did you measure e2e speed-up using something like speed-bench (available in tools/ in this repo)

[jadidbourbaki]

Great question @am17an! I ran it yesterday based on your advice.

I started the server with

llama-server \
  -m qwen2.5-0.5b-instruct-q4_k_m.gguf \
  -c 8192 --port 9931 -ngl 99 -fa on -np 1 --jinja \
  --spec-type ngram-cache -lcs static-legacy-full.bin

and then I used speed-bench like you suggested in the following way:

python tools/server/bench/speed-bench/speed_bench.py \
  --url localhost:9931 --bench qualitative --category all \
  --osl 512 --concurrency 1 --limit 8 --output master-1.json

With my setup i.e qwen2.5 0.5B with Q4K_M on my Macbook M4 Pro, the decode for the baseline was 247 tokens per sec. The decode with this change was 250.9 tokens per second i.e about a 1.6%-ish increase end to end? Also saw an analogous decrease in latency. I am assuming this is because drafting isn't the bottleneck for most workloads.

Thank you!

Also thank you @ggerganov for approving!