[Piggidragon]

Closes #37 (combining drafters, after the choice between prompt lookup and MTP in Strata by @Niko1221, MIT).

With several --spec-type (e.g. ngram-simple,draft-mtp) the first drafter that has a draft wins today. An n-gram draft of copied text makes copy-heavy answers (a config file or code given back with a few edits) much faster, but where its drafts are rejected it makes answers slower: a hybrid model pays a checkpoint restore and a replay for every partial acceptance, so a code rewrite lost 9-15%.

--spec-select chooses per step and sequence between the n-gram draft (cut to its best length), the model drafter's draft, or none:

• Acceptance per draft position (decayed hits and trials per position bucket), learned per slot: from the drafts used, from n-gram drafts that were not used and from the part of a used draft that was cut off (both compared with the text that follows, as it comes).
• Step time per number of verified tokens, learned from steps of one pass; a batch whose sequences verify different numbers of tokens takes one pass per length (ubatches of recurrent and hybrid models); the model drafter's time separately.
• Choice: maximizes the sum over the step's sequences of log(tokens / step time); no sequence may get more than 10% slower than with every one on the model drafter.

Without the flag nothing changes.

Results

Qwen3.8 27B UD-Q4_K_S, 4070 + 3060, -sm tensor, MTP depth 2 with the draft vocabulary (#36), production sampling (temperature 1.0) / greedy; one slot, 4 requests per task in a row (median):

task	MTP alone	ngram-simple,draft-mtp --spec-select	n-gram first (no select, 2 requests)
docker-compose edit (German)	78.8 / 79.8 t/s	118.1 / 123.3	124.4 / 128.4
code edit	71.9 / 76.9	71.7 / 75.5	66.5 / 65.7
German letter	47.9 / 55.5	48.5 / 56.1	50.4 / 56.2
English explanation	56.9 / 62.6	57.7 / 63.6	58.2 / 63.9

(After the review fixes; before them: compose 122.9 / 125.3, the rest the same.) In the production-like concurrent prefill test the compose edit ran at 119 and 127 t/s, the second one next to a 5.5k-token prompt of the other slot.

• With 2 requests per task the compose edit reaches 98-112 t/s: the acceptance of the n-gram drafts is learned over the first requests.
• Two slots generating at once (a compose edit next to an explanation): as MTP alone, identical drafts. A long n-gram draft next to an MTP draft costs a hybrid model a second pass over the model; the 10% limit keeps the other slot from paying for it.
• The decisions depend on measured step times, so runs differ by up to 10%.

Review fixes (c37bca8): the selector learns a draft's outcome at its verification (common_speculative_verified, before a checkpoint restore whose replay reports one token more), a draft cut by the choice and accepted up to the cut no longer counts as its drafter's own outcome (ngram-mod reset its shared table, ngram-map-k shrank its drafts), the model drafter is not called when every sequence took its n-gram draft (its time decayed toward 0), and step times are learned only from batches that hold exactly the step's drafts with nothing computed alongside.

Debug output (-v): one choose: line per step with the candidate, the chosen length, the expected tokens of both drafts and whether the n-gram draft was taken.

🤖 Generated with Claude Code

[Piggidragon]

Review fixes in c37bca8, then main (#46) merged in (8b87d64, README rows only):

1. the selector learns at the verification via common_speculative_verified (before a restore), so after a restore it no longer counts one token more (checked: 12/48, 20/22, 13/15 now match the server trace);
2. a draft cut by the choice and accepted up to the cut is not reported to its drafter as its own outcome (ngram-mod: 0 table resets instead of 11, ngram-map-k: drafts stay at 48 tokens in the second request);
3. no model draft call when every sequence took its n-gram draft (t_model stays ~55 ms on CPU, no empty-batch errors with draft-simple);
4. step times only from clean batches: the 3 steps next to a 5.5k concurrent prefill (300-436 ms) are no longer learned, 153 clean ones are.
   Benchmark after the fixes (4 requests, median): compose 78.8 -> 118.1 t/s (temperature 1.0), 79.8 -> 123.3 (greedy), the other tasks within 2% of MTP alone; with main merged 115.2 / 121.9.

🤖 Generated with Claude Code