Repository navigation
feat(server): add b10621 tokenization, template, infill, and native embedding utilities #1442
Description
Parent
Problem
mlxcel tokenization currently supports only content and add_special. b10621 also supports special-token parsing and piece-return mode, and provides apply-template, prompt token-count, infill, and native embedding utilities that are missing or schema-incompatible.
Scope
- Add
parse_specialandwith_piecesbehavior to/tokenize, including byte-array pieces when text is not valid UTF-8. - Align
/detokenizespecial-token and byte reconstruction behavior. - Implement
/apply-templatewith template, message, tool, and reasoning options supported by the loaded tokenizer/template stack. - Implement b10621-compatible chat and responses input-token-count aliases.
- Implement native
/embeddingand/embeddingsrequest/response behavior separately from/v1/embeddings. - Implement
/infillonly where the tokenizer/model declares compatible FIM tokens; return a capability diagnostic otherwise. - Align the
--spm-infillprefix/suffix/middle ordering switch with the selected model's FIM token contract.
Acceptance criteria
- Tokenize/detokenize round trips cover ASCII, Unicode, special tokens, invalid UTF-8 pieces, and empty content.
- Apply-template output is compared with the model's configured template for messages, tools, and reasoning modes.
- Token-count endpoints agree with the actual prompt submitted for generation.
- Native embedding aliases return the b10621 envelope without changing
/v1/embeddings. Delivered by fix(server): separate native completion and embedding routes from OpenAI routes #1441 (PR feat(server): separate the native completion and embedding routes #1467); thePOST /embeddingandPOST /embeddingsmanifest entries are owned by fix(server): align embedding and reranking mode, pooling, and normalization #1452, not by this issue. - Infill advertises and enforces model capability.
- Prefix/suffix/middle and suffix/prefix/middle ordering are covered with a tokenizer that supports each declared form.
Validation
- Run utility tests with at least one Llama tokenizer and one Qwen tokenizer because their special-token and chat-template behavior differs.
Remaining work (kept open after the implementation PR)
POST /infill is mounted, gated on the model's fill-in-the-middle vocabulary, and covered by tests, but its manifest entry in compat/llama-server/b10621/routes.json is still deferred for two observable divergences, so this issue stays open and owns them:
- Text in
input_prefix,input_suffix,input_extra[].text,input_extra[].filenameorpromptthat contains one of the model's own FIM marker spellings is refused with a diagnostic naming the field and the marker. b10621 tokenizes those fields with special-token parsing off and serves the marker as literal text. mlxcel's generation entry point takes a prompt string that is re-tokenized with parsing on, and a string cannot express "these characters are not that token", so carrying the text through would let a file the client did not write restructure the FIM prompt and return a fluent wrong completion. Closing this needs a pre-tokenized prompt path intoModelProvider::generate, which touches shared types thexla-ireebuild also compiles. - The FIM context is not truncated to the prefill batch: b10621 keeps only the last
3*(n_batch/4)prefix tokens and the first(n_batch/4) - (2 + prompt)suffix tokens, while mlxcel serves the whole prefix and suffix, consistent with its policy elsewhere of clamping an over-long request rather than dropping prompt text. Deciding this needs a call on whether mlxcel should drop prompt text at all, and on whichn_batchthe window follows given that mlxcel's is--prefill-chunk-size(see the--batch-sizeentry, fix: Close the remaining b10621 context and batching gaps #1472).
Activity
test(server): add pinned b10621 compatibility manifest and gates
fix(compat): reconcile manifest ownership and fix trace perplexity
inureyes commentedon Aug 28, 2026
Implementation PR: #1480. It deliberately carries no closing keyword, so this issue stays open.
Eight of the nine manifest entries this issue owns in
compat/llama-server/b10621/routes.jsonflip out ofdeferredin that PR:--spm-infill,POST /tokenize,POST /detokenize,POST /apply-template, and the fourinput_tokensroutes are nowsupportedwith a conformance test each.POST /embeddingandPOST /embeddingswere listed in this issue's scope but are owned by #1452 and were already served in their native shape by #1441 (PR #1467), so nothing here touches them.POST /infillis mounted, gated on the model's fill-in-the-middle vocabulary and covered by tests, but its entry staysdeferredwith two recorded divergences, both now written into the issue body above. Closing this issue while that entry points at it would make the validator's--check-issues-opengate red for every open PR in the repository, so the issue owns the remainder instead.One correction worth recording for anyone who touches FIM discovery later: the marker spellings and their order were read out of
libllamain the pinned b10621 archive rather than transcribed from memory. Two things a from-memory list gets wrong. DeepSeek-Coder's markers use the FULLWIDTH vertical line U+FF5C (<｜fim▁begin｜>), not ASCII|, which is also how the publisheddeepseek-coder-1.3bcheckpoint spells them, so an ASCII-pipe scan refuses/infillon a model that supports it. And b10621 additionally scans<|code_prefix|>/<|prefix|>and their suffix and middle siblings, which are absent from the commonly quoted list.feat(server): add b10621 tokenize, template, and infill utilities (#1480
feat(server): implement b10621 resumable streams and completion contr…
feat(compat): add the by_design b10621 compatibility-policy state
inureyes commentedon Aug 30, 2026
Closing: this issue's implementation merged, and the entries that kept it open now carry a state that describes them.
The work for tokenize, detokenize, apply-template, token-counting, native embedding and infill utilities landed and was verified at the time. What remained were manifest entries recording behavior that differs from llama-server b10621 on purpose and permanently, which none of the four original states could express: not
supportedbecause the behavior differs, notaliasedbecause it is not equivalent, notnot_applicablebecause the concept applies and is served, and notdeferredbecause nobody intended to close the gap. The compatibility gate requires adeferredentry's issue to stay open, so this issue was pinned open by a vocabulary gap rather than by unfinished work.#1499 added the
by_designstate for exactly that case, and PR #1500 moved this issue's entries into it. Each carries arationaleobject the validator requires in full: akindseparating an architectural impossibility from a policy choice, areason, and arevisit_ifnaming what would have to change for the decision to be reconsidered. Each is pinned by a test, not by a note.Verified before closing: this issue now owns zero
deferredentries, so closing it cannot redden thellama-compat manifestjob for other pull requests. That failure mode happened twice earlier in epic #1431 and is what the check exists to prevent.feat(server): add Responses conversation compaction (phase A) (#2520)
test(server): add pinned b10621 compatibility manifest and gates
fix(compat): reconcile manifest ownership and fix trace perplexity
feat(server): add b10621 tokenize, template, and infill utilities (la…
feat(server): implement b10621 resumable streams and completion contr…