[jmccardle]

PR Review Guide: llguidance token-bytes and tokenize_fn byte-exactness

Branch: llg-spm-tokenize (worktree repos/llama-wt-spm), based on upstream 030ebb55.
Build used throughout: cmake -B build -DGGML_CUDA=ON -DLLAMA_LLGUIDANCE=ON -DCMAKE_BUILD_TYPE=Release -DCMAKE_CUDA_ARCHITECTURES=89-real -DLLAMA_BUILD_TESTS=ON.

Diff: 4 files, 152 insertions, 13 deletions. The fix itself is ~20 lines in common/llguidance.cpp;
the rest is the regression test and one stale-comment deletion.

---

1. One-paragraph summary

common/llguidance.cpp is the bridge between llama.cpp's vocab and the llguidance constraint
engine. llguidance needs an exact bytes <-> tokens mapping in both directions. Both of
llama.cpp's conversion calls apply sequence-level SentencePiece/HF decorations that are correct
for their normal callers and wrong here. Direction one (tokens -> bytes, building llguidance's
token table) used llama_detokenize(), which strips a leading space; this is a live constraint
violation on every vocab type. Direction two (bytes -> tokens, the tokenize_fn callback)
used llama_tokenize(), which prepends a space on add_space_prefix vocabs; this kills the
constraint entirely on SentencePiece. The fix uses llama_token_to_piece() for the first, and for
the second detects non-byte-exact tokenizers and hands llguidance its own token table instead.

---

2. The contract being violated

Verbatim from the vendored llguidance.h (build/llguidance/source/parser/llguidance.h:196-225):

  /**
   * A pointer to the token strings
   * The length of this the sum of all token_lens
   */
  const uint8_t *token_bytes;
...
  /**
   * Tokenization function, see LlgTokenizeFn docs.
   * It should only tokenize the bytes and not add
   * any <BOS> etc. It should also work on any byte sequence, including
   * invalid UTF-8. If this is not the case, set tokenize_assumes_string to true.
   */
  LlgTokenizeFn tokenize_fn;
  /**
   * Set to true to not use tokenize_fn and instead tokenize greedily,
   * which is often incorrect and may reduce accuracy.
   */
  bool use_approximate_greedy_tokenize_fn;

token_bytes is "the exact bytes of each token". tokenize_fn must "only tokenize the bytes and
not add any <BOS> etc.". Both requirements were being broken.

Note tokenize_assumes_string is already true in llama.cpp. Its doc says it works around
"tokenize_fn only accepting valid UTF-8 strings and possibly adding etc." -- but the
source says the <BOS> half is a TODO, not implemented (llguidance.h:208-211). It only
handles invalid UTF-8 (toktrie/src/toktree.rs:595-618, tokenize_with_greedy_fallback). So the
flag does not rescue us.

---

3. Flow of one constrained token, end to end

Follow this to see where each bug lands.

3.1 Request to sampler

tools/server            "grammar": "%llguidance {}\nstart: ..."   or   "json_schema": {...}
                          |
common/json-schema-to-grammar.cpp:1158   json_schema_to_grammar(schema, force_gbnf)
                          |              under LLAMA_USE_LLGUIDANCE and !force_gbnf, emits
                          |              "%llguidance {}\nstart: %json " + schema
                          v
common/sampling.cpp:212   grammar_str = common_grammar_value(params.grammar)
common/sampling.cpp:213   if (grammar_str.compare(0, 11, "%llguidance") == 0)
common/sampling.cpp:215       grmr = llama_sampler_init_llg(vocab, "lark", grammar_str.c_str())

common/sampling.cpp:215 is the only production caller. The other references are the sampler's
own clone (common/llguidance.cpp:80) and the tests. Anything not starting with the literal
%llguidance falls through to GBNF at common/sampling.cpp:266-270, which is why the
tools request path is unaffected by any of this -- common/chat.cpp builds tool grammars with
build_grammar(), which emits raw GBNF.

3.2 Sampler construction -- where both bugs are planted

llama_sampler_init_llg (common/llguidance.cpp:262) calls
llama_sampler_llg_new_tokenizer(vocab) (:139), which does two things:

1. Builds token_bytes / token_lens by looping over all n_vocab tokens. Bug A lives
   here.
2. Registers tokenize_fn = llama_sampler_llg_tokenize_fn (:123). Bug B lives here.

Then llg_new_tokenizer(&tinit, ...) builds llguidance's TokTrie from token_bytes, and
llama_sampler_llg_new (:17) builds the matcher via llg_new_matcher.

Worth knowing for review: llg_new_matcher's doc (parser/src/ffi.rs:973-974) says
init.ff_tokens_ok is ignored; Matcher::new (parser/src/matcher.rs:24-40) calls
start_without_prompt(), never process_prompt(). That matters in section 6.

3.3 Per-token decode loop

llama_sampler_apply(grmr, cur_p)
  -> common/llguidance.cpp:46  llama_sampler_llg_apply
       mask = llg_matcher_get_mask(ctx->grammar)
       if (mask == nullptr) llg_matcher_compute_mask(ctx->grammar)
         -> parser/src/tokenparser.rs:439  compute_mask
         -> parser/src/tokenparser.rs:449  compute_mask_inner
              let prefix = if self.can_force_bytes() {          // <-- line 454
                  ... self.ff_tokens() ...                      // <-- line 458  CALLS tokenize_fn
                  if !ff_tokens.is_empty() {
                      return singleton_token_set(ff_tokens[0])  // <-- short-circuit
                  }
              } else {
                  compute_ff_bytes_to(&mut trg); trg
              };
              let mut allowed_tokens = self.compute_bias(&prefix);   // <-- uses the TokTrie
       ... apply mask: logit = -INFINITY for disallowed ...
       on failure: LOG_ERR, llg_free_matcher, ctx->grammar = nullptr, return   // <-- silent drop

llama_sampler_accept(grmr, token)
  -> common/llguidance.cpp:39  llama_sampler_llg_accept_impl
       llg_matcher_consume_token(ctx->grammar, token)   // byte accounting from the TokTrie

Two things a reviewer should take from this:

• compute_mask_inner calls ff_tokens() whenever can_force_bytes(). ff_tokens() is not
  only for jump-forward; it is on the ordinary mask path. So bug B breaks mask computation, in a
  plain upstream build, with no feature flags.
• can_force_bytes() (tokenparser.rs:639) is
  !lexer_spec().no_forcing && token_env.tokenize_is_canonical(). And
  tokenize_is_canonical() is simply tokenize_fn.is_some() (parser/src/ffi.rs:73). This is the
  hinge the fix turns.

---

4. Bug A: token_bytes built with llama_detokenize()

4.1 The code

    llama_token token = i;
    auto        dp    = (char *) token_bytes + offset;
    auto        size  = llama_detokenize(vocab, &token, 1, dp, max_token, false, false);

4.2 Why a single-token detokenize is not a piece

llama_vocab::impl::detokenize (src/llama-vocab.cpp:3658):

    // remove the leading space
    bool remove_space = add_space_prefix;                       // :3675

    if (remove_special && add_bos) { ... remove_space = false; ... }   // remove_special is false here

    for (int32_t i = 0; i < n_tokens; ++i) {
        int32_t n_chars = token_to_piece(tokens[i], text, avail, remove_space, unparse_special);  // :3693
        remove_space = false;

remove_space is passed as token_to_piece's lstrip argument, and lstrip strips leading
spaces (:3549). Because remove_special is false, the remove_space = false reset never
happens before the loop, so with n_tokens == 1 every token gets lstrip = 1.

Then, after the loop:

    if (clean_spaces) {                                          // :3709
        // first pass: characters ?!.,
        ...  if (text[i - 1] == ' ') { if (x=='?'||x=='!'||x=='.'||x==',') total--; }

So there are two independent corruption mechanisms, and they hit different vocab families:

mechanism	gated on	affected families
lstrip = remove_space = add_space_prefix	add_space_prefix	SPM (:2376 sets it true), UGM via KV
clean_spaces passes over " ?!.,"	clean_spaces	BPE (:2119 sets it true), WPM (:2383)

Defaults, from src/llama-vocab.cpp:

vocab type	add_space_prefix	clean_spaces
BPE	false (:2117)	true (:2119)
SPM	true (:2376)	false
WPM	false	true (:2383)
RWKV	false	false
UGM	default false, usually set by tokenizer.ggml.add_space_prefix KV (:2401)	false

This is the point most likely to be missed: BPE is not safe. clean_spaces is true for
every BPE vocab.

4.3 Measured: how many tokens got the wrong bytes

Scratch tool comparing llama_token_to_piece(vocab, t, buf, n, 0, false) against
llama_detokenize(vocab, &t, 1, buf, n, false, false) for every token:

vocab fixture	differing tokens	of which lost a leading space
models/ggml-vocab-llama-spm.gguf	16410 / 32000	16395
models/ggml-vocab-llama-bpe.gguf	117 / 128256	117
models/ggml-vocab-gemma-4.gguf	88 / 262144	88
models/ggml-vocab-bert-bge.gguf	5 / 30522	5

Examples:

llama-spm   tok    278  piece=' the'   detok='the'      <- lstrip
llama-bpe   tok   5354  piece=' ..'    detok='..'       <- clean_spaces
gemma-4     tok    783  piece=' .'     detok='.'        <- clean_spaces
bert-bge    tok   1012  piece=' .'     detok='.'        <- clean_spaces

4.4 Measured: the user-visible consequence

llguidance believes token 5354 is .., so a grammar that requires .. and forbids a space still
permits it. Grammar start: /[.a-z]{2,6}/ "!", counting allowed tokens at position 0 whose real
piece contains a space:

vocab	without fix	with fix
llama-bpe	6 bad ( ., .., ..., ...., ....., ......)	0
gemma-4	6 bad	0
bert-bge	2 bad	0
llama-spm	6136 bad	0

So on Llama-3 and Gemma -- the common case, no SPM involved -- a grammar that forbids whitespace
can emit whitespace, and llguidance records the token as if it had not.

Why this stays hidden in casual testing: when can_force_bytes() is true and the grammar has a
unique continuation, compute_mask_inner returns
singleton_token_set(ff_tokens[0]) (tokenparser.rs:459-463) computed from tokenize_fn, which
is correct on BPE. The corrupted trie is only consulted at positions with genuine choice
(compute_bias) and in consume_token byte accounting. That is exactly why the test grammar above
needs a /regex/ rather than a plain literal.

4.5 The fix, and why it is behaviour-preserving

    // llama_detokenize() works on a sequence: it drops the leading space of the first token
    // when the vocab has add_space_prefix, so ask for the piece instead
    auto        size  = llama_token_to_piece(vocab, token, dp, max_token, 0, false);

llama_detokenize on one token is token_to_piece(token, ..., lstrip=remove_space, special=unparse_special)
plus optional clean_spaces. Passing lstrip = 0 and skipping clean_spaces is precisely "give me
the raw piece", which is what token_bytes is documented to hold.

The size == 0 fallback branch is unchanged in behaviour. token_to_piece returns 0 exactly when
!special && (attr & (LLAMA_TOKEN_ATTR_UNKNOWN | LLAMA_TOKEN_ATTR_CONTROL))
(src/llama-vocab.cpp:3536-3540), and single-token detokenize returns whatever token_to_piece
returned. So the same set of tokens takes the \xff special-token-marker branch before and
after. The second call likewise changes only lstrip (false -> 0) and drops clean_spaces.

---

5. Bug B: tokenize_fn = llama_tokenize()

5.1 The code and the state at failure

static size_t llama_sampler_llg_tokenize_fn(const void * user_data, const uint8_t * bytes, size_t bytes_len,
                                            uint32_t * output_tokens, size_t output_tokens_len) {
    r = llama_tokenize(vocab, (const char *) bytes, bytes_len, (int32_t *) output_tokens, output_tokens_len,
                       false /* add_special */, true /* parse_special */);

SPM branch of llama_vocab::impl::tokenize (src/llama-vocab.cpp:3348-3366):

    bool is_prev_special = true;  // prefix with space if first token

    if (add_special && add_bos) { output.push_back(special_bos_id); is_prev_special = true; }

    for (const auto & fragment : fragment_buffer) {
        if (fragment.type == FRAGMENT_BUFFER_VARIANT_TYPE_RAW_TEXT) {
            std::string text;
            // prefix with space if previous is special
            if (add_space_prefix && is_prev_special) {           // :3363
                text = ' ';
            }

is_prev_special is initialised true unconditionally, so the first raw-text fragment always
gets the space when add_space_prefix. add_special = false does not suppress it (it only gates
BOS/EOS). This matches HF: tokenizer.encode("The", add_special_tokens=False) on Llama SPM returns
[450] = _The. It is correct SentencePiece behaviour and a wrong callback for llguidance.

Concrete state, models/ggml-vocab-llama-spm.gguf:

llama-tokenize -p "The"   ->   1 -> '<s>'   450 -> ' The'

So for grammar start: "hello world", position 0:

can_force_bytes()            true  (tokenize_fn is set -> tokenize_is_canonical())
parser.force_bytes()         forced_bytes = "hello world"
tokenize_bytes(forced)       tokenize_fn -> llama_tokenize("hello world") -> [_hello, _world]
ff_tokens[0]                 22172, real bytes " hello"
compute_mask_inner           returns singleton_token_set(22172)

Measured, pre-fix, test_forced_mask_bytes on llama-spm:

  allowed: 22172 " hello"          <- the only allowed token, and it is wrong

The sampler is then forced to emit " hello"; llama_sampler_llg_accept_impl calls
llg_matcher_consume_token(22172), which fails:

Parser Error: token "quick" doesn't satisfy the grammar; forced bytes: got ' '; applying 'q'

The matcher enters an error state, the next llg_matcher_compute_mask fails, and
llama_sampler_llg_apply (common/llguidance.cpp:53-58) does:

    LOG_ERR("llg error: %s\n", llg_matcher_get_error(ctx->grammar));
    llg_free_matcher(ctx->grammar);
    ctx->grammar = nullptr;
    return;

From then on apply is a no-op: generation continues completely unconstrained, HTTP 200, no
error field. Measured on TinyLlama-1.1B with a grammar admitting exactly one string:

got: ' The quicker you can do this, the more points you will earn.\n\n2. Time your response: ...'

5.2 Why not fix the tokenizer instead

This is the first question a reviewer will ask, so it is worth having the answer ready.

• There is no per-call override. is_prev_special = true is unconditional; a special-token prefix
  re-arms it (:3378 sets is_prev_special = true for FRAGMENT_BUFFER_VARIANT_TYPE_TOKEN), so
  prefixing <s> does not help; and add_space_prefix has no public getter in llama.h
  (only llama_vocab::get_add_space_prefix() in src/llama-vocab.h:142, not reachable from
  common/).
• bool is_prev_special = add_special; is a one-line change with a large blast radius: every
  add_special = false caller on an SPM vocab would tokenize differently. Small diff, big risk --
  the wrong shape for a bug fix.
• Prefix-and-strip tricks are not safe in general: dropping the prepended byte requires that no
  vocab piece spans the boundary, which is false for large SPM vocabs (Gemma has multi-newline
  pieces).

So the byte-exact route needs a vocab-level API decision, which does not belong in a bug fix. It is
listed as an open question in section 10.

5.3 The fix

// llguidance requires tokenize_fn to return exactly the bytes it was given. SentencePiece vocabs
// prepend a space when add_space_prefix is set and llama_tokenize() has no per-call override.
static bool llama_sampler_llg_tokenize_is_exact(const llama_vocab * vocab) {
    const std::vector<std::string> probes = { "The", "a b", "1\n2" };
    ... tokenize each probe, reassemble with llama_token_to_piece, compare ...
}

    // without an exact tokenizer, llguidance falls back to its own token table. It then stops
    // forcing bytes, which loses token healing but keeps the mask correct.
    const bool is_exact = llama_sampler_llg_tokenize_is_exact(vocab);
    if (!is_exact) {
        LOG_WRN("%s: tokenizer is not byte exact, using greedy tokenization\n", __func__);
    }
...
        /* .tokenize_fn                        = */ is_exact ? llama_sampler_llg_tokenize_fn : nullptr,
        /* .use_approximate_greedy_tokenize_fn = */ !is_exact,

Probe rationale: three probes rather than one, cheap (once per vocab, cached by
vocab_cache/tokenizer_cache at :141-146). "The" catches the SPM prefix; "a b" catches
interior whitespace handling; "1\n2" catches newline mangling (the clean_spaces family). Any
failure -- including a tokenize error or an oversized piece -- returns false, so the probe fails
safe: worst case a byte-exact vocab is misclassified and loses byte forcing, never gains a
wrong mask. Note the probe deliberately reassembles with llama_token_to_piece, i.e. it tests the
round trip through both fixed conversion points.

---

6. Why the greedy fallback does not degrade constraint enforcement

llguidance's own header calls greedy tokenization "often incorrect and may reduce accuracy", so
this needs justifying rather than asserting.

Where tokenize_bytes is actually called. Exhaustive, from the vendored source:

call site	reachable from llama.cpp?
tokenparser.rs:235 in process_prompt	No. llama.cpp uses the Matcher API; Matcher::new calls start_without_prompt() (matcher.rs:32-34)
tokenparser.rs:249 in process_prompt	No, same
tokenparser.rs:683 in ff_tokens	gated: do_force = ... && tokenize_is_canonical() (:680)
tokenparser.rs:695 in ff_tokens	same gate

So on the greedy path llguidance never tokenizes bytes at all. The "reduced accuracy" warning
describes a code path llama.cpp does not take. (It also means process_prompt's
assert!(tokenize_is_canonical()) at :220 is not a crash risk here.)

Where the mask comes from instead. compute_mask_inner's else-branch:

        } else {
            let mut trg = Vec::new();
            self.compute_ff_bytes_to(&mut trg);
            trg
        };
        let mut allowed_tokens = self.compute_bias(&prefix);

compute_bias walks the TokTrie against the Earley/lexer state. That is the real constraint and
it does not involve tokenize_bytes. Enforcement is exact.

What is actually lost. can_force_bytes() goes false, so:

1. no byte forcing -- the mask is never collapsed to a singleton, so the model chooses among all
   tokens that spell a legal continuation (see the 6-token result below);
2. no token healing prefix in the mask;
3. llg_matcher_compute_ff_tokens() returns nothing.

None of these affect correctness. Item 3 is the price: a jump-forward-style feature cannot fire
on SPM vocabs. That is graceful degradation and matches the pre-fix reality, where ff_n was 0
because the matcher was dead.

Nothing changes for byte-exact vocabs. When the probe returns true, tinit is
byte-identical to before, so BPE/gemma users keep canonical tokenization and forcing. Verified: on
llama-bpe, test_forced_mask_bytes reports the same single allowed token (15339 "hello") before
and after.

---

7. Test design, and why the obvious test does not work

tests/test-grammar-llguidance.cpp has only ever been run against a BPE vocab
(tests/CMakeLists.txt:148). Three changes:

7.1 test_no_space_tokens() -- catches bug A, on the vocab CI already uses

Grammar start: /[.a-z]{2,6}/ "!"; asserts no allowed token contains a space. A /regex/ is
required rather than a literal, because a literal triggers the singleton-forcing short-circuit and
never consults the trie (section 4.4).

	llama-bpe	llama-spm
without fix	exit 134, 6 bad ( ., .., ...)	exit 134, 6136 bad
with fix	exit 0, 0 bad	exit 0, 0 bad

This is the strongest part of the test story: it fails today on ggml-vocab-llama-bpe.gguf, the
vocab already wired into CI. No new fixture needed to demonstrate the bug.

7.2 test_forced_mask_bytes() -- catches bug B

Grammar admitting one literal; asserts every allowed token is a prefix of it.

	llama-bpe	llama-spm
without fix	1 allowed: "hello" (ok)	exit 134, 1 allowed: " hello"
with fix	1 allowed: "hello"	exit 0, 6 allowed: h, he, hel, hell, hello, h

A weaker assertion would have proved nothing. My first attempt asserted only
0 < n_allowed < n_vocab -- "the mask is a real constraint". That passes on the buggy build,
because the buggy mask allows exactly one token; it is just the wrong one. The assertion has to be
about bytes, not cardinality. Worth flagging to a reviewer as the reason the test looks the way it
does.

7.3 tokenizer_is_exact() guard and the SPM ctest entry

    llama_test(test-grammar-llguidance NAME test-grammar-llguidance-spm ARGS ${PROJECT_SOURCE_DIR}/models/ggml-vocab-llama-spm.gguf)

The pre-existing string tests call common_tokenize() on their own expected strings, which applies
the same space prefix, so they cannot express "abc" on an SPM vocab and would fail spuriously.
They are skipped there. This is a genuine limitation, not laziness: it is the same missing
capability described in section 5.2, and it is fair to say so in review.

Both ctest entries pass: ctest -R llguidance -> 2/2.

---

8. Reproduction, without and with the fix

cd repos/llama-wt-spm

# unit level
cmake -B build -DLLAMA_LLGUIDANCE=ON -DLLAMA_BUILD_TESTS=ON -DCMAKE_BUILD_TYPE=Release
cmake --build build --target test-grammar-llguidance -j"$(nproc)"
./build/bin/test-grammar-llguidance models/ggml-vocab-llama-bpe.gguf   # bug A
./build/bin/test-grammar-llguidance models/ggml-vocab-llama-spm.gguf   # bug A + bug B

# to see the failures, revert just the fix and rebuild
git stash push -- common/llguidance.cpp
cmake --build build --target test-grammar-llguidance -j"$(nproc)"
# ... both abort with exit 134 ...
git stash pop

Server level (jf_eval/spm_fix_verify.py, three cases: forced literal, json_schema enum, forced
skeleton around a free region):

model	tokenizer	without fix	with fix	greedy engaged
TinyLlama-1.1B Q8_0	llama/SPM	0/3, 3 llg errors	3/3, 0 errors	yes
Mistral-7B-v0.1 Q4_0	llama/SPM	0/3 (silent)	3/3, 0 errors	yes
Hermes-2-Pro-Mistral-7B Q6_K	llama/SPM	0/3 (silent)	3/3, 0 errors	yes
gemma-4-31B-it Q5_K_S	gemma4	3/3	3/3, 0 errors	no
Qwen3.5-2B Q5_K_S	bpe/qwen35	3/3	3/3, 0 errors	no

Pre-fix outputs worth quoting, because they show the two failure modes are different:

forced_literal : ' The quicker you can do this, the more points you will earn...'   <- bug B, free text
json_enum      : ' { "colo ur": "c h artreu se", "count":10 }'                      <- bug A, spurious spaces
half_free      : ' BEG IN[iqr] END'                                                 <- bug A

The json_enum line is the one to lead with for a maintainer: it is not an error, it is
structured-looking output with a corrupted key name.

---

9. The stale llama.h comment -- yes, it belongs here

include/llama.h:1156-1158, before:

    /// @param add_special Allow to add BOS and EOS tokens if model is configured to do so.
    /// @param parse_special Allow tokenizing special and/or control tokens which otherwise are not exposed and treated
    ///                      as plaintext. Does not insert a leading space.

That last sentence is false, and has been since November 2023. History:

• 1a159553f "tokenizer : special token handling (tokenizer : special token handling ggml-org/llama.cpp#3538)", Oct 2023, introduced both the sentence
  and the behaviour: auto raw_text = (special ? "" : " ") + fragment.raw_text.substr(...). So
  parse_special = true genuinely suppressed the leading space.
• a6fc554e2 "llama : restore prefix space in llama tokenizer (llama : restore prefix space in llama tokenizer ggml-org/llama.cpp#4081)", Nov 2023, deliberately
  reverted the behaviour: if (&fragment == &fragment_buffer.front()) { raw_text = " " + raw_text; }.
  The comment was not updated.

Why it belongs in this PR rather than a separate one:

1. It is the sentence that says the naive fix exists. Any reviewer reading llama_tokenize's
   contract will ask "why not pass parse_special = true?" -- and the honest answer is "the doc is
   wrong". Leaving it in place guarantees that round trip.
2. It materially slowed down diagnosis. I initially framed the fix as "restore documented
   behaviour", which is an invalid framing -- llama : restore prefix space in llama tokenizer ggml-org/llama.cpp#4081 was intentional. Checking history is what
   killed that framing. A reviewer who does not check history will reach the same wrong conclusion.
3. It is one line, in the same conceptual area (the space prefix of llama_tokenize), and
   trivially separable.

Scope choice: delete the false sentence, do not replace it. The change made is exactly:

     /// @param parse_special Allow tokenizing special and/or control tokens which otherwise are not exposed and treated
-    ///                      as plaintext. Does not insert a leading space.
+    ///                      as plaintext.

The tempting alternative is to document what does happen, e.g. "SPM and UGM vocabs insert a
leading space when tokenizer.ggml.add_space_prefix is set, independent of both flags". That is
true, but it is a new documented behaviour claim on a public API, and documenting it invites
exactly the "should it?" discussion that ggml-org#4081 already had. Deleting a false sentence needs no such
debate. If a maintainer asks for the positive statement, it is a one-line follow-up.

Recommendation: keep it as its own commit in this PR, so a maintainer who wants it split can
drop one commit without touching the fix.

---

10. What a reviewer should push on

#	Concern	Answer / status
1	Is greedy tokenization acceptable, or should SPM fail loudly?	Section 6: greedy keeps enforcement exact because tokenize_bytes is unreachable in llama.cpp's Matcher usage. But this is a maintainer preference, and it is the open question to raise.
2	Should SPM instead get a byte-exact tokenization entry point?	Better long-term. Needs a vocab API decision (section 5.2), out of scope for a bug fix.
3	Does the token_bytes change alter behaviour for BPE users?	Yes, and that is the point: it fixes 117 wrong tokens on llama-bpe. It does not alter the \xff special-token branch (section 4.5).
4	Is the probe robust?	Fails safe in every error path; runs once per vocab; tests the round trip through both fixed conversion points (section 5.3).
5	Why is a /regex/ needed in test_no_space_tokens?	A literal triggers singleton forcing and never consults the trie (section 4.4).
6	Why are the string tests skipped on SPM?	They tokenize their own inputs; section 7.3.
7	Conflict with PR ggml-org#15837?	Likely textual. It edits the same token_bytes loop for special-token attrs and keeps llama_detokenize, so it does not fix either bug. Worth referencing.
8	Does CI cover this path?	.github/workflows/build-3rd-party.yml:30 builds -DLLAMA_LLGUIDANCE=ON, so it compiles; it just never tested an SPM vocab, and the BPE bug went unnoticed because no test asserted on token bytes.

No existing issue or PR reports either bug (searched llguidance, grammar ignored sentencepiece,
add_space_prefix grammar, grammar not enforced llguidance).

---

11. Files touched

file	why
common/llguidance.cpp	both fixes: llama_token_to_piece for token_bytes; probe + greedy fallback for tokenize_fn
tests/test-grammar-llguidance.cpp	test_no_space_tokens (bug A), test_forced_mask_bytes (bug B), tokenizer_is_exact guard
tests/CMakeLists.txt	run the existing suite against ggml-vocab-llama-spm.gguf too
include/llama.h	delete the stale "Does not insert a leading space" sentence (separate commit)

Scratch tools used for the measurements in sections 4.3 and 4.4 are not part of the PR; they live
in the session scratchpad (tokbytes.cpp, maskcheck.cpp) and are worth keeping locally to answer
review questions.