[gabe-l-hart]

Overview

Small follow-up to #30676 to increase the number of supported options in the decision template.

Additional information

The model supports up to 256 unique choices. The challenge is that each choice needs to be represented by a single vocab index so that logits can map to probabilities, but there are not 256 obvious single-token markers that make a cohesive semantic set. Since the model is based on an LLM, the markers themselves should carry real semantic meaning (vs arbitrary tokens). To handle this on a model-by-model basis, this PR uses a search process to find combinations of the standard english capital letter set that map to single tokens in the model's vocab (eg A, AA, B, BB, etc). This is done at conversion time so it has no runtime impact and is static to the converted GGUF model.

Requirements

• I have read and agree with the contributing guidelines Yes
• AI usage disclosure: No (mostly). The original token search snippet was generated by Claude by a colleague, but I adapted it and verified the logic here by hand.

[gabe-l-hart]

cc @ngxson for a small follow-up to increase the option set from 56 -> 256 for Granite Switch as a decision model.

[gabe-l-hart]

Hmm, looks like this might not actually fix the benchmark failures. Drafting until we get to the bottom of it.

[gabe-l-hart]

@CISC @ngxson It looks like there's a mirror of the option label logic between the conversion and server_decision_context::init (here). I think there are three ways to tackle this:

1. Copy the new conversion logic over to determine the set dynamically from the tokenizer (would need to be kept in sync)
2. Parse from the template string
3. Add a new hparam to explicitly hold the list

I think (3) is probably the most extensible and avoids multiple sources of truth for this (similar to historical switch from hard-coded templates to embedded jinja2), but it would be slightly more invasive. Opinions? Mine is weakly held, so I'm happy to be minimally-invasive too.

[gabe-l-hart]

It looks like the challenge of (3) is having a decision-specific array hparam somewhat violates encapsulation. The other decision-specific hparams all get pulled into the llama_model.gguf_kv, but array-types get skipped, so without the actual llama_model_loader handy, we can't easily pull out array types from the GGUF without doing it somewhere in src (llama-hparams.cpp probably) which leaks the decision encapsulation into libllama which seems to have been deliberately avoided so far except for model-specific KVs.

Given that, I think it may make more sense to just piggyback on the existing hard-coding for the other dynamic-single-token architectures.

[gabe-l-hart]

Ok, I found that the LEV case was already more sophisticated than I realized, so that allows moving all the option determination code out of generation and leaving this all in the c++ side.

Ready for re-review @CISC (sorry for the churn)

[ngxson]

not 100% sure if it fits the use case, but we recently added decision.labels array that allows having labels in gguf instead of hard-coding in cpp

[gabe-l-hart]

not 100% sure if it fits the use case, but we recently added decision.labels array that allows having labels in gguf instead of hard-coding in cpp

Oh! That's exactly the route I was trying to go. I must have been a few commits behind

[gabe-l-hart]

Given that the LEV path is both well-established and easy to support, for now I'd say let's stick with this as written rather than trying to get the labels plumbed through conversion and in the GGUF.