Repository navigation
model : add H2O-Lightning-4B decision model - #30361
Conversation
Adds the h2o-lightning decision type for h2oai/h2o-lightning-4b (Qwen3.5-4B with a merged decision LoRA). None of the existing types match this model. It answers with " A" rather than "A", uses up to 255 labels, and was trained on Python json.dumps text, so floats like 12.0 must render as 12.0. The converter writes the decision metadata and the systemone template from the model's serve_config.json and refuses a config that doesn't match. Tests skip unless the tiny test model is available locally. Assisted-by: Claude Opus 5.5
github-actions
Bot
added
documentation
|
Hi @pseudotensor, thanks for your contribution! Per our contribution guidelines, the automated PR checker found the following issue(s) that need your attention:
Please note that maintainers reserve the right to make final decisions on PRs. If you believe there is a mistake, please comment below. |
ggml-gh-bot
Bot
added
the
draft
github-actions
Bot
removed
the
draft
|
there are quite some redundant changes here, I'll have a look a bit later. will push my clean up directly to this PR |
|
should be good for review now I put labels to a dedicated gguf metadata, hence the newly-added API in |
|
Hello @pseudotensor , I see that you mentioned in https://huggingface.co/h2oai/h2o-lightning-4b-gguf#differences-from-the-shim that llama.cpp tokenizes the text as sent but the shim is using NPC normalization. If you want, feel free to use the NPC normalizer using You can add it on the set_vocab with: |
Ah, good observation, thank you! |
|
@pseudotensor / @sandrohanea Since this is actually using the regular vocab mechanism it doesn't need to hardcode this, I will add support for detecting it in |
|
@sandrohanea Someone needs to add NFC to the other tokenizers (this model is not UGM) though, want to do a PR? |
Hello @CISC , 
|
|
@CISC done. |
|
@sandrohanea Are you able to approve workflows? |
… fails load on d812350) Owner-approved validation spike: prefill + softmax over candidate label tokens, no generation on the decider path, no llama.cpp bump and no cherry-pick of ggml-org/llama.cpp#30361. Stock-Q4_K_M is qwen35 but does not load on pin d812350: check_tensor_dims reports missing blk.32.attn_norm.weight (metadata claims MTP; the file has trunk blk.0-blk.31 only). Labels are read from GGUF KV via the gguf API. The prompt mirrors #30361 (Answer: prefill, leading-space labels, JSON options, 32k state cut, 255 options). Multi-token labels use the first token — a known limitation vs the upstream shim. 204/231 stays vendor-claimed; Series S cells pending. Co-authored-by: Gianluca Mazza <gianlucamazza@users.noreply.github.com>
* model : add H2O-Lightning-4B decision model Adds the h2o-lightning decision type for h2oai/h2o-lightning-4b (Qwen3.5-4B with a merged decision LoRA). None of the existing types match this model. It answers with " A" rather than "A", uses up to 255 labels, and was trained on Python json.dumps text, so floats like 12.0 must render as 12.0. The converter writes the decision metadata and the systemone template from the model's serve_config.json and refuses a config that doesn't match. Tests skip unless the tiny test model is available locally. Assisted-by: Claude Opus 5.5 * rm redundant code * put labels to gguf --------- Co-authored-by: Jonathan C. McKinney <jon.mckinney@h2o.ai> Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
Overview
Adds a decision type for H2O-Lightning-4B (https://huggingface.co/h2oai/h2o-lightning-4b), a Qwen3.5-4B fine-tune that answers typed decisions with calibrated probabilities. With this,
llama-serverserves it on/v1/systemonedirectly, without our Python shim.I tried the existing types first. The closest,
openjev, gets the same accuracy but differs on 4/231 answers and by up to 0.33 in probability. Our model answers with " A" rather than "A", uses up to 255 labels, and expects Python-style JSON in the prompt (12.0 stays 12.0).The converter reads the model's
serve_config.jsonand refuses to write a GGUF if it doesn't match what the code expects.Additional information
Checked against our reference implementation on 231 public decision tasks: all answers match, max probability difference 2.4e-6 (BF16 and Q8_0). Images: 30/30. Prompts are byte-identical except for non-NFC input.
The tests need a small random-weight test model (Apache-2.0, 46 MB). Until it is hosted they skip.
Two small related changes are on separate branches and can be separate PRs if wanted: an optional
noul_floorrequest field, and a fix so/v1/systemonerespects--no-cache-prompt.Requirements