[bong-water-water-bong]

What

A design note for #963 option 2 (render the NPU container's own chat_template), which that issue
left as an owner decision because it looked like a container-format change plus a new Jinja dependency.
Measured on this box, it is neither — so the decision is much smaller than it looked.

Doc only: docs/npu-chat-template.md. No code, no behaviour change.

The three findings that shrink it

1. The containers already carry the template. An imported OGA pack has chat_template.jinja; a
   FastFlowLM pack has tokenizer_config.json's chat_template. serve: name a model directory's chat template too, from its tokenizer_config.json (closes #1091) #1095 (merged) already reads it to log
   and pin its sha256, so the substrate exists.
2. The renderer is already in the tree we pin. llama.cpp's fork vendors a C++ Jinja engine in
   common/jinja/ (~6.2 kLOC, the one llama-server renders with). Reusing it means no new dependency and
   the same renderer our hrx/cpu route already trusts.
3. The templates use little of Jinja. Census over the 24 templates in the appliance catalogue:
   for/if/set+namespace/filters/attribute access/string methods/macros cover all of them;
   {% generation %} and content-gated branches on a user message appear in none. Sizes 209 B – 16.7 KB.

What the note records

• the measured state (above) with the file shapes,
• what the change is: one new app/npu_template.h, one call site (app/forward_serve.cpp:176), ChatML
  kept only for a container with no template and anchors, npu: the chat route refuses a container whose tokenizer cannot represent ChatML (measured: 200 with the wrong prompt -> 400 naming the model's specials) #987's 400 kept for neither,
• the security rules the renderer must carry (no attribute indirection, bounded loops/output, no
  include/import) on top of common/jinja's input marking, plus the sha line + pin from serve: name a model directory's chat template too, from its tokenizer_config.json (closes #1091) #1095,
• the four shipped NPU catalogue rows and what changes for each — none is blocked today (all four are
  ChatML-anchored); this is about serving each container the template it declares (e.g. FastFlowLM's
  user_system_prompt branch, which the ChatML fallback silently drops) and about the next
  non-ChatML container being servable at all,
• the three decisions it asks for: reuse common/jinja; fallback order container-template → ChatML only
  with anchors → 400; and a template that fails to render is a 400 naming its sha, never a silent fall
  back to ChatML.

Why this lane

The template audit (AUDIT-1, and the load-time line + pin in #1089/#1095) established what the
containers actually carry and made the template visible and checkable. This is the other end of the same
seam: the route that runs them.

[bong-water-water-bong]

#1115 ready for review at 0a3d328 — doc-only (1 file, +125), no code and no behaviour change. It is the design note #963's option 2 says it needs: the three costs the issue assumed (container-format change, new Jinja dependency, a big subset of Jinja) measured against this box and against the containers we actually ship. Commented on #963 with the same summary so the two are linked.

[bong-water-water-bong]

VERDICT: READY at 0a3d3280 — docs only, and the one finding I could check independently is true.

Finding 1 — "the containers already carry the template" — is the one I have evidence for, from verifying #1095 on this box: for the OGA packs we import, tokenizer_config.json carries no chat_template key at all and the template lives only in chat_template.jinja beside it (I ran the tool over the imported Qwen2.5-0.5B container: JSON branch dead, .jinja 2 561 B, sha 8aa40ce1…). So the substrate for #963 option 2 does exist, and the note's premise that this is "neither a container-format change nor a new Jinja dependency" starts from a fact rather than from an assumption.

A design note that exists to shrink an owner decision — "it looked like a format change plus a dependency; measured, it is neither" — is worth having even when it ships no code, because the cost of the decision is the thing that was blocking it.

No code, no behaviour change, one new file (docs/npu-chat-template.md), so there is nothing here to regress. I did not verify findings 2 and 3 (the renderer already being in the tree) beyond reading them.

[github-actions]

PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

🎫 Ticket compliance analysis 🔶 963 - Partially compliant Compliant requirements: The chat route now refuses containers whose tokenizer has no ChatML anchors with a 400 error The fix is minimal and doesn't change working Qwen3/Qwen2-Chat rows The solution checks for ChatML anchors in the tokenizer and refuses the chat route if they're missing Non-compliant requirements: The full implementation of rendering the container's own chat template (option 2) is not included in this PR Requires further human verification: The decision on whether to implement option 2 (rendering container's own chat template) needs to be made by the owners

⏱️ Estimated effort to review: 2 🔵🔵⚪⚪⚪

🧪 No relevant tests

🔒 No security concerns identified

⚡ Recommended focus areas for review Design Note Completeness The document describes the potential implementation of rendering the container's own chat template but is presented as a design note rather than an actual implementation. While it provides a detailed analysis of what would be needed, it doesn't actually implement the feature. This could be misleading as it suggests a complete solution when it's only a proposal. <!-- Copyright 2026 bong-water-water-bong SPDX-License-Identifier: Apache-2.0 Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. --> # The NPU route's chat template: render the container's own (engine #963 option 2) `app/forward_serve.cpp` renders **ChatML** for every container it serves, and since #987 it refuses the chat route (400) for a container whose tokenizer has no `<|im_start|>`/`<|im_end|>` anchors. That closed the silent-wrongness half of #963: a caller is never told a prompt was accepted when the model cannot read it. What is still missing is the other half — a container that carries *its own* template should be served with it, not with an approximation of it. This note records what that would take, **measured on the containers we ship**, and asks for the decision #963 says is needed. Three of the assumptions in that issue turn out to be wrong in our favour. ## What is actually true on this box (2026-11-05) **1. No container-format change is needed.** #963 assumed the model's template would have to be *packed into* the container. It is already there: an imported OGA pack (/tmp/844-out-clean, Qwen2.5-0.5B): added_tokens.json chat_template.jinja config.json generation_config.json merges.txt model.q4nx special_tokens_map.json tokenizer.json tokenizer_config.json vocab.json a FastFlowLM pack (~/Qwen3-0.6B-NPU2): config.json model.q4nx tokenizer.json tokenizer_config.json The first carries `chat_template.jinja` (2561 B, `8aa40ce145adb73c…`); the second carries the template as `tokenizer_config.json`'s `chat_template` string. Both are read and **pin-able** since #1095: every `.gguf` *and* model-directory load now prints `template sha256=…` and compares it with `config/template-pins.json`. So the substrate this design needs is already merged. **2. No new dependency is needed.** The pinned llama.cpp fork — which this engine already builds against — vendors a C++ Jinja engine in `common/jinja/` (≈6.2 kLOC: lexer, parser, runtime, value, caps; upstream llama.cpp's own, introduced in ggml-org/llama.cpp#18462). It is what `llama-server` uses to render chat templates, so it is the same renderer our `hrx`/`cpu` route already trusts. Compiling that directory into `1bit` plus a small adapter (`messages` → context → `render`) is the whole of the engine change; the alternative — inventing a mini-renderer — would be a second, weaker implementation of the same thing. **3. The templates we ship are small and use little of Jinja.** Census over the 24 templates in the appliance's catalogue (17 distinct), plus the containers on this box: | construct | templates | |---|---| | `{% for %}`, `{% if/elif/else %}`, filters (`selectattr`, `map`, `join`, `tojson`, `trim`, `default`, `replace`, `split`, `first`, `last`, `string`) | 24/24 | | `{% set %}` + `namespace(...)` | 23/24 | | attribute access (`message.content`, `messages[0].role`) | 23/24 | | `is defined` / `is string` / `is none` | 23/24 | | string methods (`.split`, `.strip`, `.lstrip`, `.rstrip`, `.startswith`) | 18/24 | | macros (`{% macro %}`) | 10/24 | | `{% generation %}` (the training-mask tag) | 0/24 | | **branch on the content of a user message** (the shape of the paper's attack in AUDIT-1) | 0/24 | Sizes run from **209 B** (`LiquidAI/LFM2-1.2B`, ChatML-shaped) to **16.7 KB** (`ggml-org/gpt-oss-20b`). Everything in that table is already covered by `common/jinja`; nothing needs a language extension. ## What the change is 1. `app/npu_template.h` (new): given a model directory, load the template — `tokenizer_config.json`'s `chat_template` string, else `chat_template.jinja` (the order #1095 already uses for the line and the pin, so the two cannot disagree) — and render `messages` + `add_generation_prompt` (+ `tools`, `enable_thinking` as the request provides them) through `common/jinja`. 2. `app/forward_serve.cpp`: `if (chat)` renders with the container's template when it has one. ChatML stays as the fallback **only** for a container with no template *and* ChatML anchors; a container with neither keeps #987's 400. 3. `tests/npu_template_test.cpp` + `tests/npu_template_route.sh`: the four containers we ship as golden prompts, and the token counts compared against `llama-server`'s rendering of the same conversation (that is the #963 measurement — 33 vs 15 tokens on ZR1 — turned into an assertion). ## What it costs, and what it can get wrong - **Engine size**: `common/jinja` is ~6.2 kLOC of C++ compiled into `1bit` (+nlohmann, already a dependency of `app/serve.cpp`). Nothing is downloaded at build time. - **Security posture**: a renderer is the one component that turns data into a prompt, so the rules matter more than the code. `common/jinja` already implements input marking against special-token injection; on top of it I would (a) refuse attribute indirection (`__class__`, `__globals__`, `cycler`) — the audit's red-flag list — (b) bound loop iterations and output size, (c) never `include` or `import`, and (d) keep the sha line + pin from #1095, which makes "the template changed" visible even when the render is valid. - **The one behavioural risk**: today every NPU container is prompted with ChatML and the Qwen3 answer parser expects that shape. Rendering the vendor's template changes the *prompt*, not the parser, so the Qwen3-family rows do not move (their template is ChatML), the FastFlowLM rows gain their `user_system_prompt` extension, and a non-ChatML container moves from "refused" to "served" — which is exactly the change #963 asks for, and exactly why it is a decision rather than a patch. ## Measured for the decision: what the four shipped NPU rows would do | catalogue row | template | ChatML anchors | today (#987) | with this note | |---|---|---|---|---| | `1bit-MONSTER/Qwen3.6-35B-A3B-NPU` | `e84f32a23f…`, 7764 B | yes | 200 | 200, rendered from its own template | | `FastFlowLM/Qwen3-0.6B-NPU2` | `a136047ba140…`, 4449 B | yes | 200, **without** FLM's `user_system_prompt` branch | 200, with it | | `FastFlowLM/Qwen3-1.7B-NPU2` | `a136047ba140…` | yes | as above | as above | | `FastFlowLM/Qwen3-4B-NPU2` | `a136047ba140…` | yes | as above | as above | So no shipped row is blocked today — this is about serving each container with the template it declares instead of one that happens to work for the Qwen3 family, and about being able to serve the next non-ChatML container (Zamba2-VL, a DeepSeek-special model, anything a customer packs) at all. ## The decision this asks for 1. **Reuse `common/jinja`** (my recommendation: it is the same renderer the llama.cpp route uses, no new dependency, and its input marking is already the security half) — or vendor a second implementation, or defer. 2. **Confirm the fallback order**: container template → ChatML only with anchors → 400. Nothing else. 3. **Confirm the strictness**: a container whose template fails to render should be a 400 naming the template sha, not a silent fall back to ChatML — the fall-back-to-something-else is the failure mode #963 exists to prevent. If the answer to (1) is "yes", the work is one PR: the renderer glue plus **one** call site (`app/forward_serve.cpp:176`, the only `npu::chatml()` render on the route; the load-time check at `:772` becomes the "no template and no anchors" case), the golden-prompt tests, and the pin rows for the containers we ship (the four above).