Changing reasoning_effort re-reads the whole prompt: the effort text is the first thing the template renders #458

What happens

serve/chat_template.jinja puts the reasoning effort instructions at the very top of the first system turn (<|im_start|>system\nReasoning effort is set to xhigh. ...). The conversation cache reuses the longest common token prefix, so a request that only changes reasoning_effort (or turns thinking off) differs from the previous one a few tokens in and re-reads the entire prompt.

Agent clients do this routinely. OpenCode's variants, a "think harder" toggle, a subagent with a different effort, none for a quick tool step. On a 150K-token agent session that is a full prefill (~100 s at 1.5K tok/s on a 3090) just for switching the level.

Measured (no GPU needed)

The same 22K-token conversation (system prompt + 20 turns) rendered with this repo's template at xhigh vs. the other levels, tokenized with the Strata tokenizer; "first differing token" is the most the cache can reuse:

What I'm running locally (works, but it needs both halves)

1. Template: drop reasoning_instructions from the top system turn and emit it as a short system turn right before the generation prompt:

   {%- if add_generation_prompt %}
       {%- if reasoning_instructions %}
           {{- '<|im_start|>system\n' + reasoning_instructions + '<|im_end|>\n' }}
       {%- endif %}
       {{- '<|im_start|>assistant\n' }}

2. Engine: the turn checkpoint goes at the last <|im_start|>, which is now the assistant header after that trailing block - so with the template change alone the checkpoint holds the effort text and even a normal next turn reuses nothing (I measured 0 reused on a plain follow-up with stock 0.1.34 + the new template). A small rule moves it in front of the trailing system turn (--tail-role-token <id of "system">, 8678 for this tokenizer):

   if (turn_at > 0 && o.tail_role_token >= 0)
       for (int64_t i = turn_at - 1; i > resume; --i)
           if (ids[(size_t) i] == o.turn_token) {
               if (ids[(size_t) i + 1] == o.tail_role_token) turn_at = i;   // <|im_start|>system right before
               break;
           }

   (It could just as well be the default whenever the turn before the last header is a system turn, without a flag.)

With both, on an RTX 3090 (OrcaRouter IQ3_XXS, 262K, int8 KV): switching effort on an 8,222-token prompt reused 8,155 tokens; seven requests in a row on a ~5K prompt, each with a different effort value (extreme, minimal, none, low, high, and two non-standard ones that the template maps to its default), reused 5,005-5,047 tokens on every request after the first, and normal follow-up turns keep reusing as before.

The open question

This changes what the model sees: the effort text moves from the top of the system prompt to just before the answer. I have not measured whether that changes how well the level is followed or answer quality (the official Qwen template puts it at the top). Would you take a PR along these lines (perhaps opt-in first) or do you prefer another approach (e.g. a checkpoint right after the effort line, so only the system prompt is re-read)?