Skip to content

jinja input marking is never consumed by llama-server #26532

Closed as duplicate

Description

@professor-moody

#18462 added jinja::string with an is_input flag so special tokens coming from user content can be told apart from ones the template emitted, and listed "implement input marking on llama-server" as a follow-up. That follow-up doesn't seem to have landed.

At 3581ba0 the flag is produced but nothing reads it:

  • common/chat-auto-parser.h:73 defaults mark_input = true
  • common/chat.cpp:924 calls global_from_json(ctx, inp, inputs.mark_input)
  • tools/server/ has zero references to is_input or mark_input
  • all four tokenize_input_prompts calls in server-context.cpp pass parse_special = true

So a two-message conversation with a ChatML payload in the user content tokenizes to the same ids as a real system turn (151644, 8948 on qwen2.5). Nothing downstream can distinguish them.

I know special token injection is a known class and this is a documented TODO rather than a surprise. What made me want to bring it up, transformers has no working equivalent. apply_chat_template(tokenize=True) forges the turns, split_special_tokens isn't a parameter it accepts, and on the two-step path that flag strips the template's own delimiters too, leaving no role structure at all. Keeping the template's delimiters while dropping the content's needs origin tracking, and is_input looks like the only implementation of it anywhere.

Is the follow-up still on the roadmap, or was it dropped deliberately? Happy to help if it's just unowned.

Activity

  1. CISC commented on Aug 4, 2026

    @CISC
    Member

    I think it's still on @ngxson TODO, but it's not trivial (if at all 100% feasible) so not high priority.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions