Repository navigation
chat: replace output string with parts - #30389
Conversation
github-actions
Bot
added
documentation
| std::string out; | ||
| for (const auto & part : parts) { | ||
| out += part.text; | ||
| } |
There was a problem hiding this comment.
Should we do a reserve here first? To something reasonable, since I assume you can't find the length without traversing the parts anyway.
| const auto & msg = inputs.continue_msg; | ||
|
|
||
| data.generation_prompt = GEN_PREFIX + THINK_START + msg.reasoning_content; | ||
| if (inputs.continue_final_message == COMMON_CHAT_CONTINUATION_CONTENT) { | ||
| data.generation_prompt += THINK_END + TEXT_START + msg.render_content(); | ||
| } | ||
|
|
||
| data.prompt += data.generation_prompt; | ||
| data.prompt.append(data.generation_prompt); |
There was a problem hiding this comment.
In a follow up, maybe we can add another append() or a + operator to concat common_chat_prompt values. I can update the parsers to provide a common_chat_prompt value with the proper input marking for assistant prefilled messages.
| if (inputs.is_inference && !inputs.add_generation_prompt) { | ||
| static constexpr std::string_view return_token = "<|return|>"; | ||
| static constexpr std::string_view end_token = "<|end|>"; | ||
| if (size_t pos = prompt.rfind(return_token); pos != std::string::npos) { | ||
| prompt.replace(pos, return_token.length(), end_token); | ||
| } | ||
| prompt.replace_last(std::string(return_token), std::string(end_token)); | ||
| } |
There was a problem hiding this comment.
If we derive the generation prompt during rendering of the template, this entire block goes away. Actually, since the refactoring of llama-cli, I'm not sure it's even needed anymore. I'll revisit this later.
|
A heads up for the actual tokenization: Qwen's thinking tags are not marked as special but user defined. There is no way to avoid parsing user defined tokens like there is for special tokens, but that's what is needed to tokenize those thinking tags into multiple pieces for user input. |
|
@aldehir sounds like we need some extra predicate to mark some tokens as "special" for the purposes of the parser - might be something to add to the parser output (already have "reserved tokens", which basically serve this purpose). |
Upstream ggml-org/llama.cpp#30389 turned common_chat_params::prompt into template and input parts. Chorus tokenizes the rendered prompt as one string, as upstream's own callers do, so it flattens the parts. Moves llama.cpp to 184f65360 and refreshes the resource-cleanup patch (offsets only).
* chat: replace output string with parts * reserve first
Overview
Second item of #28249
Formatted chat will now be an array of parts, not just a simple string. This allows selectively parsing special tokens for only certain parts (will be a follow-up PR)
Requirements