ui: add conversation compaction by ServeurpersoCom · Pull Request #27401 · ggml-org/llama.cpp

ServeurpersoCom · 2026-08-19T17:57:57Z

Overview

Close #20632

Type /compact or use the Compact button in the context gauge popup: the model summarizes the conversation so far, and later turns continue from that summary instead of the full history, freeing up context. The full history stays visible and navigable in the UI, with the summary shown as a collapsible card. The summarization prompt can be customized in Settings.

Automatic compaction near the context limit will follow in a separate PR, as an opt-in setting.

Additional information

Part of #24473

Requirements

I have read and agree with the contributing guidelines
AI usage disclosure: YES

ServeurpersoCom · 2026-08-20T06:38:56Z

1) Full conversation, staring to Compact:

2) Inference concludes with a context containing the entire conversation plus the summary, so statistics don't show the compacted one, but the full one:

3) The prefill, and thus the updating of statistics, only occurs when the subsequent inference is launched:

ServeurpersoCom · 2026-08-20T07:41:44Z

Honor the “Pre-fill KV cache after response” setting at the end of a compaction request

ServeurpersoCom · 2026-08-20T11:40:37Z

After compaction, a KV cache prefill is systematically launched, and the gauge updates immediately :

Compact.mp4

ServeurpersoCom · 2026-08-20T11:49:47Z

(squashing 2 commits into 1 so the history doesn't lie)

ServeurpersoCom · 2026-08-23T07:13:26Z

Rebased

cb88 · 2026-08-28T00:31:45Z

This might be a later feature but something it seems a lot of people do is use a smaller and or faster model eg 35B MOE vs 27B qwen for the summarization... so it would be nice if there was a way to do that say if you have router mode enabled there should be an option so select the summarization model.

ServeurpersoCom · 2026-08-28T03:52:42Z

Good idea. #27567 shares this exact behavior (secondary inference with a model picker), and on that PR the same gap exists: the mic button stays greyed out until the chosen model is loaded by hand, instead of autoloading it on demand. Both should go through one common path where the selected model is autoloaded when missing, with a clear error when --models-max is 1 or the model fails to load.

ServeurpersoCom · 2026-08-29T15:53:31Z

Screen_Recording_20260829_174746_Kiwi.Browser.mp4

This select still has the upstream bits-ui bug we ran into a year ago on our first model selector: on mobile the list jumps back to the top instantly instead of scrolling smoothly, and on desktop the menu content shifts as the scroll arrows appear and disappear. I never managed to get it working properly on our side, even with the wheel and touchmove workarounds already sitting in our select wrapper, so we ended up building our own picker: probably worth reusing it here.

ServeurpersoCom · 2026-08-29T16:41:45Z

This might be a later feature but something it seems a lot of people do is use a smaller and or faster model eg 35B MOE vs 27B qwen for the summarization... so it would be nice if there was a way to do that say if you have router mode enabled there should be an option so select the summarization model.

I reused the standard model selector. Compaction from another model works automatically. You can test it:)

NickM-27 · 2026-09-10T13:17:58Z

Along these lines, I think it would be great if it was also possible to select a dedicated model for the chat title generation.

ServeurpersoCom · 2026-09-10T13:48:58Z

I rebased this branch.

Along these lines, I think it would be great if it was also possible to select a dedicated model for the chat title generation.

That's a good idea. Could you open a feature request for the discussion title generation model choice ?

NickM-27 · 2026-09-10T13:55:14Z

Created at #28703

Just curious, are these current features waiting to be merged until the current UI work which looks to refactor the model selector are done?

ServeurpersoCom · 2026-09-10T20:04:49Z

Yes, we need to wait for allozaur to return, We need his UI review.

ServeurpersoCom · 2026-09-10T20:10:54Z

The model selection parameters should be streamlined; regarding the design, I feel that reusing the standard model selector for Settings doesn't look very well-integrated. Submitting the feature request is sufficient, since I can also use Opus 5 and Fable 5.1 but inside a refactoring PR.

ServeurpersoCom · 2026-10-14T06:53:12Z

This one needs a significant improvement: compaction works better when the last N messages (configurable) are kept verbatim in the new context. These N messages can be excluded from the summarization input without triggering a full prefill, which makes it more efficient to reuse the conversation model for summarization, both in compute and in quality (the input stays closer to the training distribution). Optionally, a synthetic prompt can be inserted between the summary and the last N messages (e.g. “Here are the most recent messages in full, to continue the conversation.”).

Rebase required

The models manager moved ModelsSelectorList under ModelsSelector and the model option helpers to /utils, where groupModelOptions takes the options alone. Drop the stale copy the rebase kept and read the helpers from their new home.

A summary written by another model loses the prefix the conversation model cached, so the whole history is prefilled again, and the model then resumes from text it did not write. The conversation model always writes its own summary.

A compaction now folds the steps the model reads but the last ones, a step being a user message or an assistant message with its tool results, and stores how many it kept on the compaction node. The model then reads the system prefix, the summary, the kept steps and what follows. The summary request is the request the chat sends, with its options, tools and message shaping, cut before the preserved steps: the server reuses the prefix it cached and only the instruction is new. The number of preserved steps is a setting, 10 by default.

Three callers needed the same thing - the automatic pre-encode pass, the action in the context gauge, and re-running a summary from the block - and each of them had grown its own copy of the request, the streaming and the storing. There is one path now: a summarization owns a node, streams into it, and reports itself through a ledger the rest of the app reads (`isCompacting`, which node belongs to which conversation). Re-running a summary is the same call on an existing node, so the two entry points cannot drift apart again.

The summary was a line of text in a tool-call-shaped collapsible, which is how a compaction reads as a side note rather than as the thing the rest of the conversation now stands on. The row is its own block: a divider that names the boundary, and the summary inside it with the model that wrote it, its statistics, and the same actions an assistant turn has - copy, edit, fork, regenerate and delete all apply to the summary, and it collapses to the divider once a summary is in, because the shrunk context is the headline and the prose is the detail. The divider is a variant of the shared collapsible block, which also needed the toggle and the title split into snippets so a caller can lay them out without copying the markup. The action icons no longer take a role: they render what a message offers, and a compaction offers the same as any other stored message.

The row owns the run, so the row says how it is going: the prompt progress in the header while the context is being read, the phase once the summary starts streaming, and the token count once it is in. Before the first summary token there is nothing to collapse, so the header stands on its own instead of offering an empty block.

The turns being summarized used to keep their own "Processing..." indicator and live reading stats while the summary streamed, so the conversation said the same thing twice in two places. The row reports the run, so the turns it replaces stay silent until it is done.

Compacting is a decision about the context, so the action belongs with the numbers that show the context filling up rather than inside the form: it sits at the bottom of both the hover card and the drawer, under the breakdown it acts on, quiet enough to stay a footnote and disabled - with the row's own spinner - while a summary is already running.

ServeurpersoCom · 2026-10-26T11:12:54Z

Rerebased:)

The screen scrolled itself with a controller of observers, intervals, direction heuristics and landing timers, and still managed to move the view while the reader was reading: any mutation that grew the branch was treated as a reason to keep following. The page is the scroller, so it follows when the reader sits at the bottom, and stays where it is otherwise. A resize observer plus native scroll anchoring does the rest, and `pointerdown` buys the reader a moment of grace to be caught mid-drag.

Expanding a block snapped it open: the body was mounted the moment `open` flipped, so a tall block moved the conversation around it in one frame. The body now goes through the same region the models table uses. The height interpolates and the visibility follows it, so a body is unmounted once the collapse has finished rather than the moment it starts - a closed block still costs nothing while tokens stream past it.

Toggling a summary left the reader wherever they were, so a conversation that had just compacted showed the divider at the bottom of the screen and read down from there. A toggle now scrolls the page once and lands the divider a breath under the tabs, both ways: the summary is read from its beginning, and closing it returns to the same boundary instead of wherever the page happens to come to rest. The scroll waits for the body to reach its height, because before that the page is too short to scroll the divider up there and the call would clamp. That wait is the shared collapse duration rather than a transition event: an engine without `interpolate-size` animates the region's grid row instead of its height, so a listener on `height` would never fire there and the page would not move at all.

The divider reported the phase in its title while the icon still showed a finished block, so a run in progress and a summary that had landed looked the same in the one place the eye reads first. The icon spins while the summary is running and folds back once it is in.

The summary is the thing the conversation now stands on, not a note beside it, and the conversation model always writes it, so the body drops the reasoning block's rule and indent (`flushBody` keeps them for the blocks that borrow them) and the model selector under the summary is gone - there is nothing left to swap.

The summary row no longer offers a model selector, so nothing hands regenerateCompaction a model anymore. Remove the parameter from the whole path, with the on-demand load of another model it carried: the summary always comes from the model the request funnel resolves.

The summary request forced thinking off, overriding the reasoning effort the user chose. The summary is the text the conversation resumes from, the task where reasoning helps the most, so the request now carries the very options of the chat.

The post-compaction pre-encode sent the branch without tools, while the tools render ahead of the messages, so it warmed a prefix the next turn never reads and that turn prefilled the whole compacted context again. The pre-encode now sends the tools, message shaping and reasoning handling of the chat, so the next turn only prefills its new message.

The per-turn pre-encode sent the messages of the request plus one assistant message holding the streamed text, without tools. With tools on, it replaced the slot the next turn extends with a prefix nobody reads again, and after an agentic turn its history did not match the stored calls and results. Without agentic, it never ran. Both pre-encodes now read the stored branch up to its leaf, cut at the last compaction, and send it with the tools, message shaping and reasoning handling of the chat. The plain completion path triggers it too.

The stream handler read every chunk but dropped the finish reason, so a summary cut by max_tokens folded the conversation into half a text. The handlers now hand the finish reason to onComplete, and a summary that ends on length rolls the compaction back with an error. The summary was framed through a replacement string, so $& or 32560 in it changed the text the model read; it is framed as written now.

ServeurpersoCom · 2026-10-26T11:24:05Z

Second push drops two leftover scroll hooks that broke svelte-check.

The collapsible body now unmounts once its collapse animation ends rather than as soon as the block closes, so the "folds back" assertions ran before the body was gone. Wait out COLLAPSIBLE_TRANSITION_MS first. Assisted-by: Claude (Opus 5.5 Medium)

ServeurpersoCom · 2026-10-26T13:28:35Z

Thanks for the fix

Conflicts (all in tools/ui, from ui: conversation compaction ggml-org#27401, model details drawer ggml-org#31176, shared delete dialog ggml-org#31187, reasoning toggle ggml-org#31212): - import unions; follow upstream restructures (openModelDetails, view-based tabs, CollapsibleContentBlock rewrite, use-stick-to-bottom hook rename) - drop imports upstream made unused (MessageRole, deviceStore, ROUTES, RouterService, use-auto-scroll) - keep the fork t() text layer on every reworked block - DialogModelInformation deleted upstream: accept deletion (no refs left) - window.confirm replaced by DialogConversationDelete in sidebar + search i18n: translate new UI (model drawer title/description, compaction labels, sidebar resize tooltip) and add the rewritten auto-scroll help; 866 keys, 0 missing, 0 duplicate.

* ui: add conversation compaction Add a compaction action that asks the model to summarize the conversation so far and stores the summary in a new compaction node appended to the branch. Later requests slice the branch at the last compaction node, keeping the system prefix and continuing from the summary instead of the full history, which frees up context while the complete history stays visible and navigable in the UI. The slice is applied once at the request funnel, covering the plain send, the agentic flow and the KV cache pre-encode, plus the continue path that bypasses it. The summary request is a one-shot outside the SSE replay machinery, streamed live into the node with a full rollback on abort, error or empty result. The action is exposed as a /compact slash command and a Compact button in the context gauge popup, with an overridable summarization prompt in the settings. * ui: pre-fill KV cache and refresh context gauge after compaction Once the summary is stored, pre-encode the compacted branch and stamp the returned prompt timings on the compaction node: re-encoding the reduced prefix is part of the compaction itself, so the server KV cache and the context gauge reflect the summary right away. The summary stream is registered under a prefixed identity (compact::convId), so the visibility kick and byte-offset resume protect it through tab switches while the reload probe never reattaches it as a chat turn. The stamped node becomes the shared context truth for the gauge and the processing state restore, and the stale live state that kept the gauge full is cleared. * chore: format * ui: allow a dedicated model for conversation compaction An opt-in setting reveals a model selector whose options come from the live router model list. The chosen model writes the summary and is loaded on demand when it is not resident, while the default path keeps resolving to the conversation's own model. * ui: align the compaction model field with project conventions * ui: fix the model option helpers import path The compaction model selector pulls filterModelOptions and groupModelOptions from the models component folder, where they no longer live, so the build fails to resolve the module. Point the import at the navigation folder, which now owns them. Missed during the rebase: the helpers moved in a commit that does not touch any compaction file, so nothing conflicted. * ui: import the model option helpers from models/utils * ui: follow the moved models selector in the compaction model field The models manager moved ModelsSelectorList under ModelsSelector and the model option helpers to /utils, where groupModelOptions takes the options alone. Drop the stale copy the rebase kept and read the helpers from their new home. * ui: drop the dedicated compaction model A summary written by another model loses the prefix the conversation model cached, so the whole history is prefilled again, and the model then resumes from text it did not write. The conversation model always writes its own summary. * ui: keep the last steps of a compacted conversation A compaction now folds the steps the model reads but the last ones, a step being a user message or an assistant message with its tool results, and stores how many it kept on the compaction node. The model then reads the system prefix, the summary, the kept steps and what follows. The summary request is the request the chat sends, with its options, tools and message shaping, cut before the preserved steps: the server reuses the prefix it cached and only the instruction is new. The number of preserved steps is a setting, 10 by default. * ui: run every compaction summary through one path Three callers needed the same thing - the automatic pre-encode pass, the action in the context gauge, and re-running a summary from the block - and each of them had grown its own copy of the request, the streaming and the storing. There is one path now: a summarization owns a node, streams into it, and reports itself through a ledger the rest of the app reads (`isCompacting`, which node belongs to which conversation). Re-running a summary is the same call on an existing node, so the two entry points cannot drift apart again. * ui: give the compaction row its own summary UI The summary was a line of text in a tool-call-shaped collapsible, which is how a compaction reads as a side note rather than as the thing the rest of the conversation now stands on. The row is its own block: a divider that names the boundary, and the summary inside it with the model that wrote it, its statistics, and the same actions an assistant turn has - copy, edit, fork, regenerate and delete all apply to the summary, and it collapses to the divider once a summary is in, because the shrunk context is the headline and the prose is the detail. The divider is a variant of the shared collapsible block, which also needed the toggle and the title split into snippets so a caller can lay them out without copying the markup. The action icons no longer take a role: they render what a message offers, and a compaction offers the same as any other stored message. * ui: report compaction progress on the row The row owns the run, so the row says how it is going: the prompt progress in the header while the context is being read, the phase once the summary starts streaming, and the token count once it is in. Before the first summary token there is nothing to collapse, so the header stands on its own instead of offering an empty block. * ui: keep the summarized turns quiet while a compaction runs The turns being summarized used to keep their own "Processing..." indicator and live reading stats while the summary streamed, so the conversation said the same thing twice in two places. The row reports the run, so the turns it replaces stay silent until it is done. * ui: hand the compact action to the context gauge Compacting is a decision about the context, so the action belongs with the numbers that show the context filling up rather than inside the form: it sits at the bottom of both the hover card and the drawer, under the breakdown it acts on, quiet enough to stay a footnote and disabled - with the row's own spinner - while a summary is already running. * ui: follow the bottom, otherwise leave the view alone The screen scrolled itself with a controller of observers, intervals, direction heuristics and landing timers, and still managed to move the view while the reader was reading: any mutation that grew the branch was treated as a reason to keep following. The page is the scroller, so it follows when the reader sits at the bottom, and stays where it is otherwise. A resize observer plus native scroll anchoring does the rest, and `pointerdown` buys the reader a moment of grace to be caught mid-drag. * ui: expand collapsible bodies with a height transition Expanding a block snapped it open: the body was mounted the moment `open` flipped, so a tall block moved the conversation around it in one frame. The body now goes through the same region the models table uses. The height interpolates and the visibility follows it, so a body is unmounted once the collapse has finished rather than the moment it starts - a closed block still costs nothing while tokens stream past it. * ui: open a compaction summary at the top of the screen Toggling a summary left the reader wherever they were, so a conversation that had just compacted showed the divider at the bottom of the screen and read down from there. A toggle now scrolls the page once and lands the divider a breath under the tabs, both ways: the summary is read from its beginning, and closing it returns to the same boundary instead of wherever the page happens to come to rest. The scroll waits for the body to reach its height, because before that the page is too short to scroll the divider up there and the call would clamp. That wait is the shared collapse duration rather than a transition event: an engine without `interpolate-size` animates the region's grid row instead of its height, so a listener on `height` would never fire there and the page would not move at all. * ui: spin the compaction icon while the summary runs The divider reported the phase in its title while the icon still showed a finished block, so a run in progress and a summary that had landed looked the same in the one place the eye reads first. The icon spins while the summary is running and folds back once it is in. * ui: let the compaction summary speak without the reasoning trimmings The summary is the thing the conversation now stands on, not a note beside it, and the conversation model always writes it, so the body drops the reasoning block's rule and indent (`flushBody` keeps them for the blocks that borrow them) and the model selector under the summary is gone - there is nothing left to swap. * ui: drop the model override from the compaction path The summary row no longer offers a model selector, so nothing hands regenerateCompaction a model anymore. Remove the parameter from the whole path, with the on-demand load of another model it carried: the summary always comes from the model the request funnel resolves. * ui: let the compaction summary think as the chat does The summary request forced thinking off, overriding the reasoning effort the user chose. The summary is the text the conversation resumes from, the task where reasoning helps the most, so the request now carries the very options of the chat. * ui: pre-encode the compacted branch as the next turn reads it The post-compaction pre-encode sent the branch without tools, while the tools render ahead of the messages, so it warmed a prefix the next turn never reads and that turn prefilled the whole compacted context again. The pre-encode now sends the tools, message shaping and reasoning handling of the chat, so the next turn only prefills its new message. * ui: pre-encode the branch the next turn actually reads The per-turn pre-encode sent the messages of the request plus one assistant message holding the streamed text, without tools. With tools on, it replaced the slot the next turn extends with a prefix nobody reads again, and after an agentic turn its history did not match the stored calls and results. Without agentic, it never ran. Both pre-encodes now read the stored branch up to its leaf, cut at the last compaction, and send it with the tools, message shaping and reasoning handling of the chat. The plain completion path triggers it too. * ui: refuse a compaction summary the output limit cut The stream handler read every chunk but dropped the finish reason, so a summary cut by max_tokens folded the conversation into half a text. The handlers now hand the finish reason to onComplete, and a summary that ends on length rolls the compaction back with an error. The summary was framed through a replacement string, so $& or 32560 in it changed the text the model read; it is framed as written now. * ui: wait for the collapse transition in the tool call tests The collapsible body now unmounts once its collapse animation ends rather than as soon as the block closes, so the "folds back" assertions ran before the body was gone. Wait out COLLAPSIBLE_TRANSITION_MS first. Assisted-by: Claude (Opus 5.5 Medium) --------- Co-authored-by: Aleksander Grygier <aleksander.grygier@gmail.com>

ServeurpersoCom requested a review from a team as a code owner August 19, 2026 17:57

ServeurpersoCom requested a review from allozaur August 19, 2026 17:58

github-actions Bot added the server/ui label Aug 19, 2026

ServeurpersoCom force-pushed the ui/compact-conversation branch from 4324b87 to 3d2d7c6 Compare August 20, 2026 11:48

ServeurpersoCom mentioned this pull request Aug 23, 2026

Feature Request: Graceful reasoning budget termination. Avoid mid-sentence cutoff. #20632

Closed

4 tasks done

ServeurpersoCom force-pushed the ui/compact-conversation branch from 3d2d7c6 to c495801 Compare August 23, 2026 07:12

ServeurpersoCom force-pushed the ui/compact-conversation branch from c495801 to 277b042 Compare August 23, 2026 09:06

ServeurpersoCom force-pushed the ui/compact-conversation branch from 277b042 to 48062de Compare August 29, 2026 14:43

ServeurpersoCom mentioned this pull request Aug 29, 2026

UI: allow mic input when an audio model is loaded #27567

Open

ServeurpersoCom force-pushed the ui/compact-conversation branch from 761e226 to f3c0908 Compare September 10, 2026 13:44

NickM-27 mentioned this pull request Sep 10, 2026

UI: Ability to select model to be used for chat title summarization #28703

Open

4 tasks done

NickM-27 mentioned this pull request Sep 10, 2026

ui: Add option to select model to be used for title generation #28710

Closed

ServeurpersoCom force-pushed the ui/compact-conversation branch from 16242fc to eed1430 Compare September 17, 2026 14:45

allozaur self-assigned this Sep 29, 2026

ServeurpersoCom force-pushed the ui/compact-conversation branch from 8f2e8a8 to 6e24c83 Compare October 21, 2026 07:40

ServeurpersoCom and others added 8 commits October 26, 2026 12:00

ui: drop the dedicated compaction model

b07cea7

A summary written by another model loses the prefix the conversation model cached, so the whole history is prefilled again, and the model then resumes from text it did not write. The conversation model always writes its own summary.

ServeurpersoCom force-pushed the ui/compact-conversation branch from b426d3a to 6ada18e Compare October 26, 2026 11:10

allozaur and others added 10 commits October 26, 2026 12:16

ServeurpersoCom force-pushed the ui/compact-conversation branch from 6ada18e to c4e3472 Compare October 26, 2026 11:19

allozaur added the merge ready A maintainer can use this label to indicate that they consider the changes final and ready to merge. label Oct 26, 2026

ggerganov merged commit 1a97e38 into ggml-org:master Oct 27, 2026
7 checks passed

ggerganov mentioned this pull request Oct 28, 2026

llama.cpp : bump version to 0.8.0 #31309

Merged

Conversation

ServeurpersoCom commented Aug 19, 2026 • edited by allozaur Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Overview

Additional information

Requirements

Uh oh!

ServeurpersoCom commented Aug 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

1) Full conversation, staring to Compact:

2) Inference concludes with a context containing the entire conversation plus the summary, so statistics don't show the compacted one, but the full one:

3) The prefill, and thus the updating of statistics, only occurs when the subsequent inference is launched:

Uh oh!

ServeurpersoCom commented Aug 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ServeurpersoCom commented Aug 20, 2026

After compaction, a KV cache prefill is systematically launched, and the gauge updates immediately :

Uh oh!

ServeurpersoCom commented Aug 20, 2026

Uh oh!

ServeurpersoCom commented Aug 23, 2026

Uh oh!

cb88 commented Aug 28, 2026

Uh oh!

ServeurpersoCom commented Aug 28, 2026

Uh oh!

ServeurpersoCom commented Aug 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ServeurpersoCom commented Aug 29, 2026

Uh oh!

NickM-27 commented Sep 10, 2026

Uh oh!

ServeurpersoCom commented Sep 10, 2026

Uh oh!

NickM-27 commented Sep 10, 2026

Uh oh!

ServeurpersoCom commented Sep 10, 2026

Uh oh!

ServeurpersoCom commented Sep 10, 2026

Uh oh!

ServeurpersoCom commented Oct 14, 2026

Uh oh!

ServeurpersoCom commented Oct 26, 2026

Uh oh!

ServeurpersoCom commented Oct 26, 2026

Uh oh!

ServeurpersoCom commented Oct 26, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

ServeurpersoCom commented Aug 19, 2026 •

edited by allozaur

Loading

ServeurpersoCom commented Aug 20, 2026 •

edited

Loading

ServeurpersoCom commented Aug 20, 2026 •

edited

Loading

ServeurpersoCom commented Aug 29, 2026 •

edited

Loading