[adrysn]

Phase A of #2397: explicit conversation compaction for the Responses API. Phase B (automatic context_management compaction on /v1/responses, the output cap at remaining context, and the streamed mlx.compaction.started / mlx.compaction.completed events) follows in a separate PR that closes the issue. The split keeps phase B separate; phase A is still +2337 lines, of which about 980 are tests and 130 docs.

Behavior

• POST /v1/responses/compact and /responses/compact take {model, input, instructions?, tools?, max_output_tokens?, keep_tokens?} and return {id, object: "response.compaction", output: [instructions, retained items, one compaction item], usage}. usage counts the one summary pass. Nothing is stored.
• Selection keeps system/developer messages in their roles, the latest user message, and the newest whole turns within keep_tokens (default 1/16 of the per-slot context, at most 8192). Splits fall only before a user turn or after the output that closes the last open call (mlx-vlm#2501), so a history ending in an unfinished tool group keeps it whole.
• Image support, tool and image-count admission run before any fetch, so a text-only model never fetches client-named URLs. Each image URL is then read once per compaction and inlined for every count and the summary prompt (output and capsule keep the original URLs); media bodies are still counted on the worker per count. Compact output that would exceed the replay body limit is refused with 400.
• The summary is one generation on the same provider call /v1/responses uses (temperature 0, thinking off, no tools, no reasoning-echo record). Counts use the /v1/responses/input_tokens render and count path. max_output_tokens defaults to 1024, at most 16384.
• Replay: /v1/responses and /v1/responses/input_tokens accept a compaction input item. Everything before the newest one except system/developer messages is replaced by the summary (assistant message) and the retained items. It runs after chained history is assembled; a response whose history held a capsule stores the instructions, capsule and tail as its input, so multi-hop previous_response_id chains keep the summary, while conversation appends only the new turn. Exactly equal system/developer messages are kept once (hash-based, linear). ResponseOutputItem::Compaction maps back to the input item for phase B.
• Capsule: base64url of {"v":1,"summary","retained"}, not encrypted by design (documented). Bad base64/JSON, v != 1, a nested capsule, or a capsule over the body limit is 400 invalid_request_error.

Decisions

• Known limits, documented: the most recent completed tool group can be summarized away (upstream keeps it), and the summary replays as an assistant turn right after the instructions, which strict-alternation templates may reject.

• Overflow errors (history over one window, summary prompt that does not fit, summary not shorter) are HTTP 400 with type: exceed_context_size_error (the server's existing overflow class) and code: context_length_exceeded (OpenAI's). Context is never replaced on failure.

• An empty summary or one that stops at max_output_tokens is 502 server_error, as upstream does. stream, previous_response_id and conversation on the compact route are 400.

• docs/responses-api.md also stops listing /v1/responses/input_tokens as not mounted; it has been mounted since feat(server): add b10621 tokenization, template, infill, and native embedding utilities #1442.

Changes during review

• Multi-hop previous_response_id chains keep the summary, conversation appends only the new turn, and a replay-size guard refuses unusable compact output (d71e90a).
• Resent instructions are deduplicated across the replayed context, and image URLs are read once per compaction (87db02b).
• Security: media admission now runs before the cache prefetch (it previously fetched URLs for text-only models and had no image-count cap), and instruction dedupe is linear instead of quadratic (95baf78).

Test plan

• cargo test --profile test-fast -p mlxcel --lib with filters responses_compact (28: compaction unit and compact route tests), server::routes:: (493, includes 501 parity), server::responses (74), server::types (107)
• cargo clippy -p mlxcel --lib --tests -- -D warnings, cargo fmt --check
• Unverified here (no Metal/CUDA): mlx-community/Qwen3-30B-A3B-4bit with the OpenAI SDK client.responses.compact(...) then client.responses.create(input=[*compacted.output, ...])

Refs #2397

[adrysn]

Implementation Review Summary

Intent

Phase A of #2397: compaction capsule, selection, summary pass, compaction input replay, and POST /v1/responses/compact. Phase B (context_management, output cap, mlx.compaction.* events) is out of scope.

Findings Addressed

• No CRITICAL or HIGH findings, so no fix commits were pushed.

Remaining Items

• Retained items are sent twice in the compact output: once as plain items and once inside the capsule. A replay body is about 2.3x the retained bytes, so a retained image near the body limit can make [...output, new] exceed main_json_body_limit_bytes (413) (MEDIUM). routes/responses_compact.rs:118
• resolve_compaction keeps every system/developer item before the capsule. With conversation (or a previous_response_id whose stored input already had the system message), the client-resent instructions from the compact output appear twice. Upstream dedupes resent instructions (MEDIUM). responses_compaction/capsule.rs:150
• Each count re-renders the full candidate. For a media body, that re-reads image URLs and takes a worker slot about log2(boundaries)+2 times per compaction (MEDIUM). responses_compaction.rs:262,287,315
• Upstream keep_last_group=True never summarizes the last completed tool group. Here, the newest boundary can summarize the most recent tool output away. The brief allows this, but it diverges from upstream (LOW). responses_compaction.rs:86
• The summary item replays as an assistant turn directly after the instructions when the cut is a user boundary. Strict-alternation templates (Gemma, old Mistral) can then reject the replay. Upstream has the same ordering (LOW). responses_compaction.rs:313
• Docs: "a chain ... keeps replaying the summary" holds for one hop only, because stored responses keep only their own input (existing behavior). keep_tokens also counts system/developer messages, which the docs do not say (LOW). docs/responses-api.md:242,278

Verification

• All stated requirements implemented
• No placeholder/mock code remaining
• Integrated into project code flow (both prefixes, --api-prefix nest, 501 parity)
• Project conventions followed
• Existing modules reused where applicable
• No unintended structural changes
• Tests pass (responses_compact, server::routes::, server::responses, server::types, clippy --lib --tests -D warnings, fmt)

The error mapping is 400 exceed_context_size_error with code context_length_exceeded. This matches the existing overflow mapping in routes/mod.rs:203-212, upstream's ContextBudgetError (400), and OpenAI's code. Selection guarantees a shorter result only through the tokens_after >= tokens_before post-check, not through selection itself. Not verified: an integrated run with a real model (no Metal or CUDA on this host).

[inureyes]

Reviewed #2520 against phase A of #2397. Fixed summary loss after repeated previous_response_id chaining by storing a self-contained capsule continuation, while conversation transcripts append only the new request. Exact resent system/developer messages are deduplicated without conflating roles or names. Added a replay-body size guard so retained media cannot produce a successful but immediately unusable compact result; it uses the existing allocation-free JSON byte counter.

Regression coverage includes a third chained request, conversation append counts, instruction deduplication and the exact byte-limit boundary. Focused Responses/compaction/types tests: 213 passed. Workspace/all-target clippy and fmt pass. The wider CPU-only workspace run exposes existing GPU-dependent NVFP4, catalog and paged-decode assumptions; no new compaction failure appeared. Real-model Qwen/Metal inference was not available on this host.

Security: version, encoded-size and nested-capsule rejection remain enforced, and storage byte budgets still apply to the expanded response snapshots. Reliability: chain context persists without duplicating conversation history. Remaining performance limit: multimodal candidate counts still re-render/refetch through the existing input-token path; version-1 retained data still appears in both output and capsule. The body guard prevents oversize success but does not remove that overhead. Automatic compaction remains phase B, so #2397 stays open.

[inureyes]

Re-reviewed the concurrent image-cache commits and hardened their admission order in 95baf78: translation, media capability, tool guards and image-count limits now run before cache prefetch. New route regressions verify zero downloads for unsupported images and excessive image counts. Instruction and URL deduplication now use hash sets instead of quadratic scans. Successful remote images are fetched once per compaction; failed reads retain the renderer's error path.

Latest validation: 217 focused Responses/compaction/types tests pass, including the image-cache and zero-fetch regressions; workspace/all-target clippy and fmt pass. The complete CPU-only workspace library run finished with the root at 9,659 passed/7 GPU-dependent failures, and mlxcel-core aborted in its pre-existing new_stream_on_gpu test because this build has no GPU backend. The pin, surgery and XLA suites passed (26/185/364). This full run is not a green GPU validation. Final merge is waiting for the latest CI head.

[adrysn]

Security and performance review (head 95baf78)

Both HIGH findings from this pass are already fixed by 95baf78, which landed while the review ran. No extra commits were needed. On 95baf78, responses_compact (28), server::routes:: (493), server::responses (74), server::types (107), clippy --lib --tests -D warnings and fmt all pass.

Fixed (HIGH)

• MediaCache::localize fetched before the media gate and had no count bound. A text-only model fetched client-named URLs, and one body could name thousands of URLs, each held in memory as a data URI of up to 64 MB. localized_items now runs the capability, tool and validate_image_count checks first.
• resolve_compaction deduplicated instructions with Vec::contains, which is O(k²) and runs on every /v1/responses request whose history holds a capsule. It now uses a HashSet.

Verified, no action

• Capsule: a size check runs before decoding, then the version check, nested-capsule refusal and the normal ResponseInputItem parser. Expanded items go through the same gate, translation and render as input. A forged capsule cannot inject a role the client could not already send in input, and the server has no instruction policy it could bypass.
• URL fetches go through try_read_image_url_with_limits, so the existing SSRF policy, DNS-rebind recheck, --media-path confinement and payload cap still apply.
• The store budget is measured on canonical_input_items, including the capsule bytes. Snapshots grow linearly per hop, which is the same as a client resending its full input.
• I found no panic or overflow path on hostile input: boundaries exclude len(), and keep_tokens and max_output_tokens are range-checked.

Remaining

• MEDIUM: the whole-context dedupe also drops a system/developer message after the capsule when it exactly equals an earlier one. Without a capsule the same body keeps both, so a deliberately repeated reminder moves. For a server-minted capsule, canonical_compaction_input already covers the resent-instructions case. capsule.rs:161
• MEDIUM: a URL that fails to read maps to None, and every later count fetches it again (about log2(boundaries)+3 attempts against a slow origin). With the count gate this is bounded to at most max_images_per_request URLs.
• LOW: expect/unreachable! at capsule.rs:142,145 are provably unreachable but panic in library code.
• LOW: every count now translates the history twice (once for admission, once for the render). The cost is linear.

[inureyes]

Hardening fixes and focused validation are complete; merge is blocked by the shared GB10 runner environment. PR #2514 job https://github.com/lablup/mlxcel/actions/runs/39560610832/job/119090751455 passed its build and real-router browser stages, then failed before Activity performance measurement: host did not become quiet within 600s: mean runnable tasks 112.1 over the last 10 samples, threshold 5. The preceding main run failed the same gate with 113.4 runnable tasks.

PR #2520's remaining self-hosted checks use the same runner and are still pending. No performance threshold was relaxed and neither PR was merged. After the runner workload is resolved, rerun the failed checks, finish the pending checks, verify the latest heads, and merge only after validation passes. Local validation and hardware limitations are recorded in the preceding hardening comments.