Repository navigation
server : pass id_slot through the Anthropic /v1/messages conversion - #28554
Conversation
server_chat_convert_anthropic_to_oai copies a small whitelist of common params from the Anthropic request into the internal OAI-style body and left id_slot out. Every other completion endpoint honours id_slot; a proxy that pins a session to the slot holding its KV cache needs the same on /v1/messages, or it silently falls back to automatic slot selection and, on hybrid/recurrent models, pays a full re-prefill. Adds a test asserting a request with id_slot set is served by that slot, observed via GET /slots.
|
Gentle ping for review. This is a 2-line change in @ngxson @ServeurpersoCom you reviewed the recent Anthropic conversion changes (#22536, #21793); would you have a moment for this one? @aldehir cc since you merged the analogous #22154. It also covers #26758 (@lobo235, same fix, open since August). Happy to close this one in favour of that, or the other way round, whichever is easier to land. |
|
Makes sense, it lets a client pin the slot on the Anthropic route too, since on SWA or hybrid models a wrong LCP pick under concurrent sessions can't reuse a partial prefix and costs a full re-prefill. #26758 is the same fix, but it doesn't follow the PR template, so I'm going with this one. I wonder if the Anthropic conversion could copy the whole body and only rewrite the Anthropic specific keys, like the Responses conversion does, I'm really not sure it holds up against every Anthropic field, but if it works it would be simpler than adding keys to the pass-through list one at a time. I'm closing and reopening to retrigger the CI, the pending runs reference a workflow file that was removed from master and the approval button fails on them, no action needed on your side. |
ServeurpersoCom
left a comment
ServeurpersoCom
left a comment
There was a problem hiding this comment.
LGTM, it's a one-line change plus a test, it lets a client pin the slot on the Anthropic route too, since on SWA or hybrid models a wrong LCP pick under concurrent sessions can't reuse a partial prefix and costs a full re-prefill. The base is old, so the red checks are stale runs from Sept 7 built before the norms fix, not this change.
|
Rebased onto current master, CI is green now. Thanks @ServeurpersoCom! |
|
@pcvelz I re-requested ngxson's review since the change request doesn't clear by itself after the fix, should be unblocked soon. |
…gml-org#28554) * server : pass id_slot through the Anthropic /v1/messages conversion server_chat_convert_anthropic_to_oai copies a small whitelist of common params from the Anthropic request into the internal OAI-style body and left id_slot out. Every other completion endpoint honours id_slot; a proxy that pins a session to the slot holding its KV cache needs the same on /v1/messages, or it silently falls back to automatic slot selection and, on hybrid/recurrent models, pays a full re-prefill. Adds a test asserting a request with id_slot set is served by that slot, observed via GET /slots. * server : shorten comments * rm test --------- Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
…gml-org#28554) * server : pass id_slot through the Anthropic /v1/messages conversion server_chat_convert_anthropic_to_oai copies a small whitelist of common params from the Anthropic request into the internal OAI-style body and left id_slot out. Every other completion endpoint honours id_slot; a proxy that pins a session to the slot holding its KV cache needs the same on /v1/messages, or it silently falls back to automatic slot selection and, on hybrid/recurrent models, pays a full re-prefill. Adds a test asserting a request with id_slot set is served by that slot, observed via GET /slots. * server : shorten comments * rm test --------- Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
Overview
server_chat_convert_anthropic_to_oaicopies a fixed list of request keys(
temperature,top_p,top_k,stream,chat_template_kwargs) into theOAI body.
id_slotis not in that list, so a slot pin that works on/completionand/v1/chat/completionsis silently ignored on/v1/messages: the request falls back to automatic slot selection and noerror is returned.
This adds
id_slotto the pass-through list so the Anthropic route behaveslike the other two.
Why it matters: I found this during development on
my own project, a
llama-swap fork that keeps each client session on the slot already holding
its KV cache by sending
id_slot. With a hybrid/recurrent model(Qwen3-Next class) a wrong slot is not a partial cache hit but a full
re-prefill; on a 99k-token session that is ~16 minutes at ~100 t/s for a
single tool call. Running Claude Code against llama-server through the
Anthropic endpoint, the proxy pinned the slot, the child still logged
selected slot by LCP similarity, and/slotsshowed the request on theother slot. With the patch the child logs
selected slot by id (N)andfollow-ups append to the existing cache.
Test:
test_anthropic_id_slotintools/server/tests/unit/test_compat_anthropic.py. Withn_slots = 2onan idle server automatic selection picks the last slot, so requesting slot 0
and asserting slot 0 served the request fails without the change.
Additional information
Same fix as #26758 by @lobo235, unreviewed since August and flagged for the
PR template; this one follows the template. Fine with either being merged.
Not in this PR: the Anthropic response serializers (
to_json_anthropic,to_json_anthropic_stream) never emitid_slot, unlike the OAI path, so aproxy cannot learn the slot from the response and has to assign it up
front. Emitting it in the non-streaming message and the
message_startevent would close that gap; I can follow up separately if wanted.
Requirements
test were drafted with AI assistance (Claude); the change was built, run
and verified by me on a real workload, and this description is mine.