Repository navigation
Conversation
Embedder sends truncate_prompt_tokens without a side, so vLLM falls back to the tokenizer default. For the pooling runner that is 'right' since vLLM 0.13, which keeps the first N tokens and drops the question that state_text appends last. The training recipe (Recipe.state_ids) keeps the tail, so send truncation_side='left' explicitly. vLLM accepts the field from 0.18 and ignores unknown fields before that.
4 tasks
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Embeddersendstruncate_prompt_tokensbut not which side to cut, so vLLM falls back to the tokenizer default. For the pooling runner that default becamerightin vLLM 0.13 (vllm/tokenizers/registry.py). Through 0.12, every runner truncated from the left. On a current vLLM (#3 is on 0.30.0), a state over 2,048 tokens keeps its first 2,048 tokens.state_textputs the question last, so the question is what gets dropped. Training cuts the other way:Recipe.state_idsintrain/embed_utils.pykeeps the tail.This PR sends
truncation_side: "left"whenevertruncate_prompt_tokensis set. The side names what gets cut, soleftdrops tokens from the start and keeps the last 2,048, question included. The change is two lines inEmbedder._fetch.vLLM accepts
truncation_sideon embedding requests from 0.18. Older versions allow unknown request fields, so they ignore it instead of rejecting the request, which means 0.13 to 0.17 still cut from the right. If you want that gap closed too, I can add avllm>=0.18pin to thevllmextra in this PR.Testing
I don't have an NVIDIA GPU, so I tested against a Qwen3-8B encoder on MLX that follows vLLM's
truncate_prompt_tokens/truncation_sidesemantics and matches the output reported in #3. The table shows cosine similarity betweenEmbedder.embed()and the same state cut to its last 2,048 tokens:The two long states come from DecisionBench medium, where 139 of the 293 question states are over 2,048 tokens. Shorter states don't change. Neither do the DeepSWE and Terminal-Bench results, because
bon_eval.pyscores precomputed embeddings. The MLX encoder in #5 always keeps the tail, so with this change the two backends agree on long states.