Repository navigation
Conversation
…py (Niko1221#1188, Niko1221#1169) Niko1221#783's batched append of a verify window's cells passed no host copy for the hybrid K8V4 format (int8 and q4_0 pass st.host, and the per-token path passes K8V4's two halves). With --kv-resident, a block that was not resident when its cells were appended came back from the host copy without them, so the tokens a window wrote were lost: free generation degenerated into repetition (Niko1221#1188, Niko1221#1135) and parked conversations lost their decode-written turns (Niko1221#1169). The batched path now passes the same host halves as the per-token path.
|
Tested this on a 2080 Ti (sm_75, CUDA 13.3, Linux): The probe is a chat where each turn asks the model to repeat the user's message word for word (greedy, thinking off). The turns are appended one at a time, so each request reads only ~40 new tokens. That's 8 turns per chain, two chains per row:
On Thanks for tracking this down. |
|
Thank you, Benjamin. This is in 0.1.40.2 with your authorship kept, unchanged. It fixes the repetition on a count-to-300 test on both CUDA (sm_86) and HIP (gfx1201). |
#783's batched append of a verify window's cells passed no host copy for the hybrid K8V4 format (int8 and q4_0 pass st.host, and the per-token path passes K8V4's two halves). With --kv-resident, a block that was not resident when its cells were appended came back from the host copy without them, so the tokens a window wrote were lost: free generation degenerated into repetition (#1188, #1135) and parked conversations lost their decode-written turns (#1169). The batched path now passes the same host halves as the per-token path.