Repository navigation
llama : try to avoid context swap #2060
Description
Currently, when the context becomes full, we pick part of the tokens and recompute the KV cache.
Instead, try to either:
- store non-RoPEd KV cache, "shift" it when the context is full and compute the RoPE over the entire cache for every new token taking into account the current positions
- store RoPEd KV cache (as we do now), "shift" it when the context is full and apply extra shift-RoPE on it (assuming RoPE is "additive")
Reacted by FNsi, lin72h, XYNTAXIS, Nobody and Frederik Vogel
Reacted by LostRuins Concedo, Alex Klinkhamer, Hassan Hayat and Nexesenex
Activity
SlyEcho commentedon Jul 2, 2023
Maybe it could also be compressed using interpolation?
cebtenzzre commentedon Sep 17, 2023
Storing non-RoPEd KV cache would allow us to implement dynamic NTK or YaRN RoPE scaling, which is the state-of-the-art for context scaling on non-finetuned models. See section 3.3 of this paper.
sd: ensure previous generation results are cleaned up on all code pat…