Add classifier-free guidance #377

Activity

1. philpax

   added

   issue:enhancementNew feature or requestNew feature or request

   on Jul 17, 2023
2. Vermeille commented on Jul 21, 2023

   

   Vermeille

   on Jul 21, 2023

   More actions

   Make sure to remove the smooth_factor and the last log_softmax in order to remain consistent with llama.cpp's and HF's implementation ( ggml-org/llama.cpp#2280 )

   👍React with 👍1Reacted by Stevie Weiss
3. KerfuffleV2 commented on Aug 12, 2023

   

   KerfuffleV2

   on Aug 12, 2023

   Contributor

   More actions

   I'm going to try to look at how to add this to llm-samplers. It will need the CFG logits though, so llm will need to handle that itself. I guess it can be supplied as a sampler resource similar to the RNG and last tokens. I'd like to figure out a more general way to handle resources but in the worst case I can just add another type of resource to that trait.

   llama.cpp CFG sampler for reference (doesn't look too complicated): https://github.com/ggerganov/llama.cpp/blob/b19edd54d51cef5e3616c18b1d0d8626895b2cba/llama.cpp#L2709-L2740

   On the llm side it looks like you have to maintain a guidance context and run the model for both contexts every token — so using CFG means evaluating the model is twice as slow (also, I think you need two K/V caches). Main relevant sections from llama.cpp's main example: https://github.com/ggerganov/llama.cpp/blob/b19edd54d51cef5e3616c18b1d0d8626895b2cba/examples/main/main.cpp#L208-L215 and https://github.com/ggerganov/llama.cpp/blob/b19edd54d51cef5e3616c18b1d0d8626895b2cba/examples/main/main.cpp#L484-L523

   👍React with 👍1Reacted by Philpax