Run models from local folder #132

Activity

1. vehystrix commented on Nov 3, 2026

   

   vehystrix

   on Nov 3, 2026

   Contributor

   More actions

   Look into the --custom-gguf option I added recently. just point it at your downloaded gguf, and hope it works :)
2. nfomin99 commented on Nov 5, 2026

   

   nfomin99

   on Nov 5, 2026

   Last edited by nfomin99

   Author

   More actions

   So, I'm back with some results:

   I managed to get the model running.
   +10-20% performance boost, reaching 63.7 t/s on my hardware (AMD Ryzen 9 6900HX, RX6850m 12GB VRAM, 32GB RAM, Arch Linux).
   I had to switch the model to https://huggingface.co/Unrestricted/Gemma4-26B-A4B-Uncensored-HauhauCS-Balanced (which gets 45-50 t/s in llama.cpp).
   However, I still have some questions:
   In llama.cpp, you can split the total context into N parts using the -np parameter, where the total context is divided equally among users. I previously had it set to 220,000, which was split into two equal parts, and my hardware handled it fine.

   But now, using this launch command:
   ./setup.sh --custom-gguf "~/polystrata_models/gemma-4-26B-A4B-it-UD-Q4_K_M.gguf" --family gemma4 --port 8099 --host 0.0.0.0 --parallel 2 --vram-reserve-mib 100 --kv-streaming on --low-ram mmap
   and setting the context to 128,000, I get the following error:
   strata-poly: no VRAM for the expert staging buffer (529 MiB): out of memory

   Is there any way to bypass this? now i set 64 000 context and all run nomally