You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Just some small thoughts on what other models might fit PolyStrata, there are other models that claims to be pretty strong. Probably not gonna pick Nemotron tho
I looked into both. LFM2.5's only MoE is the 8B-A1B, ~4.8GB at Q4_K_M, so on an 8GB card it fits whole and llama.cpp runs it fine. But it could make sense for smaller cards, like 2-4GB, where it doesn't fit and splitting experts between VRAM and CPU would actually help. I'll consider adding it for that.
Ling-2.6-flash (104B, 7.4B active) is also a good fit, that's exactly the size where this helps on normal cards. I need solid GGUF support for it in llama.cpp first, since the engine reads the normal GGUF files. I'll keep this open.
Thanks for the suggestion!
I looked into both. LFM2.5's only MoE is the 8B-A1B, ~4.8GB at Q4_K_M, so on an 8GB card it fits whole and llama.cpp runs it fine. But it could make sense for smaller cards, like 2-4GB, where it doesn't fit and splitting experts between VRAM and CPU would actually help. I'll consider adding it for that.
Ling-2.6-flash (104B, 7.4B active) is also a good fit, that's exactly the size where this helps on normal cards. I need solid GGUF support for it in llama.cpp first, since the engine reads the normal GGUF files. I'll keep this open.
there is a new lfm2.5 24b-a2b which is apparently comparable to qwen3.8-27b , i'd appreciate if you consider adding it .
Thanks for the suggestion!
I looked into both. LFM2.5's only MoE is the 8B-A1B, ~4.8GB at Q4_K_M, so on an 8GB card it fits whole and llama.cpp runs it fine. But it could make sense for smaller cards, like 2-4GB, where it doesn't fit and splitting experts between VRAM and CPU would actually help. I'll consider adding it for that.
Ling-2.6-flash (104B, 7.4B active) is also a good fit, that's exactly the size where this helps on normal cards. I need solid GGUF support for it in llama.cpp first, since the engine reads the normal GGUF files. I'll keep this open.
there is a new lfm2.5 24b-a2b which is apparently comparable to qwen3.8-27b , i'd appreciate if you consider adding it .
LFM2.5-24B-A2B is in 0.1.44.1-poly.3: --family lfm25, five sizes (UD-Q4_K_XL, UD-Q3_K_XL, QAD-Q4_0, UD-Q6_K_XL, Q8_0). Details and numbers are in docs/LFM.md.
With UD-Q4_K_XL on an RTX 4070 laptop (8 GB), a 32,000-token prompt was read at 3,540 tokens/s and the answer written at 71 to 73; llama.cpp, tuned on the same laptop, read it at 530 and wrote at 53. On a 5090 a 30,000-token prompt is read at about 23,400 tokens/s and a short chat written at about 480.
On Windows the installer downloads the ready-made engine. An earlier install gets it with UPDATE.bat, and then START-HERE.bat --setup --family lfm25 adds the model.
MiniCPM-V 4.7 has a PR now (#20). Keeping this open for Ling.
MiniCPM-V 4.7 35B-A3B is in 0.1.44.1-poly.5 too, thanks to iMilnb: --family minicpmv, three sizes, and it reads pictures. Still keeping this open for Ling.
Activity
VecSzn commentedon Oct 10, 2026
Thanks for the suggestion!
I looked into both. LFM2.5's only MoE is the 8B-A1B, ~4.8GB at Q4_K_M, so on an 8GB card it fits whole and llama.cpp runs it fine. But it could make sense for smaller cards, like 2-4GB, where it doesn't fit and splitting experts between VRAM and CPU would actually help. I'll consider adding it for that.
Ling-2.6-flash (104B, 7.4B active) is also a good fit, that's exactly the size where this helps on normal cards. I need solid GGUF support for it in llama.cpp first, since the engine reads the normal GGUF files. I'll keep this open.
TomLucidor commentedon Oct 12, 2026
Noted with https://huggingface.co/AtomicChat/Ling-3.0-flash-GGUF
Not sure about bigger models like https://huggingface.co/models?sort=downloads&search=LongCat-2.0
kamikazem commentedon Oct 15, 2026
there is a new lfm2.5 24b-a2b which is apparently comparable to qwen3.8-27b , i'd appreciate if you consider adding it .
VecSzn commentedon Oct 15, 2026
Will Do! Thanks for the advise.
TomLucidor commentedon Oct 15, 2026
Some more for the bucket list (MiniCPM and BitCPM doing a lot of good works too) but please prioritize the more popular ones https://huggingface.co/openbmb/MiniCPM-V-4.7-35B-A3B-gguf
VecSzn commentedon Oct 15, 2026
LFM2.5-24B-A2B is in 0.1.44.1-poly.3:
--family lfm25, five sizes (UD-Q4_K_XL, UD-Q3_K_XL, QAD-Q4_0, UD-Q6_K_XL, Q8_0). Details and numbers are in docs/LFM.md.With UD-Q4_K_XL on an RTX 4070 laptop (8 GB), a 32,000-token prompt was read at 3,540 tokens/s and the answer written at 71 to 73; llama.cpp, tuned on the same laptop, read it at 530 and wrote at 53. On a 5090 a 30,000-token prompt is read at about 23,400 tokens/s and a short chat written at about 480.
On Windows the installer downloads the ready-made engine. An earlier install gets it with
UPDATE.bat, and thenSTART-HERE.bat --setup --family lfm25adds the model.MiniCPM-V 4.7 has a PR now (#20). Keeping this open for Ling.
VecSzn commentedon Oct 15, 2026
MiniCPM-V 4.7 35B-A3B is in 0.1.44.1-poly.5 too, thanks to iMilnb:
--family minicpmv, three sizes, and it reads pictures. Still keeping this open for Ling.VecSzn commentedon Oct 17, 2026
LFM2.5 and MiniCPM-V are in. I'm closing this and moving Ling to the vote in #16, so the remaining model requests are all in one place.
poly: a task of the pool that shares out work does it alone