Repository navigation
[Benchmark] Running 81GB IQ3_S on 32GB RAM via massive Virtual Memory + 3080 12G (Peak 89.3 T/s!) #144
Description
Hello everyone, I want to share an extreme local deployment case. I successfully ran the Qwen3.8 IQ3_S (approx. 81GB) model on a PC with only 32GB of physical RAM and an RTX 3080 12G, achieving a peak decode speed of 89.3 T/s!
💻 My Extreme Hardware Setup:
- CPU: Intel Core i5-14490F (10 cores / 16 threads)
- GPU: NVIDIA RTX 3080 12G
- RAM: 32GB DDR5 6000MHz
- Virtual Memory (The Secret Weapon): Nearly 500GB of Virtual Memory (Formatted three old SATA SSDs: 120GB + 128GB + 256GB entirely dedicated to pagefile.sys)
- Network: 2.5G LAN (Desktop acts as a pure headless server, laptop as the client)
📊 Benchmark Results (From cold start to extreme limits):
| Scenario | Context (Tokens) | Prefill Speed | Output Length | Decode Speed | Expert Cache Hit | Notes |
|---|---|---|---|---|---|---|
| Cold Start | ~4,000 | 575 t/s (7s) | 5,653 | 71.2 T/s | 95.7% | Initial burst, instantly fast |
| Incremental | ~8,700 | 1,250 t/s (7s) | 5,306 | 75.5 T/s | 96.6% | Cache hit, speed climbing |
| Long Text | ~13,800 | 1,985 t/s (7s) | 3,011 | 79.1 T/s | 96.9% | Very stable |
| Extreme Long | ~91,400 | 1,270 t/s (72s) | 4,242 | 89.3 T/s (Peak!) | 94.8% | Reached near 90T while writing code |
| Near Full | ~118,100 | 1,217 t/s (97s) | 8,128 | 65.4 T/s | 95.3% | Pushing 118K context limit |
| High Freq Edit | ~50,000 | Instant (6s) | 200~1600 | 60 ~ 93 T/s | 91% ~ 95% | Agent editing, extremely smooth |
⚙️ Key JSON Configuration:
"--max-context", "128000",
"--kv", "int8",
"--kv-grow",
"--expert-cache", "auto",
"--adapt-async", "1",
"--prefill", "auto:32768",
"--spec", "4",
"--spec-min-p", "0.3",
"--suffix-draft", "3",
"--draft", "dflash2",
"--prompt-cache", "16",
"--conversation-cache-mib", "2048"
Activity
[-]Running 110GB UD-Q5 on 32GB RAM via massive Virtual Memory + 3080 12G (89.3 T/s peak!)[/-][+][Benchmark] Running 81GB IQ3_S on 32GB RAM via massive Virtual Memory + 3080 12G (Peak 89.3 T/s!)[/+]