the engine stopped unexpectedly (exit code None) #2118

Hi

I can't get the unsloth-ud-q4_k_xl to run - ISTA IQ3_S works.
After calibration the benchmark eventually crashes and if i start up the engine without the benchmark, it crashes within 2 seconds of the first prompt without ever outputting a token.

Any idea where to look?

Thanks!

ZFS with 3 mirrored VDEVs (essentially RAID 10 across 6 Enterprise "Read Intensive" SATA SSDs)
128 GB RAM (DDR4-2133)
2x E5-2640v4 (only 8 cores exposed to VM)
4x 5060 Ti 16 GB

Strata - Qwen3.8-Flash-Next on a normal PC (a GPU + system RAM + CPU)

=== Step 1: checking your PC ===
Your NVIDIA GPUs:
GPU 0: NVIDIA GeForce RTX 5060 Ti, 16 GB VRAM - can be used
GPU 1: NVIDIA GeForce RTX 5060 Ti, 16 GB VRAM - can be used
GPU 2: NVIDIA GeForce RTX 5060 Ti, 16 GB VRAM - can be used
GPU 3: NVIDIA GeForce RTX 5060 Ti, 16 GB VRAM - can be used

Strata can run the model on one GPU, or share it across these: then each card holds the
experts of its own layers, so together they hold about twice as many, and prompts are read about 20%
faster (details: docs/MULTI_GPU.md). A much slower extra card can also make it slower.

1. GPU 0 (NVIDIA GeForce RTX 5060 Ti, 16 GB) + GPU 1 (NVIDIA GeForce RTX 5060 Ti, 16 GB) together (recommended)
2. GPU 0 (NVIDIA GeForce RTX 5060 Ti, 16 GB) + GPU 1 (NVIDIA GeForce RTX 5060 Ti, 16 GB) + GPU 2 (NVIDIA GeForce RTX 5060 Ti, 16 GB) + GPU 3 (NVIDIA GeForce RTX 5060 Ti, 16 GB) together
3. GPU 0 (NVIDIA GeForce RTX 5060 Ti, 16 GB) only
4. GPU 1 (NVIDIA GeForce RTX 5060 Ti, 16 GB) only
5. GPU 2 (NVIDIA GeForce RTX 5060 Ti, 16 GB) only
6. GPU 3 (NVIDIA GeForce RTX 5060 Ti, 16 GB) only
   Which GPUs? [1]: 2
   [ok] GPUs: GPU 0 (NVIDIA GeForce RTX 5060 Ti, 16 GB) + GPU 1 (NVIDIA GeForce RTX 5060 Ti, 16 GB) + GPU 2 (NVIDIA GeForce RTX 5060 Ti, 16 GB) + GPU 3 (NVIDIA GeForce RTX 5060 Ti, 16 GB) together (the model's layers are split across them)
   [ok] GPU: NVIDIA GeForce RTX 5060 Ti, 15.9 GB VRAM, compute capability 12.0, driver 610.57.04
   [ok] RAM: 121 GB
   [ok] CPU: Intel(R) Xeon(R) CPU E5-2640 v4 @ 2.40GHz (AVX2)

=== Step 2: your choices ===

1. Qwen3.8-Flash-Next Qwen; GSQ-RCO quants by ISTA-DASLab - the original model

2. Swift 1.5 UkisAI's fine-tune of Qwen3.8-Flash-Next - thinks much shorter (-63% thinking tokens, 1.8x sooner answers by its authors' numbers)

3. Qwen3.8-Flash-Next Coder ISTA-DASLab's coding version - half the experts (code, tools, images kept): needs ~32 GB of RAM, faster; weaker outside coding

4. Qwen3.8-Flash-Next (Unsloth) Unsloth's ~4-bit quantizations - UD-IQ4_XS: a 94 GB download; with less than ~80 GB of RAM part of its experts are read from the SSD (UD-Q4_K_XL, 111 GB: experimental)
   Which model? [1]: 4
   [ok] model: Qwen3.8-Flash-Next (Unsloth)

5. UD-IQ4_XS ~4-bit i-quant (Unsloth Dynamic), between IQ3_S and UD-Q4_K_XL in quality; on a PC with less than ~80 GB of RAM part of its experts are read from the SSD; download 94 GB, keeps ~55 GB of its 60 GB of experts in RAM

6. UD-Q4_K_XL 4-bit (Unsloth Dynamic), EXPERIMENTAL: the best quality, but most experts come from the SSD on a 64 GB PC (7-8.5 tokens/s measured); download 111 GB, keeps ~71 GB of its 77 GB of experts in RAM
   Which size? [1]: 2
   [!] UD-Q4_K_XL is EXPERIMENTAL (docs/UNSLOTH_Q4.md): most of its experts are read from the SSD while it answers, so it is several times slower than the 2-3-bit models; quality checked against llama.cpp

UD-Q4_K_XL runs with a RAM budget of its experts, on one GPU or on GPU 0 (NVIDIA GeForce RTX 5060 Ti, 16 GB) + GPU 1 (NVIDIA GeForce RTX 5060 Ti, 16 GB) + GPU 2 (NVIDIA GeForce RTX 5060 Ti, 16 GB) + GPU 3 (NVIDIA GeForce RTX 5060 Ti, 16 GB) together:

1. GPU 0 (NVIDIA GeForce RTX 5060 Ti, 16 GB) only (recommended: the tested setup)
2. GPU 0 (NVIDIA GeForce RTX 5060 Ti, 16 GB) + GPU 1 (NVIDIA GeForce RTX 5060 Ti, 16 GB) + GPU 2 (NVIDIA GeForce RTX 5060 Ti, 16 GB) + GPU 3 (NVIDIA GeForce RTX 5060 Ti, 16 GB) together: each card caches the experts of its own layers (A dual-GPU fork for 32 GB PCs (resident mode on a layer split, pipelined windows) #642)
   UD-Q4_K_XL: which GPUs? [1]: 2
   [ok] UD-Q4_K_XL on GPU 0 (NVIDIA GeForce RTX 5060 Ti, 16 GB) + GPU 1 (NVIDIA GeForce RTX 5060 Ti, 16 GB) + GPU 2 (NVIDIA GeForce RTX 5060 Ti, 16 GB) + GPU 3 (NVIDIA GeForce RTX 5060 Ti, 16 GB), with its RAM budget (each card caches its own layers' experts, the RAM the hottest of the rest; A dual-GPU fork for 32 GB PCs (resident mode on a layer split, pipelined windows) #642)
   [ok] RAM budget: 71 GiB of UD-Q4_K_XL's experts in RAM, the rest read from the model files on the SSD
   [ok] size: UD-Q4_K_XL

Context length = how much text the model can see at once (your chat, files, tool output).
Longer needs more VRAM for it, so fewer experts fit on the GPU:

1. 8K tokens
2. 32K tokens (recommended for your GPU)
3. 64K tokens
4. 128K tokens
5. 200K tokens
6. 256K tokens
7. 384K tokens (experimental: setup adds rope scaling)
8. 512K tokens (experimental: setup adds rope scaling)
   Context? [2]: 2
   [ok] context: 32768 tokens

KV cache precision (the model's memory of the conversation):

1. 8-bit (recommended: what every published number was measured with)
2. 4-bit half the memory (about 4% faster at 128K), but measurably less precise on long
   documents; long-context lookups (needle tests) still pass
   KV cache? [1]: 1
   [ok] KV cache: 8-bit

Images: the model can also read pictures (screenshots, photos, scanned pages). This adds a 0.9 GB
download and keeps ~1.4 GB of VRAM free for the image encoder, so text is a few % slower.
Do you want images? [n]: n
[ok] images: off

=== Step 3: Python packages ===
[ok] numpy, jinja2, regex, pyyaml, tqdm, requests, cmake, ninja, pillow, psutil already installed

=== Step 4: the Strata engine ===
[ok] llama.cpp 3cf0325 (gguf-py, ggml, mtmd)
[ok] engine already built for this PC
[ok] engine: /home/neumluc/Strata/engine/strata

=== Step 5: downloading Qwen3.8-Flash-Next (Unsloth) UD-Q4_K_XL ===
[ok] Qwen3.8-Flash-Next-UD-Q4_K_XL-00001-of-00004.gguf already downloaded
[ok] Qwen3.8-Flash-Next-UD-Q4_K_XL-00002-of-00004.gguf already downloaded
[ok] Qwen3.8-Flash-Next-UD-Q4_K_XL-00003-of-00004.gguf already downloaded
[ok] Qwen3.8-Flash-Next-UD-Q4_K_XL-00004-of-00004.gguf already downloaded
[ok] model files present

=== Step 6: preparing the model for Strata ===
[ok] model prepared: /home/neumluc/Strata-data/packs/unsloth-ud-q4_k_xl
[ok] MTP draft layer: /home/neumluc/Strata-data/mtp/rt

=== Step 7: writing the start script ===
[ok] layer split across GPUs [0, 1, 2, 3] (auto)
tip: with 121 GB of RAM, --prefill auto:32768 in the config's args read prompts 21-35% faster in community benchmarks (#433 #440 #834); not set, nothing changes
tip: for several agents or clients at once, --conversation-cache-mib 8192 in the config's args keeps each one's conversation; without it they were measured re-reading ~90% of their prompts (#882 #440; docs/DETAILS.md, Multiple conversations)
the earlier run config is kept as strata-unsloth-ud-q4_k_xl.json.bak; engine options it had that this one has not (setup chooses those): --kv-resident --vision --vram-reserve-mib --pcie-frac --pool-workers --adapt-every --adapt-swaps --adapt-decay
Tune Strata for this PC now? It measures a few engine settings (about 15-30 minutes, longer on a slow card; the PC is busy meanwhile; later: START-HERE --calibrate) [y]:

Tuning Strata for this PC: the output speed is measured with a few engine settings (the PCIe share, the
draft depth, the CPU threads, the expert cache). It takes roughly 15-30 minutes (3 rounds of each value); the PC is busy meanwhile.
Loading the model for the measurements ...
[strata] starting the engine: reading the model's weights ...
[strata] loading the most-used experts into RAM (up to 71 GiB; the rest are read from the model files as needed) and locking part of them for the GPU.
YOUR PC CAN BE SLOW OR STOP RESPONDING FOR 1-3 MINUTES NOW - this is normal.
Please wait and don't close this window; the browser opens when it is ready.
[strata] filling the GPU's expert cache (3386 experts, 10.12 GiB of VRAM) ...
[strata] still starting (38 s) - please wait ...
[strata] still starting (58 s) - please wait ...
[strata] still starting (78 s) - please wait ...
[strata] still starting (98 s) - please wait ...
[strata] still starting (118 s) - please wait ...
[strata] almost ready ...
[strata] still starting (141 s) - please wait ...
[strata] still starting (161 s) - please wait ...
[strata] still starting (181 s) - please wait ...
[strata] still starting (201 s) - please wait ...
round 1/3 PCIe share: 0 56.0 0.19 60.2 0.2 58.5 0.35 59.1 0.55 49.6 0.75 45.0 0.9 48.8 1 42.7
round 2/3 PCIe share: 0 56.7 0.19 60.0 0.2 58.8 0.35 55.7 0.55 49.7 0.75 52.9 0.9 47.6 1 42.9
round 3/3 PCIe share: 0 55.9 0.19 60.8 0.2 57.4 0.35 51.9 0.55 51.6 0.75 48.9 0.9 48.2 1 42.3
PCIe share, median of 3: 0: 56.0 (55.9-56.7), 0.19: 60.2 (60.0-60.8), 0.2: 58.5 (57.4-58.8), 0.35: 55.7 (51.9-59.1), 0.55: 49.7 (49.6-51.6), 0.75: 48.9 (45.0-52.9), 0.9: 48.2 (47.6-48.8), 1: 42.7 (42.3-42.9) -> best 0.19
round 1/3 draft floor: 0.3 50.0 0.5 54.1 0.7 63.7
round 2/3 draft floor: 0.3 49.5 0.5 54.2 0.7 51.3
round 3/3 draft floor: 0.3 47.9 0.5 56.3 0.7 56.9
draft floor, median of 3: 0.3: 49.5 (47.9-50.0), 0.5: 54.2 (54.1-56.3), 0.7: 56.9 (51.3-63.7) -> best 0.7
Measuring with 7 CPU workers (restarts the engine) ...
[strata] starting the engine: reading the model's weights ...
[strata] loading the most-used experts into RAM (up to 71 GiB; the rest are read from the model files as needed) and locking part of them for the GPU.
YOUR PC CAN BE SLOW OR STOP RESPONDING FOR 1-3 MINUTES NOW - this is normal.
Please wait and don't close this window; the browser opens when it is ready.
[strata] filling the GPU's expert cache (3386 experts, 10.12 GiB of VRAM) ...
[strata] still starting (34 s) - please wait ...
[strata] still starting (54 s) - please wait ...
[strata] still starting (74 s) - please wait ...
[strata] almost ready ...
[strata] still starting (104 s) - please wait ...
[strata] still starting (124 s) - please wait ...
[strata] still starting (144 s) - please wait ...
[strata] still starting (164 s) - please wait ...
7 workers: 59.7 tok/s
Measuring with 5 CPU workers (restarts the engine) ...
[strata] starting the engine: reading the model's weights ...
[strata] loading the most-used experts into RAM (up to 71 GiB; the rest are read from the model files as needed) and locking part of them for the GPU.
YOUR PC CAN BE SLOW OR STOP RESPONDING FOR 1-3 MINUTES NOW - this is normal.
Please wait and don't close this window; the browser opens when it is ready.
[strata] filling the GPU's expert cache (3386 experts, 10.12 GiB of VRAM) ...
[strata] still starting (34 s) - please wait ...
[strata] still starting (54 s) - please wait ...
[strata] still starting (74 s) - please wait ...
[strata] still starting (94 s) - please wait ...
[strata] still starting (114 s) - please wait ...
[strata] almost ready ...
[strata] still starting (147 s) - please wait ...
[strata] still starting (167 s) - please wait ...
[strata] still starting (187 s) - please wait ...
[strata] still starting (207 s) - please wait ...
5 workers: 63.1 tok/s
Measuring with 4 CPU workers (restarts the engine) ...
[strata] starting the engine: reading the model's weights ...
[strata] loading the most-used experts into RAM (up to 71 GiB; the rest are read from the model files as needed) and locking part of them for the GPU.
YOUR PC CAN BE SLOW OR STOP RESPONDING FOR 1-3 MINUTES NOW - this is normal.
Please wait and don't close this window; the browser opens when it is ready.
[strata] filling the GPU's expert cache (3386 experts, 10.12 GiB of VRAM) ...
[strata] still starting (34 s) - please wait ...
[strata] still starting (54 s) - please wait ...
[strata] still starting (74 s) - please wait ...
[strata] still starting (94 s) - please wait ...
[strata] almost ready ...
[strata] still starting (126 s) - please wait ...
[strata] still starting (146 s) - please wait ...
[strata] still starting (166 s) - please wait ...
[strata] still starting (186 s) - please wait ...
4 workers: 62.7 tok/s
Measuring with 2 CPU workers (restarts the engine) ...
[strata] starting the engine: reading the model's weights ...
[strata] loading the most-used experts into RAM (up to 71 GiB; the rest are read from the model files as needed) and locking part of them for the GPU.
YOUR PC CAN BE SLOW OR STOP RESPONDING FOR 1-3 MINUTES NOW - this is normal.
Please wait and don't close this window; the browser opens when it is ready.
[strata] filling the GPU's expert cache (3386 experts, 10.12 GiB of VRAM) ...
[strata] still starting (32 s) - please wait ...
[strata] still starting (52 s) - please wait ...
[strata] still starting (72 s) - please wait ...
[strata] still starting (92 s) - please wait ...
[strata] almost ready ...
[strata] still starting (124 s) - please wait ...
[strata] still starting (144 s) - please wait ...
[strata] still starting (164 s) - please wait ...
[strata] still starting (184 s) - please wait ...
2 workers: 61.1 tok/s
Measuring the expert tier (the engine default) (restarts the engine) ...
[strata] starting the engine: reading the model's weights ...
[strata] loading the most-used experts into RAM (up to 71 GiB; the rest are read from the model files as needed) and locking part of them for the GPU.
YOUR PC CAN BE SLOW OR STOP RESPONDING FOR 1-3 MINUTES NOW - this is normal.
Please wait and don't close this window; the browser opens when it is ready.
[strata] filling the GPU's expert cache (3386 experts, 10.12 GiB of VRAM) ...
[strata] still starting (34 s) - please wait ...
[strata] still starting (54 s) - please wait ...
[strata] still starting (74 s) - please wait ...
[strata] still starting (94 s) - please wait ...
[strata] still starting (114 s) - please wait ...
[strata] almost ready ...
[strata] still starting (148 s) - please wait ...
[strata] still starting (168 s) - please wait ...
[strata] still starting (188 s) - please wait ...
[strata] still starting (208 s) - please wait ...
58.9 tok/s
Measuring the expert tier every 1, 80 swaps, decay 0.97 (restarts the engine) ...
[strata] starting the engine: reading the model's weights ...
[strata] loading the most-used experts into RAM (up to 71 GiB; the rest are read from the model files as needed) and locking part of them for the GPU.
YOUR PC CAN BE SLOW OR STOP RESPONDING FOR 1-3 MINUTES NOW - this is normal.
Please wait and don't close this window; the browser opens when it is ready.
[strata] filling the GPU's expert cache (3386 experts, 10.12 GiB of VRAM) ...
[strata] still starting (32 s) - please wait ...
[strata] still starting (52 s) - please wait ...
[strata] still starting (72 s) - please wait ...
[strata] still starting (92 s) - please wait ...
[strata] still starting (112 s) - please wait ...
[strata] almost ready ...
[strata] still starting (145 s) - please wait ...
[strata] still starting (165 s) - please wait ...
[strata] still starting (185 s) - please wait ...
[strata] still starting (205 s) - please wait ...
58.9 tok/s
Measuring the expert tier every 1, 160 swaps, decay 0.97 (restarts the engine) ...
[strata] starting the engine: reading the model's weights ...
[strata] loading the most-used experts into RAM (up to 71 GiB; the rest are read from the model files as needed) and locking part of them for the GPU.
YOUR PC CAN BE SLOW OR STOP RESPONDING FOR 1-3 MINUTES NOW - this is normal.
Please wait and don't close this window; the browser opens when it is ready.
[strata] filling the GPU's expert cache (3386 experts, 10.12 GiB of VRAM) ...
[strata] still starting (32 s) - please wait ...
[strata] still starting (52 s) - please wait ...
[strata] still starting (72 s) - please wait ...
[strata] still starting (92 s) - please wait ...
[strata] still starting (112 s) - please wait ...
[strata] almost ready ...
[strata] still starting (145 s) - please wait ...
[strata] still starting (165 s) - please wait ...
[strata] still starting (185 s) - please wait ...
[strata] still starting (205 s) - please wait ...
58.9 tok/s
[ok] tuned for this PC: --pcie-frac 0.19, --spec-min-p 0.70, --pool-workers 5 (58.9 tok/s)
[ok] start script: run-unsloth-ud-q4_k_xl.sh

All set.
API (OpenAI): http://127.0.0.1:8080/v1 (any API key; model name: anything)
API (Anthropic): http://127.0.0.1:8080/v1/messages
Next time: just run ./setup.sh (or run-unsloth-ud-q4_k_xl.sh) - it starts right away
Several requests at once (opt-in): --parallel 4 decodes up to 4 together instead of one after the other (each takes ~0.6 GB of VRAM from the expert cache; docs/BATCHING.md).
Run a quick benchmark now? It measures this install for about 10 minutes and writes a report you can share (later: ./bench.sh) [n]: y
Starting strata-unsloth-ud-q4_k_xl.json on port 8080 (loading takes 1-3 minutes; the PC may be slow meanwhile) ...
still loading (0 s) ...
still loading (30 s) ...
still loading (60 s) ...
still loading (90 s) ...
still loading (120 s) ...
still loading (150 s) ...
still loading (180 s) ...
Running the suite (about 3-10 minutes) ...
warming up (not measured) ...
calibrated: about 1.10 tokens per word
chat: 3 short questions x 3 runs, up to 100 new tokens
run 1 q1: decode 50.4 tok/s, first token 174.475 s, 100 tokens
run 1 q2: decode 45.6 tok/s, first token 0.611 s, 100 tokens
run 1 q3: decode 49.3 tok/s, first token 0.608 s, 100 tokens
run 2 q1: decode 65.9 tok/s, first token 0.51 s, 100 tokens
run 2 q2: decode 61.2 tok/s, first token 0.424 s, 100 tokens
run 2 q3: decode 54.2 tok/s, first token 0.441 s, 100 tokens
run 3 q1: decode 74.1 tok/s, first token 0.476 s, 100 tokens
run 3 q2: decode 63.5 tok/s, first token 0.413 s, 100 tokens
run 3 q3: decode 57.1 tok/s, first token 0.439 s, 100 tokens
4K prompt: 3 fresh prompts of about 4096 tokens, 24 new tokens
FAILED: {'type': 'server_error', 'message': 'the engine stopped unexpectedly (exit code None); the next request restarts it'}