[Engine] --expert-cache above the setup default kills the engine on Metal (Apple Silicon)

Problem: on the Metal backend, --expert-cache above the 64 that metal/setup_mac.py writes makes the engine exit 1 on any prompt shorter than the prefill chunk.

Re-tested on v0.2.0-beta (engine 0.1.45.3-metal) on 2026-10-19: still reproduces. This is not a 0.1.45-only problem.

Environment: v0.2.0-beta, engine 0.1.45.3-metal, built from the tag with ./setup.sh --update and no local patches. macOS 27.0.1. Apple M1 Max, 24-core GPU, 64 GB unified memory. Qwen3.8-Flash-Next Q2_0 (full 512 experts, --mmap-experts), --spec 2, context 32768, --prefill auto.

Reproduce:

cd ~/Documents/AI/Strata-mac
# change only the expert cache in the shipped config
python3 -c "import json;c=json.load(open('strata-q2_0.json'));a=c['args'];a[a.index('--expert-cache')+1]='2048';json.dump(c,open('/tmp/cfg.json','w'))"
./.venv/bin/python ./serve/server.py --engine strata --config /tmp/cfg.json --port 8081
# then send any prompt shorter than the prefill chunk (it comes out at 5376 here)

Expected: the request is served. --expert-cache 2048 is a legal value and the engine accepts it at boot (filling the GPU's expert cache (2048 experts, 2.64 GiB of VRAM)).

Actual: POST /v1/chat/completions returns 400, the engine exits 1, and the next request gets 503 while the server restarts it:

lending the prompt path its slots failed: prefill: relayout needs borrowed buffers and a chunk of at most 0

Which prompts fail (this build, cache 2048, chunk 5376):

target tokens	actual prompt_n	result
1,900	—	400, `lending the prompt path its slots failed ... at most 0`
2,400	—	503 (engine restarting after the previous 400)
3,000	—	400 (same message)
3,800	—	503
4,800	—	400 (same message)
7,140	—	503
9,000	7,251	OK, prefill 174 tok/s
13,000	10,460	OK, prefill 284 tok/s

The boundary is the chunk itself: below it the engine asks for a relayout and fails, above it the prompt fills the chunk and never asks. --expert-cache 64 is unaffected and never enters that path (the prompt path allocates its own buffers (too few cache slots to borrow)).

Which condition fails: instrumenting the guard in Prefill::relayout names the clause:

DIAG relayout: clause=borrowed==false chunk=5120 T_max=0 borrowed=0 borrow_spill=0 borrow=0x178bfa1000

So it is !m.borrowed, not chunk > m.T_max. borrow is non-null while borrowed is 0: the caller passed the region, and Prefill::init() did not reach m.borrowed = borrow != nullptr.

Not pack-specific: ben0112 reports the same failure on an M5 Max 128 GB with UD-Q4_K_XL; this report is an M1 Max 64 GB with Q2_0. Different chip, memory and quant, same message and same threshold.

A tested fix exists: PR #3030 (a successful no-op Prefill::relayout on Metal, plus guarding the lend slot-marking loop; +16 lines, entirely inside STRATA_GPU_METAL) was closed in the 0.2.0-beta tracker reset together with this issue. It was verified on this exact configuration before the reset — cherry-picked onto v0.1.45, --expert-cache 6144 with chunk 8192: before, a ~6,000-token prompt died 3 of 3; after, prompts of 1,235 / 2,439 / 4,043 / 4,845 / 5,648 / 6,605 / 10,462 tokens all completed with the engine up and no at most 0 or given back. The Metal branch is all I can vouch for; I have not built the CUDA, HIP or SYCL backends.

Mode 2 (given back SIGABRT) is not claimed as fixed. I never reproduced it on this machine (at --expert-cache 64 the lend path is never entered) and it did not appear in this re-test either.

Reproduction status: reproduced locally, on v0.2.0-beta, three times per prompt length, with a fresh nonce each time so cache_n is 0.

Follow-up: reproduced, and it is a different trigger from #2090

I read the #2090 thread and the triage note on #2131 before writing this. Short version: the
#2090 fix is in the binary I am running and it does not cover this case, and I have narrowed the
failing clause to one of the four.

The #2090 fix is present in 0.1.44.1

src/program/generate.cpp:11509-11527 — the loan starts where the request's chunk needs it, and a
chunk that cannot be lent is halved:

int64_t need_p = part_slots(p, want_p);
while ((need_p <= 0 || need_p + 128 > p.cache->slots()) && want_p > 256) {
    want_p = std::max<int64_t>(256, want_p / 2 / 256 * 256);
    need_p = part_slots(p, want_p);
}
if (need_p <= 0 || need_p + 128 > p.cache->slots()) { ... "#2090" ... }
...
first = (int32_t) (p.cache->slots() - need_p);
It does not fire here, because the failure is inside relayout(), called on the next line (generate.cpp:11530) — not in the loan sizing that fix touched.

Reproduction (deterministic, 3 of 3)
M1 Max / 64 GB / macOS 27.0.1 / Metal, 0.1.44.1-metal (commit 296b59d), Q2_0 pack:

--expert-cache 6144, --prefill auto → at start it picks auto chunk 8192, borrows 2513 slots
a request whose chunk is below 8192 (I used a ~6,000-token prompt; the failing chunk is 5120) → 400, engine exits 1
a request whose chunk is above 8192 (~16,000-token prompt) → succeeds
That matches the shape #2090 described ("a request's chunk is smaller than the auto chunk"), which is why I am adding it to this thread rather than opening a separate one.

It is not simply "any borrow": --expert-cache 2048 borrows 1714 slots (auto chunk 5376) and works, and 2048 / 4096 / 8192 all work when the request's chunk does not fall below the auto chunk. --expert-cache 64 never enters this path at all (the prompt path allocates its own buffers (too few cache slots to borrow)).

The failing clause, measured
I put a temporary print in the guard in Prefill::relayout (prefill.cpp:1349) that names the clause instead of reporting T_max for all four (patch below — it is a print, not a behaviour change). On the reproducing case it prints, every time:

代码块


DIAG relayout: clause=borrowed==false chunk=5120 T_max=0 borrowed=0 borrow_spill=0 borrow=0x178bfa1000
So the clause is !m.borrowed — not chunk > m.T_max.

This corrects my own first report. There I speculated that m.T_max had stayed at its initialiser. That was wrong: chunk=5120 and T_max=0, so if borrowed were true the guard would pass. The chunk size is irrelevant here; the layout flag is the whole problem.

m.borrowed is written in exactly one place, Prefill::init() (prefill.cpp:1082, assignment at 1122: m.borrowed = borrow != nullptr;), while the server has just announced that it did arrange a loan (the prompt path borrows 2513 CUDA0 cache slots). So init() did not reach 1122, and relayout() was then called believing the path was borrowed.

That is the inconsistency worth fixing: the caller decides a loan exists from its own bookkeeping (p.first, part_slots), while relayout() decides from Impl::borrowed. Nothing reconciles the two, and the guard reports the mismatch as a chunk-size limit of zero.

Directions (not tested — I stopped at the diagnosis)
relayout() rejecting with a message that says the path is not in the borrowed layout, instead of reporting T_max for four different conditions; and
either making borrowed / T_max impossible to leave stale when init() returns early, or having the caller not call relayout() unless init() succeeded.
I did not go further because both routes change engine state, and from the outside I cannot see what makes init() return early — that is the next thing I would instrument.

Reproduction, in full
bash


git clone --depth 1 --branch v0.1.44.1 https://github.com/Niko1221/Strata.git
cd Strata && ./setup.sh --yes --no-start        # picks Q2_0 on a 64 GB Mac
# in strata-q2_0.json:  "--expert-cache", "64"  ->  "6144"
.venv/bin/python serve/server.py --engine strata --config strata-q2_0.json --port 8081
# then a prompt of roughly 6,000 tokens; the engine exits and the server returns 400
Engine log at the failure:

代码块


strata generate: expert cache 6144 slots, 7.91 GiB of VRAM
strata generate: pre-filled 6144 of 6144 slots from the profile in 1.9 s (4409 MB/s); slot 0 verified
strata serve: prompt chunk auto: 8192 tokens, a 96-slot ring
strata serve: the prompt path borrows 2513 CUDA0 cache slots (3.24 GiB)
DIAG relayout: clause=borrowed==false chunk=5120 T_max=0 borrowed=0 borrow_spill=0 borrow=0x178bfa1000
ERR lending the prompt path its slots failed: prefill: relayout needs borrowed buffers and a chunk of at most 0
strata [server] 400 invalid request: lending the prompt path its slots failed: ...
strata [server] the engine stopped unexpectedly (exit code 1)
Diagnostic patch (for your use, not a proposed change)
diff


--- a/src/prefill/prefill.cpp
+++ b/src/prefill/prefill.cpp
@@ -1347,6 +1347,17 @@ bool Prefill::relayout(int64_t chunk, void* borrow, uint64_t borrow_bytes, std::
         return false;
     }
     if (!m.borrowed || borrow == nullptr || chunk <= 0 || chunk > m.T_max) {
+        const char* why = !m.borrowed       ? "borrowed==false"
+                        : borrow == nullptr  ? "borrow==nullptr"
+                        : chunk <= 0         ? "chunk<=0"
+                                             : "chunk>T_max";
+        std::fprintf(stderr,
+                     "DIAG relayout: clause=%s chunk=%lld T_max=%lld borrowed=%d borrow_spill=%d "
+                     "borrow=%p\n",
+                     why, (long long) chunk, (long long) m.T_max, (int) m.borrowed,
+                     (int) m.borrow_spill, borrow);
         err = "prefill: relayout needs borrowed buffers and a chunk of at most " + std::to_string(m.T_max);
         return false;
     }
I can test a candidate on this machine (Metal, M1 Max, ~1 minute incremental rebuild) and report back — including whether it also covers the --prefill 2048 variant from my first report.

Unrelated to the bug, in case it is useful for the Mac side: with the shipped --expert-cache 64 this M1 Max / 64 GB gives 27.0 tok/s decode and 283 / 289 tok/s prompt at 4K / 16K tokens (thinking off), against 13.3 / 83 / 80 for a dense 27B on MLX on the same machine — about 2.0x decode and 3.4-3.6x prompt. Everything else on the Metal path behaved correctly across roughly 30 engine starts today.

I have a tested fix for both failure modes on this machine family (M1 Max 64 GB): opened #3030, which addresses this issue - Prefill::relayout becomes a successful no-op on Metal (the guard clause this thread diagnosed) and the lend loop takes no slots, so the post-read refill no longer writes into the arena's purged region. Measured there: 4,643- and 10,797-token prompts complete, engine stays up, logs clean.

Independent check on a different machine/artifact: M1 Max 64 GB with the Q2_0 pack (not the IQ3_S in this PR's runs). The fix holds here too, and I have one caveat about what a larger cache is worth.

Build: v0.1.45 (477ac13) + #2895 (the Metal build fix for #2860) + this PR cherry-picked on top.
Hardware: Apple M1 Max, 64 GB, macOS 27.0.1. Pack: Q2_0, the full 512-expert GSQ-RCO build, --mmap-experts.

Before the fix, on this machine

--expert-cache 6144 + --prefill auto announces prefill_cap 8192 and borrows slots. A ~6,000-token prompt (failing chunk 5120) then dies deterministically, 3 of 3. The engine prints:

ERR lending the prompt path its slots failed: prefill: relayout needs borrowed buffers and a chunk of at most 0
strata [server] 400 invalid request: lending the prompt path its slots failed: ...

and exits 1.

The guard itself is if (!m.borrowed || borrow == nullptr || chunk <= 0 || chunk > m.T_max)
(src/prefill/prefill.cpp:1518 in v0.1.45). A temporary print names which clause fired — this is the
follow-up diagnosis the PR description refers to:

DIAG relayout: clause=borrowed==false chunk=5120 T_max=0 borrowed=0 borrow_spill=0 borrow=0x178bfa1000

So it is !m.borrowed, not chunk > m.T_max — the same clause this PR guards out.

Worth reading the numbers: borrow is non-null (0x178bfa1000) while borrowed is 0 — the caller
had already handed the borrowed region over, and Prefill::init() did not reach
m.borrowed = borrow != nullptr (prefill.cpp:1122). T_max = 0 is not "the chunk is too large";
it is the same incomplete init() failing to set it.

After the fix

Same --expert-cache 6144 (prefill_cap 8192), one prompt at each length, engine kept alive:

target	actual prompt_n	prefill tok/s	decode tok/s	result
1,500	1,235	55 (cold first request)	20.2	pass
3,000	2,439	254	27.1	pass
5,000	4,043	293	27.9	pass
6,000	4,845	287	28.2	pass
7,000	5,648	291	26.4	pass
8,192	6,605	286	26.8	pass
13,000	10,462	285	26.6	pass

No at most 0, no given back, engine alive throughout. The 4,845-token case sits below the 8,192 auto chunk — the exact shape that used to exit 1 — so mode 1 is closed here too.

Caveat on mode 2: I never reproduced the given back SIGABRT before the fix with this pack (because --expert-cache 64 never enters the lend path, see below), so I cannot claim to have verified that mode — only that nothing of that kind appeared over seven prompts up to 10.5k with the cache raised to 6144.

A caveat that may matter for the PR/release text

A larger expert cache buys nothing on this machine. Same protocol (unique nonce, cache_n = 0 asserted, 400 tokens generated, two rounds each):

`--expert-cache`	slots	VRAM free	prefill ~4K	decode ~4K	prefill ~16K	decode ~16K
64 (shipping default)	64	10,273 MiB	276	25.8	289	27.1
2,048	2,048	7,658 MiB	261	26.4	287	26.4

Both rows use the same protocol. For 6,144 I ran the length sweep above instead (prompts of 4,043–10,462
tokens, same 400-token generations): prefill 285–293, decode 26.4–28.2 — the same band.

All three are inside the same run-to-run spread. Likely reason: the pack is memory-mapped (--mmap-experts, ~38 GB) and the OS page cache already holds most of it on a 64 GB machine, so the Metal-side slot count is not the limiter.

Two consequences worth saying out loud:

The value of this fix on this class of machine is letting the cache be raised without a crash, not a speed-up by itself.
Users who never raised --expert-cache off 64 were never exposed to this bug — at 64 the engine says the prompt path allocates its own buffers (too few cache slots to borrow) and never enters the lend path. Only people who raised it hit either mode.

Suggested wording

Metal: prompts of any length no longer stop the engine when the prompt path borrows expert-cache slots (#2671). The shipping --expert-cache 64 was never affected.

Thanks for taking the time to report this.

We're resetting the tracker for 0.2.0-beta. There are more than 1,300 open issues and pull requests, too many for us to read properly, so we're closing everything that isn't being worked on right now. Closed here does not mean rejected.

To reopen it takes about two minutes:

Re-test on 0.2.0-beta. It may already be fixed. Write down what you see.
Start the title with its area: [Engine], [GUI], [TUI], [Benchmark], [Docs] or [Other].
Fill in the parts of the issue form: what happened, the Strata version, your graphics card, memory and system.
Comment /reopen. A check runs and either reopens the issue or lists what is still missing.

The rules are in CONTRIBUTING.md.

/reopen

Re-tested on 0.2.0-beta as the reset asked. Mode #1 from this thread still reproduces; mode #2 (SIGABRT) does not.

Build: v0.2.0-beta-9-gf57c0adc (2026-10-19, current main)
Hardware: Apple M5 Max, 128 GB unified memory
Model: Qwen3.8-Flash-Next UD-Q4_K_XL, --mmap-experts --prefill auto --max-context 65536

Mode #1 reproduces

With --expert-cache 2048 (above the 64 that metal/setup_mac.py writes), every request whose prompt fits in a single prefill chunk fails:

prompt_tokens	result
1,904	400 `lending the prompt path its slots failed: prefill: relayout needs borrowed buffers and a chunk of at most 0`
2,380	400 (same)
2,975	400 (same)
3,808	400 (same)
7,140	400 (same)

--expert-cache 64 never borrows slots and works normally, which matches the table at the top of this thread.

A refinement this thread does not have

The trigger is not prompt length per se — it is the prompt needing a smaller chunk than the initial layout. With --prefill auto (chunk 8192) a prompt longer than 8192 takes the full chunk every time, so no relayout is requested and it succeeds:

prompt_tokens	result
9,078	OK, 6.7 s
10,780	OK, 7.6 s
13,174	OK, 8.9 s (73.8 tok/s)

So Prefill::relayout is only reachable for prompt_len <= chunk, which is the common short-prompt case — a normal curl smoke test will not hit it, an agent workload will. Splitting the combined guard at prefill.cpp:1638 into separate messages (as suggested above) would make this visible instead of reporting a limit of 0 for a condition unrelated to 0.

Mode #2 (SIGABRT) not reproduced

A 13,174-token first request completed in 8.9 s, engine stayed up, log clean — no SIGABRT, no a copy touches a device allocation ... that was given back. #3030 reports mode #2 on M1 Max 64 GB with the Q2_0 pack; this is M5 Max 128 GB with the UD-Q4_K_XL pack, so memory pressure may be what separates them. I have not verified that, and I am not claiming mode #2 is fixed.

Logs from every run are available on request.

/reopen

Re-tested on 0.2.0-beta as the reset asked. Mode 1 reproduces. Mode 2 does not.

What happened

--expert-cache 2048 (above the 64 that metal/setup_mac.py writes) makes every prompt shorter than the prefill chunk kill the engine. With --prefill auto the chunk comes out at 5,376, so:

target tokens	actual prompt_n	result
1,900	—	400 `lending the prompt path its slots failed: prefill: relayout needs borrowed buffers and a chunk of at most 0`
2,400	—	503 (engine restarting after the previous 400)
3,000	—	400 (same message)
3,800	—	503
4,800	—	400 (same message)
7,140	—	503
9,000	7,251	OK, prefill 174 tok/s
13,000	10,460	OK, prefill 284 tok/s

The threshold is exactly the chunk: below 5,376 the relayout is requested and fails, above it the prompt fills the chunk and never asks. That is the same boundary ben0112 reports, and it matches the original diagnosis at the top of this thread.

One addition: the engine does not only return 400. It exits code 1 and the server restarts it, which is why the next request gets 503:

[strata] the engine stopped unexpectedly (exit code 1). The engine exited (code 1) after it reported:
lending the prompt path its slots failed: prefill: relayout needs borrowed buffers and a chunk of at most 0

So in an agent workload (many short prompts) the engine is restarting more or less permanently, not just refusing one request.

Memory pressure is not involved here: this is a 64 GB machine with the Q2_0 pack, VRAM free well above the arena, and it dies on a 1,900-token prompt.

Version

v0.2.0-beta (engine 0.1.45.3-metal), built from the tag with ./setup.sh --update and no local patches.

Graphics card, memory, system

Apple M1 Max, 24-core GPU, 64 GB unified memory, macOS 27.0.1. Model: Qwen3.8-Flash-Next Q2_0 (full 512 experts, --mmap-experts), --spec 2, ctx 32768.

This is not pack-specific

ben0112 reports the same failure on M5 Max 128 GB with UD-Q4_K_XL. Here it is M1 Max 64 GB with Q2_0. Different chip, different memory, different quant, same message and same threshold. --expert-cache 64 is fine in both — it never borrows slots (the prompt path allocates its own buffers (too few cache slots to borrow)).

A tested fix exists, and it was closed in the same reset

PR #3030 (a successful no-op Prefill::relayout on Metal, plus guarding the lend slot-marking loop; +16 lines, entirely inside STRATA_GPU_METAL) fixes both modes. I verified it on this exact configuration before the reset, cherry-picking it onto v0.1.45:

before: --expert-cache 6144, chunk 8192, a ~6,000-token prompt died 3 of 3
after: prompts of 1,235 / 2,439 / 4,043 / 4,845 / 5,648 / 6,605 / 10,462 tokens all completed, engine stayed up, no at most 0, no given back. The 4,845-token case is below the 8,192 chunk — the exact shape that used to fail.

The CUDA/HIP/SYCL branches are byte-identical by construction, but I can only speak for the Metal build; I have not built the other backends.

Caveat on mode 2: I never reproduced the given back SIGABRT on this machine (at --expert-cache 64 the lend path is never entered), and it does not appear in this re-test either. I am not claiming it is fixed — only that mode 1 is measured live in 0.2.0-beta and mode 2 is not.

Suggestion

If the relayout guard is going to stay as-is for now, splitting the combined condition in prefill.cpp (line 1638 in this tree) into separate error messages would at least stop it reporting a chunk of at most 0 for a condition that has nothing to do with 0.

Not reopened yet. Still missing:

Start the title with its area: [Engine], [GUI], [TUI], [Benchmark], [Docs] or [Other], then a short description. Example: "[Engine] decode stops after 2 minutes".
Re-test on Strata 0.2.0-beta and name that version in the issue. If it no longer happens, say so in a comment instead.

Edit the issue and comment /reopen again. Details: CONTRIBUTING.md

Reopened, thank you. A maintainer will pick it up from here.

Thank you, and thanks to DarkJoll and ben0112 for the diagnosis. Fixed in 0.2.3: Metal no longer lends expert-cache slots to the prompt path (Prefill::relayout), and the automatic expert cache is capped at 1500 slots, so --expert-cache above the setup default no longer stops the engine on prompts shorter than the chunk.

Release: https://github.com/Niko1221/Strata/releases/tag/v0.2.3

Navigation Menu

[Engine] --expert-cache above the setup default kills the engine on Metal (Apple Silicon) #2671

Follow-up: reproduced, and it is a different trigger from #2090

The #2090 fix is present in 0.1.44.1

Before the fix, on this machine

After the fix

A caveat that may matter for the PR/release text

Suggested wording

Mode #1 reproduces

A refinement this thread does not have

Mode #2 (SIGABRT) not reproduced

What happened

Version

Graphics card, memory, system

This is not pack-specific

A tested fix exists, and it was closed in the same reset

Suggestion

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

[Engine] --expert-cache above the setup default kills the engine on Metal (Apple Silicon) #2671

Description

Activity

ailabjeremy-ctrl commented on Oct 16, 2026

Follow-up: reproduced, and it is a different trigger from #2090

The #2090 fix is present in 0.1.44.1

DarkJoll commented on Oct 18, 2026

ailabjeremy-ctrl commented on Oct 18, 2026

Before the fix, on this machine

After the fix

A caveat that may matter for the PR/release text

Suggested wording

Niko1221 commented on Oct 19, 2026

ben0112 commented on Oct 19, 2026

Mode #1 reproduces

A refinement this thread does not have

Mode #2 (SIGABRT) not reproduced

ailabjeremy-ctrl commented on Oct 19, 2026

What happened

Version

Graphics card, memory, system

This is not pack-specific

A tested fix exists, and it was closed in the same reset

Suggestion

github-actions commented on Oct 19, 2026

github-actions commented on Oct 19, 2026

Niko1221 commented on Oct 29, 2026

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions