memory: Gemma 4 E4B Q4_K_M peaks at 11.7 GiB on cpu_x86 and 10.6 GiB on Metal (4.6 GiB file, load_from_memory)

[geisten]

Summary

With Gemma 4 E4B Q4_K_M (4.64 GiB GGUF) loaded through geist_model_load_from_memory, peak RSS goes far beyond one copy of the weights on the CPU and Metal backends. The model bytes are a single verified, read-only, page-aligned anonymous mapping owned by the caller. The header says these bytes are aliased zero-copy. The extra memory therefore has to come from backend-side copies.

Backend	Host	Peak RSS (whole process)
file size		4.64 GiB
vulkan	Ryzen 9 9950X + RTX 2080 Ti, Linux	6.74 GiB
auto (cpu_x86, 16 threads)	same host	11.66 GiB
metal	M1 Max (64 GB)	10.6 GiB

Measured with HELIO (--ai-bench, peak ru_maxrss of the process including its ~1 GiB game state) at geistlib 7a0b421.

Impact

HELIO targets the Steam Deck (16 GB, shared with the GPU) and Macs with 16 GB. Above ~6 GiB the game cannot ship this model there. The CPU path is the Deck fallback.

Hypotheses (unverified)

• cpu_x86 repacks Q4_K/Q6_K into W4A8 / Q4K×8 layouts (q4k_to_w4a8, q4k_to_q4kx8) and keeps the original tensors as well. That would roughly double the weights.
• metal creates MTLBuffers with copies (newBufferWithBytes) rather than newBufferWithBytesNoCopy on the page-aligned host memory. On unified memory both are resident.
• vulkan uploads to device-local memory, so host RSS stays near one copy (cf. transformer/vulkan: arena-skip predicate, lazy embed-table resolve and resident mmap after upload (weights_device_copy follow-ups) #468 for resident-mmap follow-ups).

Requests

• Document the expected resident footprint per backend for load_from_memory and for load(path).
• CPU: either drop the source tensors after repacking, or repack lazily or in place, so one layout stays resident.
• Metal: wrap page-aligned caller memory with newBufferWithBytesNoCopy where alignment allows.
• Add peak RSS for Gemma 4 E4B to the performance matrix (bench: populate a same-protocol model × system performance matrix #364).

Related: #468 (vulkan resident mmap), #504 (cpu_x86 prefill).

[geisten]

Metal part fixed in #594: read-only load_from_memory bytes are now wrapped with newBufferWithBytesNoCopy (also across the 128 MiB VM entries macOS splits large mappings into). HELIO, Gemma 4 E4B Q4_K_M, M1 Max: peak RSS 9.9 → 5.3 GiB. The cpu_x86 items are tracked by #584/#590.

[geisten]

Documentation item: #611 adds a "Resident memory per backend" section to docs/BACKENDS.md. It covers repacked copies per backend and dtype, the env switch for each, how load(path) and load_from_memory count, and the per-session KV formula.

While writing it I found a real bug: a session's KV cache was sized from the model's cap, not the session's max_seq_len. Any consumer that loads with a large cap and opens small sessions paid for the full cap per session. #611 fixes it and adds a test.

Open questions:

1. cpu_x86 Q4_K on Zen 5 / AVX-512: the Q4_Kx8 repack still doubles Q4_K there (the 9950X in the table above). GEIST_Q4K_RAW=1 halves it at about half the prefill speed. Should the Steam Deck build (Zen 2, no AVX-512, so it reads raw already) be enough, or should the default flip to raw when the model is loaded via load_from_memory, since the caller's pages can never be released?
2. Peak RSS row for bench: populate a same-protocol model × system performance matrix #364: this needs a run of HELIO or geist-bench on the target hosts. I can't measure it here.

---

Generated by Claude Code