[austindixson]

Summary

Adds an opt-in Apple Silicon serving path using MLX:

• clm-embed-mlx serves Qwen3 last-token embeddings (default mlx-community/Qwen3-8B-4bit) over the same /v1/embeddings API CLM already uses
• clm-serve --device mlx runs the released projection head on Metal
• .[mlx] extra, scripts/setup_mlx.sh, and serve_mlx.sh for one-command Mac setup
• Unified-memory preflight so loads fail loudly instead of getting jetsam'd next to a large Metal server

Linux / CUDA / vLLM defaults are unchanged. On Darwin, vllm is skipped via a platform marker (same idea as #7).

Relation to other Mac work

• Complementary to the unquantized PyTorch MPS path in Add PyTorch MPS serving with the original Qwen3-8B weights #7 (original weights on MPS).
• Different from the closed MXFP8 converter approach in Add an optional Apple Silicon encoder path for CLM #5: this uses a public mlx-lm 4-bit checkpoint and integrates heads/cache as a first-class mlx device.

Test plan

• On Apple Silicon: bash scripts/setup_mlx.sh && bash serve_mlx.sh
• POST /v1/system_one with a typed Choice/Noul question returns a sensible distribution
• clm-serve --device cpu / CUDA path still behaves as before on Linux
• pip install -e '.[mlx]' does not pull vLLM on macOS

[widike]

Thanks for this, it made running CLM on a laptop practical. I tested it at 7bc936a on a MacBook Air M5 with 32 GB: setup_mlx.sh with Python 3.12, then serve_mlx.sh. Qwen3-8B-4bit downloaded and loaded in about 70 s, left about 8 GiB reclaimable, and /v1/systemone answered in about 320 ms warm. I needed three changes to get there, the first of which also affects results.

1. Long states lose their end (src/clm/mlx_embed.py:76)

self.tokenizer.encode(t, add_special_tokens=True)[: self.max_tokens]

This keeps the first max_tokens tokens. Training keeps the last tokens of a state, because the question sits at the end (train/embed_utils.py:43-49, state_ids returns ids[-self.cap:]). #6 fixes the same issue on the vLLM path. With head truncation, any state over 2048 tokens can have its question cut off.

-            self.tokenizer.encode(t, add_special_tokens=True)[: self.max_tokens]
+            self.tokenizer.encode(t, add_special_tokens=True)[-self.max_tokens:]

Training truncates action texts from the head (text_ids(keep="head")). The embedding endpoint cannot tell states from actions, so tail is the better single default: states are the inputs that realistically exceed the limit.

2. serve_mlx.sh resolves the wrong root (serve_mlx.sh:6)

The script lives at the repo root, but ROOT is computed as if it were in scripts/:

ROOT="$(cd "$(dirname "$0")/.." && pwd)"

So it cds into the parent of the checkout and never sources .venv/bin/activate. It only works if the venv is already active in the calling shell.

-ROOT="$(cd "$(dirname "$0")/.." && pwd)"
+ROOT="$(cd "$(dirname "$0")" && pwd)"

3. The memory preflight ignores inactive pages (src/clm/memory.py:98-99)

snapshot() counts free + speculative + purgeable pages. macOS keeps file cache in inactive pages and hands them back on demand, so on a machine that has been up for a while the preflight badly underestimates what is available. Mine reported:

[clm-mem] need 10.0 GiB reclaimable, have 0.8 GiB (pressure=unknown)

while memory_pressure reported 70% free and vm_stat showed about 11.3 GiB inactive. serve_mlx.sh exited with code 2 and never started the embedder.

-    # reclaimable-ish: free + speculative + purgeable
-    free_pages = (pages.get("free", 0) + pages.get("speculative", 0) + pages.get("purgeable", 0))
+    # reclaimable-ish: free + speculative + purgeable + inactive (macOS keeps file cache
+    # in inactive pages and hands it back on demand)
+    free_pages = (pages.get("free", 0) + pages.get("speculative", 0) + pages.get("purgeable", 0)
+                  + pages.get("inactive", 0))

With this change it reported 12.3 GiB and started normally.

Relatedly, pressure is always unknown: _pressure_level() looks for the words normal/warn/critical, but memory_pressure on macOS 27 only prints page counts and a "free percentage" line, with none of those words. sysctl -n kern.memorystatus_vm_pressure_level returns the level directly (1 = normal, 2 = warn, 4 = critical).

[austindixson]

Thanks @widike — all three items from your review are addressed, plus a follow-up on the memory free estimate. Tip is now 5ba5d35.

1. Tail truncation — mlx_embed.py keeps the last max_tokens ids via keep_tail() (same idea as training state_ids / vLLM truncate_prompt_tokens). Falsy / <= 0 still disables truncation.

2. serve_mlx.sh ROOT — ROOT="$(cd "$(dirname "$0")" && pwd)" (script lives at repo root; dropped the erroneous /..).

3. Memory preflight — free count includes inactive pages; pressure prefers kern.memorystatus_vm_pressure_level (1/2/4 → normal/warn/critical) with the existing memory_pressure parse as fallback.

Follow-up (double-count): an earlier tip summed free + speculative + purgeable + inactive. On macOS purgeable pages already sit on the active/inactive queues, so that overstated reclaimable free (~1.1–1.3 GiB on a live Mac) and loosened the jetsam guard. In 5ba5d35 the formula is free + speculative + inactive (purgeable dropped), matching psutil’s available on macOS. Inactive is still counted as you asked.

Live Mac check (Apple Silicon, tip 5ba5d35): sysctl kern.memorystatus_vm_pressure_level readable without root (1 → normal); snapshot().free_bytes matched a one-pass vm_stat recomputation of free+speculative+inactive; keep_tail(range(10), 3) → [7, 8, 9]; serve_mlx.sh ROOT resolves to the checkout.

Appreciate the careful review.