rMLX

Rust-native, single-binary MLX inference backend for Apple Silicon.

Warning

rMLX uses the MLX that Homebrew installs. On M5 with macOS 26, Homebrew's MLX has no Neural Accelerator kernels; rMLX warns and tells you how to build MLX with them: docs/MLX_PAIR.md. See which Mac gets what.

A native, no-Python local LLM server for Apple Silicon. It serves MLX-format models over OpenAI- and Anthropic-compatible HTTP APIs, so any client of those APIs can use it as a local backend. One cargo build --release artifact; no Python runtime and no GGUF.

What works

• Text generation: OpenAI /v1/chat/completions and Anthropic /v1/messages, streaming (SSE), temperature, top-k/p, penalties, a thinking budget, and schema-constrained decoding.
• Image input through image_url parts (HTTP(S) URL, data URI, file path or raw base64): Gemma 4, the encoder-free Gemma 4 unified checkpoint, Gemma 3 and Qwen3-VL-MoE.
• Audio input: Gemma 4 audio towers take input_audio parts. Whisper serves /v1/audio/transcriptions and /v1/audio/translations, and the rmlx transcribe CLI writes txt, vtt, srt or json. Long audio is walked in 30 s windows.
• Speech output: /v1/audio/speech runs Qwen3-TTS.
• Embeddings: /v1/embeddings, text and image, with jina-embeddings-v4.
• Tool calling: OpenAI tool_calls and Anthropic tool_use, multi-turn. The Qwen XML, Qwen JSON and Gemma call formats are parsed.
• Multi-model registry: load on demand, unload on idle, a cap on resident models, and a multimodal encoder-output cache keyed per model.
• Weight formats: bf16, affine 2, 3, 4, 5, 6 and 8 bits, mxfp4, mxfp8, nvfp4, ternary and ParoQuant.
• KV cache: 8-bit affine, mixed-width affine, and the rotation families TurboQuant, IsoQuant, PlanarQuant and RotorQuant; paged KV; asymmetric K/V; an SSD tier. Most codecs decode from a bf16 mirror, so they do not shrink the cache; docs/KV_QUANT.md says which do.
• Speculative decoding (--draft-kind): an MTP head (Qwen 3.5 family, Gemma 4 assistant), DFlash, DFlash 2, EAGLE-3, or a smaller full model of the verifier's family.
• Prompt caching: automatic prefix caching with block hashing.

There is no rmlx convert command: MLX-to-MLX conversion is not implemented.

Golden-token tests (temperature 0, exact token ids; make model-check-full) cover these snapshots:

Family	Snapshot	Arch
Gemma 4	gemma-4-e4b-it-mxfp8	Gemma4ForConditionalGeneration
Qwen 3.8	Qwen3.8-27B-4bit	Qwen3_5ForConditionalGeneration
Bonsai 2	Ternary-Bonsai-2-27B-mlx-2bit	prism_hadamard_qwen35 (model_type)
BitNet	bitnet-b1.58-2B-4T	BitNetForCausalLM

Qwen 3.6, the first-generation Bonsai checkpoints and MedGemma (Gemma 3) are supported and keep their goldens; they are not test targets.

docs/MODELS.md covers every architecture, including the Gemma 4 QAT checkpoints and their complex-image limit.

Performance

The decode anchors of the three test-target models, with the build and the command that measured them, are in docs/PERF_BASELINE.md. The same doc names the scripts that measure a build against them and that run one cell against mlx-lm, oMLX, ParoQuant or llama.cpp.

Requirements

• Apple Silicon Mac (M-series). Metal only — no CUDA / ROCm / x86.
• Rust stable (1.95+).
• MLX + mlx-c installed locally. rMLX links the stable mlx-c C ABI; it does not vendor or build MLX itself.

brew install mlx-c          # provides the MLX + mlx-c libraries

The build finds MLX via brew --prefix (or /opt/homebrew/opt/…) on its own. Point it elsewhere only if your install is somewhere else:

export MLX_C_PREFIX="$(brew --prefix mlx-c)"   # dir containing lib/libmlxc.dylib + include/
export MLX_PREFIX="$(brew --prefix mlx)"

MLX on each Mac

rMLX uses the mlx and mlx-c that Homebrew installs, on every Mac, and ships no MLX formula of its own. The install command is the same everywhere (Install); what differs is MLX's Neural Accelerator (NAX) kernels. MLX builds them only for a deployment target of macOS 26.2 or later, and only M5 and later chips have the GPU Neural Accelerator that runs them. Without them on such a chip, prompt processing (prefill) is slower. Token generation (decode) and the output do not change.

Mac	After brew install rmlx	To get the NAX kernels
M1 to M4, any macOS	Nothing is missing: these chips have no Neural Accelerator. No warning.	Not applicable.
M5 and later, macOS 27 or later	The Homebrew bottle of MLX for macOS 27 has the kernels. No warning.	Nothing to do.
M5 and later, macOS 26.2 to 26.x	The Homebrew bottle of MLX for macOS 26 has no NAX kernels. rMLX serves, prefill is slower, and rMLX warns at startup.	Build MLX from source for a macOS 26.2 target.
M5 and later, macOS 26.0 or 26.1	The same bottle, the same warning.	Update macOS: MLX does not build the kernels for a lower target.

The steps for each Mac, and the source build, are in docs/MLX_PAIR.md.

At startup, rMLX scans the mlx.metallib of the MLX it loaded. It warns only when the chip has a Neural Accelerator and the kernels are missing. The warning names the file it read and the fix for that macOS version. To check by hand:

strings "$(brew --prefix mlx)/lib/mlx.metallib" | grep -c steel_gemm_fused_nax
# M5 and later: a non-zero count. 0 means this MLX has no NAX kernels.

crates/rmlx-mlx/mlx-pin.txt names the MLX / mlx-c pair rMLX is developed and measured on. It does not limit rmlx serve. On M5 and later, and on a Mac whose chip cannot be identified, rmlx bench and rmlx baseline refuse to measure when the loaded pair is not the pinned one with the NAX kernels. rmlx healthcheck and make mlx-preflight report the same verdict.

rMLX builds against the mlx-c 0.6 and the 0.7 C API. When Homebrew moves mlx-c to the other C API under an installed rmlx, rMLX does not make the attention call whose argument list differs. It returns an error that names the fix, brew reinstall rmlx, and rmlx bench and rmlx baseline refuse on every Mac. rMLX also warns at startup when the MLX it loaded has another version than the one it was built against. See docs/FFI.md.

Extra tooling for kernel work (contributors only)

Not needed to build or run rMLX — only to compile-check or profile the Metal kernels:

• Full Xcode (not just the Command Line Tools), selected with sudo xcode-select -s /Applications/Xcode.app/Contents/Developer.
• The Metal Toolchain component, which Xcode does not install by default:

xcodebuild -downloadComponent MetalToolchain

With both present, make check-metal-compiles compiles every .metal kernel natively, and make profile-gputrace captures a GPU trace for Xcode (docs/PROFILING.md). Without them the compile gate reports SKIP; hosted CI runs it strictly.

Install

Each path builds from source and links the Homebrew MLX and mlx-c, so MLX must be present (brew install mlx-c).

Script (ensures Rust + MLX, then builds):

curl -fsSL https://raw.githubusercontent.com/Pushkinist/rMLX/main/install.sh | bash

Prefer to inspect first (recommended for any curl | bash):

curl -fsSL https://raw.githubusercontent.com/Pushkinist/rMLX/main/install.sh -o install.sh
less install.sh && bash install.sh

Homebrew (via tap):

brew tap Pushkinist/rmlx
brew trust Pushkinist/rmlx   # one-time: Homebrew loads third-party taps only once trusted
brew install rmlx

Cargo:

brew install mlx-c
MLX_C_PREFIX="$(brew --prefix mlx-c)" \
  cargo install --git https://github.com/Pushkinist/rMLX --bin rmlx rmlx-cli

Build

For development / from a clone:

git clone https://github.com/Pushkinist/rMLX
cd rMLX
cp .env.example .env          # set RMLX_O_MODELS_ROOT to your models folder
cargo build --release        # → target/release/rmlx

Or use the Makefile wrapper:

make build      # cargo build --workspace --release
make ci         # the pre-merge gate

Run

Serve an MLX-format model directory (the mlx-community safetensors layout). The model id is the directory name:

target/release/rmlx serve --model /path/to/mlx-community__gemma-4-e4b-it-mxfp8 --port 8080

Then call it like any OpenAI endpoint:

curl http://localhost:8080/v1/chat/completions \
  -H 'content-type: application/json' \
  -d '{
    "model": "mlx-community__gemma-4-e4b-it-mxfp8",
    "messages": [{"role": "user", "content": "Say hello in one word."}],
    "temperature": 0
  }'

Inspect a model's architecture + quantization without serving:

target/release/rmlx info --model /path/to/snapshot

See docs/CLI.md for the full subcommand / flag reference.

Documentation

Doc	Topic
docs/CLI.md	CLI subcommands, flags, env vars
docs/SERVER.md	HTTP server: OpenAI / Anthropic compat, routes, tool calling
docs/MODELS.md	Per-architecture model reference
docs/WEIGHT_QUANTS.md	Weight quantization formats
docs/KV_QUANT.md	KV-cache quantization contract; links the per-topic KV docs
docs/KV_CACHE.md	KV cache architecture
docs/SPECULATIVE.md	Speculative decoding (MTP / DFlash / Eagle3)
docs/PROMPT_CACHE.md	Prompt + automatic prefix caching
docs/SAMPLING.md	Per-token sampling + constrained decoding
docs/FFI.md	rmlx-mlx ↔ mlx-c FFI bridge
docs/METRICS_DB.md	Metrics DB: ingest, identity + rmlx metrics
docs/METRICS_SCHEMA.md	Metrics DB schema + metric registry

CLAUDE.md carries the architecture overview and the workspace crate graph.

Non-goals

• Not a GGUF runtime. rMLX reads MLX-format snapshots only.
• No training, fine-tuning, fusing or LoRA merging.
• No per-request LoRA hot-swap: fuse externally and load the merged snapshot.
• Apple Silicon only — no CUDA, ROCm, or x86 SIMD paths.

Releasing

The version lives in [workspace.package].version in the root Cargo.toml. The release flow is in docs/RELEASING.md.

License

Dual-licensed under either of

• Apache License, Version 2.0 (LICENSE-APACHE)
• MIT license (LICENSE-MIT)

at your option.

Sibling projects

rMLX stands on a lot of other people's work: the MLX ecosystem, the rotation-KV quantization research it ports, and the servers it learned its API shape from. Many thanks to:

MLX foundation

• ml-explore/mlx — the MLX array framework.
• ml-explore/mlx-c — the stable C ABI rMLX links against.
• ml-explore/mlx-lm — reference loader + numerics.
• oxiglade/mlx-rs — community Rust binding over mlx-c.
• safetensors/safetensors — the weight format + Rust crate.

KV / weight quantization research

• aivrar/multi-turboquant — TurboQuant KV toolkit.
• scrya-com/rotorquant — RotorQuant.
• ParaMind2025/isoquant — IsoQuant / PlanarQuant.
• z-lab/paroquant — ParoQuant weight rotation.
• TheTom/llama-cpp-turboquant — TurboQuant KV Metal kernels (llama.cpp).
• TheTom/turboquant_plus — TurboQuant+ KV cache + multi-axis fidelity scoring.

Servers & multimodal

• Blaizzy/mlx-vlm — vision-language reference.
• jundot/omlx — multi-model MLX server (API-shape reference).
• EricLBuehler/mistral.rs — fast, flexible Rust LLM inference engine.
• ai-dynamo/dynamo — NVIDIA datacenter-scale distributed inference framework.