DGPP is a C++ and CUDA inference engine for NVIDIA DGX Spark (GB10) systems. It serves large mixture-of-experts and dense language models through an OpenAI-compatible HTTP API on one Spark, or on two or four Sparks joined by tensor parallelism over RoCE. The repository holds the whole stack: CUDA kernels, RDMA collectives, tokenizer, Jinja chat-template interpreter, scheduler, prefix cache, HTTP service, the cluster launcher and the deployment templates.
- Built for the Spark. The GB10's 128 GB of unified memory, its tensor cores and its single 200 Gb RoCE port shape every design decision: the memory plan, the kernels, the collectives and the NVMe tier are written for this box rather than adapted to it.
- Scales across Sparks. Two or four nodes serve one model through tensor parallelism over RoCE, with the collectives captured inside the decode graph. The ranks behave as one machine: rank 0 journals every admission, every peer replays the same schedule and checks a digest of its operation stream each tick.
- Serves checkpoints as published. FP8, NVFP4, MXFP4, MXFP8, packed int4 and int8, GPTQ codebooks, AutoRound and EXL3 trellis weights are read by kernels written for each format. There is no conversion step and no dequantized copy, and BF16 tensors are kept in a lossless 12-bit form that rebuilds the exact bits in registers.
- Exact. The default chain computes what the checkpoint specifies. A greedy request returns the same tokens run after run, on every rank, with speculative decoding on or off. Reduced-precision levers are explicit keys, off by default and named in the startup log.
- Fast where it counts. Graph decode, speculative decoding with MTP layers, DeepSeek's DSpark blocks or the DFlash2 drafter, continuous batching with latency-adaptive prefill, exact prefix caching with an NVMe cold tier, KV pools up to 2M tokens and YaRN ramps to 768K positions.
- Complete and operable. The OpenAI Chat Completions surface with tools, structured output, reasoning, logprobs, images and PDF files. Metrics as JSON and Prometheus. A setup wizard, a cluster launcher, versioned releases, a startup memory plan and fast boot from a resident image cache. No Python at serving time.
Decode throughput counts the engine's decode steps only. Cold prefill is the median of three uncached prompts at each length. Ranges span five prompt classes, and C is the number of concurrent client requests.
| Model / weights | Nodes | C1 decode tok/s | C8 decode tok/s | Cold prefill s at 2K / 8K / 32K |
|---|---|---|---|---|
| GLM-5.3-Flash NVFP4/FP8 | 2 | 31-37 | 58-62 | 2.22 / 7.14 / 29.6 |
| GLM-5.3-Flash NVFP4/FP8 | 4 | 59-65 | 104-113 | 1.17 / 4.63 / 20.5 |
| GLM-5.3-Flash FP8 | 4 | 47-52 | 1.33 / 5.69 / 24.5 | |
| Qwen3.8-Flash-Next NVFP4 | 1 | 44-52 | 1.36 / 4.52 / 18.1 | |
| Qwen3.8-Flash-Next NVFP4 | 2 | 62-75 | 2.13 / 7.87 / 28.8 | |
| Qwen3.8-Flash-Next FP8 | 4 | 73-87 | 1.58 / 5.61 / 23.0 | |
| Qwen3.8-Flash-Next FP4-Local | 1 | 35-76 | 66-133 | 1.62 / 5.75 / 23.0 |
| Qwen3.8-27B NVFP4 | 1 | 25-60 | 119-244 | 1.20 / 3.94 / 16.6 |
| Qwen3.8-27B NVFP4 | 2 | 45-105 | 175-369 | 1.20 / 4.23 / 17.3 |
| Qwen3.8-27B NVFP4 | 4 | 64-166 | 245-431 | 1.06 / 3.90 / 15.6 |
| Qwen3.8-27B FP8 | 4 | 58-137 | 219-406 | 1.25 / 4.39 / 17.6 |
| DeepSeek-V4.1-Flash-1017 MXFP4/FP8 | 4 | 54-89 | 84-139 | 1.01 / 2.64 / 10.0 |
| DeepSeek-V4.1-Flash-1017 EXL3 3.5 bpw | 4 | 70-134 | 165-248 | 1.19 / 3.45 / 13.3 |
Rows with every cell filled were measured on the shipped templates at the current scheduler defaults. Rows with an empty C8 cell were measured earlier, with fixed prefill budgets and before the decode rate at concurrency was kept, so their C1 and prefill figures stand and their service-level rates at C2 to C8 are in the wall table of the benchmarks. Benchmarks is the source of truth: every deployment, the wall rates with prefill and queueing included, the quality set, the long-context matrix, the decode-mode comparisons, and each row's source revision, binary hash and configuration. One command reproduces a template's rows:
scripts/serve_campaign.py run --config deploy/<template>.jsonflowchart LR
client["OpenAI-compatible client"] -->|HTTP| http
subgraph rank0["Rank 0"]
http["HTTP server and<br/>OpenAI service"] --> sched["Scheduler and<br/>admission journal"]
sched --> step0["Graph decode step"]
step0 --> mem0[("Resident weights,<br/>KV pool, prefix cache")]
mem0 --> nvme0[("NVMe cold tier")]
end
subgraph peers["Ranks 1 to 3"]
step1["Graph decode step"] --> mem1[("Resident weights,<br/>KV pool, prefix cache")]
mem1 --> nvme1[("NVMe cold tier")]
end
sched -.->|"journal over TCP"| step1
step0 <-->|"collectives over RoCE"| step1
One dgpp-serve process runs per Spark. Rank 0 serves HTTP, prepares
each request off the HTTP loop, and admits it through a journal that
every peer follows. A prompt whose token prefix matches a cached snapshot
attaches to it, in memory or from the NVMe tier, and prefills only its
suffix in chunks sized to the latency targets. Decode is one captured
graph replay per step for all live requests, with the attention, the
expert layers, the RoCE collectives and the speculative verification
inside the graph. Every rank holds its shard of the weights resident in
the checkpoint's own format and boots from a per-rank resident image.
DESIGN.md has the architecture and its contracts.
Each row is a checkpoint with a shipped deployment template. The node counts link to the templates and the model names link to their notes. Context is the request ceiling the template serves, the lesser of the checkpoint's positions and the KV pool. Draft names the speculative mechanism and depth the template ships with.
Templates are named cluster_<model>_<quant>_w<nodes>.example.json. Copy
one without .example and the launcher fills the node addresses, SSH user
and ports from the site's .env. A _yarn512k or _yarn768k variant is
the same deployment with a YaRN rope ramp that raises the request ceiling
past the checkpoint's 262K positions, accepted by the
release check. The
deployment catalogue explains the naming, the shared
scheduler defaults and the launcher flags that select plain decode, other
draft depths, cache formats and slot counts on any template.
To serve the FP8 GLM-5.3-Flash release, copy the four-node hybrid
template, set model to unsloth/GLM-5.3-Flash-FP8 and lower
engine.kv_capacity to 393216. The FP8 experts take 31 GiB more per rank,
and the startup memory plan refuses the hybrid's larger context. The
hybrid itself joins the NVFP4 routed experts of
dabsLabs to the
remaining tensors of the Unsloth FP8 release, and
the composition tool rebuilds it from
those sources.
Image inputs use each checkpoint's own vision encoder, loaded when the family supports it. See image inputs.
Each node is a DGX Spark on DGX OS or another Ubuntu 24.04 aarch64 install, with the NVIDIA driver, rdma-core and Python 3.10 or newer. Rank 0 also builds, which needs CUDA 13, CMake 3.25 or newer and a C++20 compiler. Multi-node deployments need RoCE between the Sparks and passwordless SSH from rank 0 to every peer. Each node holds the full checkpoint in its Hugging Face cache on local NVMe, from tens of gigabytes for the 27B models to several hundred for the largest. To build on an x86 workstation instead, use the Docker cross-build.
- One Spark. Qwen3.8-27B NVFP4 is the general-purpose choice: tools, images, the DFlash2 drafter and a 1M-token KV pool. Qwen3.8-Flash-Next NVFP4 is the mixture-of-experts alternative, with the single-Spark guide behind it.
- Two Sparks. GLM-5.3-Flash or Qwen3.8-27B NVFP4 with a 2M-token pool.
- Four Sparks. DeepSeek-V4.1-Flash-1017 at 1M tokens of context, or GLM-5.3-Flash at 768K with the NVMe tier ready to enable.
On an internet-connected Spark that will be rank 0:
git clone https://github.com/HawkBearPig/dgpp.git
cd dgpp
./scripts/setup.shThe wizard offers the templates by node count, saves the site settings,
checks SSH and dependencies on every node, walks through RoCE lane
selection, builds the release server, downloads the checkpoint once on
rank 0 and syncs the peers, then runs the serving preflight. Reruns
reuse the build and complete caches. Add --start to boot the world
once every check passes, and wait for READY before sending requests.
The API listens on 0.0.0.0:18080 by default and has no authentication
or TLS. Bind to 127.0.0.1 and put a reverse proxy in front for
anything beyond a trusted network. The
nginx example is a starting point.
Reasoning tokens count toward max_tokens, so this short example
requests low reasoning effort:
MODEL=$(curl --fail -s http://127.0.0.1:18080/v1/models | jq -r '.data[0].id')
jq -n --arg model "$MODEL" \
'{model:$model,max_tokens:160,reasoning_effort:"low",messages:[{role:"user",content:"Name three primary colors."}]}' \
| curl --fail http://127.0.0.1:18080/v1/chat/completions \
-H 'Content-Type: application/json' --data-binary @-python3 scripts/dgpp-cluster status --config /path/to/deployment.json
python3 scripts/dgpp-cluster down --config /path/to/deployment.jsonFor unattended setup, read-only checks, offline cache sync and the manual walkthrough, see getting started.
Any OpenAI-compatible client works. With the official Python SDK:
from openai import OpenAI
client = OpenAI(base_url="http://127.0.0.1:18080/v1", api_key="unused")
model = client.models.list().data[0].id
stream = client.chat.completions.create(
model=model,
messages=[{"role": "user", "content": "Name three primary colors."}],
max_tokens=160,
reasoning_effort="low",
stream=True,
)
for chunk in stream:
print(chunk.choices[0].delta.content or "", end="", flush=True)| Endpoint | What it serves |
|---|---|
POST /v1/chat/completions |
Chat with streaming, function and custom tools under constrained decoding, JSON Schema structured output, reasoning effort levels with reasoning_content, logprobs, stop strings, n, logit bias, images and PDF files |
POST /v1/completions |
Legacy completions |
GET /v1/models |
The served model with its input modalities, context limits and reasoning effort mapping |
GET /health |
Readiness |
GET /metrics |
JSON counters: throughput, prefill, graph batches, speculative acceptance and the caches. Also at /v1/metrics |
GET /metrics/prometheus |
The same counters in Prometheus text format. Each peer can expose its own on ports.metrics |
Agent clients such as opencode and Hermes run against it, and assistant history forwarded through LiteLLM is accepted with its thinking parts. The API profile has the full contract, the extensions and the model-dependent limits.
The default serving chain computes what the checkpoint specifies, the same way every time. A greedy request returns the same tokens run after run and on every rank, with speculative decoding on or off. Performance work keeps the cache, rank and speculative contracts exact. Representation changes such as the packed tensor-core prefill or the 12-bit BF16 form carry bitwise regression gates. Changes to accumulation order are judged with numerical accuracy checks and matched teacher-forced scoring.
Reduced-precision levers exist, such as FP8 activations for the tensor
cores or BF16 expert partial sums, but never silently. Each is an
explicit engine.* key, off by default, carried in the config digest so
every rank runs the same setting, and named in the startup log. A
template turns one on only when its measured quality cost on real
prompts sits inside the default chain's run-to-run band, and the
benchmarks list the default and levers-on rows separately.
Optimizing performance says which
levers pay on which model and how to measure a change on your own
workload.
A release is a versioned tarball installed once per node. It carries the server, the CUDA runtime it was built against, the launcher scripts, the templates and a checksum manifest. Published builds are on GitHub Releases, each with its SHA-256 in the notes.
CONFIG=/path/to/deployment.json
gh release download --pattern 'dgpp-*.tar.zst' # the latest published build
scripts/release.sh # or build one from this tree
scripts/dgpp-cluster install dgpp-VERSION.tar.zst --config "$CONFIG"
scripts/dgpp-cluster up --release VERSION --config "$CONFIG"
scripts/dgpp-cluster releases --config "$CONFIG"The version is stamped from the tree at build time and travels on the journal's settings record, so a world of mixed versions refuses to form. Versions install side by side. Rolling back is stopping the service and selecting an older one. See release and install.
Start from the template for your model and node count, then use the configuration reference for every setting, its default and its effect on memory, latency and output. Operations covers booting, stopping and watching a world, memory on a node, failure semantics and health checks. Tools lists the serving, diagnostic and validation commands.
| Path | What |
|---|---|
src/serve/ |
the HTTP server, the OpenAI-compatible service, the admission journal, the image and file frontends, metrics |
src/sched/ |
the scheduler, the prefix cache index and the NVMe tier's disk cache |
src/engine/ |
the session and engine interfaces every model implements, the eager and graph engines, speculative decoding, prefill tasks and batches, memory plans, prefix arenas |
src/models/ |
one directory per family: glm (GLM-5.3-Flash), glm_dsa (GLM-5.3), glm4 (GLM-4.7), qwen (Qwen3.8-Flash-Next, Qwen3.8-27B, Qwen3.5-122B-A10B), dsv41, dsv4, mimo, flint and the dflash2 drafter, with the KDA and DSA layer libraries they share |
src/kernels/ |
the CUDA kernels: GEMV and GEMM for each weight format, attention, MoE, norms, speculative selection and the sampler |
src/net/ |
the RoCE collective bus, TCP coordination and the roster |
src/text/ |
the tokenizer, the Jinja chat template, the tool and JSON Schema grammars and their parsers |
src/sample/ |
the sampler's host math |
src/loaders/, src/core/, src/common/ |
safetensors and the Hugging Face cache, arenas and tracing, logging and process memory |
apps/ |
dgpp-serve and the per-family load, forward and bind checks |
scripts/ |
the cluster launcher, setup, release packaging and the serving, benchmark and fabric operations |
tools/ |
the Python checkpoint and reference tooling the tests use |
deploy/ |
the deployment templates and their catalogue |
tests/ |
the unit, host, loopback and GPU suites |
| Page | What |
|---|---|
| Getting started | guided and manual setup, node prerequisites, checkpoint preparation |
| Cross-compiling | building Spark binaries on an x86 workstation with Docker |
| Networking | node addresses, SSH, RoCE lanes and cache paths |
| Dependencies | the vendored and system libraries |
| Deployment templates | the template catalogue, naming, shared defaults and launcher flags |
| Configuration | every site and deployment setting, its default and its effect |
| Operations | booting, stopping and watching a world, memory on a node, failure semantics, health |
| Release and install | the packaged release on a fresh cluster |
| Tools | serving, diagnostic and validation commands |
| OpenAI API compatibility | the request and response contract, extensions, limits and live metrics |
| Image inputs | the vision path per family, preprocessing, caching and memory |
| Tokenization | how prompts are encoded |
| HTTP output delivery | the streaming flush modes |
| Speculative decoding | MTP execution and correctness |
| Continuous batching | admission, shared prefill and the adaptive policy |
| Prefix cache | prefix reuse and memory sizing |
| NVMe cold tier | evicted prefix entries on local NVMe: persistence, explicit reuse across upgrades (nvme_cache.reuse_binding), pins, bounds and metrics |
| Optimizing performance | the levers, which pay on which model, and how to measure a change |
| Numerics | judging a numerics change |
| Benchmarks | every measured serving number with its method and reproduction |
| Benchmark procedures | the platform, kernel and transport probes |
| YaRN release check | the long-context acceptance run |
| Model cards | per-checkpoint serving notes and measurements |
| Model notes | architecture and port records per family: GLM-5.3-Flash NVFP4, GLM-5.3, GLM-4.7, Qwen3.8-Flash-Next, Qwen3.5-122B-A10B, DeepSeek-V4.1-Flash, DeepSeek EXL3, MiMo-V2.6-Flash, Flint |
| Testing | the suites and what each proves |
| Adding a model | the checklist for a new family |
| DESIGN.md | the serving architecture and its contracts |
| PLAN.md | implementation status by milestone |
| CHANGELOG.md | the history by milestone |
CONTRIBUTING.md describes the build presets, the test
discipline, the evidence a fabric change needs, the code style and where
things go. Changes are developed against real hardware: a Spark runs the
single-node suites and the four-node fabric runs the multi-node checks
and the serving benchmarks. Every push runs the Python host suites and
the documentation checks, and pushes to master cross-compile the server
for the GB10. Bug reports, model requests and feature requests have
issue templates, and
SECURITY.md says how to report a vulnerability privately.
The engine serves checkpoints published by Zhipu AI (GLM), the Qwen team, DeepSeek, Xiaomi MiMo, RecursiveSI (Flint) and NVIDIA, and quantized exports by Unsloth, dabsLabs, RadixArk, Saren, local-inference-lab, kaushikvira, LibertAIDAI and HawkBearPig. The DFlash2 drafter for Qwen3.8-27B is published by z-lab. The EXL3 weight format comes from exllamav3. Image decoding uses the vendored stb_image and simplewebp, and regular expressions use PCRE2.
Apache License 2.0. See LICENSE and NOTICE.
Third-party licenses are recorded in docs/licenses/.