[Defilan]

What

Samples and a guide section for running DeepSeek-V4.1-Flash EXL3 3.5bpw at tensor-parallel 3 across three DGX Sparks cabled in a ring (no switch), on the llmkube-vllm-cuda-gb10-dsv41-exl3 runtime image:

• config/samples/inferenceservice_multinode_vllm_three_sparks_ring.yaml: the InferenceService exactly as it is serving on the lab ring today (multiNode fabric block with the management IP as the bootstrap plane and all four ConnectX ports as the data plane, NCCL_IB_SUBNET_AWARE_ROUTING=1, the measured memory knobs).
• config/samples/model_dsv41_flash_exl3.yaml: the Model (the 54 files from an S3-compatible store; every member needs all of them locally, including the two 101 GB Engram tables).
• config/samples/jobs/dsv41-tp3-config.yaml and config/samples/jobs/drop-page-cache.yaml: the two checkpoint-side Jobs a TP3 boot needs (the 64 to 72 head padding in config.json, and a page-cache drop before the group boots).
• docs/site/guides/multi-node-inference.md: a "Three-Spark ring" section explaining why fabric.address and ibHCA are two different planes on a ring, the one NCCL variable that makes a P0 to P1 ring work, and the measured numbers.
• scripts/validate-samples.py: Job added to the standard kinds so the samples validate.

Why

The multi-node field docs describe a switched fabric. On a cabled ring no fabric address is reachable from every member, and NCCL's default NIC pairing (by channel index on both ends) cannot satisfy an odd cycle, so a group dies with ncclSystemError during channel setup. Both facts were found the hard way and are now written down next to a working example. The samples give a reproducible three-Spark deployment for the largest open model that fits three 128 GB Sparks.

Refs #1423, Refs #1752 (both stay open; the CRD-level follow-up is a separate sub-project).

How

The InferenceService is the lab's measured configuration copied verbatim (node names and management IPs are one lab's, the header says so), trimmed to the built image's contract, with DSpark k=5 speculative decoding and a 131,072-token context. Measured 2026-09-13 on the ring with the built image: three-rank NCCL all-reduce 23.2 GB/s bus bandwidth; prefill 1,094 to 1,334 tok/s from 3.4k to 49.8k prompt tokens; decode 26.5 tok/s single-stream without speculation, 31 to 35 with DSpark k=5 on prose and 65 on code; 82.5 GiB per Spark with the drafter (79.5 without). Design decisions worth noting: gpuMemoryUtilization: 0.78 is the floor that leaves a KV pool at TP3 (0.72 is negative); MAX_JOBS=1 bounds any first-serve JIT to one nvcc; the guide tells readers to turn swap off on the members, because a swapping Spark pages kubelet out instead of OOM-killing the worker.

Merge order: after defilantech/llmkube-runtimes PR #41, which publishes the :stable tag the sample references; draft until then.

Assisted-by: Claude Code (drafted the sample manifests from the measured InferenceService, the guide section, and this PR text; verified with make validate-samples, server-side dry-runs against the lab cluster, and the ring boot the numbers come from).

Checklist

• Tests added/updated (sample validation covers the new manifests; no Go code changed)
• make test passes locally (not affected: no Go changes; make validate-samples reports 34/34 valid)
• make lint passes locally (no Go changes)
• Commit messages follow conventional commits
• All commits are signed off (git commit -s) per DCO
• AI assistance (if any) is disclosed above, per CONTRIBUTING.md
• Documentation updated (if user-facing change)

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!