I’m Naresh, a distributed systems and GPU networking engineer. I work on the infrastructure that large-scale training and inference run on: kernel networking, RDMA/RoCE, DPDK, SR-IOV, KVM and NCCL. When a 512-GPU job stalls, the cause is usually one bad link, NIC or GPU, while every rank reports the same timeout. I build the infrastructure that finds that one component and gets training back fast, and that moves bytes between GPUs without wasting the hardware.
Everything here is built in public. A number only appears once it’s measured and linked to its run.
|
goodput: fault attribution & recovery for multi-node training Problem. One bad GPU, NIC, cable or switch port stalls a collective, and every rank times out the same way. Stock tooling detects in minutes, attributes nothing, and restarts the whole job. Approach. Fault injection with a ground-truth ledger, NCCL flight-recorder signatures, DCGM and NIC telemetry joined on the topology to name the culprit rank and component, then the smallest safe recovery: in-process communicator re-init, single-node replacement, peer in-memory checkpoints. Status. Downstream. infra-incident-intel: goodput's labeled incidents train two Hugging Face models that plug back in as its classifier: a calibrated, evidence-citing ModernBERT triage tier and a grounded Qwen3 + LoRA root-cause tier. |
kvwire: RDMA KV-cache transport for disaggregated inference Problem. Prefill is compute-bound and decode is memory-bound. Colocated, they interfere: prefills spike TPOT, decodes spike TTFT. Approach. Split them onto separate GPU pools, and make the part in between fast and safe: a GPUDirect RDMA transport, a Triton KV re-layout kernel (kept only if the data justifies it), and a coordinator that stays correct when the network fails mid-transfer. Plugs into vLLM through its KV-connector interface. Status. |
In NCCL, nearly every hardware fault ends the same way: a watchdog timeout on every rank. The engineering is in the last two columns: naming the one component, then choosing the cheapest recovery that is still safe.
| Fault | What every rank reports | Evidence that names the component | Smallest safe recovery |
|---|---|---|---|
| GPU off the bus (Xid 79) | Watchdog timeout, same collective sequence number | Xid in kernel log / DCGM; the one rank missing from the flight recorder's last completed op | Swap in a spare node, restore from peer in-memory checkpoint |
| Uncorrectable ECC / row-remap failure (Xid 48, 64) | Hang or NaN loss, then timeout | DCGM ECC and row-remap counters on one GPU | Drain node, resume from last checkpoint |
| NVLink fault (Xid 74) | Intra-node collectives stall | Xid 74, NVLink CRC / replay counters | Swap node |
| RoCE link flap or bad optic | Inter-node all-reduce stalls on all ranks | link_downed, symbol / CRC errors on one NIC port, IB async events |
Communicator re-init once the link recovers; else swap node |
| PFC pause storm | All ranks slow, no errors anywhere | Pause counters traced hop by hop to the port that paused first (FIG. 3 below) | Fix the origin (ECN / DCQCN thresholds, isolate port); restarting changes nothing |
| Straggler (thermal / power throttling) | Step-time skew, no timeout | DCGM clock-throttle reasons, per-rank step time | Cordon at the next checkpoint boundary |
goodput's fault injector knows exactly what it broke, so every incident it produces is labeled by construction. Those incidents become a public benchmark and two models that plug back into goodput as its classifier, and have to beat its hand-written rules to earn the slot. Code: infra-incident-intel.
| Artifact | What it does | Stage |
|---|---|---|
infra-failure-bench · dataset |
Multi-host incident windows with root cause, culprit and evidence lines. Test splits hold out cluster topologies, cascades (root ≠ loudest symptom) and real incidents | schema + split code, 27 tests · awaiting goodput export |
infra-triage-modernbert · Space |
Fast tier: root-cause class, the lines that prove it, calibrated resolve-or-escalate; milliseconds on CPU via ONNX | router, calibration, goodput Verdict adapter |
infra-rca-qwen3-4b-lora · Space |
Slow tier: grounded RCA naming culprit, causal chain and cited lines; GRPO on rewards a script can check (citations exist, category and culprit match the ledger) | output schema, grounding checks, rewards |
Each one runs in the browser, with its model and assumptions stated. No hand-waved benchmarks.
| Model | What it shows | |
|---|---|---|
| 💥 | One dead link, 512 idle GPUs | Cut a link in a live ring all-reduce and watch every rank time out the same way |
| 🔎 | Triaging an NCCL timeout | Four incident drills: find the culprit from flight-recorder, Xid and NIC evidence |
| 🧊 | PFC: the lossless network that can freeze itself | Push a toy RoCE fabric into deadlock, then turn on ECN |
| 🛤️ | Where NCCL traffic actually goes | Rail-optimized vs fat-tree for DP, TP and MoE all-to-all, with and without PXN |
| 📦 | The KV-cache bill | Size the prefill→decode handoff for real model configs and link speeds |
| 💾 | How often should a 16K-GPU job checkpoint? | Young/Daly with sliders: which lever buys the most goodput |
| 🧵 | The packet path | Step one packet through the kernel stack, XDP, DPDK and RDMA |
No self-ratings: every row links to the work and states how far along it is.
| Area | Evidence | Stage |
|---|---|---|
| Collectives & NCCL | goodput · distributed-training-framework-nccl · ring all-reduce post | design + harness |
| RDMA / RoCE / PFC | nicprof · kvwire · PFC post | runnable |
| Linux kernel performance | kptk | runnable |
| Inference serving | kvwire · high-performance-llm-inference-engine · KV-cache post | early |
| GPU kernels (CUDA / Triton) | custom-cuda-fused-attention-triton | early |
| eBPF / XDP / DPDK | kernel-level-ai-traffic-shaper · packet path post · DPDK essay | early / write-up |
| Virtualization (SR-IOV / KVM) | virtualization essay | write-up |
| Incident triage models | infra-incident-intel (calibrated classifier + grounded RCA, verifiable rewards) | contract + verifiers |
| ML for infra ops | 6 RCA & failure-prediction model cards (NCCL, GPU, network, Linux, Kubernetes logs) | model cards (no weights yet) |
- nicprof: which NIC counter explains which millisecond of training step time. Aligns RDMA/ethtool/SR-IOV counters to each rank’s steps and traces PFC pause back to its origin, so a victim port isn’t blamed.
nicprof demo· 69 tests - kptk: explains why a Linux workload is slow from kernel evidence: run-queue delay, NUMA locality, THP fallback, and
perf_event_open(2)called directly. Standard library only · 43 tests
Open to GPU cluster networking, distributed-training infrastructure and inference-platform roles
USA · Canada · Europe