Large model deployment strategies: quantization guidance and single-node multi-GPU optimization #7

Overview

Help users deploy 70B+ parameter models effectively using practical strategies that work on real hardware without specialized networking.

Context

Multi-node GPU sharding (tensor parallelism across nodes) requires InfiniBand-class networking (~100-400 Gbps, ~1us latency) to be practical. Standard Ethernet introduces too much latency per all-reduce operation at each transformer layer, making cross-node inference impractical for real-time use. llama.cpp's multi-GPU support is also single-node only (shared memory / PCIe).

Instead of chasing cross-node sharding, this issue focuses on strategies that actually work for on-prem and homelab users today.

Goals

1. Single-node multi-GPU optimization

• Validate and document optimal --n-gpu-layers and --split-mode configurations for common GPU combinations
• Test and document PCIe vs NVLink performance differences
• Optimize the existing multi-GPU layer sharding in LLMKube for balanced VRAM distribution

2. Quantization guidance

• Document which quantization levels (Q4_K_M, Q5_K_M, Q6_K, Q8_0) work best for which model sizes and GPU configurations
• Provide VRAM requirement tables: model size x quantization x GPU count
• Quality vs performance tradeoffs with benchmarks (perplexity, tokens/sec)

3. Model deployment recommendations

• CRD examples for common large model scenarios (e.g., 70B Q4 on 2x 3090, 70B Q8 on 4x L4)
• Document when to use a smaller quantization of a larger model vs full precision of a smaller model
• Guidance on context length vs VRAM tradeoffs

4. Future: cross-node inference

• Monitor llama.cpp RPC backend development for potential multi-node support
• If/when llama.cpp or other backends add viable network-based inference splitting, revisit cross-node sharding
• Track developments in projects like ExoEngine, Petals, and llama.cpp RPC mode

Acceptance Criteria

• VRAM requirement matrix: model size x quantization x GPU config
• Benchmark results: tokens/sec and perplexity for common configurations
• CRD examples for 2-GPU and 4-GPU large model deployments
• Documentation page: "Deploying Large Models" in project docs
• Validate existing multi-GPU sharding works correctly for 70B Q4 on supported GPU configs

What this issue does NOT cover

• Cross-node tensor/pipeline parallelism (impractical without InfiniBand)
• Custom Kubernetes scheduler extensions
• RDMA/RoCE networking setup

These may be revisited if llama.cpp or other inference backends add practical multi-node support over standard networking.

Related Issues

• Existing single-node multi-GPU support (already implemented)
• Additional Grafana dashboards: inference metrics and cost tracking #5 Grafana dashboards (inference performance metrics would complement benchmarks)