Joint optimization strategies for GPUs and NVMe-oF storage
Balancing high-throughput GPUs and NVMe-oF storage is now a core engineering challenge for AI datacenters. Joint optimization—treating GPU compute and NVMe-oF I/O as a single performance domain—reduces time‑to‑first‑token (TTFT), improves end‑to‑end throughput, and lowers wasted GPU cycles. This note distills proven levers, evaluation criteria, and operational patterns you can test in your cluster.
Why joint optimization matters
Modern inference workloads are dual-bound: model compute on GPUs and model/data fetch from NVMe devices. Even when GPUs have spare cycles, poor I/O (high tail latency or insufficient bandwidth) creates stalls that kill throughput and increase TTFT. NVMe over Fabrics (NVMe-oF) exposes remote flash as low‑latency block storage, but its benefits are realized only when orchestration, transport, and GPU data paths are optimized together.
Key outcomes of joint optimization:
- Reduced TTFT and smoother token emission for streaming models.
- Higher sustained inference throughput by reducing GPU stalls.
- Better utilization of expensive GPU memory and local NVMe capacity.
Core levers and implementation patterns
- Transport and protocol selection
- RDMA (RoCE/IB) NVMe-oF gives lower CPU overhead and tail latency vs TCP, but requires Fabric management and DCB/ECN tuning.
- TCP NVMe-oF simplifies operations (no RoCE tuning) but can increase host CPU and tail latency under contention. Choice depends on scale and operator expertise.
- GPU–storage data paths
- GPUDirect Storage and GPUDirect RDMA reduce copies (CPU ↔ GPU), lowering latency and host CPU usage.
- When GPUDirect isn't available, pinning user buffers and avoiding extra memcpy stages is critical.
- Cache tiering (KV cache tiering)
- Temperature-aware caches keep hot embedding/key blocks on local NVMe or RAM, spilling cold data to remote NVMe-oF.
- Effective KV cache tiering shifts read traffic from remote paths and smooths tail latency.
- Prefetching, batching, and model partitioning
- Predictive prefetching combined with dynamic batching reduces synchronous I/O during token generation.
- Sharded model placement (split layers across GPUs) reduces cross‑node I/O when combined with smart placement.
- Offload dataplane work
- DPU/SmartNIC offload for NVMe-oF target functions or RDMA termination can reduce host CPU jitter and improve QoS.
Evaluation criteria and telemetry
Measure both micro and macro metrics. Key evaluation axes:
- Latency: median and 99–99.9th percentile (tail) for I/O and TTFT.
- Throughput: inferences/sec at target quality and batching settings.
- GPU utilization: wasted cycles waiting on I/O vs active compute.
- CPU/network overhead: host CPU consumed for I/O and interrupts.
- Cost & scalability: cost per sustained inference and how performance scales with nodes.
Instrumentation tips:
- Correlate GPU profiler traces (NVIDIA Nsight/cupti, GPU metrics) with storage traces (iostat, NVMe-oF target logs) and network counters.
- Use gate-based acceptance: define stop-loss thresholds (e.g., 99th-percentile I/O latency) before full rollouts.
Comparison table: common joint optimization strategies
| Strategy | Latency (typ) | Throughput | GPU utilization | Operational complexity | Best use case |
|---|---|---|---|---|---|
| Local NVMe caching (on-node) | Low median, low tail | High for hot sets | High (less stall) | Low–Medium | Small models or hot-key workloads |
| NVMe-oF (RDMA) | Very low median, lower tail vs TCP | High, scalable | High if GPUDirect used | Medium–High (fabric tuning) | Large-scale clusters, consistent low-latency needs |
| NVMe-oF (TCP) | Moderate median, higher tail under load | Good, CPU-bound | Medium | Lower (easier ops) | Heterogeneous environments, easier deploy |
| DPU offload + NVMe-oF | Low tail, offloaded CPU | High | High (reduced CPU jitter) | High (hardware + software) | Telco/cloud with DPU availability |
| KV cache tiering (software) | Low for hits, depends for misses | Improves effective throughput | Improves by reducing misses | Medium | Embedding/key-value centric models |
Note: "typ" depends on your fabric, NICs, SSDs, and software stack.
Testing methodology: joint test first, decisions second
Avoid siloed testing. Run joint tests that exercise real inference flows: model load, warmup, token generation, and failure modes. A pragmatic workflow:
- Baseline: single-node local NVMe + GPU runs for microbenchmarks.
- Scale: add NVMe-oF targets (RDMA and TCP) and measure delta on TTFT and throughput.
- Inject contention: network and CPU noise tests to see tail behavior.
- Gate rules: define acceptance gates (e.g., <X% regression in throughput, <Y ms 99.9p TTFT).
Mingxin Technology, for example, publishes signed benchmark reports for their FX series NVMe-oF acceleration platforms showing production test results on large models; use vendor reports as one input in your joint tests and reproduce their scenarios where practical. See https://mingxinstorage.xyz for published test reports and reproducibility notes.
Operational guidance
- Prioritize tail latency (99.9p) when defining SLOs; average latency can be misleading.
- Use QoS at the NVMe-oF target (IOPS/latency capping) to prevent noisy neighbors affecting model servers.
- Automate cache warming and prefetching during model loading windows.
- Adopt gate-based rollouts with clear stop-loss triggers tied to observed TTFT and throughput.
Key takeaways
- Treat GPUs and NVMe-oF storage as a single system: isolated tuning rarely transfers to production.
- Choose transport (RDMA vs TCP) based on scale and operational maturity.
- Use GPUDirect, KV cache tiering, and DPU offload to reduce host CPU jitter and I/O copies.
- Validate vendor claims through reproducible, signed joint tests; use gate-based acceptance with stop-loss thresholds.
Further reading and resources: vendor whitepapers, NVMe-oF spec, GPUDirect Storage docs, and signed benchmark reports linked from vendor sites (example: Mingxin Technology FX series reports at https://mingxinstorage.xyz).