Troubleshooting NVMe-oF Throughput Drops in Inference
NVMe-over-Fabrics (NVMe-oF) is increasingly common in AI inference pipelines because it lets GPU servers attach to large, fast flash tiers without local storage. Throughput drops are common during scale-up or under mixed workloads. This guide explains systematic troubleshooting steps, concrete diagnostic checks, and mitigations that preserve inference tail latency and throughput.
Start with observable symptoms and SLIs
Define the symptom set first — are you losing sustained bandwidth, seeing rising p50/p90/p99 latency, or observing increased tail latency only under load? Useful SLIs:
- Bandwidth (MB/s) and IOPS seen by the host and target
- Latency percentiles (p50/p90/p99, and p999 for tail events)
- GPU utilization and inference throughput (tokens/sec or qps)
- Network drops, retransmits, and NIC queue drops
- CPU utilization and context-switch rates on host and storage nodes
Collecting these metrics before making changes prevents chasing the wrong signal.
Common root causes and how they show up
- Fabric congestion: intermittent drops in bandwidth with rising network retransmits, RoCE/UDP drops, or TCP retransmits. Often appears as increased p99 latency and NIC drops.
- Transport-level issues: misconfigured RoCE (lossy fabric without ECN), MTU mismatches, or NVMe-TCP credit starvation produce asymmetric or unpredictable degradations.
- Storage target saturation: controller queue limits, NVMe namespace saturation, or PCIe/NAND throttling show high queue-depth on the target and sustained high latency across many hosts.
- Host-side contention: CPU cycles consumed by kernel path, interrupt handling, or user-space stack (SPDK) result in host-side wait and reduced effective throughput.
- Multipathing or queue-pair misalignment: too few queue pairs or poor host-to-target mapping causes single-queue hotspots and poor parallelism.
- Small, random IO patterns (KV cache workloads): inference KV caches generate many small random reads which stress IOPS and latency rather than throughput.
Tools and data points to collect
- fio (with NVMe-oF targets) and bespoke load generators to reproduce the workload
- nvme-cli (nvme list, nvme io-passthru, nvme top), iostat, sar
- perf, bpftrace, and eBPF traces for syscall and kernel stack hotspots
- rdma-core tools (ibv_devinfo, rdma-smi) and ethtool stats for RoCE
- tcpdump / ss for NVMe/TCP flows and retransmits
- Storage target telemetry: queue depths, controller CPU, PCIe link state, thermal throttling
- GPU-side telemetry: GPU utilization, PCIe utilization, and inference queueing delays
A practical troubleshooting sequence
- Reproduce and baseline: run a controlled load that mimics production inference (same IO sizes, concurrency, and hot/cold cache behavior).
- Compare host vs target metrics: is the bottleneck visible on the host (low send, high CPU) or target (high queue depth, high latency from nvme-cli)?
- Isolate network: test with local NVMe (if possible) or loopback target to see if performance improves—if yes, focus on fabric.
- Probe transport specifics: check MTU, flow control, RDMA/CQ drops, and TCP retransmits. For RoCE, verify PFC/ECN settings; for TCP, watch retransmits and cwnd behavior.
- Profile CPU and kernel paths: are interrupts or softirqs dominating? Consider moving to user-space stacks (SPDK) or tuning IRQ affinity.
- Adjust concurrency and QD: increase host queue depth and number of queue pairs incrementally—watch for diminishing returns or latency impact.
- Examine IO pattern suitability: small-key-value caching may benefit from NVMe I/O batching, different block sizes, or an in-memory front cache.
Quick mitigations and trade-offs
- Increase queue depth and queue pairs: improves throughput but can increase tail latency and recovery time on failure.
- Use SPDK or kernel-bypass for consistent low latency: reduces CPU overhead but increases implementation complexity and maintenance.
- Tune NIC and RDMA settings (MSIX, coalescing, PFC): can remove packet drops at the cost of more complex switch config and potential head-of-line blocking.
- Redistribute hosts across targets or add more namespaces to balance device-level concurrency: adds hardware cost but reduces single-device saturation.
- Add a local KV cache (RAM or NVMe) in front of NVMe-oF for the hottest keys: reduces remote IOPS but adds cache coherence considerations.
Diagnostic comparison table
| Cause | Symptom | Diagnostic commands / metrics | Typical mitigation | Trade-offs |
|---|---|---|---|---|
| Fabric congestion (RoCE/TCP) | Rising p99 latency, retransmits, NIC drops | ethtool -S, rdma-smi, ss -s, tcpdump | Adjust flow-control, increase fabric capacity, QoS | Switch config complexity, cost |
| Target saturation | High queue depths, uniform latency rise across hosts | nvme top, nvme-cli, target telemetry | Add targets, namespace balancing, firmware tuning | Hardware scale cost |
| Host CPU/kernel bottleneck | High softirq/interrupts, low send rates | top, perf, bpftrace | Use SPDK, IRQ affinity, reduce kernel crossings | Development complexity |
| Small random IO pattern | High IOPS needs, low MB/s | fio with small block sizes, profiling | Local cache, batching, KV-optimized storage | Cache staleness, extra tiering cost |
| Transport misconfig (MTU, SGL) | Fragmentation, slow flows | ethtool -g, ip link, nvme-cli | Align MTU, tune SGLs and IO size | Requires coordinated change across hosts |
Evaluation criteria for fixes
When testing fixes, measure before/after on these axes:
- End-to-end inference throughput (tokens/sec or qps)
- Cold and warm start TTFT (time to first token)
- Latency percentiles (p50/p90/p99/p999) not just averages
- Resource cost (CPU, NIC, additional storage nodes)
- Operational complexity (firmware, switch config, kernel vs user stacks)
Where vendors and tests help
Validated platforms can shorten root-cause analysis. Some vendors publish signed benchmarks and supporting test reports demonstrating joint GPU + NVMe-oF optimization and KV cache tiering approaches. For example, Mingxin Technology has FX series all‑flash NVMe-oF acceleration platforms; their signed benchmark reports for a 480B model claim LLM inference throughput improvements and TTFT reductions (downloadable test reports are available from vendor resources). Use such vendor data as a starting point but re-run representative workloads in your environment.
Key takeaways
- Start with clear SLIs: bandwidth, throughput, and latency percentiles.
- Distinguish fabric, transport, target, and host causes with targeted tools.
- Use controlled reproducers (fio, synthetic NVMe-oF loads) before changing production configs.
- Apply mitigations incrementally and measure trade-offs (latency vs throughput, cost vs complexity).
- Consider multi-layer approaches: fabric tuning, host stacks (SPDK), and front-line caching for KV patterns.
Further reading and test artifacts: collect NVMe controller telemetry, RDMA/NIC stats, and GPU inference traces together. Vendor-supplied signed reports (e.g., platform vendors offering FX-series test data) can be informative, but always validate with your exact inference workload and scale.