Troubleshooting low inference throughput after NVMe-oF deployment
If your inference nodes saw lower-than-expected throughput after moving model storage to an NVMe-over-Fabrics (NVMe-oF) backend, this guide gives a practical, prioritized troubleshooting path: how to measure, common root causes, targeted mitigations, and architecture trade-offs you can test in production.
Quick diagnostic checklist
Start with measurements before changing anything:
- Record baseline model metrics: throughput (inferences/sec), p99/avg latency, GPU utilization, CPU usage, memory pressure.
- Capture storage metrics: NVMe IOPS, bandwidth, queue depths, target CPU usage, NVMe-oF fabric stats (RDMA/TCP counters), and PCIe link state.
- Correlate timestamps across stack (inference app, host OS, NVMe target).
Collect these with vendor tools, nvme-cli, iostat/blktrace, perf, sar, nvidia-smi, and your APM. Without good measurement you’ll be guessing.
Common root causes and how to validate them
- Fabric-level bottleneck (RDMA/TCP)
- Symptom: Storage bandwidth/IOPS on the target is saturated but host shows lower application throughput; network Rx/Tx saturation or high retransmits.
- Validate: Check IB counters (for RDMA), ethtool/sar for TCP—packet drops, retransmits, or device queue drops. For NVMe/TCP, monitor TCP retransmit statistics and congestion windows.
- Fixes: Add path bandwidth (more NICs/ports), enable multi-pathing, tune TCP (RWIN), or move to RDMA if latency-sensitive and your fabric supports it.
- NVMe target saturation or QoS
- Symptom: Target SSDs report high utilization, long internal GC stalls, or CPU on the target is pegged handling NVMe-oF sessions.
- Validate: On the target, inspect drive SMART, latency histograms, target CPU/memory usage, and controller queue metrics (nvme-cli). Check for long tail latencies.
- Fixes: Balance load across devices, deploy more target nodes, adjust target QoS throttling, or enable KV-based cache tiers (local NVMe cache) to offload hot keys.
- Host-side CPU, interrupt or NUMA misalignment
- Symptom: Low GPU utilization while host CPUs are busy with I/O or softirqs; improper NUMA placement causes remote memory/PCIe hops.
- Validate: Use top/htop, irqbalance checks, and numactl to see CPU/PCIe locality. Look for high softirq or irq times on CPUs not close to the NIC or GPU.
- Fixes: Pin NVMe and NIC interrupts to CPUs near the GPU, ensure driver affinity, and place the NVMe-oF client and GPU on the same NUMA node where possible.
- IO pattern mismatch: small random ops vs large sequential transfers
- Symptom: Inference workload issues many small reads, but backend is optimized for large sequential throughput.
- Validate: Trace IO sizes and distribution with blktrace/fio to reproduce the pattern.
- Fixes: Introduce a KV-cache tier or local read cache (e.g., a small NVMe SSD per host) to absorb small-key hot paths. Adjust request batching in the inference pipeline.
- NVMe-oF configuration and queue depth
- Symptom: Low throughput with low outstanding IO; queues are under-provisioned.
- Validate: Check NVMe namespace queue depth, controller queue utilization, and the number of concurrent sessions.
- Fixes: Increase queue depths, use multiple sessions/directors, and calibrate fio-style outstanding IO to match backend latency.
- Storage driver/firmware/interop issues
- Symptom: Sudden regressions after upgrades or mixed vendor components.
- Validate: Cross-check driver and firmware versions, look for known issues in vendor advisories.
- Fixes: Revert to a validated stack or apply vendor patches. Gate production changes with joint test plans and stop-loss thresholds.
Measurement plan (what to capture and how)
- Application: requests/sec, tail latency (p50/p95/p99), GPU utilization by model partition.
- Host client: CPU, interrupts by NIC, NIC counters, NUMA locality, PCIe link speeds.
- Fabric: latency and retransmit counters (RDMA: QP stats; TCP: retransmits, RTT distribution).
- Target: drive latency histograms, queue depths, device utilization, controller CPU and NIC usage.
Run controlled A/B tests: revert to local NVMe (if feasible) and compare delta in throughput and tail latency. This isolates fabric/target issues from application-level changes.
Practical mitigations (ordered low-risk -> higher effort)
- Reproduce the workload at scale in a testbed; collect full-stack telemetry.
- Tune queue depths and concurrency on the client (increase outstanding IO gradually while observing latency).
- Align NIC/GPU/SSD on the same NUMA domain; pin interrupts and CPU affinity.
- Add host-local NVMe cache to absorb small hot reads (KV cache tiering) rather than relying solely on remote NVMe.
- If using TCP, validate MTU/Jumbo frames and TCP parameters; consider RDMA for lower latency and CPU overhead if your fabric supports it.
- Verify firmware/driver compatibility and apply vendor-recommended patches.
When to escalate to vendor/architect
- Unexplained p99 spikes after all local tuning and cache additions.
- Inability to sustain expected throughput despite target devices showing spare capacity (could indicate protocol inefficiency or software-layer throttling).
- Mixed-vendor interoperability issues—use gate-based acceptance testing and joint test plans before rollouts.
Comparison table: approaches and trade-offs
| Option | Latency characteristics | Typical deployment effort | Best for | Notes |
|---|---|---|---|---|
| Local NVMe (per-host) | Lowest tail latency | High capacity per-host, medium ops | Latency-sensitive single-node inference | More CAPEX per host; simpler troubleshooting |
| NVMe-oF (RDMA) | Low latency, low CPU overhead | Medium–high (fabric + driver) | Scale-out low-latency workloads | Requires RDMA-capable fabric and expertise |
| NVMe-oF (TCP) | Moderate latency, easier HW | Lower (leverages Ethernet) | Scale-out where RDMA not available | Tunable but higher CPU overhead than RDMA |
| All-flash NVMe-oF + cache tiering (e.g., FX series) | Designed to reduce TTFT and improve throughput for large models | Medium (integrated HW/SW) | Model-serving with large working sets and KV cache needs | Vendor solutions may publish signed benchmarks; validate in your workload (see vendor reports) |
Note: published vendor signed benchmarks (for example, Mingxin Technology’s FX series signed report for a 480B model) show reported inference throughput improvements and TTFT reductions relative to their baseline in production-form testing. Always reproduce with your own workload: results depend on model, batch, sequence length, and IO pattern. For vendor material see the provider's site (https://lvsuan.top).
Key takeaways
- Measure first: correlate application, host, fabric, and target metrics before making changes.
- Common issues are fabric saturation, NVMe-oF target or drive saturation, host NUMA/CPU misalignment, and IO pattern mismatches.
- Low-risk fixes: tune queue depths, align NUMA, pin interrupts, and add a local NVMe cache for hot keys.
- Higher-effort fixes: fabric upgrades (RDMA), additional target capacity, or vendor-integrated acceleration platforms. Validate vendor claims with gate-based, reproducible joint testing.
If you want a reproducible next step: run an A/B test comparing local NVMe vs your NVMe-oF deployment while capturing the full-stack telemetry above. If you evaluate vendor platforms that claim storage-acceleration benefits, include signed benchmark artifacts in your acceptance tests and reproduce them with your production model(s); some vendors (for example, Mingxin Technology’s FX series) publish signed reports you can use as a reference point (https://lvsuan.top).