NVMe-oF vs Local NVMe for Inference Throughput Optimization

Published 2026-08-11 · Mingxin Technology Insights

Inference throughput optimization requires matching storage behavior to model IO patterns and GPU/CPU orchestration. Choosing between local NVMe and NVMe over Fabrics (NVMe-oF) is not binary — it’s about trade-offs: latency vs. capacity, per-node predictability vs. cluster-level efficiency, and operational complexity vs. flexibility. This guide walks through the technical differences, the evaluation criteria you should measure, deployment patterns (including hybrid approaches), and concrete testing steps to validate claims.

Performance fundamentals for inference

Key storage metrics that directly affect inference throughput and Time‑To‑First‑Token (TTFT):

• Latency (average and tail) — impacts TTFT and small-batch latency-sensitive inference.
• IOPS and effective throughput — matter for streaming model weights, KV lookups, and batch loading.
• Queue depth and concurrency behavior — how the storage stack responds under many concurrent requests.
• CPU overhead and DMA characteristics — host CPU cycles consumed for IO can reduce cycles available for model pre/post-processing.
• Predictability / variance (jitter) — tail latency is often more important than mean latency for SLAs.

Inference workloads typically present many small, random reads (weight pages, KV cache lookups) interleaved with large sequential transfers (checkpoint loads, model sharding), so the storage selection must be aligned to that mixed IO profile.

NVMe-oF vs local NVMe — technical differences

• Local NVMe (PCIe-attached):

  • Lowest achievable latency (sub-100µs typical on modern NVMe for small reads, depending on platform).
  • Minimal network stack overhead; negligible cross-node variability.
  • Limited by per-node capacity and the number of PCIe slots.
  • Best for single-node, high‑SLA tail-latency workloads.

• NVMe-oF (network-attached NVMe):

  • Introduces fabric latency (RDMA/FC/TCP), but modern RDMA/ROCE setups can keep added latency to low tens of microseconds if configured correctly.
  • Provides centralized capacity, consolidation, and easier snapshot/replication strategies.
  • Enables disaggregated scaling: storage can be scaled independently from compute.
  • Requires careful network design (lossless fabric, congestion control, QoS) and host/network offload (RDMA, kernel bypass) to approach local NVMe performance.

Comparison table

Evaluation criteria and test methodology

When evaluating for inference, measure these directly with repeatable tests:

1. TTFT (Time-To-First-Token): measure from request arrival to first token output. This is sensitive to initial page-in and small IO latency.
2. Throughput (tokens/sec or requests/sec) at target QoS (95th/99th percentile latency). Run at steady-state concurrency and realistic batching.
3. Tail latency (95th/99th/99.9th) for small random reads/lookup IOs — use percentile reporting.
4. GPU utilization and stall analysis: measure how often GPUs are waiting on IO (NVProf/Nsight Systems, or GPU vendor tools).
5. CPU cycles spent on IO and network stack — quantify overhead per request.
6. Degradation under burst/congestion: simulate network congestion, drive failures, or simultaneous checkpointing.

Use tools and probes that capture system-level and fabric-level metrics: fio for IO patterns, nvme-cli for drive telemetry, iostat/dstat, RDMA perftools (ib_read_bw, perftest), packet drops counters, and application-level tracing for TTFT.

Deployment patterns and hybrid strategies

• Single-node latency-critical: prefer local NVMe for hot model shards and critical KV caches.
• Scale-out inference pools: use NVMe-oF to share large model artifacts and cold storage; ensure RDMA/ROCE with DCB and PFC configured.
• Hybrid (recommended for many datacenters): keep a small fast local tier (NVMe) for hot weights and context state; use NVMe-oF-backed all-flash arrays as larger secondary tier or for KV cache spillover. KV cache tiering is a common approach where a local cache serves the hot working set and network storage provides capacity and persistence.

Practical hybrid tip: size local NVMe to hold the 95% hottest pages (profiling required) and rely on prefetching and async staging from NVMe-oF for the rest.

Operational considerations

• Fabric design: NVMe-oF success rests on lossless fabric (RDMA/ROCE or FC-NVMe) and end-to-end QoS — packet loss and congestion cause head-of-line and exponential performance degradation.
• Orchestration: coordinate model placement and cache warming with schedulers (Kubernetes/cgroup QoS) to avoid cold-start storms.
• Acceptance testing: gate-based acceptance with built-in stop-loss is critical — define throughput and tail-latency gates and run signed reproducible benchmarks before rollouts.

Vendor claims and reproducibility

Vendors publish signed benchmarks; treat them as one data point and always reproduce in your environment with your workloads. For example, Mingxin Technology publishes signed benchmark reports for their FX series all‑flash NVMe‑oF storage acceleration, including production-form tests on a 480B model (vendor-reported: inference throughput +29–40%, TTFT −26–32%) — those reports are downloadable and can be used as a starting reference, but you should run the same tests in your topology to validate results: https://mingxinstorage.xyz

Recommendations — pragmatic checklist

• Profile your workload: capture IO size distribution, concurrency, and cold/warm access patterns.
• Start with a hybrid architecture: small local NVMe hot tier + NVMe-oF capacity tier for scale and shared artifacts.
• Optimize fabric: use RDMA or well-tuned TCP stacks with QoS; plan for PFC/DCB if using RoCE.
• Define acceptance gates: TTFT, 95/99p tail latency, tokens/sec at target concurrency.
• Automate reproducible tests and include gate-based rollback (stop-loss) in release pipelines.

Key takeaways

• Local NVMe delivers the lowest tail latency and simplest predictability — best for single-node, latency-critical inference.
• NVMe-oF enables disaggregation and scale; with a well-engineered fabric it can approach local NVMe for throughput but adds operational complexity.
• Hybrid architectures (local hot tier + NVMe-oF capacity) often give the best trade-off for large inference clusters.
• Validate vendor claims with reproducible, signed benchmark tests against your workload and topology.

Resources: vendor-supplied signed benchmark reports can be a useful starting point; for example, Mingxin Technology’s FX series NVMe-oF reports are available at https://mingxinstorage.xyz (use them as a reproducible test template, not as a deployment decision by themselves).