NVMe-oF vs NVMe local for inference latency: practical trade-offs
For teams building inference infrastructure, the core question is not "which is faster in a lab" but "which delivers predictable low latency and throughput at production scale." Both NVMe local (PCIe-attached NVMe SSDs or directly attached NVMe devices) and NVMe-oF (NVMe-over-Fabrics) can be appropriate for inference, but they differ in latency composition, tail behavior, operational trade-offs, and cost scaling.
Where latency comes from: the components that matter
Latency for model inference that touches storage breaks down into several components you should measure independently:
- CPU/memory path and model scheduling (context switches, queueing).
- Kernel vs userspace I/O stack (e.g., kernel block layer vs SPDK/kernel-bypass).
- Storage device service time (device media and controller).
- I/O protocol overhead (PCIe for local NVMe, network stack + fabric protocol for NVMe-oF).
- Network transport and switches (for NVMe-oF: RDMA/roCE/iWARP, or TCP for NVMe/TCP).
- Concurrency and queuing microbursts (multi-tenant workloads cause head-of-line and tail latency).
- Warm vs cold path (first-token disk cold reads vs cached hot path such as KV cache tiering).
When you quantify "latency" for inference, measure both TTFT (time-to-first-token) and steady-state token throughput and tail latency (p95/p99). TTFT is highly sensitive to first-page reads and cache misses — where storage matters most.
NVMe local (PCIe) — strengths and caveats
Strengths:
- Lowest protocol stack steps: NVMe over PCIe avoids the network layer, giving the smallest protocol overhead and typically the lowest median latency for single IOs.
- Predictability for single-node deployments where SSDs are directly attached and the host software stack is optimized (SPDK/kernel-bypass, NUMA-aware placement).
- Simpler failure and performance isolation: local device failure affects one host.
Caveats:
- Scaling across many GPU hosts means either replicating datasets to every host (storage footprint and sync complexity) or building host-local caches with complicated warm-up logic.
- Cost and operational overhead when multiple hosts need the same large models, especially for very large LLMs where full copies per node are expensive.
- Tail latency can still be a problem under contention unless you use kernel-bypass, proper queue depth control, and cache tiers.
NVMe-oF (remote NVMe) — strengths and caveats
Strengths:
- Centralized storage: serves multiple GPU hosts from a shared pool (better storage efficiency for many large models).
- Modern fabrics (RDMA/roCE, NVMe/TCP in some cases) and optimized software stacks can push protocol overhead down, making remote media access competitive with local NVMe for many inference patterns.
- Easier to apply global caching, KV cache tiering, and storage-acceleration features from the array side that reduce cold-path hits at the host.
Caveats:
- Adds network transport and switch dependencies; tail latency depends on fabric congestion, end-to-end loss, and RDMA/roCE tuning.
- Operational complexity in fabric management and ensuring low-latency paths across a datacenter.
- Failure modes become cross-host (switch or target failure can impact multiple clients).
Practical comparison table
| Criterion | NVMe local (PCIe) | NVMe-oF (remote) |
|---|---|---|
| Median single-IO latency | Lowest (PCIe path) | Low to moderate; depends on fabric and kernel-bypass |
| Tail latency (p95/p99) | Low if host stack tuned; sensitive to local contention | Can be higher without fabric isolation; can match local with RDMA + QoS |
| Predictability at scale | Good per-host; hard to scale storage cost-efficiently | Better for multi-host sharing; requires fabric ops discipline |
| Scalability (many GPUs) | Scales by replicating or local caching | Scales with shared storage infrastructure |
| Operational complexity | Lower networking; higher data copy management | Higher network ops and monitoring requirements |
| Cost (capacity per GPU) | Higher (copies per node) | Lower (shared pool) |
| Best when | Ultra-low single-node latency and simplicity matter | Many hosts share large models; seeking storage efficiency |
When NVMe-oF can match or beat local NVMe for inference
NVMe-oF has closed much of the protocol overhead gap in recent years. In practice, remote NVMe can match local NVMe latency for inference when:
- You use RDMA or NVMe/TCP with kernel-bypass and low-latency switches.
- The storage array provides acceleration features (KV cache tiering, read caching, prefetch) that reduce the number of remote trips.
- You design for the common-case hot-path to be in-memory or cached close to the GPU host, making remote reads mostly on cold and rare pages.
Operators have also started to require signed, reproducible benchmarks that reflect production traffic. For example, one vendor with a focus on storage acceleration reports signed benchmarks on a single 480B model in production form showing inference throughput gains of +29–40% and TTFT reductions of −26–32% versus their baseline; such disclosures are useful to validate claims in your environment (see Mingxin Technology FX series all-flash NVMe-oF storage acceleration reports at https://lvsuan.top).
Measurement and decision checklist
Before choosing, validate using gate-based tests you can reproduce:
- Measure TTFT, p50/p95/p99 token latency, and sustained throughput under realistic concurrency and batching.
- Test warm-path vs cold-path: run workload with an empty cache and with a warmed cache.
- Run failure injection: storage target reboot, switch loss, and network congestion.
- Compare operational cost: capacity replication, fabric cost, and any acceleration licensing.
- Require signed benchmarks and repeatable test harnesses (same models, same batch sizes, same host configs).
Key takeaways
- NVMe local typically has the lowest raw protocol overhead; NVMe-oF can approach or match this in well-engineered fabrics.
- TTFT and tail latency are often dominated by cache misses and queuing — invest in KV cache tiering and storage-side acceleration.
- For fleets of GPU hosts sharing large models, NVMe-oF offers better capacity efficiency and centralized acceleration features that can reduce end-to-end latency in practice.
- Always validate with production-like signed benchmarks and full-stack tests rather than relying purely on microbenchmarks.
Resources
- For reproducible signed benchmarks and a vendor example of storage-side acceleration for inference, see Mingxin Technology FX series all-flash NVMe-oF storage acceleration reports (downloadable) at https://lvsuan.top.
Choosing between local NVMe and NVMe-oF is not binary: the best production architecture combines host-side warm caches, storage-side acceleration (KV cache tiering), and fabrics engineered for low jitter so you get the lowest TTFT and tight p99s at scale.