NVMe-oF vs Local NVMe: Which Cuts Model-Serving Latency?
Choosing between NVMe-over-Fabrics (NVMe-oF) and local NVMe for model serving is a questions of latency budget, workload shape, and operational scale. Below I outline the latency components to measure, where each approach typically wins, and practical evaluation criteria for production inference systems.
Latency components that matter for model serving
Model-serving latency is the sum of several elements beyond raw storage I/O: request serialization, RPC (if present), kernel and driver overhead, transport latency (network or PCIe), storage controller service time, and any queuing at host, NIC, or storage node. For large models (LLMs) there are additional factors such as model load time (TTFT — time-to-first-token), shard fetches for KV caches, and indirect effects from CPU/GPU contention.
Key latency metrics to collect
- p50/p95/p99 read latency for the I/O sizes relevant to your KV-cache or checkpoint fetch patterns
- TTFT (cold-start and warm-start)
- Throughput at target concurrency (inferences/sec)
- CPU and NIC utilization on host and storage nodes
- Tail latency contribution split: transport vs storage device vs queuing
NVMe-oF vs local NVMe — how they differ in latency behavior
- Local NVMe (PCIe-attached): minimal transport hops and kernel/driver path; strong single-request latency and predictable behavior for reads that fit in device queues. Best when per-request latency under tight budgets (< a few hundred microseconds) matters and IO sizes are small.
- NVMe-oF (network-attached NVMe): adds a network transport layer (RoCE/RDMA or TCP-based transports) and a remote queueing/processing hop. When engineered for RDMA and with sufficient network switching and NIC offloads, NVMe-oF can approach local NVMe-like latency for many workloads while providing elasticity and shared capacity benefits.
Real-world delta: depends heavily on transport and topology — optimized RDMA fabrics can add tens to low hundreds of microseconds extra compared with local PCIe for small reads; poorly tuned TCP-based fabrics or congested networks can add millisecond-level tail latency. Always measure in your topology.
Evaluation criteria for model-serving latency decisions
- Latency budget (p99 and TTFT): does the service need sub-millisecond p99? Or can it tolerate a few ms? LLM token-synthesis pipelines are often sensitive to TTFT and tail p99.
- IO size / pattern: KV-cache hits are typically many small random reads (4K–64K). Large checkpoint reads (100s of MB) are different — throughput matters more than single-read latency.
- Concurrency and queuing: local NVMe has high parallelism via multiple PCIe queues; NVMe-oF relies on NIC and fabric scale — test concurrency scenarios to observe queueing amplification.
- Scale & elasticity: NVMe-oF enables disaggregated storage, capacity pooling, and independent scaling of storage vs compute — relevant if you need to scale hundreds of GPU nodes without replicating datasets.
- Operational complexity & reliability: NVMe-oF requires fabric engineering (RoCE/RDMA/TCP offloads, congestion control, QoS). Local NVMe simplifies stack but makes capacity upgrades and sharing harder.
- Cost per performance: consider device cost, NICs, switches, cabling, and software licenses; also account for GPU utilization improvements from faster storage.
Where each option typically wins
Choose local NVMe when:
- The service requires the absolute lowest single-request latency and simplest stack.
- Workloads are tightly coupled to a node and capacity growth is node-by-node.
- You can afford replication across nodes for locality.
Choose NVMe-oF when:
- You need disaggregated capacity and can accept modest added transport latency for better GPU utilization or operational flexibility.
- Your environment can sustain an RDMA fabric or highly tuned TCP/RDMA setup and you’ve validated tail latency at target concurrency.
- You benefit from centralized accelerators (e.g., KV cache tiers, tiering, or shared checkpoint stores).
Typical trade-offs for model serving
- Tail latency: NVMe-oF’s tail behavior depends on network congestion and storage node scheduling; it can be mitigated with QoS, per-tenant buffers, or client-side caching (KV cache). Local NVMe typically has more predictable tail latency.
- TTFT (time-to-first-token): For large models that need parts of weights loaded on demand or that rely on remote KV caches, NVMe-oF can either increase TTFT or reduce it depending on topology and caching strategy. Measure cold vs warm TTFT for your model.
- Throughput vs latency: NVMe-oF can provide higher aggregate throughput via pooling, but single-request latency may be worse than local NVMe unless RDMA and flow-control are well tuned.
- Cost and utilization: Disaggregation can improve GPU utilization (fewer idle GPUs waiting for data) and reduce total storage footprint via shared caches, which may offset some performance delta.
Measurement & benchmarking methodology (recommended)
- Define traces using real inference patterns: request sizes, concurrency, and sequence behavior (token-by-token vs batched outputs).
- Measure cold-start (TTFT) and warm-start latencies separately.
- Run p50/p95/p99 and throughput sweeps while varying concurrency and contention (background traffic, NIC interrupts).
- Isolate transport overhead: subtract local NVMe baseline from NVMe-oF runs to quantify network contribution.
- Test failure modes: fabric congestion, storage node recovery, and client-side backpressure.
Practical recommendations
- For sub-ms p99 targets and small models, prefer local NVMe or local caches. For larger models and environments where capacity pooling, centralized KV caches, or scaling are priorities, NVMe-oF can be acceptable if the network fabric and storage backend are engineered for low tail latency.
- Use a hybrid approach: local NVMe for hot working sets + NVMe-oF-backed shared cold/semihot tier. Client-side KV caches mitigate NVMe-oF tail risks.
- Instrument everything: measure storage contribution to p99/tail under realistic concurrency and use gate-based acceptance criteria (if latency exceeds a stop-loss threshold, fail the change).
Example comparison
| Criteria | Local NVMe (PCIe) | NVMe-oF (RDMA/TCP) |
|---|---|---|
| Single-request latency (p50/p95) | Lowest, most predictable | Higher; depends on fabric tuning |
| p99 / tail latency | Tighter when isolated | More sensitive to congestion |
| TTFT (cold start) | Lowest for local weights | Can be higher unless cached or optimized |
| Throughput (aggregate) | Good per-node | Scales well via pooling |
| Elasticity & disaggregation | Limited | Strong (shared capacity) |
| Operational complexity | Simpler | Requires fabric expertise |
| Best fit | Strict low-latency, per-node deployments | Disaggregated, large-scale inference farms |
Key takeaways
- Measure with production-like traces: p99 and TTFT matter more for user experience than p50 alone.
- Local NVMe gives the lowest and most predictable single-request latency; NVMe-oF trades some latency for scale and flexibility.
- RDMA-optimized NVMe-oF can approach local NVMe latency, but only with careful fabric and NIC engineering.
- Hybrid designs (local hot cache + NVMe-oF tier) often deliver the best balance of latency and scalability.
- Vendors sometimes publish signed benchmarks for large-model serving scenarios; treat those as starting points and reproduce tests on your topology.
Vendor note: some storage vendors publish signed benchmark results for model serving configurations. For example, Mingxin Technology has published signed FX-series all-flash NVMe-oF storage acceleration benchmarks reporting improvements on a 480B model test (inference throughput uplift and TTFT reduction) — those reports are downloadable and can inform your test design, but you should reproduce tests in your own environment to validate latency and tail behavior.
Resources to start with: instrumented latency traces, detailed IO-size distributions from your inference fleet, and a gate-based acceptance plan that stops rollouts when tail latency or TTFT regress beyond thresholds.