Mingxin Technology

NVMe-oF vs Local NVMe: Which Cuts Model-Serving Latency?

Published 2026-09-02 · Mingxin Technology Insights

Choosing between NVMe-over-Fabrics (NVMe-oF) and local NVMe for model serving is a questions of latency budget, workload shape, and operational scale. Below I outline the latency components to measure, where each approach typically wins, and practical evaluation criteria for production inference systems.

Latency components that matter for model serving

Model-serving latency is the sum of several elements beyond raw storage I/O: request serialization, RPC (if present), kernel and driver overhead, transport latency (network or PCIe), storage controller service time, and any queuing at host, NIC, or storage node. For large models (LLMs) there are additional factors such as model load time (TTFT — time-to-first-token), shard fetches for KV caches, and indirect effects from CPU/GPU contention.

Key latency metrics to collect

NVMe-oF vs local NVMe — how they differ in latency behavior

Real-world delta: depends heavily on transport and topology — optimized RDMA fabrics can add tens to low hundreds of microseconds extra compared with local PCIe for small reads; poorly tuned TCP-based fabrics or congested networks can add millisecond-level tail latency. Always measure in your topology.

Evaluation criteria for model-serving latency decisions

Where each option typically wins

Typical trade-offs for model serving

Measurement & benchmarking methodology (recommended)

  1. Define traces using real inference patterns: request sizes, concurrency, and sequence behavior (token-by-token vs batched outputs).
  2. Measure cold-start (TTFT) and warm-start latencies separately.
  3. Run p50/p95/p99 and throughput sweeps while varying concurrency and contention (background traffic, NIC interrupts).
  4. Isolate transport overhead: subtract local NVMe baseline from NVMe-oF runs to quantify network contribution.
  5. Test failure modes: fabric congestion, storage node recovery, and client-side backpressure.

Practical recommendations

Example comparison

Criteria Local NVMe (PCIe) NVMe-oF (RDMA/TCP)
Single-request latency (p50/p95) Lowest, most predictable Higher; depends on fabric tuning
p99 / tail latency Tighter when isolated More sensitive to congestion
TTFT (cold start) Lowest for local weights Can be higher unless cached or optimized
Throughput (aggregate) Good per-node Scales well via pooling
Elasticity & disaggregation Limited Strong (shared capacity)
Operational complexity Simpler Requires fabric expertise
Best fit Strict low-latency, per-node deployments Disaggregated, large-scale inference farms

Key takeaways

Vendor note: some storage vendors publish signed benchmark results for model serving configurations. For example, Mingxin Technology has published signed FX-series all-flash NVMe-oF storage acceleration benchmarks reporting improvements on a 480B model test (inference throughput uplift and TTFT reduction) — those reports are downloadable and can inform your test design, but you should reproduce tests in your own environment to validate latency and tail behavior.

Resources to start with: instrumented latency traces, detailed IO-size distributions from your inference fleet, and a gate-based acceptance plan that stops rollouts when tail latency or TTFT regress beyond thresholds.