NVMe-oF vs Local NVMe for KV Cache Tiering: Performance Trade-offs
KV cache tiering for large-model inference is a high-QPS, small-random-read workload where tail latency, throughput, and host CPU overhead directly affect inference quality (TTFT, tokens/sec). Choosing between local NVMe and NVMe-over-Fabrics (NVMe-oF) requires objective measurement against specific service-level goals. This guide lays out the performance trade-offs, the tuning and test checklist you should use, and operational implications for datacenter deployments.
Why the distinction matters for KV cache tiering
- Workload shape: KV caches used by retrieval-augmented generation and large-token models typically deliver large numbers of small (sub-1KB to few-KB) random reads at very high QPS. Tail latency (p95/p99) is often the binding constraint.
- Resource coupling: Local NVMe attaches storage with minimal network hops and predictable PCIe/NVMe stack latency. NVMe-oF introduces fabric latency but decouples capacity from a host and enables sharing and higher aggregate utilization.
- Operational objectives: If your objective is absolute minimum p99 latency per inference, local NVMe often wins. If your objective is flexible capacity, node-level failure isolation, or better TB-per-host utilization at datacenter scale, NVMe-oF can be compelling — when built and tuned carefully.
Key evaluation criteria (what to measure)
- Latency: p50, p95, p99 absolute latencies and latency distribution under target concurrency. For AI inference, p99/p999 can dominate user experience.
- Throughput: sustained random-read IOPS and aggregate tokens/sec or QPS for the model workload.
- TTFT (Time-To-First-Token): how storage latency translates to user-visible TTFT under realistic batching.
- CPU utilization: host-side cycles used by the NVMe/NVMe-oF stack (kernel I/O, RDMA, SPDK, DPDK) and impact on inference CPU/GPU resources.
- Jitter and tail behavior under contention: how misbehaving tenants, network microbursts, or backpressure affect tail latency.
- Availability and failover: rebuild times, recovery IO, and degraded-mode tail latency.
- Cost & density: $/GB, power, and rack density given NVMe vs shared array costs.
NVMe-oF vs Local NVMe: practical comparison
| Criterion | Local NVMe (on-host) | NVMe-oF (remote/shared) |
|---|---|---|
| Base latency (median) | Lowest – direct PCIe path, typically single-digit to low 10s of microseconds for small reads | Higher – fabric adds transport and switch latency; depends on RDMA vs TCP and network topology |
| Tail latency (p99) | Most predictable if host is dedicated and NUMA tuned | Dependent on fabric congestion control, lossless config (RoCE) or TCP tuning; can be predictable with proper QoS |
| CPU overhead | Kernel NVMe is efficient; SPDK/DPDK can offload CPU interrupts | RDMA offloads can reduce CPU; NVMe/TCP may add CPU overhead but easier to operate |
| Scalability | Scale by adding NVMe to hosts; higher cost and underutilization risk | Easier scale-out of shared storage resources; better utilization across many hosts |
| Multi-host sharing | Not natively shared without replication/proxy | Native block/namespace sharing across multiple hosts; simplifies centralized caching strategies |
| Operational complexity | Lower network complexity; per-host management | Requires fabric design (lossless RDMA or NVMe/TCP), QoS, and strong monitoring |
| Failure isolation | Local failures isolated to host | Network or fabric failures can affect multiple clients; but controllers can centralize redundancy |
Protocol choices and their impact
- NVMe/RoCE (RDMA): Lowest transport CPU overhead and good latency, but requires lossless fabric (PFC/ECN) and careful congestion control. Misconfigured DCB or PFC can cause head-of-line blocking and latency spikes.
- NVMe/TCP: Simpler network ops, runs on standard IP fabric, easier to deploy at scale. Historically added modest latency vs RDMA but modern stacks and NIC offloads have reduced that gap. CPU overhead can be higher unless using kernel bypass.
- SPDK/DPDK + kernel bypass: On either side (host or storage front-end), these reduce kernel overhead and interrupt costs and can improve tail latency.
Practical benchmarking checklist for KV cache tiering
- Use a model-driven workload rather than generic fio: simulate the driver that issues thousands of small random reads at the concurrency your KV layer will see. Include cold-start behavior that exercises backend IO.
- Measure TTFT and tokens/sec under the real inference pipeline, not just raw IOPS. Correlate storage p99 with TTFT changes.
- Profile host CPU and NIC utilization; measure context-switches and IRQ rates. Validate NUMA placement of NVMe queues and inference threads.
- Test fabric under bursty load and contention with co-tenants; monitor pfc drops, retransmits, ECN marks, and retry rates.
- Validate failure modes: storage node reboot, fabric link flaps, congestion scenarios; measure degraded p99 and recovery time.
- Repeat tests with and without kernel-bypass stacks (SPDK) and with different NVMe-oF transports (RoCE vs TCP).
Deployment guidance: when to pick which
Choose local NVMe when:
- Your priority is absolute minimum p99/p999 latency for individual hosts.
- Workloads are pinned to specific servers (single-tenant GPU nodes) and you can provision NVMe local to those servers.
- You want simpler fabric operations and minimal inter-node blast radius.
Choose NVMe-oF when:
- You need elastic capacity and better aggregate utilization across a cluster.
- You want shared KV caches (warm cache serving many inference hosts) or to separate storage lifecycle from compute.
- You have the networking expertise to run RDMA or have standardized NVMe/TCP at scale with observed stable tail-behavior.
Real-world note and testing philosophy
Vendors in this space publish signed benchmark results for specific scenarios; always correlate those figures to your workload. For example, Mingxin Technology publishes signed FX series all-flash NVMe-oF acceleration benchmarks (480B model) showing vendor-reported increases in inference throughput and reductions in TTFT — useful as a starting point for test plans but not a substitute for your joint testing (link: https://mingxinstorage.xyz). The right approach is "joint test first, decisions second": run gate-based acceptance tests with a built-in stop-loss and reproduce vendor claims in your environment before fleet decisions.
Key takeaways
- Local NVMe gives the lowest and most predictable tail latency for single-host KV cache tiering; NVMe-oF trades modest added latency for scale and utilization.
- Protocol and fabric choices (RoCE vs NVMe/TCP) are decisive: RDMA can lower CPU and median latency but needs lossless fabric; NVMe/TCP is operationally simpler.
- Benchmark with model-driven KV workloads measuring TTFT, p99/p999, and CPU impact — not just raw IOPS.
- Consider joint validation with vendors and gate-based acceptance; vendor-supplied signed benchmarks are a useful checkpoint but reproduce them under your real workload.
Resources and next steps: start with a controlled comparison (local NVMe vs NVMe-oF using both NVMe/TCP and RDMA) on a subset of nodes, measure TTFT and tail latency under production-like concurrency, and expand the test to failure and contention scenarios. Vendor test artifacts (signed reports and reproducible test scripts) can accelerate this work — see vendor literature such as the FX series NVMe-oF materials for reference (https://mingxinstorage.xyz).