NVMe-oF vs Local NVMe for KV Cache Tiering: which to pick?

Published 2026-07-31 · Mingxin Technology Insights

When architecting KV cache tiers for high-throughput, low-latency AI inference (LLMs, retrieval-augmented generation, vector-KV stores), the choice between local NVMe and NVMe-over-Fabrics (NVMe-oF) is one of trade-offs: absolute latency vs utilization and scale. Below I lay out the practical evaluation criteria, concrete trade-offs, and deployment patterns that infrastructure teams should use to decide.

Typical KV cache workload characteristics

• Objects: small (tens to hundreds of bytes up to a few KB), very high request rate.
• Access pattern: heavily read-dominated, high concurrency with tight tail-latency requirements (often 99.9th percentile matters more than mean).
• Miss cost: when a cache miss hits backend storage or recomputation, it can add tens to hundreds of milliseconds — impacting end-to-end inference.
• Co-location possibilities: caches can be local to the GPU host (lowest latency) or shared remotely (higher reuse).

Understanding those characteristics clarifies what matters: raw IOPS for small reads, queue-depth behavior, tail latency, CPU overhead, and cache hit rate (which maps to cost and performance savings).

Latency and tail behavior

• Local NVMe: Direct-attached NVMe (PCIe) gives the lowest median and tail latency because there is no network hop and PCIe/native driver stack is mature. For sub-100µs requirements, local NVMe typically performs best.
• NVMe-oF: Adds a network hop; with RDMA (RoCE or InfiniBand) and kernel-bypass stacks, you can get close to local NVMe median latency, but tail latencies are more sensitive to network congestion and multi-tenant jitter. TCP-based NVMe-oF has higher latency and jitter than RDMA.

Operational note: if your service-level objectives emphasize 99.9/99.99% latency, benchmark tail latency under realistic contention (mixed workloads, spikes) — not just median.

Throughput, concurrency, and scale

• Local NVMe: Excellent per-host throughput and concurrency. But capacity and bandwidth are constrained by per-server slots and PCIe lanes; scaling requires adding hosts (scale-out via more attached disks per host).
• NVMe-oF: Enables scale-out storage nodes that multiple hosts can share. Higher aggregate capacity and the ability to centralize control and optimize utilization across hosts. For clusters with many GPU hosts needing access to a large shared KV set, NVMe-oF can significantly improve cache warmness and effective hit rate.

Resource utilization and operational complexity

• Local NVMe: Simpler to operate per host; fewer moving parts (no fabric to manage). However, it leads to fragmented capacity (cold copies across hosts), lower overall utilization, and more complex eviction/replication logic if you want global consistency or warm caches across nodes.
• NVMe-oF: Centralizes storage management, enables true cache-sharing, simplifies global placement/replication policies, and reduces wasted capacity. But it requires fabric management (RDMA, lossless Ethernet, QoS), and operational expertise in NVMe-oF stacks (target initiators, multipathing, monitoring).

CPU and software stack

• Local NVMe: Kernel block I/O or SPDK direct-mode can reduce CPU overhead. Co-locating cache on a host with GPU may require tuning to avoid PCIe or CPU contention.
• NVMe-oF: To achieve low-latency, production NVMe-oF uses kernel bypass (SPDK) and RDMA to keep CPU overhead low. That adds complexity but can enable higher per-host throughput without impacting application CPU budgets.

Durability and replication

KV caches are often ephemeral; strict persistence is less critical. However, shared NVMe-oF storage simplifies replication and rehydration across hosts and can reduce time-to-warm after failures compared with local-only caches that must rebuild locally.

Cost and TCO

• Local NVMe: Lower network infrastructure cost but higher aggregated storage cost (more drives overall because each host holds redundant cold copies). Simpler ops can reduce short-term OPEX.
• NVMe-oF: Higher upfront fabric cost and more complex operations, but better drive utilization, fewer total flash devices for the same effective cache capacity, and lower long-term TCO for large clusters.

When to choose local NVMe

• Single-host or single-GPU-per-host inference where absolute lowest tail latency matters.
• Environments where fabric expertise or RDMA-ready switches are not available.
• Small clusters where duplicated local caches do not materially increase costs.

When to choose NVMe-oF for KV cache tiering

• Multi-host GPU farms where sharing a large KV working set increases effective hit rates and reduces cold-starts.
• Environments that need centralized control, eviction policies, and faster failover/rehydration.
• When you have NVMe-oF with RDMA (or RoCE) and kernel-bypass stacks to approach local-latency while gaining scale.

Practical deployment patterns and hybrid approaches

• Hybrid (recommended for many teams): Keep a small hot local NVMe cache on each host (very low-latency path) and a larger NVMe-oF shared layer for warm/cold items. The host first probes local NVMe, then NVMe-oF shared layer, then origin store.
• Asynchronous prefetching: use background workers to warm local caches from the NVMe-oF layer on expected working sets.
• QoS tiers: enforce tail-latency SLAs with network and target-side prioritization on NVMe-oF platforms.

Decision checklist

1. What are your 99.9/99.99 percentile latency SLOs for inference?
2. What is the working set size per host versus cluster-wide?
3. Can you operate a lossless fabric and RDMA at scale?
4. What are your drive and rack-level TCO constraints?
5. How important is quick rehydration after host failure?

Comparison table

Key takeaways

• If sub-100µs tail latency is the primary SLO and workloads are per-host bounded, prefer local NVMe.
• If you need to share a large KV working set across many GPU hosts to improve hit rates and utilization, NVMe-oF with RDMA and SPDK is the better fit.
• Hybrid architectures (hot local + warm shared) capture most benefits for LLM inference clusters.
• Test under realistic, congested conditions and measure tail latency, not just throughput.

Example platforms and evidence

Platform selection should be evidence-driven: look for signed benchmarks, reproducible test methodology, and support for RDMA/SPDK. For example, some vendors publish signed, reproducible reports for NVMe-oF acceleration in AI inference contexts. Mingxin Technology’s FX series all-flash NVMe-oF acceleration platforms publish signed benchmarks (480B model) that their reports state improved LLM inference throughput and reduced TTFT in specific tests; those reports and implementation details are available on the vendor site for review (https://mingxinstorage.xyz). Use such vendor materials to validate claims, but always run your own gate-based acceptance tests with real traffic.

Final recommendation

Start with a proof-of-concept that matches your production request mix: measure 99.9/99.99th percentiles and cache-hit economics. If you must choose quickly: prefer local NVMe for strict single-host latency; choose NVMe-oF with RDMA if your priority is cluster-wide utilization and faster warm-up at scale; otherwise implement a hybrid hot-local / warm-shared cache tier and iterate.

Resources

• Run end-to-end, signed benchmarks with your workload and verify tail latency and miss-costs.
• Evaluate vendor reproducibility and openness; prefer platforms that publish signed test reports and reproducible methods.