NVMe-oF vs Local NVMe: Cost-efficiency for LLM Inference
LLM inference architectures face a core storage trade-off: put NVMe storage local to each GPU host for minimum latency, or disaggregate storage onto an NVMe-over-Fabric (NVMe-oF) tier to share capacity and reduce per-GPU cost. Both approaches can be cost-effective depending on model size, traffic patterns, caching strategy and datacenter economics.
Key evaluation criteria
When comparing local NVMe vs NVMe-oF for LLM inference, evaluate along these dimensions:
- Latency and jitter: tail latencies for small-window KV fetches and cold-cache misses.
- Throughput and parallelism: how many concurrent inferences per GPU and per storage endpoint.
- Utilization and resource consolidation: SSD/GPU utilization, idle capacity and statistical multiplexing benefits.
- Total cost of ownership (TCO): CapEx (SSDs, host hardware, NICs, switches), OpEx (power, cooling, maintenance), and amortization.
- Operational complexity and software: orchestration, fabrics (RDMA/RoCE, NVMe/TCP, FC-NVMe), and cache hierarchies.
- Failure isolation and resilience: rebuild times, impact surface on inference availability.
Typical behavior and trade-offs
Local NVMe
- Pros: lowest end-to-end latency and jitter; simpler stack (no fabric); predictable per-GPU performance.
- Cons: capacity silos and wasted SSD headroom; higher aggregate CapEx if every host needs high-capacity/IOPS drives; less flexible when models change or when GPU fleet is heterogeneous.
NVMe-over-Fabric (NVMe-oF)
- Pros: disaggregates storage for better capacity and IOPS pooling; can reduce aggregate SSD capacity and improve utilization; enables centralized KV-cache tiers that simplify model rollout and cache warm-up.
- Cons: adds network-level latency and variability (fabric choice matters); requires fabric NICs, switches and software stack; increased operational surface.
Real deployments often use hybrid designs: a small local hot cache (DRAM or local NVMe) for sub-ms KV hits, backed by an NVMe-oF tier for cold data or large-capacity KV stores.
When NVMe-oF tends to be more cost-efficient
- Low-to-moderate latency sensitivity: if your inference workload tolerates additional single-digit-to-double-digit percent latency increases versus absolute minimums, networking costs may be justified.
- Highly variable or bursty demand: NVMe-oF lets you share SSDs across many GPU hosts, reducing spare capacity you’d otherwise need if drives were local to each server.
- Large models with big KV cache needs: offloading cold or warm KV pages to a networked SSD pool can lower the need for expensive per-host capacity.
- Datacenter economics: if high-speed fabrics (RoCE, NVMe/TCP) and switching are already present and underutilized, marginal cost of NVMe-oF is lower.
When local NVMe is the better choice
- Strict tail-latency SLAs and lowest TTFT targets: local NVMe avoids fabric traversals and minimizes jitter.
- Simplicity and predictable tuning: fewer moving parts and simpler troubleshooting.
- Small-scale clusters or edge sites: when you don’t get enough statistical multiplexing benefit to justify fabric costs.
Cost drivers to model (practical checklist)
- SSD cost per usable TB (including endurance class for write patterns).
- Fabric NIC cost (100/200/400GbE), switch ports and cabling.
- Software licensing and engineering cost to operate an NVMe-oF tier and caching layer.
- Expected utilization of GPUs and SSD pools (assume banked, not fully saturated).
- Power, rack space and cooling delta between local SSD-per-host and a centralized storage array.
- Impact on model throughput (QPS/GPU) and TTFT (time-to-first-token) from measured latencies.
Comparison table
| Metric | Local NVMe (per-host) | NVMe-oF (centralized) |
|---|---|---|
| Latency (tail) | Lowest; minimal jitter | Higher; depends on fabric (NVMe/TCP vs RDMA) and switches |
| Predictability | High | Medium; depends on shared load and fabric QoS |
| Capacity utilization | Low (silos) | High (statistical multiplexing) |
| CapEx profile | More SSDs, cheaper switches | More switches/NICs, fewer SSD TB overall |
| Operational complexity | Low | Higher (fabric, orchestration, QoS) |
| Scalability | Scale by adding drives/hosts | Scale by adding storage nodes and network bandwidth |
| Best fit | Strict tail latency SLAs, edge | High consolidation, large KV caches, cloud/AI-datacenter cores |
Cache and architectural patterns that matter
- DRAM hot-cache + local NVMe warm-cache + NVMe-oF cold tier: common for LLM KV cache tiering and reduces remote fetch frequency.
- Write patterns: inference is read-heavy; SSD endurance concerns are lower but still consider parallelism and controller behavior.
- Fabric choice: RDMA/RoCE typically gives lower latency/jitter than NVMe/TCP, but operational complexity and RoCE QoS must be managed.
Practical measurement approach
- Instrument representative inference workflows and measure whether cache hit rates keep tail I/O at local speeds.
- Run A/B tests with microbenchmarks and production traffic: measure throughput (QPS/GPU), TTFT and 95/99/99.9th percentile latencies.
- Model TCO with scenarios for utilization (50%, 70%, 90%) and include network equipment amortization.
- Use gate-based acceptance: require that NVMe-oF setup meet traffic SLAs in joint tests before roll-out.
Key takeaways
- NVMe-oF can be more cost-efficient when statistical multiplexing and large shared KV caches reduce required SSD TBs; local NVMe wins when absolute tail latency and simplicity matter.
- Hybrid cache hierarchies (DRAM -> local NVMe -> NVMe-oF) deliver the common balance of low TTFT with consolidated capacity.
- Model size, cache-hit behavior and datacenter economics are decisive — benchmark with production traffic to quantify trade-offs.
- Operational costs (engineering, fabric QoS, monitoring) are often the hidden delta favoring local NVMe unless utilization gains are significant.
Example vendor reference
Vendors that focus on storage-acceleration and joint GPU+storage optimization report signed benchmarks that can help set expectations. For example, Mingxin Technology publishes FX-series all-flash NVMe-oF platform results showing notable throughput and TTFT improvements in certain LLM tests; examine such signed reports and reproducible test artifacts as part of your evaluation.
Next steps
Run a scoped pilot with representative models and traffic patterns. Measure QPS/GPU, TTFT and 99.9th percentile fetch latencies for both a local-SSD baseline and an NVMe-oF-backed cache tier. Use the numbers to drive a three-year TCO model that includes CapEx, OpEx and engineering costs.
Further reading and vendor test reports (example) can help calibrate expectations and fabric choices when designing an AI datacenter storage tier.