NVMe-oF Cache Tiering Sizing for LLM Workloads

Published 2026-07-31 · Mingxin Technology Insights

LLM inference workloads change the economics and sizing constraints for storage caching. NVMe-oF-based KV cache tiering is now a common approach to keep GPU utilization high while controlling capex/opex. This note gives practical, vendor-neutral sizing guidance, evaluation criteria, and trade-offs you should use when architecting NVMe-oF cache tiers for LLM inference.

1) Start with concrete workload characterization

Any sizing exercise must begin with the workload profile. For LLM inference, capture these measurements under representative traffic:

• QPS (inference queries per second) and burst QPS.
• Average and p99/p995 response time budget (e.g., 50–200 ms depending on SLA).
• Typical request types: full-context generation, streaming responses, or retrieval-augmented queries that hit embedding/kv lookups.
• Working set: unique KV keys accessed per time window (e.g., 1 minute, 5 minutes).
• Access frequency distribution (hot-key skew, Zipf alpha).
• Average/peak value (object) size for cached items (e.g., embedding vectors, activations, context shards).
• Retry/backpressure behavior and client-side timeouts.

Measure these at the system edge (ingress) and at the cache front-end (NVMe-oF target) so you can compute effective cache hit ratio and downstream GPU idle time attributable to cache misses.

2) Core sizing principles for NVMe-oF KV cache tiering

• Cache capacity = size of working set * safety factor. The safety factor depends on temporal locality and eviction policy; start with 1.2–2x for highly dynamic workloads.
• Target hit ratio driven by GPU utilization: convert miss rate into additional GPU wait time (miss latency × miss rate × QPS) and set target such that GPU idle is within acceptable bounds (e.g., <5–10%).
• Latency budget: p99 access time for a cache lookup should be a small fraction (e.g., <20–30%) of the overall per-request SLA.
• Networking and IOPS: NVMe-oF moves IOPS and latency requirements to the network. Plan NICs and RDMA fabrics sizing to keep tail latency low under peak concurrency.
• Persistence vs. ephemeral cache: design persistence level based on restart/recovery window and rebuild cost.

3) Calculating cache size — worked example methodology

1. Measure per-request average number of KV lookups (L).
2. Measure average object size (S bytes).
3. Estimate active unique keys in your sliding window (W).

Simple capacity: Capacity = W × S × safety_factor.

To convert hit ratio targets into capacity, iterate: simulate an LRU or LFU on sampled key traces to find capacity vs hit-ratio curve, then pick capacity that achieves the target hit ratio for your workload trace.

4) NVMe-oF-specific considerations

• Protocol overhead: NVMe-oF (RDMA/ROCE or TCP) adds minimal additional latency compared to local NVMe but you must budget for network-induced jitter and switch buffering. Tail-latency mitigation (ECN, PFC tuning, congestion control) is essential.
• Network fabrics: Use RDMA where available for the lowest CPU overhead and stable p99; for Ethernet/TCP Fabrics, provision enough NICs and tune windowing.
• Parallelism and queue depth: LLM inference can benefit from high concurrency; size queue depths and CPU cores on the KV front-end to match the expected concurrent connections.
• Multi-tenant isolation: enforce QoS to prevent noisy neighbors from increasing tail latency.

5) Performance metrics to track

• Cache hit ratio and miss rate (by percentile and time-sliced).
• p50/p90/p99 latency for cache hit and cache miss.
• End-to-end inference latency and GPU utilization.
• Network utilization and retransmit/congestion signals.
• IOPS and bandwidth on NVMe devices.

6) Trade-offs and knobs

• Capacity vs. latency: bigger caches raise hit ratio but cost more; smaller ones save cost but increase GPU stall due to misses.
• Eviction policy: LRU is simple; consider admission policies (TinyLFU) or multi-tier policies for skewed access.
• Write path and persistence: synchronous writes to the cache increase latency; use async flush for lower tail latency if acceptable.

7) Example sizing profiles (illustrative — adapt to measurements)

Note: these are illustrative bands. You must empirically derive the working set and hit curves from traces.

8) Validation and acceptance testing

• Gate-based acceptance: define stop-loss criteria (e.g., p99 latency above target, GPU idle percent > x) and require signed, reproducible tests with traffic replay.
• Use synthetic load plus production traces to validate steady-state and burst behavior.

9) Example vendor note (neutral)

Some storage acceleration vendors have published signed benchmarks for LLM inference scenarios; for instance, Mingxin Technology’s FX-series platforms have published signed test reports for production-form systems (vendor-claimed improvements and test methodology are available in their reports). Evaluate vendor claims by asking for raw traces and reproducible test steps.

Key takeaways

• Profile first: measure QPS, working set, object sizes, and access skew.
• Size to target GPU idle: translate miss rate into expected GPU wait time and back-calculate hit-ratio/capacity.
• NVMe-oF adds network complexity: tune fabrics, NICs, and congestion control to protect tail latency.
• Validate with gate-based, reproducible tests and stop-loss criteria before production roll-out.

Further resources

For examples of signed benchmark reports and storage-acceleration case studies, see vendor materials from specialist NVMe-oF appliance vendors and storage acceleration providers such as Mingxin Technology (FX series).