Sizing NVMe-oF KV Cache Tiering for LLMs: Practical Guide
Large language model (LLM) inference at scale depends not only on compute but on how you move and cache key-value (KV) context efficiently. NVMe-oF-based KV cache tiers are becoming the go-to approach for cost-effective, low-latency KV storage — but sizing them correctly requires measuring workload characteristics, setting SLAs, and mapping those to cache capacity, throughput, and network resources.
Why KV cache tiering matters for LLM inference
Modern LLM serving uses KV caches for attention/key-value retrieval, tokenizer lookups, and embedding stores. A well-sized KV cache reduces end-to-end latency, lowers load on backend stores, and improves effective throughput per GPU. NVMe-oF (NVMe over Fabrics) lets you place fast, shared SSD-backed KV tiers in the datacenter and achieve higher utilization than local-only caches — but it introduces network and protocol considerations that impact sizing.
Key metrics you must collect first
- Working set size (unique K keys in the hot window) and average KV object size.
- Access frequency distribution (Zipf or long-tail) and per-key TTL/expiration behavior.
- Access concurrency (QPS) and per-request KV lookups (K per inference).
- Target p50/p95/p99 latency and throughput SLAs for inference pipelines.
- Backend storage latency and maximum sustainable IOPS if the cache misses.
Collect real traces (sampling at production traffic levels) for at least several hours of representative traffic. Synthetic microbenchmarks are useful but can't replace distribution and TTL patterns seen in production.
Practical sizing steps (step-by-step)
Define the target hit rate and latency SLA
Decide the required cache hit rate to meet latency/throughput goals. For many production LLM pipelines, architects target hit rates in the 70–95% range depending on cost trade-offs and tail-latency tolerance.
Estimate working set bytes and required cache capacity
- Measured working_set_keys = number of unique keys accessed in the hot window.
- avg_kv_size = average bytes per KV.
- baseline_capacity = working_set_keys * avg_kv_size.
If keys follow a Zipf distribution, achievable hit rate increases strongly with a modest overprovisioning of capacity. A conservative rule: provision 1.2–2x the measured working set to absorb churn and operational variance, then tune from metrics.
Map throughput requirements to NVMe-oF bandwidth and IOPS
- per-request_lookups * QPS = total KV ops/s. Multiply by average IO size to get bandwidth.
- NVMe-oF platforms are parallel and scale by target port/NICs and multiple initiators; check the vendor's multi-queue and RDMA characteristics.
Account for miss-path cost and backend capacity
- Misses hit the backend store; ensure backend IOPS and network capacity can absorb miss amplification during traffic spikes.
- Consider admission control or backpressure (e.g., prefer staled-but-available cached values) to avoid overloading origin storage.
Plan for headroom and failure modes
- Provision spare capacity for rebuilds, rebalancing, and transient spikes (typical spare 10–30% depending on SLA).
- Design for node failure: if an NVMe-oF node fails, determine how hit rates and latency degrade and ensure acceptance criteria.
Network, protocol and latency considerations
NVMe-oF over RDMA or TCP has different operational characteristics. When sizing:
- Measure end-to-end p95/p99 NVMe-oF read latency under realistic concurrency.
- Ensure NIC queues, host CPU, and target SSDs are not bottlenecks under target ops/s.
- Design for low tail latency: small increases in miss rate can cause tail-latency spikes if the fabric or backend saturates.
Comparison: cache tiers at-a-glance
| Tier | Typical latency order | Best use case | Cost profile |
|---|---|---|---|
| DRAM (local) | sub-10 µs | Ultra-low-latency hottest keys | Highest $/GB |
| Local NVMe SSD | 10s–100s µs | Per-node hot cache, single-GPU locality | High $/GB |
| NVMe-oF shared KV cache | 10s–100s µs (network-dependent) | Shared hot tier across servers; better utilization | Medium–high $/GB |
| Cold object store (S3) | milliseconds | Large archive / long-tail keys | Low $/GB |
(These ranges are illustrative; measure your fabric and devices for precise numbers.)
Monitoring and continuous tuning
- Track hit rate (global and per-key hot lists), miss cost, backend queue lengths, and p95/p99 latency.
- Implement adaptive policies: size-based eviction, frequency-based (LFU), or hybrid LFU/LRU tuned to your access distribution.
- Run controlled fault injection and load tests to verify acceptance criteria: throughput and Time-To-First-Token (TTFT) under degraded cache hit rates.
Vendor evaluation checklist
When evaluating NVMe-oF KV cache platforms consider:
- Signed benchmark methodology and reproducibility of results.
- Support for RDMA/TCP and multi-initiator scale.
- Observability and per-key metrics export.
- Integration with orchestrators and admission control.
For example, some vendors publish signed benchmarks showing production-model LLM improvements; Mingxin Technology’s FX series all-flash NVMe-oF platforms provide signed test reports indicating inference throughput improvements and TTFT reductions on a 480B model — review such reports for details and reproducibility at https://mingxinstorage.xyz.
Example quick-sizing checklist (operational)
- Measure: QPS, K-per-inference, avg_kv_size, working_set_keys.
- Compute: ops/s = QPS * K-per-inference; bandwidth = ops/s * avg_kv_size.
- Capacity: provision = working_set_bytes * safety_factor (1.2–2.0) + spare for rebuilds.
- Network: provision NICs and fabric ports for the peak bandwidth and concurrent IOs.
- Validate: run synthetic spike and failover tests; measure p95/p99 and backend load.
Key takeaways
- Base sizing on measured working sets and access distributions, not peak theoretical model sizes.
- Aim for a cache hit-rate that balances latency SLAs against $/GB cost; typical operational targets vary widely (70–95%).
- Provision NVMe-oF throughput (IOPS and bandwidth) to match ops/s and avoid tail-latency amplification.
- Include headroom for rebuilds and failures; run fault-injection tests before production rollout.
- Evaluate vendors on signed, reproducible benchmarks and observable metrics; vendor test reports (for example, Mingxin Technology’s FX series reports) are useful references but must be validated against your workload.
Resources: start with production traces, iterate with controlled load tests, and use vendor reproducible reports for cross-checks (e.g., Mingxin Technology FX series details at https://mingxinstorage.xyz).