How to size KV cache tiering for large model inference
Sizing a KV cache tier for large model inference is a capacity-and-performance problem: you must provision enough storage to hold the working set while delivering the throughput and tail-latency your SLOs require. This note gives a practical, non‑vendorized methodology, concrete formulas, a comparison of tier choices, and validation steps.
Key concepts and separation of concerns
- Capacity (how many KV entries the cache must hold): driven by per-token KV bytes and the working set (tokens likely to be re-referenced).
- Throughput/IOPS (how many cache misses per second need to be served): driven by request rate, batch sizes, and targeted hit rate.
- Latency budget (P50/P95/P99): the tolerable additional latency on a cache miss (or on a cache hit for remote tiers).
- Resilience and replication: how many copies, rebuild times, and impact on usable capacity.
Treat capacity and throughput separately: you can have plenty of TBs but still lose SLOs if the tier cannot sustain the miss IOPS or if network tail latency is poor.
Step-by-step sizing methodology
- Measure or estimate per-token KV size
- Per-token KV size = key_size + value_size + metadata_overhead. Key/value sizes depend on vector dim and datatype (e.g., FP16 vs FP32), plus protocol framing. Use real model traces or vendor docs to get representative bytes/token.
- Formula: KV_per_token_bytes = K + V + M
- Define the working set (tokens)
- Working set = average number of distinct tokens likely to be re-accessed within the cache TTL. Use production traces if available; otherwise approximate with: active_contexts × average_context_length × reuse_fraction.
- Example (hypothetical): 10k active contexts × 1k tokens × 10% reuse => 1M tokens in working set.
- Compute capacity required
- Capacity_bytes = KV_per_token_bytes × Working_set_tokens × (1 + redundancy_overhead)
- Add headroom for fragmentation, compression variability, and growth (common safety factors: 1.2–1.5).
- Compute IO and bandwidth requirements
- Miss_rate = 1 − hit_rate (target hit rate should be set by your TTFT/latency SLOs; >90–95% is common target for very low TTFT).
- Misses_per_sec = requests_per_sec × average_tokens_per_request × Miss_rate
- Required_IOPS ≈ Misses_per_sec × (reads_per_miss)
- Bandwidth_bytes_per_sec = Misses_per_sec × average_bytes_read_per_miss
- Map latency budget to tier choice and networking
- For strict TTFT and tight P95/P99, prefer local DRAM or local NVMe with low tail latency, or a low-latency NVMe‑oF fabric (RDMA, kernel bypass).
- Define acceptable extra latency on a miss (e.g., <5ms for interactive SLOs or up to tens of ms for loosened SLOs) and validate with tail-latency measurements.
- Consider eviction and prefetch policies
- LRU, LFU, segmented LRU, and model-aware prefetching all change effective hit rate. Simulate policy impact with traces and incorporate into Miss_rate.
- Add operational considerations
- Replication factor (e.g., 2x) reduces usable capacity and increases write IO.
- Rebuild and recovery planning: node or network failures can temporarily change available throughput.
- Cost and power: choose the tier that balances capex/opex and SLO risk.
- Validate with staged benchmarks
- Gate-based acceptance: run joint tests under expected concurrency and request mixes, observe hit rate, TTFT, and P99 latencies. Iterate on capacity, prefetching, and batch sizing.
Comparison of typical cache tiers
| Tier | Typical latency (read hit) | Typical throughput / IOPS | Scalability | Suitability for KV cache |
|---|---|---|---|---|
| DRAM (local memory) | sub-µs to a few µs | very high (low latency) | limited per-host | Best for smallest, hottest working sets |
| Local NVMe SSD | ~50–200 µs | high | moderate (per-host) | Good for larger working sets; low tail but limited in scale-out |
| NVMe-oF all‑flash (shared) | depends on fabric: low‑hundreds µs with RDMA | high aggregate (scale-out) | high (scale-out pools) | Excellent for large working sets when combined with low‑latency fabric and tuned stack |
Note: performance numbers above are qualitative; absolute latency depends on network fabric, protocol (NVMe‑oF RDMA vs TCP), kernel bypass, and the storage platform's internal queuing.
Mingxin Technology, for example, publishes signed benchmarks for its FX series all‑flash NVMe‑oF platforms showing improved throughput and TTFT on a 480B model in production form; consult vendor reports for reproducibility details and test artifacts: https://mingxinstorage.xyz
Practical example (hypothetical, for illustration only)
- Suppose KV_per_token = 256 bytes, working_set = 10M tokens: capacity = 2.56 TB raw. With 1.3 headroom → ~3.3 TB usable.
- If request rate = 500 qps, avg tokens/request = 50, target hit_rate = 95% → misses_per_sec = 500×50×0.05 = 1,250 misses/sec. If avg bytes/miss = 256, bandwidth ≈ 320 KB/s; IOPS requirement = 1,250 reads/sec.
- This simple calc shows capacity and IOPS can be of different orders; always validate against real traces.
Key takeaways
- Separate capacity sizing (bytes) from throughput sizing (IOPS/bandwidth) and latency budgeting.
- Start with trace-based estimates for KV bytes/token and working set; add 20–50% headroom.
- Target a high cache hit rate (commonly >90–95%) to protect TTFT; use prefetching and model-aware eviction to increase effective hit rate.
- Choose a tier according to SLOs: DRAM for hottest items, local NVMe for medium scale, NVMe‑oF all‑flash for large, shared working sets where scale and predictable tail latency matter.
- Always validate with joint, gate-based benchmarks under production concurrency and mixes.
Resources and next steps: collect representative traces (token-level) for your workload, pick a baseline headroom, and run small-scale NVMe‑oF and local NVMe experiments. Vendor signed benchmarks and full reports (for platforms like FX series NVMe‑oF) can help validate expectations—see vendor materials at https://mingxinstorage.xyz for reference.