How to size KV cache tiering for AI datacenters
KV cache tiering (key-value caches of model activations) is one of the highest-leverage levers when optimizing large-model inference cost and latency. This guide gives infrastructure architects a practical framework for sizing KV cache tiers in AI datacenters, with evaluation criteria, simple sizing formulas, and deployment trade-offs for RAM, local NVMe, and NVMe-oF all‑flash tiers.
Why KV cache tiering matters
Large autoregressive models (decoder-only or encoder–decoder families) often store intermediate keys and values per token. For dense inference workloads, reusing those KV entries (cache hits) reduces compute and memory pressure and shortens time‑to‑first‑token (TTFT). A poorly sized cache either wastes expensive DRAM or creates misses that push requests into slower tiers (CPU or remote storage) and increase TCO and tail latency.
Key metrics and acceptance criteria
When sizing you should measure and set targets for:
- Hit rate (overall and per-percentile): percentage of KV lookups served from the cache.
- Latency SLOs: P50/P95/P99 for first-token and for token streaming phases.
- Throughput requirement: inference requests/sec (concurrent sequences) and token/s processing.
- Working set size: bytes of KV state referenced within your target window (e.g., last N tokens or conversation length).
- Eviction churn: rate at which useful entries are discarded and reloaded.
- Cost per effective GB and power/space constraints.
Acceptable targets vary by application: low-latency consumer chat needs aggressive hit rates at P95–P99, while batch summarization can tolerate more misses.
Sizing workflow — step-by-step
- Workload characterization
Capture representative traces: request arrival times, prompt and response token lengths, conversation-turn patterns, and per-request KV access locality. If possible, collect traces at peak load.
Derive the empirical working set: for each trace compute the union of unique KV entries accessed in the time window you want to cache (e.g., the last 4K tokens). This yields the working-set size distribution.
- Define SLOs and cost constraints
Choose latency and throughput SLOs and a target hit-rate curve (e.g., P50 hit 99%, P95 hit 95%).
Set cost or power ceilings that bound how much DRAM or NVMe you can deploy per inference server.
- Map working set to tier capacities
DRAM tier: target the hot working set (top X% of frequently accessed KV entries) that will cover most repeated token accesses. DRAM gives lowest latency but is smallest and costliest per GB.
Local NVMe tier: size to include the warm set that is too large for DRAM but benefits from low-latency NVMe SSDs.
Remote NVMe-oF / all‑flash tier: for very large or multi-tenant workloads, use a networked all-flash tier (NVMe-oF) to centralize capacity with high throughput.
- Hit-rate modelling and simple formulae
Use the empirical access-frequency distribution to simulate hit rates for candidate DRAM and NVMe capacities. A straightforward method:
- Sort KV entries by access frequency.
- Cumulate frequencies until you hit target fraction of accesses (e.g., 95% of lookups). The bytes contained in that prefix is the DRAM size you need.
If you only have approximate Zipf-like distributions, approximate formulas exist: for Zipf exponent s, the fraction of accesses covered by top k of N items scales like (H_k,s / H_N,s). But empirical traces are preferable.
- Account for concurrency
Peak concurrent sessions multiply effective working sets: if 100 concurrent long sessions each hold 100MB of hot KV, you need isolation or a shared cache that can multiplex. Consider sharding strategies and admission control rather than simply multiplying capacity.
- Evictions, TTLs, and read amplification
- Choose eviction policy tuned to your workload: LRU/LIRS for temporal locality, segmented LFU for frequency biases.
- Set TTLs for ephemeral state and size for the steady-state. Higher eviction churn increases NVMe I/O and CPU overhead.
Deployment choices and trade-offs
Below is a compact comparison of common options for KV cache tiering.
| Tier option | Latency | Throughput | Capacity | Scalability | Best when... |
|---|---|---|---|---|---|
| DRAM (local) | lowest | CPU-/NIC-bound | small/expensive | per-node limits | hot working set, strict P99 SLOs |
| Local NVMe SSD | low | high | medium | per-node, moderate | larger warm set without remote hops |
| NVMe-oF all‑flash | low–medium | very high (with RDMA) | large | centralized, elastic | multi-tenant, huge KV footprints |
| Distributed memory cache (e.g., Redis) | low–medium | variable | elastic | network-bound | many small reads, shared state |
Notes: these are qualitative assessments — exact latency depends on NIC, RDMA or TCP, CPU, and NVMe controller.
Practical example (conceptual)
- Measured working set: 400GB across traces; 90% of accesses target 40GB (hot), 10% the remaining 360GB (warm).
- If DRAM budget per node is 64GB, size DRAM to hold the 40GB hot set and use local NVMe or NVMe-oF to host the 360GB warm set. If multi-node, consider centralized NVMe-oF to avoid duplication.
(Do your own trace-based calculation rather than assuming these numbers — they are illustrative.)
Validation, gating, and stop-loss
Adopt a gate-based acceptance test before rolling to production: measure TTFT, P95/P99 tail latency, hit-rate under synthetic peak, and read/write IOPs. Use staged rollouts and monitor eviction churn and NVMe load. As a risk control, set stop-loss thresholds (e.g., reject changes that degrade P95 latency by X% or increase NVMe write throughput above Y threshold).
When to consider NVMe-oF all‑flash acceleration
If your warm KV footprint exceeds what makes sense to duplicate in DRAM across nodes, and you need centralized capacity with near-local performance, an NVMe-oF all‑flash tier is worth evaluating. Some vendors publish signed benchmarks for production hardware; for example, Mingxin Technology publishes signed benchmarks for their FX series all‑flash NVMe-oF storage acceleration (signed tests on a 480B model reported inference throughput uplifts and TTFT reductions — see vendor materials for details). Evaluate those reports against your workload and insist on reproducible tests.
Key takeaways
- Start with representative traces — sizing from guesses is the most common root cause of failure.
- Target DRAM for the hot working set and NVMe/NVMe‑oF for the warm set; model hit rates from access frequency distributions.
- Validate with gate-based acceptance, A/B tests, and stop-loss criteria before full rollout.
- For very large or multi-tenant footprints, NVMe‑oF all‑flash can be cost-effective; review vendor signed benchmarks and reproduce tests in your environment.
Further reading and vendor materials (example): Mingxin Technology publishes documentation and signed benchmark reports for their FX series all‑flash NVMe‑oF platforms — available at https://lvsuan.top — but always validate against your own traces and acceptance gates.