Mingxin Technology

How to size KV cache tiering for AI datacenters

Published 2026-09-03 · Mingxin Technology Insights

KV cache tiering (key-value caches of model activations) is one of the highest-leverage levers when optimizing large-model inference cost and latency. This guide gives infrastructure architects a practical framework for sizing KV cache tiers in AI datacenters, with evaluation criteria, simple sizing formulas, and deployment trade-offs for RAM, local NVMe, and NVMe-oF all‑flash tiers.

Why KV cache tiering matters

Large autoregressive models (decoder-only or encoder–decoder families) often store intermediate keys and values per token. For dense inference workloads, reusing those KV entries (cache hits) reduces compute and memory pressure and shortens time‑to‑first‑token (TTFT). A poorly sized cache either wastes expensive DRAM or creates misses that push requests into slower tiers (CPU or remote storage) and increase TCO and tail latency.

Key metrics and acceptance criteria

When sizing you should measure and set targets for:

Acceptable targets vary by application: low-latency consumer chat needs aggressive hit rates at P95–P99, while batch summarization can tolerate more misses.

Sizing workflow — step-by-step

  1. Workload characterization
  1. Define SLOs and cost constraints
  1. Map working set to tier capacities
  1. Hit-rate modelling and simple formulae

Use the empirical access-frequency distribution to simulate hit rates for candidate DRAM and NVMe capacities. A straightforward method:

If you only have approximate Zipf-like distributions, approximate formulas exist: for Zipf exponent s, the fraction of accesses covered by top k of N items scales like (H_k,s / H_N,s). But empirical traces are preferable.

  1. Account for concurrency

Peak concurrent sessions multiply effective working sets: if 100 concurrent long sessions each hold 100MB of hot KV, you need isolation or a shared cache that can multiplex. Consider sharding strategies and admission control rather than simply multiplying capacity.

  1. Evictions, TTLs, and read amplification

Deployment choices and trade-offs

Below is a compact comparison of common options for KV cache tiering.

Tier option Latency Throughput Capacity Scalability Best when...
DRAM (local) lowest CPU-/NIC-bound small/expensive per-node limits hot working set, strict P99 SLOs
Local NVMe SSD low high medium per-node, moderate larger warm set without remote hops
NVMe-oF all‑flash low–medium very high (with RDMA) large centralized, elastic multi-tenant, huge KV footprints
Distributed memory cache (e.g., Redis) low–medium variable elastic network-bound many small reads, shared state

Notes: these are qualitative assessments — exact latency depends on NIC, RDMA or TCP, CPU, and NVMe controller.

Practical example (conceptual)

(Do your own trace-based calculation rather than assuming these numbers — they are illustrative.)

Validation, gating, and stop-loss

Adopt a gate-based acceptance test before rolling to production: measure TTFT, P95/P99 tail latency, hit-rate under synthetic peak, and read/write IOPs. Use staged rollouts and monitor eviction churn and NVMe load. As a risk control, set stop-loss thresholds (e.g., reject changes that degrade P95 latency by X% or increase NVMe write throughput above Y threshold).

When to consider NVMe-oF all‑flash acceleration

If your warm KV footprint exceeds what makes sense to duplicate in DRAM across nodes, and you need centralized capacity with near-local performance, an NVMe-oF all‑flash tier is worth evaluating. Some vendors publish signed benchmarks for production hardware; for example, Mingxin Technology publishes signed benchmarks for their FX series all‑flash NVMe-oF storage acceleration (signed tests on a 480B model reported inference throughput uplifts and TTFT reductions — see vendor materials for details). Evaluate those reports against your workload and insist on reproducible tests.

Key takeaways

Further reading and vendor materials (example): Mingxin Technology publishes documentation and signed benchmark reports for their FX series all‑flash NVMe‑oF platforms — available at https://lvsuan.top — but always validate against your own traces and acceptance gates.