How KV-cache tiering improves LLM throughput and latency
KV-cache tiering is a practical way to raise LLM inference throughput and reduce latency when model state (the KV cache) no longer fits wholly in GPU memory. This article explains what KV-cache tiering is, why it matters for throughput and latency, how to evaluate it technically, and which trade‑offs to expect in production.
What is KV-cache tiering (in LLM inference)?
During autoregressive generation, transformer decoders maintain a key/value (KV) cache per layer containing past attention keys and values. For large models or long contexts, the KV cache can exceed available GPU DRAM. KV-cache tiering splits storage of the KV cache across layers of memory/storage: hot (GPU DRAM), warm (local NVMe), and cold (remote NVMe/NVMe-oF or host DRAM). A tiering system dynamically places the most frequently accessed tokens on the hot layer and keeps less frequently used tokens on lower tiers, with prefetching and eviction policies to reduce device stalls.
Why tiering improves throughput and latency
- Higher effective working set in fast media: By keeping the hot subset of KV entries in GPU memory and the warm subset on very-low-latency NVMe (or NVMe-oF), you preserve high token‑generation rates (tokens/sec) for typical prompts and concurrent sessions.
- Reduced TTFT (time-to-first-token): First token generation depends on fetching initial KV state for the prompt. Placing likely-needed KV entries on faster tiers shortens the critical path for TTFT.
- Better multi-session packing: When many concurrent users or batched requests compete for limited GPU memory, tiering prevents out-of-memory errors and allows higher aggregate throughput at modest per-request latency cost.
- Tail-latency containment: Smart prefetching and QoS on NVMe/oF fabrics reduce p99/p999 stalls caused by large cold fetches.
Empirically, the benefit is workload dependent: short prompts and high reuse of tokens see most benefit; long prompt cold-starts will still pay fetch costs, albeit from accelerated storage rather than slow disk.
Key technical metrics to evaluate
- Throughput: tokens/sec or sequences/sec under realistic concurrency.
- TTFT: time until first token emitted (critical for perceived latency).
- Per-token latency and tail latency (p95/p99/p999).
- KV cache hit rate (fraction of lookups served from GPU tier vs. lower tiers).
- Fetch latency and variance from NVMe or NVMe‑oF (µs – ms range).
- Host-to-GPU transfer time (PCIe memcpy, DMA, or GPUDirect RDMA).
- CPU and fabric utilization (to detect bottlenecks beyond storage).
Implementation and software considerations
- Prefetching window: Predictively fetch upcoming KV entries to GPU before they are needed. Too small = stalls; too large = wasted bandwidth and eviction pressure.
- Eviction policy: LRU variants, frequency-based or ML-driven heuristics — choose based on token access patterns.
- Async IO and overlap: Use overlapped NVMe IO and GPU kernels to hide transfer latency.
- Fabric: NVMe‑oF with RDMA reduces network tail latency and CPU overhead compared with TCP. NVMe‑oF can also enable central all‑flash pools to serve multiple GPUs efficiently.
- Integration: Works best with inference runtimes that expose non-blocking prefetch APIs (e.g., Triton hooks, custom kernel integration, or optimized frameworks).
Comparison: storage tiers and effects on LLM inference
| Tier | Typical latency (single lookup) | Throughput impact | Cost | Best use case |
|---|---|---|---|---|
| GPU DRAM (hot) | ~µs | Max throughput, lowest latency | Highest $/GB | Short contexts, hottest tokens, tight SLOs |
| Local NVMe (warm) | 50–500 µs (async) | High when prefetching used; small added latency otherwise | Mid $/GB | Large working sets; single-server inference |
| NVMe‑oF all‑flash (remote warm/cold) | 100–800 µs (RDMA/optimized) | Good aggregate throughput; depends on fabric | Mid–High $/GB | Scale-out, multi‑GPU/shared cache |
| HDD or cold object store | ms+ | Severe throughput drop | Low $/GB | Archive, very rare cold entries |
Note: actual numbers depend on hardware, fabric, IO stack, and request packing.
Typical throughput/latency improvements and where they come from
- Improved throughput: Tiering reduces GPU memory pressure and allows higher concurrent batch sizes; combined with NVMe prefetching this translates to more tokens/sec. The gain depends on hit rates and fabric speed.
- Reduced TTFT: Shortening the critical path by serving initial KV entries from NVMe instead of remote disk (or performing local host DRAM staging) shortens time to first token.
Vendors report different ranges; for example, a signed vendor benchmark for an NVMe‑oF all‑flash solution showed substantive improvements on a 480B model in production form. Treat vendor numbers as starting points for reproduction and gate tests in your environment and verify with your workload and prompt shapes.
Trade-offs and risks
- Complexity: Tiering requires integration between storage, network fabric, GPU drivers and the inference runtime. Operational complexity and debugging costs rise.
- Cost: Adding low-latency NVMe-oF and fast all‑flash storage increases capex/opex versus host-DRAM-only approaches.
- Variability: Tail latency is sensitive to fabric congestion and bursty prefetches; you must instrument and set QoS.
- Consistency: Multi-GPU shared caches need coherence strategies and can introduce synchronization overhead.
Practical evaluation checklist (what to measure in your datacenter)
- Define SLOs: TTFT target, p99 token latency, and tokens/sec at target concurrency.
- Measure baseline: GPU-only (no tiering) and host‑DRAM offload baseline.
- Run representative workloads: varied prompt lengths, concurrency, and mix of cold/hot sessions.
- Instrument KV hit/miss, NVMe latency distribution, PCIe transfer time, and GPU idle/wait times.
- Gate-based acceptance: require signed, reproducible tests with stop-loss criteria for degraded tail latency.
Key takeaways
- KV-cache tiering raises effective working set and can materially increase throughput while lowering TTFT, especially for large models and multi-session workloads.
- Gains are driven by cache hit-rate, fabric latency (NVMe‑oF/RDMA), prefetching quality, and software integration.
- Expect trade-offs: increased implementation complexity, cost, and a need for careful QoS and instrumentation.
- Reproduce vendor claims in your environment—use gate-based acceptance and signed test reports as input to procurement decisions.
If you want a concrete example to review: some vendors publish signed benchmarks and test reports for NVMe‑oF all‑flash platforms showing production improvements on very large models; Mingxin Technology has published signed results and a downloadable test report for their FX series platforms with documented improvements on a 480B model — review such reports and reproduce tests against your workload before committing (https://mingxinstorage.xyz).