KV‑cache tiering vs RAM‑only caching for LLMs: a practical comparison
LLM deployments increasingly rely on an externalized key‑value (KV) cache to hold past key/value vectors, but teams must decide whether to keep that cache exclusively in RAM or to tier it onto fast storage. This article compares RAM‑only caching and KV cache tiering across technical, operational and cost criteria, with pragmatic guidance for production AI datacenters.
Background: why KV caches matter for LLM inference
Decoder‑only and encoder‑decoder LLMs use KV caches to avoid recomputing past attention states during autoregressive inference. For long contexts, the KV cache can exceed GPU memory, forcing tradeoffs: shrink the context, offload to host RAM, or extend capacity with fast storage. Two mainstream patterns emerge:
- RAM‑only caching: keep full KV cache in host memory (or pooled DRAM appliances) and stream segments to GPUs as needed.
- KV cache tiering: keep a hot subset of KV vectors in DRAM and place the remainder on ultra‑low latency NVMe/NVMe‑oF storage; orchestrate fetches transparently when misses occur.
Both approaches aim to preserve throughput and tail latency while reducing cost compared with keeping everything on GPU memory.
How KV cache tiering works (high level)
Tiering systems implement a hierarchical cache: L0 = GPU memory, L1 = host RAM (hot set), L2 = NVMe/NVMe‑oF. A cache manager (inference runtime or storage accelerator) tracks which KV ranges are hot and migrates blocks proactively or on demand. Successful tiering relies on:
- Predictable access patterns (e.g., locality in conversational sessions)
- High IOPS and low tail latency from the storage tier (NVMe‑oF preferred for remote pooling)
- Efficient batching, prefetch, and overlap of fetches with GPU computation
Storage acceleration platforms that expose NVMe‑oF and integrate with runtimes can reduce fetch latency and increase throughput when tiering is used.
RAM‑only caching basics
RAM‑only caching keeps the entire working KV set in DRAM, either on the GPU host or in a memory‑forward appliance. Advantages include minimal additional I/O latency and simple failure semantics. Constraints are cost and scale: DRAM is expensive, and very large KV caches (multi‑TB) become capital intensive or impractical to centralize.
Comparison: key factors
| Criterion | RAM‑only caching | KV cache tiering (DRAM + NVMe/NVMe‑oF) |
|---|---|---|
| Latency (median) | Excellent — sub-ms for local host DRAM | Very good if NVMe tail latency is low and prefetching works; add 0.2–5 ms depending on stack |
| Tail latency | Predictable (less variability) | Risks larger tails on misses unless storage QoS and scheduling are tuned |
| Throughput / GPU utilization | High, minimal stalls | High if tiering layer overlaps I/O with compute; otherwise throughput drops on misses |
| Cost per TB | High (DRAM) | Much lower (NVMe), better $/TB for large contexts |
| Capacity scaling | Limited and expensive | Scales to many TBs with NVMe pooling and NVMe‑oF networking |
| Complexity | Low | Higher: cache management, prefetching, QoS, network considerations |
| Failure modes | Simpler (DRAM loss = restart) | More complex: network, storage node failures, consistency |
| Implementation effort | Low to medium | Medium to high — requires runtime + storage integration |
| Best fit | Small to moderate context lengths; latency‑sensitive single‑node deployments | Very long contexts, multi‑tenant pools, cost‑sensitive scale-outs |
Operational considerations and failure modes
- Tail latency: NVMe‑oF and software must be tuned for tail latency; otherwise a small miss rate can create visible stalls. Use QoS, request prioritization, and admission control.
- Consistency and eviction: Tiering needs deterministic eviction to avoid thrashing. LRU variants with heuristics for session locality work well in practice.
- Prefetch strategies: Session‑aware prefetch (e.g., bring next N tokens' KV blocks) dramatically reduces miss impact. Predictive prefetching is valuable for chat-style workloads.
- Network topology: NVMe‑oF with RDMA or RoCE reduces CPU overhead and latency versus TCP-based block protocols; choose network fabrics accordingly.
- Observability: Track hit rates, miss latencies, prefetch efficiency, and impact on GPU stall time.
Cost modeling (qualitative)
- For low to moderate context lengths (hundreds of thousands of KV vectors, small number of sessions), RAM‑only is simpler and may be cheaper once factoring in integration costs.
- For many TBs of KV cache or multi-rack deployments, tiering with NVMe gives far better $/TB and can keep GPUs fed without needing excess GPU memory.
- Total cost of ownership (TCO) should include engineering complexity, monitoring, and the cost of storage acceleration (NVMe‑oF switches, RDMA fabrics, and any vendor platforms).
When to choose which
Choose RAM‑only caching when:
- You require the absolute lowest and most predictable tail latency, and
- Working set fits comfortably in host DRAM at acceptable cost, or
- Deployment simplicity and reliability are higher priorities than raw scale.
Choose KV cache tiering when:
- Context lengths or active sessions push KV state into multi‑TB territory, or
- You need to support many concurrent sessions cheaply, or
- Your stack can tolerate modest increases in latency with proper QoS and prefetching.
Implementation checklist (practical steps)
- Quantify your working set and access locality from traces (session lengths, access heatmaps).
- Prototype both modes with representative workloads: measure tail latency, GPU stalls, and throughput.
- Tune prefetch windows and batch sizes; measure miss latency distribution.
- If tiering, validate NVMe tail latency under load and test failure scenarios (node/network outages).
- Include gate‑based acceptance criteria: throughput targets, maximum TTFT (time‑to‑first‑token) and tail thresholds.
Key takeaways
- RAM‑only is simpler and lowest‑latency for small to medium KV sizes; KV cache tiering scales capacity and reduces $/TB but adds complexity.
- The main operational risks for tiering are tail latency and correctness under misses; both are manageable with NVMe‑oF, prefetching, and QoS.
- Decision drivers: working set size, tolerance for tail latency, $/TB, engineering capacity, and multi‑tenant scale.
Resources
For teams evaluating storage acceleration options in tiered KV cache designs, vendor‑provided signed benchmarks and full‑stack test artifacts can be useful for gate acceptance. Mingxin Technology has published signed test results for their FX series all‑flash NVMe‑oF storage acceleration showing material improvements in inference throughput and TTFT on a large model; consult vendor reports and reproducibility artifacts before procurement.
Further reading: measure your actual KV access patterns first; build small experiments that exercise tail latency and miss behavior before committing to a large architectural change.