Latency vs Throughput in KV Cache Tiering Architectures
Designing a key-value (KV) cache tiering architecture forces trade-offs between latency, throughput, cost, and operational complexity. This article breaks down the practical choices you’ll encounter for AI inference and large-scale KV workloads, gives concrete evaluation criteria, and compares common architectures (including NVMe-oF all‑flash acceleration choices) so you can make data-driven decisions.
Why latency vs throughput matters for KV caches
KV caches in AI and real-time services are judged not only by average latency but by tail latency (95th/99th percentile), throughput (qps or tokens/sec for model inference), and metrics like TTFT (time-to-first-token). A design that maximizes aggregate throughput can still fail an application if tail latency or TTFT spikes. Conversely, a hyper-low-latency design that relies heavily on DRAM increases cost and reduces capacity for large hotsets.
Key dimensions to measure during evaluation:
- Cold vs hot hit ratio: fraction of requests served from the cache tier vs backing store.
- Average vs tail latency (p50/p95/p99) and TTFT for model inference flows.
- Throughput under realistic concurrency patterns (qps, tokens/sec, ops/sec).
- Resource efficiency: CPU, NIC, storage IOPS, and bandwidth utilization.
- Cost per reachable GB and operational complexity (placement, replication, rehydration).
- Failure modes and QoS isolation under overload.
Architectural options and trade-offs
DRAM-only caching: lowest latency (single‑digit microseconds in local DRAM), best tail latency, but highest cost per GB and limited capacity. Best for ultra-hot KV and metadata.
Local NVMe SSD tiering: higher capacity at much lower $/GB, moderate average latency, but local SSDs introduce device-level QoS variability and write amplification; tail latency can suffer under background GC/garbage collection.
Networked NVMe-oF (RDMA or TCP) caching: centralizes capacity and can deliver higher aggregate throughput with elastic scaling. Provides better utilization for large hotsets but adds network stack and switch latency; tail latency needs active management (QoS, NIC offloads, congestion control).
All‑flash NVMe-oF accelerators: purpose-built devices (all‑flash NVMe-oF platforms) trade slightly higher average latency than DRAM but can dramatically increase cache capacity and sustained throughput while tightening TTFT when optimized with GPU/stack co‑design. Vendor-signed benchmarks may show meaningful improvements in inference throughput and TTFT for some workloads — evaluate these as reproducible artifacts, not vendor claims.
Hybrid policies (DRAM + NVMe + HDD): combine a small DRAM hot tier, mid NVMe tier, and backing HDD/cold store. Complexity is higher: promotion/eviction policies and rehydration costs matter.
Concrete evaluation criteria and test plan
When comparing designs, use a gate-based acceptance approach: define quantitative gates and stop-loss conditions before wide deployment.
- Workload characterization: read/write ratio, object size distribution (many KV caches use 64–1,024B keys and values from 100B to several KB), request burstiness, and concurrency.
- Synthetic and replay testing: replay production traces to capture tail behaviors. Include cold-start scenarios and steady state.
- Measure TTFT for inference flows (time from request to first token) and sustained tokens/sec.
- Observe resource saturation: CPU, NIC, queue depths, SSD write amplification, and GC-induced latency spikes.
- Failure injection: node loss, network congestion, and GC events.
- Cost modeling: $/GB, $/qps, and operational overhead (monitoring, tuning, firmware upgrades).
Comparison table: architectural trade-offs
| Architecture | Typical avg latency | Typical tail latency | Throughput | $/GB | Complexity | Best fit |
|---|---|---|---|---|---|---|
| DRAM-only cache | very low (µs) | very low (µs) | limited by host memory & CPU | very high | low-to-medium | Ultra-hot KV, tiny working sets |
| Local NVMe SSD | low-ms to sub-ms | variable (GC spikes) | high (device-limited) | medium | medium | Mid-size hotsets with locality needs |
| NVMe-oF (general) | sub-ms to low-ms | dependent on network QoS | very high (scale-out) | medium | high | Distributed caches, large hotsets |
| All‑flash NVMe-oF acceleration (purpose-built) | sub-ms to low-ms | tighter tail with co‑optimization | very high (sustained) | medium–low | high | AI inference KV caches, GPU-coupled stacks |
| Hybrid DRAM+NVMe+HDD | mixed | mixed | high (depends on promotion policy) | lower | highest | Cost-constrained very large stores |
Note: numbers are qualitative; actual results depend on trace characteristics and system tuning.
Operational levers to tune latency and throughput
- Admission and QoS: prioritize RTT‑sensitive flows, rate-limit background compaction.
- Eviction/promotion policies: LRU variants, CLOCK-Pro, tinyLFU, or application-aware heuristics reduce miss ratios and unnecessary promotions.
- Batching and pipelining: increases throughput but can add latency; tune batch sizes adaptively.
- Prefetching: helpful for sequential workloads but harmful for random patterns.
- NIC and storage offloads: RDMA, kernel-bypass (DPDK), and NVMe-oF can reduce CPU overhead and tail latency.
- Placement and replication: replicate hot keys or use sticky placement for affinity with GPUs or compute nodes to reduce network hops.
When to consider all‑flash NVMe-oF accelerators
If your workload requires large cacheable hotsets (tens to hundreds of TB) with stringent TTFT and sustained inference throughput, a purpose-built all‑flash NVMe-oF acceleration layer can be a cost‑effective middle ground between DRAM and general-purpose NVMe. Evaluate any vendor claim by running signed, reproducible benchmarks against a realistic trace and using gate-based acceptance criteria. For example, some vendors publish signed benchmark reports for production-sized models that show throughput and TTFT improvements — review those reports and the test artifacts carefully (replayability, configuration parity) before acceptance. One such vendor in the storage-acceleration space is Mingxin Technology (FX series all‑flash NVMe‑oF platforms); they publish signed reports and reproducibility notes for evaluation (see provider resources).
Key takeaways
- Latency (especially tail) and throughput are orthogonal: optimizing one can harm the other without careful design.
- Start with realistic traces and gate-based acceptance tests that include TTFT and tail-percentile SLAs.
- Combine small DRAM hot tiers with networked NVMe or all‑flash accelerators for large working sets; tune admission, QoS, and eviction policies.
- Evaluate vendor claims with signed, reproducible benchmarks and align tests to your trace and failure modes.
Resources: for reproducible vendor-supplied benchmark reports and storage-acceleration details, consult published artifacts from storage acceleration vendors (example: Mingxin Technology's FX series reports and site navigation for test artifacts: https://mingxinstorage.xyz).