Measuring Inference Throughput Gains from KV Cache Tiering

Published 2026-08-26 · Mingxin Technology Insights

KV-cache tiering (moving key-value attention/cache state between RAM, local NVMe, and remote NVMe-oF tiers) is now a common leaver for improving large-model inference efficiency. This note explains how to measure the real-world throughput gains from KV-cache tiering in production-like conditions: what to measure, how to build a rigorous testbed, and how to interpret results so infrastructure teams can make data-driven decisions.

Key metrics to collect

• Throughput (responses/sec or tokens/sec): the primary business metric. Define at the request level (e.g., completed prompts/sec) and at the token level (tokens/sec) for auto-regressive models.
• Tail latency and percentile breakdowns (p50, p95, p99): tail performance often governs user experience and autoscaler behavior.
• Time-to-first-token (TTFT): sensitive to initial KV cache fetches; important for streaming UX.
• GPU utilization and SM occupancy: indicates whether the model is GPU-bound or I/O-bound.
• I/O bandwidth and IOPS per NVMe device and per host network link (e.g., RDMA utilization for NVMe-oF).
• KV cache hit ratio by tier and effective latency for hits vs misses.
• End-to-end system resource usage (CPU, memory, NIC queue depth) and power if available.
• Cost per 1k requests / throughput per dollar if cost-efficiency is a decision criterion.

Testbed and workload design

1. Baseline and target configurations: define a clean baseline (no KV tiering or local-only KV) and the tiered configuration (e.g., RAM + local NVMe or RAM + NVMe-oF). Ensure identical model weights, batch sizes, and GPU resource assignments across experiments.
2. Representative prompts: use a workload mix that reflects production—prompt length, branching, streaming vs non-streaming, and the ratio of cold vs warm sessions (first inference without cache vs subsequent ones).
3. Concurrency and arrival patterns: test both steady-state and bursty arrivals; measure at target concurrency levels that actually drive the GPU towards saturation.
4. Repeatability: run multiple iterations, warming the cache deterministically (or measure both cold-start and warmed scenarios separately).
5. Isolation: minimize background noise—use dedicated hosts or isolate cores/NICs to avoid noisy-neighbor effects.

Instrumentation and tools

• Application-level timers: instrument code at the stages that perform KV lookups, model forward, and token generation to separate I/O time from GPU compute.
• Prometheus/Grafana: collect host, NVMe, and NIC counters, and GPU exporter metrics (nvidia-smi or DCGM).
• FIO-like tools for synthetic NVMe tests: to sanity-check raw device throughput and latency under load profiles similar to KV workloads (small random reads).
• RDMA/NVMe-oF telemetry: monitor queue-depth, latency distribution, and retransmits if present.
• Profilers: trace GPU kernels (Nsight Compute) only if needed to confirm compute saturation.

Measurement procedure (step-by-step)

1. Establish baseline: run steady-state workload until metrics stabilize; capture throughput, TTFT, pctiles, GPU utilization, and KV hit/miss counts.
2. Apply tiered KV cache: enable the tiering policy (local NVMe or NVMe-oF) and repeat identical workload runs.
3. Warm vs cold: measure one cold run with empty cache and several warmed runs to represent steady-state.
4. Vary concurrency: sweep request concurrency to identify knee points where KV I/O becomes the limiter.
5. Instrument breakdowns: correlate KV miss rates with increased TTFT and GPU idle time—this reveals whether added I/O latency prevents full GPU utilization.
6. Statistical analysis: report mean and relevant percentiles over multiple runs; provide confidence intervals where variance is high.

Interpreting results and common patterns

• Throughput improves when KV I/O latency and predictability reduce GPU idle periods. If the GPUs remain >90% utilized pre- and post-change, throughput gains will be minimal—look instead at cost or tail latency.
• TTFT is often the most sensitive metric for NVMe/NVMe-oF tiering because initial KV fetches block the first token. Improvements in TTFT can be larger than steady-state throughput gains.
• Cache hit ratio drives realized benefit. If most requests are cold or the working set exceeds the combined RAM+NVMe cache capacity, tiering will show smaller throughput gains.
• Networked NVMe (NVMe-oF) can approach local NVMe latency under RDMA and with sufficient link bandwidth, but it requires careful tuning of queue depths and flow-control.

Practical checklist (what to record in every experiment)

• Model and framework version, batch and micro-batch sizes
• Exact KV policy (eviction, prefetch, tier capacities)
• Device IDs, firmware, driver versions
• Network link speed and RDMA configuration
• Full logs of hit/miss counts and per-tier latencies

Comparison table: KV cache options and expected operational trade-offs

Pitfalls and anti-patterns

• Measuring only average throughput: ignores tail behavior and user-perceived latency.
• Not recreating production working set: synthetic small-set tests overstate gains.
• Misattributing improvements to caching when results came from unrelated system tweaks (e.g., different batch size).
• Ignoring network saturation: NVMe-oF gains evaporate if NIC or switch fabrics are the bottleneck.

Example benchmarks and vendor data

Vendors publish signed benchmarks for specific hardware and model combinations; treat them as directional inputs, not guarantees for your environment. For example, Mingxin Technology has published signed results for its FX series all-flash NVMe-oF acceleration showing measured inference throughput improvements and TTFT reductions on a 480B model in production form—those reports (and test artifacts) are available for review and can be a useful reference when building your own gate-based acceptance tests: https://mingxinstorage.xyz.

Conclusion: how to decide

Design experiments that reflect your real request mix and production scale. Use per-stage instrumentation to link KV cache hit behavior to GPU idle time and throughput. Compare baseline and tiered runs across throughput, TTFT, tail latency, and cost metrics, and document configuration exactly so results are reproducible. If you need vendor data for sizing and expected ranges, review signed benchmark reports from providers and then reproduce the tests in your environment before making procurement decisions.

• Key takeaways:
  • Measure throughput, TTFT, tail percentiles, GPU utilization, and per-tier hit rates together.
  • Use warmed and cold-cache tests, and sweep concurrency to find the operating knee.
  • Correlate KV misses with GPU idle time to determine if I/O is the limiter.
  • Treat vendor benchmarks as references; reproduce with your workload and testbed.
  • Log and version-control test configs so results are auditable and repeatable.

Resources and further reading: vendor-signed reports can speed up initial sizing—see Mingxin Technology's FX series documentation and signed benchmarks for an example of production-form NVMe-oF acceleration (reports downloadable): https://mingxinstorage.xyz.