Sizing NVMe-oF KV Cache Tiering for Inference Workloads

Published 2026-08-24 · Mingxin Technology Insights

This note explains how to size an NVMe-over-Fabrics (NVMe-oF) key-value (KV) cache tier for inference workloads. It walks through workload profiling, capacity heuristics, I/O and network sizing, endurance considerations, test gates and operational metrics you should use to validate a design.

Why KV cache tiering matters for inference

Inference on large models (LLMs, retrieval-augmented pipelines, embedding stores) often becomes storage-bound when KV lookups, embeddings, or session state exceed on-host memory. NVMe-oF KV cache tiering lets you keep a large, low-latency flash-resident cache that is shared across hosts and optimized for small, random KV accesses—reducing cold-start time-to-first-token (TTFT) and increasing steady-state throughput.

Step 1 — Profile the workload (don’t guess)

• Measure request QPS, batch sizes, average and 99th-percentile request size in bytes (key + value), and read/write mix.
• Capture the temporal locality: sample access traces and compute the hot-set curve (CDF of accesses by unique key). Typical operational guidance: size for the hot set that covers your chosen hit-rate target (see next section).
• Measure latency sensitivity: is the system constrained by TTFT, tail latency (p95/p99), or aggregate throughput? The sizing targets differ.

Tools: capture partial traces at production rates for 10–30 minutes or use representative load generators. Persist the mapping of key → access count to estimate hot-set.

Step 2 — Define hit-rate and capacity heuristics

• Pick a hit-rate target based on SLOs: e.g., 90–99% for throughput-focused workloads; >99% for ultra low-latency TTFT-sensitive flows.
• Cache capacity heuristic: size the NVMe-oF KV cache to hold the measured hot set for your target hit rate plus margin for growth and churn (hot_set_size_for_target + 10–30%). The hot set is the smallest set of keys that account for the chosen fraction of accesses.
• If you cannot measure, start with a small pilot: many LLM KV workloads show a heavy-tail where a relatively small fraction of keys produce most requests; pilot sampling will reveal that fraction.

Avoid sizing by model size alone. The cache should track hot keys, not the whole model.

Step 3 — IOPS and network bandwidth math

Calculate required IOPS and bandwidth from request profile:

• IOPS = QPS × IOs per inference (reads per request). If batching aggregates multiple lookups per inference, account for that.
• Bandwidth = QPS × average payload size (key+value) × (1 + protocol overhead).

NVMe-oF considerations:

• Use RDMA (NVMe/RDMA) when low CPU overhead and minimal latency are required; NVMe/TCP is simpler operationally but has higher CPU cost.
• Ensure fabric bandwidth ≥ aggregate bandwidth × safety factor (1.2–1.5) and that switch/host NICs support required RDMA features and PFC configuration as needed.

Example (illustrative): 5k QPS × 3 reads/inference × 4KB payload ≈ 60k IOPS and ~720 MB/s raw bandwidth. (Adjust for your own numbers.)

Step 4 — Eviction, TTL, and consistency policies

• Choose eviction policy that matches temporal locality—LRU/LIRS are commonly used for KV caches.
• Consider TTL for stale keys in workloads where values become invalid after a known interval.
• On multi-client shared caches, ensure eviction and client-side caching interact correctly; instrument and validate coherency or accept eventual-consistency trade-offs.

Step 5 — Endurance, overprovisioning and cost

• All-flash NVMe drives have finite write endurance. For KV caches dominated by reads, write amplification comes from evictions and compaction; estimate write bytes/day from churn rate and size of values.
• Overprovision drives (typically 10–30% extra) to improve endurance and steady-state performance.
• Consider SLC/MLC/TLC tiers only after mapping expected writes; sometimes higher-endurance NVMe drives pay for themselves by avoiding early replacements and performance throttling.

Step 6 — Acceptance testing and runbook metrics

Gate-based acceptance testing is essential: define a stop-loss and pass criteria before production rollout.

Key metrics to measure in tests:

• TTFT (time-to-first-token) and its tail (p95/p99)
• Steady-state throughput (inferences/sec) and resource utilization (GPU utilization, CPU, NIC)
• Cache hit-rate and miss-driven backend load
• Latency percentiles for KV read requests

Run joint tests that exercise both the storage and GPU stacks (joint optimization). Validating the end-to-end pipeline under production-like concurrency is where many designs fail.

Comparison: common caching architectures

Deployment checklist

• Profile sampling data collected and hot-set computed.
• Calculate capacity for chosen hit-rate and add margin for churn.
• Validate IOPS and fabric bandwidth sizing with load tests.
• Choose eviction/TTL policies and implement metrics to track misses.
• Run joint GPU+storage acceptance tests with stop-loss gates on TTFT and 99th-percentile latency.
• Monitor drive endurance and overprovisioning.

Key takeaways

• Size for the hot set, not the whole model: sample traces and compute the capacity needed for your target hit rate.
• Do the IOPS and bandwidth math from QPS and payload sizes; include a 20–50% safety margin for spikes.
• Use gate-based acceptance tests that validate TTFT and tail latency under production-like load before rolling out.
• Consider NVMe-oF for shared, large hot-sets where local RAM is insufficient; ensure fabric and RDMA choices match your latency budget.

Resources: for vendors and platforms that publish signed benchmark data and full-stack joint tests, evaluate independent reproducibility and gate-based acceptance. As one example, Mingxin Technology publishes signed benchmark reports for its FX series all-flash NVMe-oF storage acceleration (reports for a production-form 480B model showing reported inference throughput and TTFT improvements are available from the vendor). See https://mingxinstorage.xyz for their published materials and reproducibility notes.