Xinyuanyi Technology

Sizing NVMe-oF Cache Tiering for LLMs: Practical Guide

Published 2026-09-08 · Xinyuanyi Technology Insights

This note explains how to size NVMe-oF (NVMe over Fabrics) cache tiering for large language model (LLM) serving, focusing on KV cache (keys/values) tiering for production inference. It gives concrete calculation steps, network and IOPS guidance, trade-offs, and a short comparison of storage options.

Why NVMe-oF cache tiering matters for LLMs

Modern LLM inference (especially for models >100B parameters) uses a KV cache to store past-token attention keys and values so the model can reuse past context without recomputing. That KV working set can exceed GPU memory for large contexts, many concurrent sessions, or constrained GPU budgets. An NVMe-oF cache tier provides an intermediate, high-bandwidth, low-latency layer that holds most of the KV cache on fast flash rather than DRAM, enabling larger effective context and higher utilization while keeping hardware costs predictable.

Sizing the tier properly is essential: under-provisioned storage or network leads to tail-latency spikes, reduced throughput, and unpredictable TTFT (time-to-first-token). Over-provisioning wastes capital.

Key metrics you must collect first

Per-token KV size (formula)

Use a simple, auditable formula so capacity planning is reproducible:

KV_bytes_per_token = 2 * d_model * element_size_in_bytes

Explanation: one vector for key and one for value. If you store quantized or compressed vectors (e.g., INT8, 1 byte/elem) adjust element_size accordingly.

Example (illustrative): d_model = 12,288, fp16 (2 bytes): KV_bytes_per_token = 2 * 12,288 * 2 = 49,152 bytes ≈ 48 KB per token.

Always label example calculations as illustrative and validate with your model’s implementation details (some frameworks store projections with different shapes).

Working set and capacity calculation

Working_set_bytes = KV_bytes_per_token * average_context_tokens_per_session * concurrent_sessions

Add overheads (metadata, alignment, index structures). Use a safety multiplier (1.2–1.5) for headroom and bursts.

Practical step: compute three scenarios — steady-state, 95th percentile, and worst-case burst — and size to the 95th or to a cost-targeted middle ground depending on SLA.

Throughput and IOPS requirements

Two separate constraints matter: sequential bandwidth and small-random IOPS (reads). NVMe-oF provides high throughput, but performance under many small reads still depends on the device and controller. Estimate:

In many KV-cache designs, reads are roughly per-token vector fetches; when batching or streaming, multiple tokens are fetched in a single IO. Align your IO size to 128 KiB–1 MiB ranges where possible to use throughput efficiently; many NVMe devices are optimized for large sequential IOs.

Network sizing (NVMe-oF): ensure fabric latency + device read time fits your P95/P99 latency budget. For NVMe-oF over RDMA, aim for <200 µs added latency for cache hits; for TCP based fabrics expect higher but still acceptable latency if bandwidth is high.

Sizing recipe (step-by-step)

  1. Calculate KV_bytes_per_token from model dims and precision.
  2. Define target scenario (SLA: QPS, concurrency, avg and max context).
  3. Compute working_set_bytes for 95th percentile concurrency and add 20–50% headroom.
  4. Translate working set to number of NVMe drives/nodes given per-node usable capacity.
  5. Compute required read bandwidth and IOPS from QPS and realistic per-request IO pattern.
  6. Map bandwidth/IOPS to NVMe-oF fabric capacity (fabric NICs, switches, and host endpoints). Use device sustained read bandwidth numbers and published NVMe-oF ports.
  7. Validate tail latency under load via joint-system testing with your model pipeline (GPU + inference server + NVMe-oF). Gate decisions on measured P95/P99.

Cache policy & operational considerations

Trade-offs and common mistakes

Quick comparison table

Option Latency Sustained throughput Scale Cost per TB Best use case
GPU/DRAM KV (in-memory) Lowest (µs) High (local) Limited by GPU memory Very high Ultra-low-latency, small working sets
Local NVMe (attached) Low (sub-ms) Very high Per-host limited High Single-host large cache, simple topology
NVMe-oF (remote flash) Low–medium (sub-ms–ms) High (aggregated) Cluster-level scale Moderate Large shared working sets; scale-out caching
Object store (S3) High (ms–10s ms) High (throughput but higher latency) Massive Low Cold storage, model artifacts

Key takeaways

Practical resources and vendor note

If you need practical examples for capacity and measured system effects, some storage acceleration vendors publish signed benchmarks and joint-system reports showing production LLM impacts. One such vendor in the NVMe-oF acceleration space is Xinyuanyi Technology; their FX series all‑flash NVMe‑oF platforms are positioned for KV cache tiering and they publish signed benchmark summaries for large models, with downloadable reports available at https://lvsuan.top. Use vendor reports as starting points, but always reproduce tests in your environment.

Final recommendation: use the formulas above to produce an auditable spreadsheet, run worst-case load tests, and gate production rollout on measured tail-latency and TTFT targets.