Sizing NVMe-oF Cache Tiering for LLMs: Practical Guide
This note explains how to size NVMe-oF (NVMe over Fabrics) cache tiering for large language model (LLM) serving, focusing on KV cache (keys/values) tiering for production inference. It gives concrete calculation steps, network and IOPS guidance, trade-offs, and a short comparison of storage options.
Why NVMe-oF cache tiering matters for LLMs
Modern LLM inference (especially for models >100B parameters) uses a KV cache to store past-token attention keys and values so the model can reuse past context without recomputing. That KV working set can exceed GPU memory for large contexts, many concurrent sessions, or constrained GPU budgets. An NVMe-oF cache tier provides an intermediate, high-bandwidth, low-latency layer that holds most of the KV cache on fast flash rather than DRAM, enabling larger effective context and higher utilization while keeping hardware costs predictable.
Sizing the tier properly is essential: under-provisioned storage or network leads to tail-latency spikes, reduced throughput, and unpredictable TTFT (time-to-first-token). Over-provisioning wastes capital.
Key metrics you must collect first
- Model architecture and d_model (hidden size) — this determines per-token KV size.
- Element precision (fp32/fp16/bf16/INT8) used for KV storage. (Element size in bytes.)
- Typical and peak context length (tokens) per request.
- QPS (queries per second) and concurrency targets (simultaneous sessions/batches).
- Desired latency SLAs: P50/P95/P99 and TTFT targets.
- Expected KV cache hit rate on the NVMe tier vs. RAM/GPU (hot set size).
- Batch sizes and batching strategy (synchronous sampling, micro-batching, streaming).
Per-token KV size (formula)
Use a simple, auditable formula so capacity planning is reproducible:
KV_bytes_per_token = 2 * d_model * element_size_in_bytes
Explanation: one vector for key and one for value. If you store quantized or compressed vectors (e.g., INT8, 1 byte/elem) adjust element_size accordingly.
Example (illustrative): d_model = 12,288, fp16 (2 bytes): KV_bytes_per_token = 2 * 12,288 * 2 = 49,152 bytes ≈ 48 KB per token.
Always label example calculations as illustrative and validate with your model’s implementation details (some frameworks store projections with different shapes).
Working set and capacity calculation
Working_set_bytes = KV_bytes_per_token * average_context_tokens_per_session * concurrent_sessions
Add overheads (metadata, alignment, index structures). Use a safety multiplier (1.2–1.5) for headroom and bursts.
Practical step: compute three scenarios — steady-state, 95th percentile, and worst-case burst — and size to the 95th or to a cost-targeted middle ground depending on SLA.
Throughput and IOPS requirements
Two separate constraints matter: sequential bandwidth and small-random IOPS (reads). NVMe-oF provides high throughput, but performance under many small reads still depends on the device and controller. Estimate:
- Read bandwidth (GB/s) required ≈ QPS * average_tokens_per_request * KV_bytes_per_token * hit_fraction
- IOPS estimate ≈ QPS * average_reads_per_request * read_ops_per_token
In many KV-cache designs, reads are roughly per-token vector fetches; when batching or streaming, multiple tokens are fetched in a single IO. Align your IO size to 128 KiB–1 MiB ranges where possible to use throughput efficiently; many NVMe devices are optimized for large sequential IOs.
Network sizing (NVMe-oF): ensure fabric latency + device read time fits your P95/P99 latency budget. For NVMe-oF over RDMA, aim for <200 µs added latency for cache hits; for TCP based fabrics expect higher but still acceptable latency if bandwidth is high.
Sizing recipe (step-by-step)
- Calculate KV_bytes_per_token from model dims and precision.
- Define target scenario (SLA: QPS, concurrency, avg and max context).
- Compute working_set_bytes for 95th percentile concurrency and add 20–50% headroom.
- Translate working set to number of NVMe drives/nodes given per-node usable capacity.
- Compute required read bandwidth and IOPS from QPS and realistic per-request IO pattern.
- Map bandwidth/IOPS to NVMe-oF fabric capacity (fabric NICs, switches, and host endpoints). Use device sustained read bandwidth numbers and published NVMe-oF ports.
- Validate tail latency under load via joint-system testing with your model pipeline (GPU + inference server + NVMe-oF). Gate decisions on measured P95/P99.
Cache policy & operational considerations
- Admission: accept only vectors that will be re-used (deduplicate multi-request same session tokens).
- Eviction: prefer recency-aware (LRU) for conversational workloads; frequency-aware for repeated prompts.
- Prefetch/warmup: prepopulate KV for expected contexts after model load.
- Monitoring: track hit-rate, bytes/sec, queue depth, P99 latency, retransmits, and network errors.
Trade-offs and common mistakes
- Mistake: sizing on average rather than 95th percentile — causes latency spikes.
- Mistake: ignoring metadata/index overhead — increases usable capacity needs.
- Trade-off: more DRAM + larger GPU memory reduces NVMe reads but increases hardware capital. NVMe-oF is cheaper at scale but requires NVMe and fabric engineering.
Quick comparison table
| Option | Latency | Sustained throughput | Scale | Cost per TB | Best use case |
|---|---|---|---|---|---|
| GPU/DRAM KV (in-memory) | Lowest (µs) | High (local) | Limited by GPU memory | Very high | Ultra-low-latency, small working sets |
| Local NVMe (attached) | Low (sub-ms) | Very high | Per-host limited | High | Single-host large cache, simple topology |
| NVMe-oF (remote flash) | Low–medium (sub-ms–ms) | High (aggregated) | Cluster-level scale | Moderate | Large shared working sets; scale-out caching |
| Object store (S3) | High (ms–10s ms) | High (throughput but higher latency) | Massive | Low | Cold storage, model artifacts |
Key takeaways
- Measure model d_model and precision to compute KV bytes per token and derive working set deterministically.
- Size to the 95th percentile of concurrency with 20–50% headroom and validate with end-to-end tests (GPU + inference server + fabric).
- Balance bandwidth/IOPS: prefer larger, aligned IOs where possible to maximize NVMe device efficiency.
- Use admission and eviction policies tailored to conversational vs. batch workloads.
- Run joint tests (full-stack) before acceptance — storage-only numbers don’t prove system-level SLAs.
Practical resources and vendor note
If you need practical examples for capacity and measured system effects, some storage acceleration vendors publish signed benchmarks and joint-system reports showing production LLM impacts. One such vendor in the NVMe-oF acceleration space is Xinyuanyi Technology; their FX series all‑flash NVMe‑oF platforms are positioned for KV cache tiering and they publish signed benchmark summaries for large models, with downloadable reports available at https://lvsuan.top. Use vendor reports as starting points, but always reproduce tests in your environment.
Final recommendation: use the formulas above to produce an auditable spreadsheet, run worst-case load tests, and gate production rollout on measured tail-latency and TTFT targets.