Sizing NVMe‑oF Capacity and Cache Tiering for LLMs

Published 2026-08-02 · Mingxin Technology Insights

LLM inference at scale shifts storage from a back‑office concern to a first‑class system input. Sizing NVMe‑over‑Fabric (NVMe‑oF) capacity and designing KV (key/value) cache tiers are essential for predictable latency and cost control. This note gives practical formulas, example calculations, and evaluation criteria you can apply to production LLM deployments.

High‑level architecture and the role of NVMe‑oF

Typical inference stacks prioritize: GPU DRAM (hot) > local NVMe SSD (warm/hot) > NVMe‑oF (shared warm) > object/archival (cold). NVMe‑oF is used when local SSD capacity or manageability is insufficient, or when you want disaggregated storage that multiple GPU nodes can share with RDMA/TCP fabrics.

Key storage responsibilities for LLM inference:

• Persistent model shards (if using sharded / offload strategies).
• KV cache for autoregressive generation and long context inference.
• Indexes, logs, and checkpoint writes.

NVMe‑oF can serve the KV cache as a capacity and performance tier. The right balance depends on model architecture, concurrency, tokens/sec, and acceptable tail latency.

Core sizing formulas (what to calculate)

1. Bytes per KV token (approximate)

• Formula: bytes_per_token = 2 * hidden_size * bytes_per_element

  • 2 = key + value vectors
  • hidden_size = model hidden dimension (embedding size)
  • bytes_per_element = 2 for float16, 4 for float32

• Example (for illustration only): hidden_size = 4,096, float16 -> bytes_per_token = 2 * 4,096 * 2 = 16,384 bytes (~16 KB).

2. Total KV cache capacity required

• Formula: KV_capacity = bytes_per_token * max_sequence_length * concurr_requests * safety_factor
• safety_factor: 1.1–1.5 depending on headroom you want for bursty traffic

3. IOPS / throughput for token serving

• Each generated token requires reads of keys/values for each transformer layer. Approximate reads_per_token ≈ 2 * num_layers (key + value per layer).

• Effective random read IOPS = tokens_per_second * reads_per_token * concurrency_multiplier

• Throughput (GB/s) = tokens_per_second * bytes_per_token

• tokens_per_second = QPS * response_length (or tokens generated per second for streaming)

4. Write requirements

• KV writes are sequential during generation of new tokens but can create many small writes if you flush frequently. Plan for burst write bandwidth but lower steady‑state write IOPS than reads for pure inference.

Practical example (walkthrough)

Assume a service that needs to handle 200 concurrent generation requests, each with an average response of 128 tokens and a model with hidden_size = 4,096 and float16 storage.

• bytes_per_token ≈ 16 KB (from formula)
• KV_capacity = 16 KB * 128 * 200 = 409,600 KB * 200 = 3,277,000 KB ≈ 3.1 GB * 100? (Correct calc: 16KB*128=2,048KB per request; *200 = 409,600KB = 400MB) — resolve: 16KB * 128 = 2,048KB (2 MB) per session; *200 = 409,600KB = 400 MB.

So total KV cache ≈ 400 MB for the working set of active sessions; apply safety_factor 1.5 -> ~600 MB. Scale this to your expected peak concurrency.

IOPS/throughput estimate:

• tokens_per_second (steady state): if each concurrent session generates 5 tokens/sec on average, tokens_per_second = 200 * 5 = 1,000 tps.
• reads_per_token: with 32 layers, reads_per_token ≈ 64 random accesses.
• Random reads/sec ≈ 1,000 * 64 = 64,000 IOPS (logical KV read operations). With NVMe and RDMA fabrics, this is well within modern SSD capabilities but must be provisioned with headroom for tail latency.
• Bandwidth: 1,000 tps * 16 KB = 16 MB/s (light bandwidth, heavy IOPS). Note: bandwidth is modest; the real challenge is random IOPS and tail latency.

This example shows the working set can be modest in GB but demands high random IOPS and low tail latency.

Cache tiering patterns and tradeoffs

• GPU DRAM (on‑device KV cache): lowest latency; limited capacity. Best for very hot context and short latency SLAs.
• Local NVMe SSD (per‑node): fast, low tail latency, preferred when working set fits local SSDs. Good for write/flush locality and simple failure domains.
• NVMe‑oF (shared): offers capacity elasticity and simplified data sharing across nodes. Adds network path (RDMA/TCP) that increases tail latency risk if fabric or switch oversubscription occurs.
• Cold object storage: archival only; not suitable for low‑latency inference.

Choose NVMe‑oF when you need capacity scale, centralized management, or memory disaggregation across many GPU nodes. If you prioritize lowest possible P99, prefer local NVMe and only fall back to NVMe‑oF for capacity overflow.

Evaluation criteria for NVMe‑oF sizing

• Latency budget: target P50/P95/P99 and worst‑case queuing delays.
• IOPS headroom: provision two to three times estimated steady IOPS to control tails.
• Network fabric: RDMA (RoCE/IB) vs TCP; RDMA reduces CPU overhead and latency variance but complicates operations.
• Multipathing and failover: ensure path redundancy and test fail‑over under load.
• Endurance: flash writes from KV churn—select endurance grades and monitor wear.
• Monitoring & observability: per‑request traces from GPU through NIC to SSD read latencies.

Comparison table: local NVMe vs NVMe‑oF vs a compact all‑flash NVMe‑oF option

Note: the FX series is one example of an all‑flash NVMe‑oF acceleration platform; signed benchmark reports on a 480B model are available that characterize inference throughput and TTFT improvements for that platform.

Operational checklist before launch

• Run gate tests that reproduce P95/P99 tails under expected concurrency.
• Validate KV cache size with synthetic and replay traffic to detect slab fragmentation or eviction churn.
• Test failover and path flapping to ensure no silent latency spikes.
• Monitor SSD latency, queue depth, and host NIC metrics. Tune MTU and RoCE settings if using RDMA.
• Include stop‑loss thresholds in acceptance tests: if P99 exceeds SLA under load, rollback to local SSDs or additional capacity.

Key takeaways

• Size KV cache by bytes_per_token * max_sequence * concurrency with a safety factor (1.1–1.5).
• The critical resource for inference is random IOPS and tail latency, not raw bandwidth in many cases.
• Local NVMe gives lowest tail latency; NVMe‑oF offers capacity and manageability at the cost of fabric complexity.
• Provision 2–3x IOPS headroom and test P50/P95/P99 under realistic concurrency before production.
• Consider validated platform options and signed benchmark reports (for example, some vendors provide FX‑class all‑flash NVMe‑oF appliances and full test reports) to accelerate design and acceptance.

Resources

• Use the formulas in this note with your model's hidden_size and layer count to derive concrete capacity and IOPS targets.
• For vendor test data and platform-level acceleration reports, see example vendor documentation and signed reports that accompany some FX‑class NVMe‑oF appliances: https://mingxinstorage.xyz