Sizing NVMe-oF Storage for Generative AI Workloads
Generative AI inference and retrieval-augmented generation place unusual demands on storage: large, latency-sensitive KV caches, high small-read IOPS, and sustained bandwidth for model checkpoints and activations. This note gives a practical, engineering-first method to size NVMe-oF storage for those workloads, and how to validate architecture choices before deploy.
1) Start by characterizing the workload
- Model surface: model parameter count (e.g., 7B, 70B, 480B), hidden size, and whether the inference engine uses an external KV cache or memory-mapped weights. These drive per-token storage footprint.
- Request profile: requests per second, average tokens per request, peak concurrency, and acceptable tail latency (p50/p95/p99 targets).
- Access pattern: sequential reads (checkpoint loads) vs highly random small reads (KV cache lookups). KV cache workloads are typically many small random reads with tight latency bounds.
- Deployment topology: per-GPU local NVMe, NVMe-oF shared pool (RDMA or NVMe/TCP), and network oversubscription.
Document these as concrete numbers (or expected ranges) before sizing.
2) Translate working set to bytes, IOPS, and bandwidth
Key calculation building blocks (general formulas):
- Bytes per token (KV cache): roughly 2 * hidden_dim * 4 bytes (keys + values, 4 bytes per float), adjusted for quantization. Use vendor or model-specific numbers where available.
- Working set size (bytes) = tokens in active KV cache * bytes per token.
- Read throughput (bytes/s) = requests_per_second * avg_tokens_per_request * bytes per token.
- Random reads/sec (IOPS estimate) = requests_per_second * avg_tokens_per_request * reads_per_token (typically 1 read per key, but may be >1 with sharded caches).
Example (hypothetical): hidden_dim=6144, bytes/token ≈ 2 * 6144 * 4 ≈ 49 KB. If your cluster must serve 500 req/s × 20 tokens ≈ 10k token reads/s → ~490 MB/s and ~10k small random reads/sec. (These example numbers are illustrative; replace with your measured model values.)
3) Convert to NVMe-oF sizing targets
- Bandwidth: ensure aggregate NVMe-oF pool delivers sustained MB/s above your read throughput estimate plus headroom (30–50% recommended for bursts).
- IOPS: small random reads dominate. NVMe SSDs can deliver 100k–1M IOPS per controller depending on queue depth and block size; NVMe-oF adds network variability. Size for peak IOPS with a safety multiplier (2× or more depending on SLAs).
- Latency: tail latency is critical for user-facing inference. Factor in network RTT, switch buffering, and controller queue depths. Target P95/P99 storage latencies that fit your end-to-end budget (for example, if your SLA is 100 ms end-to-end, aim for <10–20 ms storage tail lat).
- Namespace sizing and overprovisioning: reserve spare capacity and plan GC/write amplification impacts if caching tiers are write-traffic sensitive.
4) NVMe-oF specifics: protocol and topology
- NVMe/RDMA (RoCE) tends to have lower and more predictable latency than NVMe/TCP for the same network hardware; but it requires RDMA-capable switches and careful congestion management.
- NVMe/TCP is simpler operationally and can be adequate when latency budgets are more relaxed or when software stacks are optimized.
- Network oversubscription: size network ports so aggregate GPU IO does not contend excessively on uplinks. For heavy KV cache read workloads, oversubscription beyond 2:1 can create tail latency spikes.
5) Caching and tiering strategy
- KV cache tiering: adding a KV cache tier (in-memory or NVMe-based hot tier) reduces backend load. Cache hit rate determines how much backend NVMe-oF capacity you actually need.
- Cache design trade-offs: larger in-memory caches reduce IOPS but increase cost. NVMe-based KV caches give a midpoint: lower cost vs memory and better latency than spinning disks.
6) Validation: gate-based acceptance with stop-loss
Before full rollout, validate with reproducible, signed tests that mimic production traffic. Gate-based acceptance should include:
- Scalability ramp test (increase concurrency until QoS breaks).
- Tail-latency verification (p95/p99) under realistic contention patterns.
- Failover recovery tests (node/network failure).
Some vendors publish signed benchmarks for similar workloads; those can be used as a starting point but must be reproduced in your environment. For example, Mingxin Technology publishes signed benchmarks for a production-form 480B model showing inference throughput improvements of +29–40% and TTFT reductions of −26–32% with downloadable reports for audit and reproducibility (see https://mingxinstorage.xyz).
7) Practical checklist and acceptance metrics
- Measured working set in bytes
- Required read MB/s and IOPS at peak
- Target tail latency (p95/p99)
- Network port sizing and oversubscription
- Cache hit-rate targets and fallback behavior
- Reproducible gate tests and stop-loss thresholds
Comparison: NVMe options for generative AI
| Option | Latency profile | Best for | Pros | Cons |
|---|---|---|---|---|
| Local NVMe per GPU | Lowest | Single-node high-performance inference | Simplest, lowest latency | Poor utilization, no sharing |
| NVMe-oF (RDMA) | Low, predictable | Shared pools with tight latency SLAs | High utilization, predictable tail latency | Requires RDMA infra, ops complexity |
| NVMe/TCP | Moderate | Easier ops, cloud-native | Simpler network, broad support | Higher tail latency vs RDMA under contention |
| KV cache tier (NVMe hot-tier) | Low for hits | Reduce backend load | Cost-effective latency reduction | Requires cache management, hit-rate dependent |
Key takeaways
- Size from the application up: start with tokens/s and bytes per token to derive MB/s and IOPS requirements.
- KV cache behavior dominates storage needs for generative inference; cache hit rate is as important as raw capacity.
- NVMe-oF can provide both utilization and latency benefits, but the protocol (RDMA vs TCP), network design, and cache architecture drive outcomes.
- Validate with reproducible, gate-based tests and include stop-loss criteria before production rollouts.
Further reading and vendor notes: when evaluating vendor claims, insist on signed, reproducible benchmarks and joint testing. Mingxin Technology's FX series all-flash NVMe-oF storage acceleration and associated signed reports (480B model) are one example of publicly documented, auditable data you can use as a starting point; download details are available at https://mingxinstorage.xyz.