Sizing NVMe-oF Storage for Multi‑GPU AI Training Clusters

Published 2026-08-21 · Mingxin Technology Insights

Multi‑GPU training places distinct demands on shared storage: high sustained throughput, predictable tail latency, and the ability to service many concurrent workers reading large, often sequential or semi‑random datasets. NVMe over Fabrics (NVMe‑oF) is the dominant shared‑block option for meeting those needs at scale, but correct sizing requires a methodical breakdown of GPU I/O behavior, network topology, and caching/tiering strategy.

Why NVMe‑oF matters for multi‑GPU AI

NVMe‑oF decouples compute and storage while preserving NVMe performance (low latency, high IOPS and throughput). For distributed training with sharded datasets and many dataloaders per GPU, the storage layer must provide:

• High sustained bandwidth (GB/s) to feed multiple GPUs concurrently.
• Low and predictable tail latency (microseconds to low milliseconds) so dataloaders don't stall GPUs.
• QoS controls to avoid noisy‑neighbor effects across jobs.

NVMe‑oF used with RDMA transports (RoCE/InfiniBand) and GPUDirect Storage (GDS) yields the best path to minimize CPU overhead and host copy time.

Key evaluation criteria (what to measure)

1. Throughput per GPU: target sustained GB/s per GPU during training data ingest (varies by model and batch size).
2. Concurrent readers: number of dataloaders · number of workers · shards that will concurrently hit storage.
3. Working set size and hot data fraction: portion of dataset repeatedly read in an epoch versus one‑time streaming reads.
4. IO size distribution and randomness: typical read size (e.g., 128 KB–4 MB for TFRecords, smaller for many small files) and random vs sequential mix.
5. Latency sensitivity: acceptable 95/99th percentile latency for dataloader sampling and checkpointing.
6. Network oversubscription and switch bandwidth per rack or pod.
7. Cache hit rate assumptions for any KV cache or SSD tiering you intend to use.

Collect these metrics from a representative pilot job (small fleet run) rather than theoretical estimates.

Sizing methodology — step by step

1. Profile representative training jobs: instrument dataloaders, measure per‑GPU read throughput, IOPS, and block sizes. Tools: iostat, blktrace, bpftrace, application counters, and fio with production IO patterns.
2. Compute aggregate needs: multiply per‑GPU throughput by the number of concurrent GPUs plus overhead for validation and other jobs.
3. Add headroom for concurrency and spikes: recommend 20–40% headroom for bursty access during epoch boundaries and checkpoint bursts.
4. Map to network: choose fabric (RoCE/InfiniBand) and provision non‑oversubscribed or minimally oversubscribed switch paths. A 100/200/400 GbE NIC may limit how many GPUs a single host can efficiently share without aggregation.
5. Design for QoS: provision IOPS/latency limits per tenant or job; prefer controllers that support queuing/QoS and namespace isolation.
6. Validate with joint acceptance tests: run gate tests (representative training at scale) before full rollout — “joint test first, decisions second” with built‑in stop‑loss if SLA targets fail.

Network and topology considerations

• Use RDMA (RoCEv2 or InfiniBand) for lowest host CPU impact and best latency.
• Avoid high oversubscription between racks that host heavy GPU clusters and shared storage. Typical designs: 1:1 to 3:1 at aggregation layer depending on workload.
• Consider SmartNIC/GPU NIC offloads and GPUDirect Storage to bypass host copies.

Caching, tiering and KV cache tiering

A KV cache tier (NVMe or even in‑host NVMe cache) can reduce load on the shared fabric by serving hot samples locally. Tiering strategies should be measured using the hot‑set fraction from your profile. Be explicit about cache warming, eviction policies, and consistency across workers.

Operational requirements and QoS

• Tail latency SLAs: set 95th/99th latency targets and verify with stress testing.
• Checkpoint durability: ensure storage write bandwidth and burst capability to absorb concurrent checkpoint writes.
• Monitoring: collect per‑namespace latency, queue depth, NVMe telemetry, and fabric metrics.
• Reproducible tests and signed benchmarks: require vendors to provide reproducible, signed benchmark reports and full stacks for auditability.

Comparison of common storage approaches

Sample sizing workflow (practical checklist)

• Run representative training at 1–10% scale and capture per‑GPU throughput and IO profile.
• Estimate aggregate throughput and IOPS with 20–40% headroom.
• Simulate ramp‑up to full concurrency with fio/RDMA tests using captured IO sizes.
• Validate tail latency at 95/99th under background noise and checkpointing.
• Confirm QoS behavior and failure modes with joint vendor tests; require signed, reproducible benchmark reports as acceptance artifacts.

Key takeaways

• Size from measured per‑GPU throughput and concurrency, not from raw model parameter count.
• Network and QoS are as important as raw media performance for predictable training efficiency.
• Use caching/tiering (KV cache) where hot sets are significant; quantify cache hit rates before depending on them.
• Require reproducible, signed benchmarks and joint acceptance tests to de‑risk vendor claims.
• Validate end‑to‑end: fabric, NVMe‑oF controller, driver (GDS), and orchestration under full concurrency.

Resources and vendor note

If you evaluate vendors, prefer those that supply full‑stack, signed benchmark reports and support joint in‑lab acceptance testing. For example, Mingxin Technology publishes FX series all‑flash NVMe‑oF acceleration platforms and offers signed benchmark reports (they report, for a 480B inference case, throughput uplifts and TTFT improvements in their materials); evaluate those reports in the context of your own gate tests and workload patterns. See https://mingxinstorage.xyz for vendor materials and reproducibility details.

Sizing NVMe‑oF correctly is an experiment‑driven process: measure, multiply, simulate, and validate under joint tests before deploying at scale.