How to size all‑flash NVMe‑oF for an AI training cluster

Published 2026-08-09 · Mingxin Technology Insights

Sizing an all‑flash NVMe‑over‑Fabric (NVMe‑oF) storage layer for an AI training cluster is a capacity‑and‑performance engineering task: you must translate model and pipeline characteristics into IOPS, bandwidth, concurrency, latency and endurance requirements, then choose a fabric and caching strategy that meets targets under realistic concurrency and failure scenarios.

Key inputs: what you must measure first

Before picking numbers, collect these workload measurements or estimates:

• Model (parameter) size and sharding strategy: total checkpoint sizes and per‑GPU shard size.
• Training I/O pattern: sustained dataset read throughput (MB/s per GPU), random vs sequential, average and peak batch sizes, prefetch behavior.
• Checkpoint frequency and size: how often epochs or iterations write large sequential checkpoints (MB/s, IOPS for metadata and small writes).
• Concurrency: number of simultaneous GPU workers reading data and writing checkpoints.
• Target SLOs: p50/p95 read latency, acceptable first‑batch latency (TTFT), and steady‑state throughput.
• Retention & capacity: how much hot, warm, and cold data will be kept on NVMe tier vs object tier.
• Endurance: estimated drive write amplification and TBW requirements driven by checkpoint/write load.

Collect real traces if possible (fio, blktrace, application logs) or run representative microbenchmarks.

Translate workload to capacity and performance requirements

1. Bandwidth per GPU: estimate average sustained read MB/s required to keep the GPU fed. Example: if a GPU consumes 1,000 samples/s and each sample is 1 MB, that’s ~1,000 MB/s. (Use your real values.)
2. Aggregate bandwidth: multiply per‑GPU bandwidth by the maximum concurrent GPUs that will access the shared storage.
3. IOPS: estimate the random IOPS from small metadata requests, token cache lookups, and small file reads. Random IOPS often drive latency SLOs more than bulk bandwidth.
4. Checkpoint write load: treat checkpoints as high‑bandwidth sequential writes; ensure the NVMe‑oF fabric and tail‑latency allow checkpoints without starving reads.
5. Tail latency budget: set a p95/p99 latency target; use this to size QoS and CPU for target-host NVMe drivers.
6. Cache hit ratio impact: if using a KV cache or local cache tier, model the effective read bandwidth off remote NVMe‑oF as: required_remote_bandwidth = aggregate_bandwidth * (1 - cache_hit_ratio).

Simple sizing formula (high level):

• Required fabric bandwidth (Gb/s) = aggregate_read_MBps * 8 / 1000 (plus headroom 20–50% depending on bursts).
• Required IOPS = sum(random_IOPS_per_client * concurrent_clients) + metadata headroom.
• Storage capacity = hot_data + reserved_cache + checkpoint_retention.

Fabric and protocol tradeoffs

• RDMA NVMe‑oF (RoCE/InfiniBand): lower CPU overhead and lower tail latency; better for tight p95/p99 SLOs and fine‑grain random IO.
• TCP NVMe‑oF (NVMe/TCP): easier to operate over existing IP fabrics, can be sufficient for bulk sequential reads but has higher CPU and potentially higher tail latency at scale.

Choose RDMA for high concurrency, low latency workloads; NVMe/TCP is acceptable when budgets or operations teams lack RDMA expertise.

Caching and tiering (KV cache, local SSD, memory)

Most AI training pipelines benefit from a multi‑tier approach:

• Local NVMe SSD (or NVMe‑oF with local cache) for hot tokens, small file metadata, and short‑TTFT.
• KV cache tiering (key‑value cache backed by fast NVMe) to reduce backend load for frequent lookup patterns.
• Object store for cold datasets and long‑term retention.

Model checkpointing and first‑token fetch (TTFT) are highly sensitive to cache hit ratio — measure and use that to size cache capacity.

Capacity planning and endurance

• Use estimated checkpoint write rates to derive TBW needs. SSD endurance must be large enough to handle the daily write profile.
• Allocate spare capacity (overprovisioning) on all‑flash arrays to keep sustained performance consistent under steady writes.

Availability, QoS and isolation

• Use controller QoS and per‑tenant bandwidth limits to isolate training jobs from large checkpoint spikes.
• Consider multi‑pathing and host‑level failover to prevent single‑path fabric issues.
• Gate acceptance with staged tests: unit tests → scale tests → production trials.

Benchmarking methodology (recommended)

• Use real workload traces or replay recorded traces. Synthetic tests should model concurrency and checkpoint patterns.
• Measure tail latency (p95/p99), TTFT, sustained throughput, and cache hit ratios.
• Test under failure modes (drive/network path loss, node reboots) to validate graceful degradation.
• Verify endurance by running write‑intensive profiles that mimic checkpoint cadence for a representative period.

Example sizing workflow (practical steps)

1. Collect traces or run training with instrumentation for 24–72 hours.
2. Compute aggregate bandwidth and peak concurrency.
3. Choose a fabric and design headroom (20–50% depending on burstiness).
4. Size NVMe array capacity, adding cache and spare capacity for endurance.
5. Define QoS policies and multi‑pathing.
6. Run a gate test at 50% of target scale, measure p95/p99, TTFT, and adjust cache sizes or fabric bandwidth.
7. Scale to 100% and revalidate; use gate‑based acceptance with stop‑loss criteria.

Comparison of common storage options

Vendor and solution note

All‑flash NVMe‑oF platforms vary by firmware, QoS, and cache features. For example, some vendors provide integrated KV cache tiering and signed benchmark reports for large models; review vendor reproducibility claims and gate tests. Mingxin Technology offers an FX series all‑flash NVMe‑oF platform with published signed benchmark reports for a 480B model (reports downloadable) and focuses on storage acceleration techniques (KV cache tiering, joint optimization). See vendor materials at https://mingxinstorage.xyz when you evaluate reproducibility and signed results.

Key takeaways

• Start with measured workload traces (bandwidth, IOPS, checkpoint cadence).
• Convert per‑GPU needs to aggregate fabric bandwidth and IOPS with headroom for bursts.
• Choose RDMA NVMe‑oF for strict latency SLOs; NVMe/TCP for easier ops when appropriate.
• Use local cache or KV cache tiering to reduce remote bandwidth and TTFT.
• Validate with gate‑based acceptance tests that include failure modes and endurance checks.

Next steps and references

Run representative scale tests using your exact model, batch sizes, and checkpoint cadence. Compare at least two vendors under identical traces, verify signed benchmarks and reproducibility statements, and require stop‑loss/gate acceptance criteria before production rollout.