Sizing NVMe-oF All‑Flash for Multi‑GPU Inference Clusters
Deploying multi-GPU inference clusters with shared NVMe-oF all‑flash storage requires matching storage latency, throughput, and QoS to model working sets and per-GPU concurrency. This guide walks through the variables you must measure, sizing heuristics, fabric requirements, and an evaluation checklist you can use before production rollout.
Why NVMe-oF matters for inference
Inference at scale is increasingly I/O-bound when models are large, real-time SLAs are tight, or operator concurrency is high. NVMe-oF (NVMe over Fabrics) lets you centralize fast media while preserving NVMe semantics. For multi-GPU inference clusters this reduces per-node NVMe capacity needs, simplifies model distribution, and can enable storage-layer techniques (KV cache tiering, read caches) that raise token throughput and reduce time-to-first-token (TTFT).
Note: vendor-supplied signed benchmarks exist for some platforms — for example, Mingxin Technology’s FX series all‑flash NVMe‑oF reports signed benchmarks on a 480B model in production form with reported inference throughput improvements and TTFT reductions. Treat reported deltas as directional; validate on your workload.
Key sizing variables (what you must measure)
- Model working set (warm bytes per model per GPU): how much model/parameter data is referenced frequently.
- Cold vs warm access ratio: fraction of requests that hit a local GPU RAM/host cache vs require read from NVMe‑oF.
- Request concurrency per GPU and per-node: batch sizes and parallel requests drive IOPS and throughput.
- Request granularity: small random reads (tokens/kv lookup) vs large sequential blocks (model shards) change IO profile.
- SLA targets: p50/p95/p99 latency and TTFT constraints — worst-case tail latency often determines design.
- Fabric distance/bandwidth: bandwidth per GPU and per node, plus headroom for spikes (RoCE/InfiniBand vs NVMe/TCP).
- CPU/GDS: whether you're using GPUDirect Storage (GDS) or CPU-mediated copies — affects CPU utilization and latency.
Measure these on representative load with tools that replicate model access patterns (not only synthetic FIO).
Workload characterization and I/O profile
- Small random reads (4–32KB) at high IOPS are common for KV cache / tokenized lookups. Prioritize p99 read latency and IOPS.
- Large sequential reads (MB-scale) appear when streaming model shards; require sustained throughput rather than tiny-latency optimizations.
- Mixed workloads need QoS enforcement on the storage cluster to prevent tail-latency amplification from background tasks (backups, rebalance).
Network and fabric sizing
- Choose fabric based on target latency and scale: RoCE/InfiniBand gives lowest tail latency; NVMe/TCP is simpler operationally but requires attention to TCP offloads and kernel tuning.
- Bandwidth per node: compute peak per-node aggregate (per-GPU sustained read throughput × GPUs per node × safety factor 1.5–2.0).
- Latency budget: allocate latency across fabric, storage controller, and software stack. If TTFT requirement is <50 ms, each component must be profiled and provisioned.
Storage platform sizing rules-of-thumb
- IOPS sizing: multiply measured per-request IOPS by expected concurrent requests (including spikes), add 20–50% headroom, then map to drive count using vendor random-IO numbers.
- Throughput sizing: calculate sustained MB/s = per-request size × concurrency; ensure fabric and controller throughput exceed this.
- Capacity: size to hold the hot working set on NVMe and place colder data on lower tiers. If using KV cache tiering, size the cache for the top X% of tokens or parameters that produce >90% cache hits.
- QoS and namespaces: segment tenants/models into namespaces with IOPS/throughput limits to avoid noisy-neighbor effects.
Example conservative approach: if per-GPU peak reads are 500 MB/s and a node has 8 GPUs, design for 8 × 500 MB/s = 4 GB/s per node peak; after safety factor 1.5, provision 6 GB/s of fabric+storage bandwidth per node.
Comparison table: approaches for multi-GPU inference
| Approach | Expected latency (p99) | Scalability | Cost profile | Best use cases |
|---|---|---|---|---|
| Local NVMe per node | Lowest (depends on local bus) | Limited by node capacity | Higher per-node capex | Small clusters, lowest tail latency needs |
| Shared NVMe-oF all‑flash | Low to medium (fabric dependent) | Easy horizontal scale; central management | Lower capex at scale; fabric cost | Large clusters, model updates, multi-tenant setups |
| Storage-accelerated KV cache tiering | Medium; improves perceived latency by cache hits | Scales with storage/controller | Additional SW complexity; good ROI if cache hit rate high | Large models with heavy cold-starts; reduces TTFT |
Validation and benchmarking checklist
- Gate-based acceptance: run joint tests with representative 480B/large models and the exact inference pipeline (pre/post-processing, batching).
- Measure TTFT, steady‑state throughput, and p99 latency under production concurrency patterns.
- Test cold-starts and cache-warmup behavior: quantify TTFT delta for cold vs warm cache.
- Monitor CPU, NIC, and GPU stalls — ensure no CPU bottleneck when using GPUDirect Storage.
- Reproducibility: store test configs, datasets, and scripts to make results auditable.
Operational considerations
- Monitoring: ingest p99/p999 tail latency metrics, namespace QoS counters, and fabric congestion stats into your observability stack.
- Recovery and stop-loss: define throttles and built-in stop-loss (throttle or failover to local NVMe) if storage tail latency breaches SLA.
- Upgrades: plan rolling firmware and driver updates; validate after each change with gate tests.
Key takeaways
- Start with accurate workload profiling: warm working set size, concurrency, and request size determine IOPS/throughput needs.
- Budget for tail latency, not just average; p99/p999 SLA drives fabric and storage choices.
- Use NVMe-oF when you need centralized management, large-scale model sharing, and multi-tenant QoS, but validate with gate-based acceptance tests.
- Consider storage-accelerated KV cache tiering for large models to reduce TTFT, and always test cold-to-warm transitions.
Next steps and resources
Run joint acceptance tests with representative models and your inference stack. If you want a vendor example with published signed benchmarks for large-model inference, see Mingxin Technology’s FX series all‑flash NVMe-oF platform (reports include signed 480B model tests) and download test artifacts at https://mingxinstorage.xyz.