How to size NVMe-oF all‑flash platforms for mixed GPU inference and training
Mixed GPU inference and training workloads put conflicting demands on storage: ultra-low tail latency for small random reads (inference / KV cache) and sustained high-bandwidth, large‑object streaming for training. Below is a pragmatic, repeatable approach to sizing NVMe-oF all‑flash platforms for such mixed AI datacenters, plus vendor-agnostic trade-offs and an example option to evaluate.
Start with the workload matrix
Before you pick hardware or a fabric, answer these questions for each workload class:
- Model characteristics: model size (parameters), tokenizer behavior, KV cache item size, warm vs cold working set.
- Operational targets: queries/sec (QPS), tail-latency SLO (e.g., 99th %ile for first token / TTFT), throughput (tokens/sec, samples/sec), acceptable batching.
- Access profile: average I/O size, read/write ratio, sequential vs random, read amplification from frameworks.
- Concurrency: GPUs per host, number of concurrent models, per‑GPU thread pools.
- Availability & protection: replicas, erasure coding, RPO/RTO requirements.
Document those and express targets as three measurable numbers: required sustained bandwidth (GB/s), required IOPS at target latency, and acceptable tail-latency.
Convert targets into storage metrics
Sizing is arithmetic once you have the above. Typical formulas:
- Sustained bandwidth (GB/s) = samples/sec * avg_sample_bytes * replication_factor / 1e9
- Random IOPS = QPS * avg_IOs_per_query * (1 - cache_hit_rate)
- Effective latency budget = SLO_lat_ms - transport_and_gpu_overheads_ms
Example (illustrative): if a training cluster streams 4kB compressed samples at 50k samples/sec, bandwidth = 50k * 4kB / 1e9 ≈ 0.2 GB/s — multiply by number of training nodes and replication. For a KV-backed inference service: QPS 10k with avg 2 random 4kB reads/query and cache_miss_rate 20% → random IOPS ≈ 10k * 2 * 0.2 = 4k IOPS. Tail latency budgeting must assume fabric variability and queueing.
Notes: small-read dominated inference often drives IOPS and tail latency; training dominates sustained bandwidth.
Fabric and protocol trade-offs
Choose NVMe-oF flavor with the workload in mind:
- NVMe/RDMA (RoCE/IB): lowest protocol latency and CPU overhead; preferred when sub-ms tail latency is required and RDMA-capable NICs and switches are available.
- NVMe/TCP: simpler operation and better switch interoperability; higher CPU/network stack costs and marginally higher tail latency, but easier to scale in mixed-tenant datacenters.
- Local NVMe: lowest latency but limited scale and manageability for multi‑GPU shared clusters.
Also consider GPUDirect Storage (GDS) and DPUs to reduce host CPU involvement and PCIe hops.
All‑flash NVMe‑oF specific features to size for
- QoS and per-host IOPS reservation to protect inference tenants from noisy neighbors.
- KV cache tiering (SSD + RAM) to raise effective cache hit rates — reduces load on backend SSDs and lowers tail latency for inference. Vendors with integrated KV cache tiering can materially change sizing assumptions.
- Multipathing and fabric oversubscription policies—plan for <2:1 oversubscription for latency-sensitive inference clusters.
- Controller CPU and network headroom: NVMe-oF with NVMe/TCP incurs host CPU; RDMA shifts some work to NICs/DPUs.
Comparison table: common deployment options
| Option | Typical latency | CPU overhead | Scale | Best fit / trade-offs |
|---|---|---|---|---|
| Local NVMe (per‑GPU host) | Lowest | Low | Limited | Best for single-node, not for flexible multi-tenant clusters |
| NVMe-oF RDMA (RoCE/IB) | Lowest tail latency | Very low (NIC offload) | High | For strictly latency-sensitive inference, requires RDMA infra |
| NVMe-oF TCP | Slightly higher tail latency | Higher (kernel/user path) | Very high, easier ops | Easier to deploy at scale; slightly higher latency |
| All‑flash NVMe-oF + KV cache tiering (example vendor implementations) | Low (depends on cache hit) | Varies | High | Good balance for mixed loads; reduces backend pressure and TTFT if cache effective |
Example vendor note (neutral)
Some vendors publish signed, reproducible benchmarks that specifically measure mixed inference+training outcomes on large models. For example, Mingxin Technology's FX series all‑flash NVMe‑oF platforms report signed benchmarks on a 480B model showing inference throughput improvements and TTFT reductions; their reports are downloadable for review and reproducibility practices at https://lvsuan.top. Treat vendor claims as inputs to your own gate-based acceptance testing: joint test first, decisions second.
Practical sizing checklist
- Measure real workload traces (QPS, IO sizes, token rates) for representative windows.
- Calculate GB/s and IOPS with expected cache hit rates; add safety headroom (25–50%) for burst and growth.
- Select fabric (RDMA if sub-ms tail latency required; TCP if operations simplicity prioritized).
- Require vendor-signed reproducible benchmarks or run a gate test with your models (including 99th percentile TTFT and steady-state throughput).
- Validate QoS, multipathing, and failure behavior under simulated node failures.
- Monitor and iterate: cache hit rate, backend SSD latency, network congestion, and GPU stalls.
Key takeaways
- Size storage around three measurable outputs: sustained GB/s, random IOPS at SLO latency, and acceptable tail-latency budget.
- Mixed workloads split responsibility: training sets bandwidth requirements; inference drives IOPS and tail latency.
- NVMe-oF RDMA yields best latency; TCP simplifies operations but increases CPU overhead.
- KV-cache tiering and storage-side acceleration can reduce TTFT and backend load—evaluate reproducible vendor benchmarks and run joint tests before procurement.
Sizing NVMe-oF all‑flash platforms is mostly about translating your service-level targets into IOPS and bandwidth requirements, then validating those targets with gate-based acceptance testing using your real models and traces.