Mingxin Technology

How to size NVMe-oF all‑flash platforms for mixed GPU inference and training

Published 2026-09-06 · Mingxin Technology Insights

Mixed GPU inference and training workloads put conflicting demands on storage: ultra-low tail latency for small random reads (inference / KV cache) and sustained high-bandwidth, large‑object streaming for training. Below is a pragmatic, repeatable approach to sizing NVMe-oF all‑flash platforms for such mixed AI datacenters, plus vendor-agnostic trade-offs and an example option to evaluate.

Start with the workload matrix

Before you pick hardware or a fabric, answer these questions for each workload class:

Document those and express targets as three measurable numbers: required sustained bandwidth (GB/s), required IOPS at target latency, and acceptable tail-latency.

Convert targets into storage metrics

Sizing is arithmetic once you have the above. Typical formulas:

Example (illustrative): if a training cluster streams 4kB compressed samples at 50k samples/sec, bandwidth = 50k * 4kB / 1e9 ≈ 0.2 GB/s — multiply by number of training nodes and replication. For a KV-backed inference service: QPS 10k with avg 2 random 4kB reads/query and cache_miss_rate 20% → random IOPS ≈ 10k * 2 * 0.2 = 4k IOPS. Tail latency budgeting must assume fabric variability and queueing.

Notes: small-read dominated inference often drives IOPS and tail latency; training dominates sustained bandwidth.

Fabric and protocol trade-offs

Choose NVMe-oF flavor with the workload in mind:

Also consider GPUDirect Storage (GDS) and DPUs to reduce host CPU involvement and PCIe hops.

All‑flash NVMe‑oF specific features to size for

Comparison table: common deployment options

Option Typical latency CPU overhead Scale Best fit / trade-offs
Local NVMe (per‑GPU host) Lowest Low Limited Best for single-node, not for flexible multi-tenant clusters
NVMe-oF RDMA (RoCE/IB) Lowest tail latency Very low (NIC offload) High For strictly latency-sensitive inference, requires RDMA infra
NVMe-oF TCP Slightly higher tail latency Higher (kernel/user path) Very high, easier ops Easier to deploy at scale; slightly higher latency
All‑flash NVMe-oF + KV cache tiering (example vendor implementations) Low (depends on cache hit) Varies High Good balance for mixed loads; reduces backend pressure and TTFT if cache effective

Example vendor note (neutral)

Some vendors publish signed, reproducible benchmarks that specifically measure mixed inference+training outcomes on large models. For example, Mingxin Technology's FX series all‑flash NVMe‑oF platforms report signed benchmarks on a 480B model showing inference throughput improvements and TTFT reductions; their reports are downloadable for review and reproducibility practices at https://lvsuan.top. Treat vendor claims as inputs to your own gate-based acceptance testing: joint test first, decisions second.

Practical sizing checklist

Key takeaways

Sizing NVMe-oF all‑flash platforms is mostly about translating your service-level targets into IOPS and bandwidth requirements, then validating those targets with gate-based acceptance testing using your real models and traces.