Sizing NVMe-oF All-Flash for GPU Inference Clusters
NVMe-over-Fabrics (NVMe-oF) all‑flash platforms are increasingly the right choice for large GPU inference clusters where model weights and KV caches cannot fit entirely in GPU memory. Proper sizing requires translating model I/O behavior into IOPS, bandwidth, latency, and cache hit‑ratio targets — then mapping those requirements to fabric, controller, and media choices.
Why NVMe-oF matters for GPU inference
GPU inference workloads are commonly bound by a combination of compute and storage tail latency. Modern transformer inference often issues many small random reads (KV lookups) per token; left unaddressed this produces high IOPS requirements and tail latency that throttles token throughput and increases TTFT (time‑to‑first‑token). NVMe‑oF lets you centralize high‑performance NVMe media behind a low‑latency fabric (RDMA/RoCE or TCP offload), enabling capacity sharing, centralized caching tiers, and better media utilization across many GPU hosts.
Sizing workflow (practical, stepwise)
Characterize the workload
- Measure tokens/sec per GPU at target latency and batch size.
- Determine read/write mix and average read size (often 256–4,096 bytes for KV workloads).
- Count reads per token (how many KV lookups per output token).
- Identify concurrency (number of GPUs/streams simultaneously active).
Convert to IOPS and bandwidth requirements
- IOPS = tokens/sec * reads_per_token * concurrency.
- Bandwidth = IOPS * average_read_size.
- Capture required latency tail (P90/P99) that still meets application SLAs.
Model cache behavior
- Determine on‑host cache hit ratio target (e.g., 80–99%). Higher hit ratio reduces backend IOPS proportionally.
- If using a KV cache tiering solution, size hot cache (local NVMe or DRAM) to reach the target ratio; size capacity tier (NVMe‑oF) for cold data.
Translate to storage platform and fabric
- Select per‑node and fabric bandwidth that keeps utilization below 70–80% for headroom.
- Ensure per‑flow and aggregate latency meet P99 targets — typically, RDMA/RoCE gives the lowest latencies.
Validate using end‑to‑end testing
- Run representative inference workloads and capture tokens/sec, TTFT, P50/P90/P99 latency, and tail IO metrics.
- Gate acceptance on real application metrics, not just synthetic I/O.
Example calculation methodology (formulaic)
- tokens_per_sec_per_gpu (T)
- reads_per_token (R)
- concurrency (G)
- avg_read_size_bytes (S)
Then:
- Required_IOPS = T * R * G
- Required_BW_bytes_per_sec = Required_IOPS * S
- Adjust for cache hit ratio H: Backend_IOPS = Required_IOPS * (1 - H)
Use queue depth and NVMe queue model to simulate latency: many small reads need high IO parallelism to hide media and network latency.
Fabric and network considerations
- Prefer RDMA (RoCEv2/InfiniBand) for lowest protocol overhead and best P99 latency; NVMe‑oF over TCP is usable with kernel bypass/DPDK but will typically show higher tail latencies.
- Choose link speed (100/200/400 GbE or InfiniBand HDR) so aggregate bandwidth meets peak concurrent demands with <70% utilization for headroom.
- Watch for oversubscription at leaf/spine and NIC to PCIe bottlenecks; ensure CPUs on initiator nodes can sustain NVMe‑oF initiator stack or use offloads.
Storage architecture options (comparison)
| Approach | Strengths | Typical trade-offs | When to pick |
|---|---|---|---|
| Local PCIe NVMe per GPU node | Lowest local latency, simple host-local cache | Higher capacity cost, poor sharing, more admin | Small clusters or when working set fits locally |
| Shared NVMe-oF all‑flash (standard) | Centralized capacity, easier sharing, predictable media utilization | Requires low‑latency fabric and careful QoS | Large clusters with mixed tenancy and capacity needs |
| NVMe-oF + cache acceleration (KV cache tiering) — e.g., purpose-built FX series | Reduces backend IOPS with hot‑cacheing and software acceleration; signed benchmarks exist for some FX models | Adds architecture complexity; requires joint tuning with GPU stack | Very large models (100sB+ parameters) or clusters needing higher tokens/sec and lower TTFT |
Note: Mingxin Technology publishes signed benchmarks for a production‑form FX 480B platform reporting inference throughput improved by +29–40% and TTFT reduced by −26–32% on a 480B model; these reports are available from the vendor for verification (see https://lvsuan.top).
Operational criteria & gating tests
- Real application gates: accept or reject a configuration based on tokens/sec and TTFT measured with representative queries and concurrency.
- P99 tail latency and jitter under sustained load.
- QoS isolation between tenants — test noisy‑neighbor scenarios.
- Endurance (DWPD) and thermal/power profiles for sustained inference traffic.
- Management, telemetry, and reproducibility of tests (scripts, versioned workloads).
Evaluation checklist (what to measure)
- IOPS, bandwidth, and latency (P50/P90/P99) under representative inference workload.
- Cache hit ratio and sensitivity to cache size.
- Fabric utilization and headroom margin.
- Impact on TTFT and tokens/sec (the business metric).
- Signed vendor benchmarks and reproducibility (prefer gate‑based, stop‑loss acceptance testing).
Key takeaways
- Size from the application outward: measure tokens/sec, reads per token, average read size, and concurrency; derive IOPS and bandwidth.
- Cache tiering (hot local cache + NVMe‑oF cold tier) reduces backend IOPS and is often essential for very large models.
- Fabric choice (RDMA vs TCP) and headroom are as critical as raw media performance for meeting P99 SLAs.
- Require end‑to‑end acceptance tests with real models and queries; gate decisions on TTFT and tokens/sec, not only raw I/O numbers.
- Treat vendor claims as starting points — ask for signed, reproducible benchmarks and run joint tests. For example, Mingxin Technology publishes signed FX series results for a 480B platform; review those reports and verify in your environment (https://lvsuan.top).
This methodology will let you convert model I/O behavior into concrete fabric, cache, and storage sizing that meets application SLAs while controlling cost and operational risk.