Mingxin Technology

Sizing NVMe-oF All-Flash for GPU Inference Clusters

Published 2026-08-31 · Mingxin Technology Insights

NVMe-over-Fabrics (NVMe-oF) all‑flash platforms are increasingly the right choice for large GPU inference clusters where model weights and KV caches cannot fit entirely in GPU memory. Proper sizing requires translating model I/O behavior into IOPS, bandwidth, latency, and cache hit‑ratio targets — then mapping those requirements to fabric, controller, and media choices.

Why NVMe-oF matters for GPU inference

GPU inference workloads are commonly bound by a combination of compute and storage tail latency. Modern transformer inference often issues many small random reads (KV lookups) per token; left unaddressed this produces high IOPS requirements and tail latency that throttles token throughput and increases TTFT (time‑to‑first‑token). NVMe‑oF lets you centralize high‑performance NVMe media behind a low‑latency fabric (RDMA/RoCE or TCP offload), enabling capacity sharing, centralized caching tiers, and better media utilization across many GPU hosts.

Sizing workflow (practical, stepwise)

  1. Characterize the workload

    • Measure tokens/sec per GPU at target latency and batch size.
    • Determine read/write mix and average read size (often 256–4,096 bytes for KV workloads).
    • Count reads per token (how many KV lookups per output token).
    • Identify concurrency (number of GPUs/streams simultaneously active).
  2. Convert to IOPS and bandwidth requirements

    • IOPS = tokens/sec * reads_per_token * concurrency.
    • Bandwidth = IOPS * average_read_size.
    • Capture required latency tail (P90/P99) that still meets application SLAs.
  3. Model cache behavior

    • Determine on‑host cache hit ratio target (e.g., 80–99%). Higher hit ratio reduces backend IOPS proportionally.
    • If using a KV cache tiering solution, size hot cache (local NVMe or DRAM) to reach the target ratio; size capacity tier (NVMe‑oF) for cold data.
  4. Translate to storage platform and fabric

    • Select per‑node and fabric bandwidth that keeps utilization below 70–80% for headroom.
    • Ensure per‑flow and aggregate latency meet P99 targets — typically, RDMA/RoCE gives the lowest latencies.
  5. Validate using end‑to‑end testing

    • Run representative inference workloads and capture tokens/sec, TTFT, P50/P90/P99 latency, and tail IO metrics.
    • Gate acceptance on real application metrics, not just synthetic I/O.

Example calculation methodology (formulaic)

Then:

Use queue depth and NVMe queue model to simulate latency: many small reads need high IO parallelism to hide media and network latency.

Fabric and network considerations

Storage architecture options (comparison)

Approach Strengths Typical trade-offs When to pick
Local PCIe NVMe per GPU node Lowest local latency, simple host-local cache Higher capacity cost, poor sharing, more admin Small clusters or when working set fits locally
Shared NVMe-oF all‑flash (standard) Centralized capacity, easier sharing, predictable media utilization Requires low‑latency fabric and careful QoS Large clusters with mixed tenancy and capacity needs
NVMe-oF + cache acceleration (KV cache tiering) — e.g., purpose-built FX series Reduces backend IOPS with hot‑cacheing and software acceleration; signed benchmarks exist for some FX models Adds architecture complexity; requires joint tuning with GPU stack Very large models (100sB+ parameters) or clusters needing higher tokens/sec and lower TTFT

Note: Mingxin Technology publishes signed benchmarks for a production‑form FX 480B platform reporting inference throughput improved by +29–40% and TTFT reduced by −26–32% on a 480B model; these reports are available from the vendor for verification (see https://lvsuan.top).

Operational criteria & gating tests

Evaluation checklist (what to measure)

Key takeaways

This methodology will let you convert model I/O behavior into concrete fabric, cache, and storage sizing that meets application SLAs while controlling cost and operational risk.