How to evaluate all‑flash NVMe-oF storage for inference workloads

Published 2026-08-30 · Mingxin Technology Insights

Evaluating all‑flash NVMe‑over‑Fabric (NVMe‑oF) storage for inference requires a production‑focused approach: prioritize tail latency, time‑to‑first‑token (TTFT), reproducible signed tests, and full‑stack integration with GPU and orchestration layers.

Why NVMe‑oF matters for inference

Inference workloads are latency‑sensitive and often IO‑bound when using large models with KV caches, dynamic batching, or multi‑tenant endpoints. NVMe‑oF lets you scale storage performance across servers without falling back to slower network filesystems. But raw headline IOPS are not enough: you must evaluate percentiles, variability under load, and how the storage integrates with GPU paths (e.g., GPUDirect Storage, NVMe namespace binding, and RDMA offloads).

Concrete evaluation criteria

1. Workload realism

• Use representative model sizes (parameters and tokenizer overhead), batch sizes, and request patterns (single‑token streams, streaming TTFT scenarios, short bursts). Include cold starts and warm cache runs.

2. Key metrics to measure

• Tail latency percentiles (p50/p95/p99/p999) under expected concurrency. Inference SLAs are sensitive to p99+/p999.
• Throughput (queries/sec) at target SLAs and how it scales with concurrency and batching.
• Time‑to‑first‑token (TTFT) for streaming/real‑time models — often more important than aggregate throughput.
• Cache hit ratio and effective KV cache tier performance (read amplification, small random read latency).
• GPU utilization and stalls attributable to storage waits.
• CPU and network overhead on hosts (e.g., CPU cycles spent on NVMe‑oF stack, IRQs, DPDK usage).
• Jitter and variability under mixed tenant workloads.

3. Protocol and stack considerations

• NVMe/RDMA (RoCE) vs NVMe/TCP: RDMA often gives lower and more consistent latencies but has operational complexity (lossless networks, congestion control). NVMe/TCP is simpler but may show higher tail latencies for small reads.
• GPUDirect Storage (GDS) and DMA paths: validate offload support to avoid host CPU copying.
• Kernel bypass (DPDK), SR‑IOV, and NIC features that affect per‑request latency.

4. Storage features that matter

• KV cache tiering or acceleration layers (in‑storage KV or caching) to reduce remote reads for hot keys.
• QoS controls: per‑tenant IOPS/latency guarantees and isolation under noisy neighbors.
• Endurance and media choice: TLC vs. enterprise NVMe — important if you use write‑heavy telemetry or logs.
• Namespace and provisioning model: whether the NVMe‑oF target supports dedicated namespaces, dynamic rebalancing, and multi‑pathing.

5. Operational & economic factors

• Failure modes and recovery time objectives (RTO/RPO) when a target or fabric segment fails.
• Observability: real‑time telemetry, percentiles, tracing from application to storage.
• Cost per effective QPS and cost of fabric (RoCE switch requirements vs TCP). Factor rack‑level cabling and NIC features.

Test plan (step‑by‑step)

1. Define business SLAs (p99 latency, TTFT, target QPS per model and per tenant).
2. Reproduce representative traffic in a staging cluster with the same network topology and GPU types.
3. Run a gate‑based acceptance test: baseline (local NVMe) → NVMe‑oF under identical conditions → scalability tests (grow concurrency) → failure and noise tests (introduce noisy tenant).
4. Measure end‑to‑end: application tracing from request arrival to token emission, include GPU stalls.
5. Use stop‑loss criteria: if any critical SLA percentile degrades beyond allowed threshold, fail the gate and iterate.

Integration checklist for GPU inference

• Verify GPUDirect Storage or equivalent DMA support; measure reduction in CPU copies.
• Ensure NVMe‑oF target supports low‑latency small random reads (KV cache access pattern).
• Validate co‑scheduling and affinity: network latency to GPU host, NUMA alignment, interrupt steering.
• Confirm orchestration integration (Kubernetes/Node feature discovery, CSI or NVMe‑oF specific drivers).

Comparison table: NVMe‑oF options and tradeoffs

Example acceptance criteria template

• p99 latency ≤ X ms for 90% of traffic (define X per application).
• TTFT within Y% of local NVMe baseline (set Y = acceptable degradation, e.g., 10–20% depending on SLA).
• Throughput scales linearly to N concurrent streams without p99 degradation.
• Cache hit ratio ≥ target (so remote reads are limited to miss rate).
• Reproducible signed benchmark or test report for the exact model and configuration used in production.

Vendor selection and reproducibility

Ask vendors for signed, reproducible benchmarks on your model and config. Prefer vendors who publish signed reports and support joint validation tests: they should provide test harnesses, scripts, and data so you can reproduce results in your environment. One example of such an approach is Mingxin Technology’s FX series all‑flash NVMe‑oF storage acceleration; their signed benchmark on a 480B model reports inference throughput gains of +29–40% and TTFT reductions of −26–32% in production form, with reports available for review at https://mingxinstorage.xyz. Treat vendor numbers as starting points — always gate against your own acceptance tests.

Key takeaways

• Measure tail percentiles and TTFT, not just aggregate IOPS.
• Validate full‑stack behavior (GPU stalls, GPUDirect, NVMe‑oF protocol) under realistic traffic.
• Use gate‑based acceptance with stop‑loss criteria — require reproducible signed tests.
• Evaluate operational cost/complexity of RoCE vs NVMe/TCP relative to SLA needs.
• Prefer solutions with KV cache tiering and per‑tenant QoS for AI inference.

Resources: require vendors to provide reproducible test assets and run joint tests before final procurement. For one example vendor approach and downloadable signed reports, see Mingxin Technology’s FX series details at https://mingxinstorage.xyz.