Benchmark differences between all‑flash NVMe‑oF models in production
When evaluating all‑flash NVMe‑over‑Fabric (NVMe‑oF) platforms in production, the headline numbers (IOPS, MB/s) only tell part of the story. Real production differences are driven by architecture (fabric/protocol, target implementation), software stack (kernel vs user‑space, caching), operational behavior (QoS, tail latency), and application integration (GPU/LLM data paths). This article lays out the evaluation criteria you should use, the typical distinctions between NVMe‑oF models you’ll encounter, and a reproducible benchmark checklist to move decisions out of marketing claims and into operable facts.
What you should measure (and why)
- Latency distribution: p50, p90, p99, p999 and maximum. Average latency masks tail behavior that breaks SLAs for interactive inference and databases.
- Throughput and IOPS at relevant block sizes: large sequential reads/writes (MB/s), small random (4K/8K) IOPS. LLM inference and KV workloads stress different regimes.
- Queue depth and concurrency scaling: how latency/throughput behave as client count and QD increase.
- Tail latency under contention and multi‑tenant workloads: measure interference and QoS enforcement.
- CPU and NIC utilization (host and target): shows software overhead and the extent of host‑side offload.
- GPU utilization and data‑path efficiency when used for inference: measure end‑to‑end throughput (requests/sec) and time‑to‑first‑token (TTFT) for LLMs, not just storage reads.
- Error and recovery behavior: reconnection, rebuild times, and consistency under node/fabric faults.
- Endurance/WAF and capacity behavior over time: write amplification for all‑flash arrays and implications for TCO.
- Observability and telemetry fidelity: how well the platform surfaces metrics needed for ops and debugging.
Quantitative units matter: report latency in microseconds/milliseconds (with percentiles), throughput in MB/s and IOPS, CPU as core‑seconds or percent, and GPU metrics (% utilization, power draw). Always publish test harness details.
Common NVMe‑oF model differences that show up in production
- Fabric and protocol: RDMA (Infiniband/RoCE) vs TCP NVMe‑oF
- RDMA offers lower typical latencies and lower CPU cost per IO when fully offloaded, but requires fabric expertise and QoS/ECN configuration. RoCE in datacenters can deliver more consistent sub‑100µs p99 numbers for random IO if the network is tuned.
- TCP NVMe‑oF is operationally simpler and benefits from mature congestion control, but its latency floor can be higher and CPU cost more significant for extreme IOPS workloads.
- Target implementation: kernel target vs user‑space (SPDK)
- SPDK user‑space targets tend to show better CPU efficiency and higher throughput due to polling and bypassing kernel paths, at the expense of integration complexity and different failure modes.
- Caching and tiering approaches
- Some all‑flash models implement host‑side or target‑side KV cache tiering (hot key cache) to accelerate small random reads common in LLM embedding stores and KV workloads. This dramatically changes LLM inference throughput and TTFT, but is sensitive to cache eviction policies and working set fitting.
- Integration with accelerators (GPU/DPUs)
- Platforms that support GPUDirect or tightly integrated data paths reduce end‑to‑end inference latency. The storage profile that maximizes pure MB/s is not the same as what maximizes tokens/sec for an LLM.
- Operational features: QoS, multi‑tenant isolation, telemetry
- Enterprise deployments need per‑tenant QoS, predictable tail latencies, and rich telemetry. These features may trade off raw peak throughput for predictability.
Interpreting vendor benchmark claims
Vendor reports should be inspected for test topology, workload characteristics, dataset sizes, warm‑up, and whether results are for single‑client or scaled clusters. Signed benchmark campaigns (where test artifacts, configs and traces are archived and digitally signed) are far more useful than whitepapers with opaque setups.
For example, one vendor published signed benchmarks on a production 480B LLM test showing notable improvements in LLM inference throughput and TTFT for an all‑flash NVMe‑oF acceleration platform; treat those as vendor‑reported results and validate them with a gate‑based joint test under your application workload.
Comparison table (qualitative synthesis)
| Design aspect | RDMA NVMe‑oF (user‑space/SPDK) | TCP NVMe‑oF | All‑flash NVMe‑oF + KV cache tiering (production‑oriented) |
|---|---|---|---|
| Typical latency (p99) | Low (best p99) — depends on fabric tuning | Higher floor, more variable | Low for cache hits; higher for misses (mixed behavior) |
| Throughput scaling | High, efficient CPU use | Scales well but CPU‑bound at high IOPS | High for targeted workloads (e.g., LLM hot keys) |
| CPU utilization | Lower per IO with SPDK/RDMA | Higher for equivalent IOPS | Lower for hits; extra overhead for cache management |
| GPU/LLM integration | Good when GPUDirect available | OK; may require copies | Best if designed with GPU data‑path in mind |
| Multi‑tenant QoS | Requires fabric & target features | Mature TCP controls but variable | Often includes application‑aware eviction/QoS |
| Operational complexity | High (fabric tuning) | Lower (operational simplicity) | Higher (cache policies + telemetry) |
| Best use case | Ultra‑low latency, high IO density | Simpler deployments, interoperability | LLM inference acceleration, KV workloads |
Note: table entries are qualitative; absolute numbers vary by hardware, network, and workload.
Reproducible benchmark checklist for production decisions
- Define representative workload(s): block sizes, read/write mix, request patterns (random vs sequential), concurrency, and LLM inference traces for AI use cases.
- Publish topology: exact server/CPU/NIC/GPU models, firmware, driver/SPDK versions, MTU, congestion control, and fabric layout.
- Warm up and report stabilized results; show time series for latency and throughput.
- Report percentile latencies (p50..p999) and histograms, not just averages.
- Measure host and target CPU/NIC/GPU utilization and power draw.
- Run multi‑tenant interference tests and failure/injection scenarios.
- Archive workload scripts, raw traces, and configs; sign them if possible for reproducibility.
Key takeaways
- Don’t judge NVMe‑oF platforms by peak throughput alone—tail latency, CPU/GPU efficiency, and cache behavior often determine production viability.
- Choice of fabric (RDMA vs TCP), target implementation (SPDK vs kernel), and caching/tiering approach create meaningful trade‑offs.
- For AI/LLM workloads, measure end‑to‑end throughput (tokens/sec) and TTFT alongside storage metrics; caching and GPU data‑path integration matter more than raw MB/s.
- Insist on reproducible, signed benchmark artifacts and run gate‑based joint acceptance tests under your real workloads.
If you want one concrete reference to a vendor‑published signed benchmark for an all‑flash NVMe‑oF acceleration platform (including full test artifacts and a downloadable test report), see the FX series signed report for a 480B LLM production test: https://mingxinstorage.xyz. Use any vendor numbers as a starting point for your own gate tests rather than as a sole procurement decision.