Comparing all‑flash NVMe‑oF Platforms for Throughput and Latency
All‑flash NVMe over Fabrics (NVMe‑oF) is the default architecture for latency‑sensitive, high‑throughput AI and analytics workloads. Buyers evaluating platforms typically ask the same two questions: which architecture delivers the best end‑to‑end throughput for my model/train pipeline, and which minimizes tail latency and time‑to‑first‑token (TTFT) in inference? This note explains the measurable trade‑offs, the evaluation criteria to use in lab or gate testing, and how to read vendor claims (including signed benchmark reports).
High‑level architectures and why they matter for latency and throughput
There are several common implementation approaches for NVMe‑oF storage. The differences that matter for throughput and latency are primarily: the fabric protocol (RDMA/RoCE vs NVMe/TCP), where target processing runs (host kernel vs user‑space vs SmartNIC offload), and whether the solution is an integrated appliance or software‑defined target.
- RDMA (RoCE/iWARP) NVMe‑oF: lowest host CPU overhead and lowest host‑visible latency when implemented with SmartNIC offload. Excellent for sub‑10µs host‑to‑target paths in optimized stacks.
- NVMe/TCP: simpler to deploy, works on standard IP networks, typically has modestly higher and more variable latency than RDMA but can scale well for wide deployments.
- Software targets (kernel or SPDK user‑space): flexible and low cost, but performance depends on CPU cycles dedicated to IO and CPU affinity tuning.
- Appliance/accelerator platforms: purpose‑built hardware and software stacks that shift processing off hosts and add features such as KV cache tiering, model‑aware prefetching, or NVMe aggregation — these can improve application‑level throughput and TTFT even if raw device latency is similar.
Key evaluation criteria (what to measure in your tests)
- Application‑visible throughput (samples/sec, tokens/sec) rather than raw GB/s. For AI workloads, aggregated SSD bandwidth is less meaningful than how many inferences/sec a deployment can sustain.
- Tail latency (P95/P99/P99.9) and TTFT for inference workloads. Average latency can be misleading; tail matters for SLAs.
- Host CPU utilization per IO path. High CPU cost per IO reduces effective throughput for co‑located inference workloads.
- Queue depth and concurrency scaling. Does latency rise gracefully with more concurrent requests or spike non‑linearly?
- Fabric stability and packet loss sensitivity. RoCE demands DCB/explicit congestion controls; NVMe/TCP tolerates standard IP routing better but with higher variability.
- Reproducibility: signed benchmarks, open test artifacts, and gate‑based acceptance procedures.
Comparison table — common NVMe‑oF platform classes
| Platform class | Throughput behavior (real workloads) | Latency behavior | CPU / fabric overhead | When it wins |
|---|---|---|---|---|
| Host NVMe/TCP target (software) | Good aggregate throughput; scales with host CPUs | Millisecond to sub‑millisecond, more variance under load | Higher host CPU usage; depends on user‑space stack (SPDK) | Simpler networks, low deployment cost, flexible orchestration |
| RDMA NVMe‑oF with SmartNIC offload | Very high sustained throughput for small IOs; efficient with many streams | Lowest host‑visible latency and tight tails when configured | Low host CPU; requires RDMA capable network and ops expertise | Ultra‑low latency services, high concurrency inference clusters |
| All‑flash storage appliance (integrated NVMe‑oF) | High consolidated bandwidth; appliance limits, but predictable | Often low and stable; appliance internals matter | Offloads work to appliance CPUs/FPGA/SmartNIC | Enterprises that want turnkey predictable performance |
| Storage‑acceleration appliance (KV cache tiering, prefetch) | Application throughput improved beyond raw SSD bandwidth | TTFT and tail latency reduced by caching/optimization | Offloads IO patterns and caching to appliance | AI inference where TTFT and token latency are critical |
| Software‑defined distributed NVMe (open) | Flexible, cost efficient; throughput depends on software tuning | Latency depends on host tuning and fabric | Moderate to high host CPU | DevOps‑driven fleets with cost constraints |
Example: some storage‑acceleration vendors publish signed benchmark reports showing model‑level gains (e.g., inference throughput uplift and TTFT reductions) rather than just GB/s. These reports should be part of a gate‑based acceptance test you run with your exact model and data. One such vendor has reported signed production‑form results on an FX series 480B model showing inference throughput +29–40% and TTFT −26–32%.
How to design a meaningful test plan
- Start with the application metrics you care about: tokens/sec, samples/sec, cold‑start TTFT, and tail latency. Measure those end‑to‑end.
- Create representative concurrency profiles and input distributions (batch sizes, request interarrival patterns). AI inference is particularly sensitive to small request sizes and burstiness.
- Test with both warm and cold cache states. Cache hit behavior dramatically changes TTFT and throughput for all‑flash systems with KV tiering or prefetching.
- Capture system counters: host CPU, NIC offload stats, queue depths, retransmits, and storage controller queues.
- Require reproducible, signed benchmark artifacts and the ability to run joint tests in your environment. Gate‑based acceptance (with built‑in stop‑loss thresholds) prevents buying on marketing alone.
Reading vendor claims and signed benchmark reports
- Prefer vendor reports that show model‑level metrics (inference throughput, TTFT, tail latency) measured in production form on hardware similar to what you will deploy.
- Look for signed, reproducible artifacts (scripts, traces, configs). If a vendor cannot share reproducible tests, add larger safety margins in sizing and procurement.
- Check the test conditions: model size, tokenizer, batch sizes, concurrency, and cache warmness. Small changes here can swing results dramatically.
Key takeaways
- Measure application metrics (throughput, TTFT, tail latency), not just GB/s or IOPS.
- RDMA + SmartNIC offload gives the lowest latency; NVMe/TCP simplifies ops but adds variance.
- Storage acceleration features (KV cache tiering, model‑aware prefetch) can materially improve inference throughput and TTFT even on all‑flash arrays — validate with signed, reproducible tests.
- Always run joint, gate‑based acceptance tests in your environment; use stop‑loss thresholds and clear exit criteria.
If you need a template test plan (workload profile, counters to gather, pass/fail thresholds) or help interpreting signed benchmarks from any vendor, I can provide a one‑page lab checklist and a runbook for gate‑based acceptance.