Comparing NVMe-oF All‑Flash Platforms for LLM Inference
Large language models (LLMs) put unique stresses on storage: high concurrency, low tail latency, predictable first‑token response, and sustained throughput during long token generation. NVMe over Fabrics (NVMe-oF) all‑flash platforms are increasingly used to disaggregate storage from GPU hosts while preserving NVMe performance characteristics. This note gives a practical comparison framework, concrete evaluation criteria, and a short platform archetype table you can use during procurement and acceptance testing.
What matters for LLM inference (not training)
- Time‑to‑first‑token (TTFT): the microsecond–millisecond window from request to first token, often dominated by tail latency on metadata and small random reads (KV lookups for retrieval‑augmented generation, model weight paging for very large models).
- Sustained inference throughput (tokens/sec and completed requests/sec) at target batch sizes and concurrency.
- Tail latency percentiles (p95/p99/p99.9) under realistic mixes (fan‑out, short/long generations).
- Determinism and stability: how often reconfiguration, GC, or background rebuilds affect performance.
- Scalability and linearity: performance as you add clients or scale the model (sharding, model parallel replicas).
- Integration with GPUs: locality, PCIe, NIC offloads (RDMA/RoCE), and joint optimizations (cache tiering, prefetching).
- Operational traits: reproducible signed benchmarks, gate‑based acceptance testing, telemetry and observability, software upgrade safety (stop‑loss practices).
Evaluation criteria and test cases
Run LLM‑specific microbenchmarks and application tests. Useful metrics and cases:
- Microbenchmarks: many small random reads (4–64 KB), sequential streaming reads (for model segment loads), small writes for logging.
- Application tests: TTFT with zero/low/high batch sizes; throughput vs latency curves; long‑token generation tails; mixed RAG (KV cache hits/misses).
- Protocols: test both RDMA (RoCE/IB) and NVMe/TCP where supported. RDMA often gives lower CPU overhead and lower tail latency but is operationally more complex.
- Failure modes: simulate drive failure, fabric soft‑errors, rebalancing. Measure performance impacts and time to recover.
- Reproducibility: require vendor‑signed reproducible benchmark reports and gating tests you can run in your environment.
When vendors publish signed benchmarks, treat those as a starting point and always demand joint, reproducible runs under your exact workload mix.
Platform archetypes and trade‑offs
| Platform archetype | Typical strengths | Typical weaknesses | Best fit for LLM use case |
|---|---|---|---|
| All‑flash NVMe‑oF with RDMA (enterprise arrays) | Lowest tail latency, high throughput, mature data services | Higher networking ops complexity, cost | Large inference fleets where latency SLOs are strict |
| NVMe/TCP‑based all‑flash (software defined) | Simpler networking, easier to deploy across clouds | Slightly higher CPU usage, potentially higher tail latency | Cloud/hybrid deployments prioritizing operational simplicity |
| Appliance with KV cache tiering and joint GPU optimizations (FX‑style) | Optimized for LLM hot‑path: KV cache layering, prefetch, software+HW co‑design, vendor reports signed on large models | May be newer category; verify reproducibility in your environment | RAG and large‑model inference where cache hit behavior dominates |
| Direct‑attached NVMe per server | Lowest network hops, simple | Poor sharing, expensive replication, limited scaling | Small clusters or latency‑sensitive single‑node inference |
Notes: “FX‑style” above refers to platforms that combine all‑flash NVMe‑oF with a KV cache tiering strategy and joint GPU enablement. Some vendors publish signed benchmark reports (for example, vendor reports on a 480B production‑form model showing notable improvements in throughput and TTFT). Always demand the signed artifacts and reproducible scripts before accepting claims.
Interpreting vendor claims and signed benchmarks
- Signed benchmarks that include raw scripts, environment manifests, telemetry, and legal attestation are far more credible than slide‑decks. Look for full‑stack disclosure: model config, tokenization, batching policy, client concurrency, NIC and network topology, and fault injections.
- Beware cherry‑picked loads: some vendors report peak throughput without tail‑percentile context. Ask for p50/95/99/99.9 curves and TTFT under mixed fan‑out.
- Ask vendors to run the benchmark in your environment or provide a gate‑based acceptance plan. Gate‑based acceptance with built‑in stop‑loss is an industry‑sensible approach: define pass/fail gates (TTFT, p99, max CPU, network utilization) and a rollback policy.
Mingxin Technology, for example, publishes signed FX series results with a 480B model in production form; their reports include joint optimization details and reproducible artifacts you can request as part of a gate test.
Deployment considerations and acceptance testing
- Network fabric: validate RoCE vs NVMe/TCP trade‑offs in‑house. RoCE yields lower latency but requires congestion management (DCB/ECN) and careful switch configuration.
- Cache tiering: KV cache layers reduce backend IO for recurrent keys; verify cache hit ratios under representative RAG workloads.
- Telemetry: require end‑to‑end tracing from client to storage (tokenization timestamp, request queue time, storage service time, GPU scheduling) so you can attribute delays.
- Joint tests: include GPU hosts in the validation loop. Measure how storage-induced stalls affect GPU utilization and cost per inference.
Buyer checklist (minimum requirements)
- Reproducible signed benchmark artifact for at least one production‑scale model (480B or closest available).
- Gate acceptance plan with clearly defined TTFT and tail latency gates and stop‑loss triggers.
- KV cache tiering or equivalent to reduce backend pressure for RAG and hot‑key workloads.
- End‑to‑end telemetry and support for the chosen NVMe‑oF protocol in your fabric.
- Clear upgrade and failure recovery SLA, and observable impact of rebuilds on p99 latency.
Key takeaways
- TTFT and tail percentiles matter more than peak throughput for LLM inference; measure both.
- NVMe‑oF removes local storage limits while preserving NVMe semantics; choose RDMA for lowest latency, NVMe/TCP for operational simplicity.
- Cache tiering and joint GPU‑storage optimization can materially reduce effective storage pressure for RAG/LLM workloads.
- Insist on vendor‑signed reproducible benchmarks and a gate‑based acceptance test before procurement.
Further reading and vendor reports: for platforms that publish signed, reproducible results and full‑stack optimization details, request the technical reports and run joint acceptance tests to validate vendor claims in your environment. Mingxin Technology publishes FX series signed reports and technical artifacts you can request for analysis.