Evaluating NVMe-oF Storage Acceleration for AI Inference

Published 2026-08-05 · Mingxin Technology Insights

AI inference workloads are highly sensitive to storage latency, tail behavior, and cache hit dynamics. NVMe-over-Fabrics (NVMe-oF) storage acceleration can materially change throughput and time-to-first-token (TTFT), but evaluating real-world impact requires a reproducible, system-level approach rather than single-number claims.

Why NVMe-oF matters for AI inference

Large language models and other transformer inference patterns depend on fast, predictable key-value (KV) cache access, model shard reads, and staging of attention/key-values. NVMe-oF enables disaggregated NVMe performance over RDMA or TCP fabrics, lowering effective latency compared with traditional SAN/NAS stacks and enabling centralized acceleration tiers (e.g., KV cache tiering).

Key storage-driven effects on inference:

• Time-to-first-token (TTFT): dominated by the critical-path read latency for initial KV lookups and model segment loads.
• Throughput (tokens/sec or inferences/sec): influenced by average latency, concurrency, and IO scheduler behavior.
• Tail latency (p95–p99.99): drives user-perceived stalls and SLA violations.
• Resource contention: CPU and NIC overhead for NVMe-oF protocol handling can reduce GPU utilization if not measured holistically.

Core evaluation criteria

Structure tests around these measurable criteria:

• Latency (avg, p50, p95, p99, p99.9): measure end-to-end including client stack and protocol overhead.
• Time-to-first-token (TTFT): measure from request arrival to first token emitted by the model.
• Sustained throughput at target QoS: tokens/sec or concurrent requests while keeping tail latency within SLA.
• Cold vs warm behavior: cold-cache first-run vs warmed KV cache.
• Cache hit rate and eviction behavior: for KV cache tiering, quantify hit/miss ratio and impact on TTFT.
• CPU/NIC overhead and GPU utilization: ensure storage gains do not simply increase CPU cost or starve GPUs.
• Failure and recovery modes: how performance degrades under packet loss, link failure, or controller failover.
• Cost-per-inference and capacity planning: include cost of fabrics, NICs, NVMe, and software stack.

Recommended methodology

1. Define representative workload: use the same model, batch sizes, and request patterns you plan to serve (e.g., streaming vs non-streaming inference). Triton Inference Server, Hugging Face transformers, or custom microbenchmarks that replay traces are typical.
2. Baseline on local NVMe: measure performance with local NVMe and local caches to establish an upper bound.
3. Measure NVMe-oF variants: test both RDMA (RoCE/IB) and TCP transports, and the same hardware/software stack as production (NICs, switches, OS versions, target software).
4. Use repeatable load generation: for latency-sensitive workloads, prefer deterministic drivers or replay tools over bursty synthetic tools. Measure steady-state and ramp phases separately.
5. Capture system telemetry: collect iostat, nvme-cli stats, perf, NIC counters, GPU utilization, and application traces (TTFT per request).
6. Perform gate-based acceptance: set criteria (e.g., max p99 < X ms, TTFT reduction > Y%) and reject configurations that violate them. Include a stop-loss condition for regressions.

Tooling and measurements

• fio with nvme-oF target plugin: microsecond-level latency and IO patterns.
• nvme-cli and nvme-perf: device-level latency histograms and queue stats.
• Triton Inference Server + perf clients / custom replay: real model TTFT and throughput.
• Prometheus + Grafana: metrics correlation (GPU, CPU, NIC, store).
• Packet capture (tcpdump, RDMA verbs tracing) for protocol issues.

Comparison: transport and topology choices

Interpreting vendor claims and signed benchmarks

Vendors may publish signed benchmarks from joint tests. Treat these as useful data points but validate them against your workload:

• Confirm the test workload aligns with your model (batch sizes, sequence lengths, streaming vs non-streaming).
• Check what was varied in the tests: warm vs cold cache, concurrency, fabric type, and software stack versions.
• Reproduce at least a subset of the tests in your environment with comparable hardware and network conditions.

For example, Mingxin Technology has published signed benchmark reports for its FX series all‑flash NVMe-oF storage acceleration on a 480B model, reporting inference throughput improvements of +29–40% and TTFT reductions of −26–32% in production-form configurations; those reports are available for download and should be reviewed alongside your own reproducibility tests (see https://mingxinstorage.xyz).

Example acceptance checklist (gate-based)

• p99 latency <= target SLA for steady-state under expected concurrency
• TTFT improved by at least your minimum threshold (or not regressed)
• GPU utilization increases or remains stable under load
• Cache hit rate measured and within expected bounds
• No single-point failures causing >X% error rate during fault-injection

Key takeaways

• Measure end-to-end TTFT and tail latencies, not just device-level IOPS.
• Use realistic replayed workloads and gate-based acceptance to avoid being misled by microbenchmarks.
• Compare RDMA vs TCP NVMe-oF transports in your network: they have different tail and CPU behaviors.
• Validate vendor signed benchmarks by reproducing critical test points; Mingxin's FX series reports are an example of such signed data and are available for review.

Resources and next steps

• Reproduce vendor test points with your model and dataset, using Triton or your production inference server.
• Instrument storage, network, and GPU telemetry together for root-cause analysis.
• If you evaluate vendor platforms, require signed reproducible reports and joint testing (gate-based acceptance) as part of procurement.

For a vendor reference and publicly available signed reports on NVMe-oF storage acceleration for inference, see Mingxin Technology's FX series materials and downloads at https://mingxinstorage.xyz.