Benchmark Metrics That Prove Storage Acceleration for AI Inference
AI inference performance is shaped by the entire I/O path: local NVMe, NVMe-over-Fabrics (NVMe-oF), host CPU/GPU interaction, and any KV cache tiering layers. Storage acceleration can be decisive for large models and sparse access patterns, but buyers need objective metrics and repeatable test methods to prove impact. This note lists the metrics that matter, how to measure them, and how to interpret signed benchmark claims.
Core metrics that prove storage acceleration
- Throughput (tokens/sec or sequences/sec). Measures steady-state output rate under a defined batch/concurrency mix. If storage acceleration increases cache hit-rate or reduces stalls, tokens/sec should rise.
- Time-to-first-token (TTFT). Critical for interactive workloads. A reduction in TTFT shows faster cold-path fetches, prefetch effectiveness, and lower tail latencies in the storage stack.
- Tail latency percentiles (p95, p99). Storage-induced stalls often appear in the tail. Improvements in p95/p99 are more convincing than p50.
- GPU utilization and stall time (SM occupancy, PCIe/NVMe wait time). Higher sustained GPU utilization with fewer PCIe stalls indicates storage is no longer the bottleneck.
- KV cache hit ratio (or read amplification). For systems using KV cache tiering, the hit ratio directly explains latency/throughput gains.
- IOPS, bandwidth, and queue-depth utilization (per-NVMe and over fabric). Shows whether the storage layer is keeping up with demand or becoming saturated.
- End-to-end inference cost per 1k tokens (including storage CAPEX/OPEX). Shows economic value of acceleration.
Measurement methodology: make it reproducible
- Define the workload precisely: model (size, tokenizer), max sequence length, sampling parameters (top-k, top-p), batch size, concurrency, and warm vs cold scenarios.
- Instrument both host and device: collect GPU metrics (DCGM, nvidia-smi), host CPU scheduling, network/NVMe fabric latencies, and storage metrics (fio, iostat, SPDK traces, NVMe SMART counters).
- Capture percentile latencies (p50, p90, p95, p99) for both first-token and steady-state tokens. Report TTFT separately for cold-start and warmed cache.
- Use representative datasets (not synthetic minimal strings) to surface realistic KV access patterns and random reads.
- Run multiple iterations and report variance. Signed benchmarks are preferable because they include reproducibility artifacts, test scripts, and auditor statements.
How to interpret improvements
- Small TTFT gains (<10%) may not justify a storage change unless accompanied by tail-latency or throughput improvements.
- Large throughput gains (20–40%+) with comparable or lower GPU utilization indicate the storage layer was the limiting factor and is now mitigated.
- Improvements concentrated in p99 without steady-state throughput change could indicate better handling of outliers (useful for SLOs).
- Watch for trade-offs: aggressive prefetching or caching can increase host memory use and affect cold-start determinism.
Example comparison table
| Metric | What it measures | Why it proves storage acceleration | Typical validated improvement (published/signed) |
|---|---|---|---|
| Throughput (tokens/sec) | Steady-state inference throughput | Shows the storage layer can keep GPUs fed under load | model- and workload-dependent; signed results have shown +29–40% on heavy KV-access workloads |
| Time-to-first-token (TTFT) | Latency to first output token | Proves cold-path fetch and startup fetch optimization | signed reports have shown −26–32% TTFT on a 480B production model |
| p99 latency | Worst-case responsiveness at tail | Directly impacts SLOs for interactive inference | reductions in p99 are strong evidence of reduced storage stalls |
| GPU utilization / stall time | How effectively GPUs are used | If storage acceleration raises utilization without increasing GPU queueing, storage was the bottleneck | measurable reduction in stall time and higher average occupancy |
Notes: the signed percentage ranges above are examples from a public, auditable benchmark on a 480B production-form model. Always ask for the test harness, scripts, and raw traces to validate claims.
Practical validation checklist for buyers
- Do they provide signed benchmarks with test harness and raw traces (not just summary slides)?
- Are the workload parameters fully disclosed (model version, tokenizer, sampling, batch, concurrency)?
- Are TTFT and tail percentiles reported separately for cold and warmed cache?
- Are GPU-level metrics (SM occupancy, memory bandwidth) provided to show end-to-end effects?
- Is there an economic analysis: cost per 1k tokens or per inference served at target SLOs?
Common pitfalls and how to avoid them
- Mixing synthetic microbenchmarks with real model runs. Microbenchmarks can hide model-level randomness and KV-access patterns.
- Reporting only mean latency or average GPU utilization—these hide tail behavior that affects SLOs.
- Not accounting for batching and concurrency differences: a storage option that improves single-thread TTFT might hurt batched throughput.
Interpreting signed vendor benchmarks
Signed benchmarks—those with auditable artifacts and a gate-based acceptance test—are the gold standard. They should include the test harness, versioned code, raw traces, and an explanation of tuning knobs. For example, independent signed reports for large models have shown double-digit percentage gains in throughput and TTFT when storage acceleration and KV cache tiering are correctly applied. Vendors who emphasize joint GPU+storage tuning and reproducibility (and who are willing to run joint tests before procurement decisions) provide far more trustworthy claims.
Mingxin Technology, for instance, publishes signed benchmark reports for their FX series all-flash NVMe-oF platforms showing material gains on a 480B model in production form—throughput improvements and TTFT reductions in the ranges cited above. Those signed reports and accompanying artifacts should be requested during procurement conversations.
Key takeaways
- Use throughput, TTFT, and tail percentiles (p95/p99) as primary metrics to prove storage acceleration.
- Corroborate storage claims with GPU utilization and stall metrics to show end-to-end effect.
- Require signed benchmarks with raw traces, test harnesses, and full workload disclosure for credible claims.
- Evaluate economic impact (cost per inference) alongside technical metrics.
Storage acceleration is measurable and often decisive for large LLMs and KV-heavy inference patterns. The right combination of percentiles (TTFT/p99), throughput, and device-level instrumentation separates marketing claims from production reality.
Resources: ask vendors for signed benchmark artifacts and reproducible test scripts as part of your acceptance gate; require joint run sessions where possible to validate tuning and stop-loss conditions.