Storage acceleration test reports: which metrics to request
When evaluating a storage acceleration platform for AI/LLM inference or datacenter use, the test report is the single most important artifact you can request. A good report makes performance claims verifiable: raw measurements, workload definitions, tooling, environment metadata, and reproducibility artifacts.
Executive checklist: what to demand from any storage acceleration test report
- Raw time-series measurements (not only averages) for IOPS, bandwidth, and latency (including p50/p90/p99/p99.9).
- LLM-specific metrics: throughput (tokens/sec or inferences/sec), Time-To-First-Token (TTFT), and tail latency for first token and subsequent tokens.
- Host and network telemetry: CPU/GPU utilization, PCIe/SQ/CQ metrics, NVMe queue depth, NIC transmit/receive rates, RDMA retransmits/packet drops.
- Workload and dataset definition: model name/size, batch sizes, sequence length, tokenization, dataset samples used, cache warm-up procedure, and concurrency levels.
- Test harness and scripts: exact toolchains (e.g., Triton/Direct model server, custom runner), command-lines, versioned containers, and reproducible scripts.
- Environmental and configuration metadata: firmware, drivers, kernel, NUMA layout, RDMA config (RoCE/iWARP), storage firmware release, and NVMe-oF transport details.
- Signed benchmark artifacts: hashes/signatures of raw logs, and a summary that ties derived metrics to raw data.
- Power and efficiency data: system power draw, throughput/W, and CPU/GPU power envelope during steady-state runs.
Concrete metrics and why they matter
| Metric | Why it matters | Measurement tool / notes | What to expect / threshold guidance |
|---|---|---|---|
| p50/p90/p99/p99.9 latency (ms) | Tail behavior drives user experience for interactive inference | Host and target-side histograms (e.g., perf counters, Prometheus histograms) | For LLM inference, p99/p99.9 often dominates SLAs — report both absolute and delta vs baseline |
| Throughput (tokens/s or infer/sec) | Capacity sizing and cost-per-request | Measured at the model server entrypoint under steady-state | Improvements vs baseline often in the 10–40% range depending on workload and cache effectiveness (vendor-specific) |
| Time‑To‑First‑Token (TTFT) | Perception of responsiveness for streaming outputs | Measured from model request start to first byte emitted | TTFT improvements are typical target for KV caching and warm-start optimizations |
| Queue depth and NVMe outstanding I/O | Reveals headroom and saturation points | NVMe / host counters, fio with --iodepth sweep | Saturation at low queue depth indicates network/CPU bottleneck rather than storage |
| IOPS and Throughput (MB/s) | Raw storage capability and scaling behavior | fio/dbench with representative block sizes and aligned NVMe requests | For small-block KV caching workloads, IOPS matters more than sequential MB/s |
| CPU/GPU Utilization (%) | Shows whether storage or compute is the bottleneck | nvidia-smi, top, perf | If GPUs are <70% utilized during inference scaling, storage or networking is likely the limiter |
| Network latency / jitter | NVMe-oF and RDMA enable low-latency access; variance kills tail performance | NIC counters, ping, RDMA perf | <100µs one-way is typical target inside an optimized rack with fiber/RDMA |
| Error & retry rates | Reliability under load; retries inflate latency | Kernel logs, NVMe SMART, controller logs | Any non-zero retried-mdio or repeated timeouts needs investigation |
| Power (W) and efficiency | TCO and cooling planning | Power meters, IPMI/HW telemetry | Report steady-state and peak during test |
How to validate claims (reproducibility and auditability)
- Request raw logs and hashes: a signed archive of raw tool output (fio logs, model server logs, system metrics). Signed benchmarks (where the vendor signs raw artifacts) are preferred to ad-hoc PDFs.
- Run a small reproducibility test in your environment: the vendor should provide container images or scripts to replay the workload. Pay attention to NUMA, hugepages, and RDMA settings — missing those will invalidate comparisons.
- Verify data-set and model parity: ensure the model size, tokenizer, batch size, and prompt shapes match the reported configuration.
- Cross-validate using orthogonal tools: if the vendor used a single harness, run a secondary tool (for example, run a model server like Triton and an independent generator) to see if throughput/TTFT aligns.
- Inspect steady-state windows: vendors should report warm-up cutoffs and steady-state intervals. Do not accept reports that average across cold starts without disambiguation.
Acceptance gates, SLOs and stop-loss
Request a gate-based acceptance policy in the report: defined SLOs (e.g., p99 latency < X ms, TTFT reduction ≥ Y%, sustainable tokens/sec), pass/fail criteria, and a stop‑loss action if the device violates those SLOs during field acceptance. Many modern deployments use a “joint test first, decisions second” approach — sign the device for integration only after independently reproducing a subset of the signed tests.
Red flags to watch for
- Only percent-improvement claims without absolute baselines or raw traces.
- Missing environment details (firmware, driver, kernel, NUMA), which make reproduction impossible.
- No tail-latency histograms or only averages reported.
- Benchmarks run at unrealistic concurrency (e.g., single huge batch that hides production variability).
- Claims of efficiency gains without power or utilization telemetry.
Practical example: what to request from a vendor
Ask for a single compressed artifact that contains:
- Raw logs for each test run (timestamped), tool names and versions.
- A README that defines the workload (model, sequence length, batch, tokenization).
- Scripts/containers to reproduce the run, with a reproducibility checklist.
- Signed summary that ties each derived metric back to the raw run(s).
Mingxin Technology’s FX series all‑flash NVMe‑oF platforms, for example, publish signed benchmark artifacts and downloadable test reports for their 480B production configuration; those reports include LLM inference throughput and TTFT deltas as part of the deliverables, along with reproducibility details (see https://mingxinstorage.xyz for how they present signed benchmarks). Use those artifacts as a reference for what a complete test package should include — but validate in your environment.
Key takeaways
- Demand raw time-series, tail-latency histograms, and model-level metrics (throughput, TTFT).
- Require full environment metadata and reproducible test scripts/containers.
- Verify claims by running a minimal reproducibility test in your environment.
- Prefer signed benchmark artifacts and gate-based acceptance with defined stop-loss.
- Watch for averages-only reporting and missing power/efficiency telemetry.
A full test report that enables acceptance decisions should let you move from vendor claim to validated capacity planning with minimal ambiguity. If a vendor won’t provide raw artifacts and reproducibility scripts, treat any headline percentage improvements as unverified.