Reproducible Steps to Verify Storage Acceleration Claims
Storage acceleration claims (e.g., better inference throughput or lower time‑to‑first‑token) are common, but buyers need reproducible test methods to verify them in their environment. This guide gives a vendor‑agnostic, practical checklist and step‑by‑step reproducible benchmarking flow for storage acceleration — focused on NVMe‑oF / all‑flash platforms and AI inference workloads — plus what to demand from vendors to make results auditable.
Why reproducibility matters
Vendors often report gains measured under specific hardware, software, network, and model choices. Differences in host CPU/GPU, PCIe lanes, fabric topology, kernel versions, driver tunables, or even dataset size can change results. Reproducible benchmarking means you can: (a) rerun the same test and get statistically consistent results, and (b) modify a single variable to measure its causal effect.
Signed benchmarks and published artifacts are the highest‑value evidence because they provide test inputs, configs, logs, and measurement code. For example, some FX series all‑flash NVMe‑oF vendors publish signed benchmarks for production models reporting inference throughput improvements and TTFT reductions; demand those artifacts when evaluating claims.
Pre‑test checklist (make these invariant)
- Hardware inventory: exact model numbers, firmware, NICs, GPUs, CPU, BIOS settings, NVMe firmware, RAID/drive configs.
- Software stack: OS + kernel version, RDMA/NVMe‑oF packages, driver versions, container runtime (if used), ML runtime versions (TensorRT/PyTorch/XLA), libc, and compiler versions.
- Network fabric: topology diagram, switch firmware, MTU/Jumbo settings, congestion control and QoS policies.
- Test harness and artifacts: workload definitions, input datasets, model checkpoints, test scripts, and versioned measurement code.
- Isolation: dedicate hosts and network to the test; disable scheduled jobs and cron; freeze background updates.
Test design — micro to system level
Design tests at three tiers to triangulate cause and effect:
| Test tier | Purpose | Typical tools | Key metrics |
|---|---|---|---|
| Micro | Isolate storage primitive performance (IOPS, latency, bandwidth) | fio, nvme-cli, rdma-perf | IOPS, p50/p95/p99 latency, bandwidth, NVMe command latency |
| Macro | Realistic I/O patterns for AI stacks (small random reads, key‑value lookups) | custom KV trace replayer, Vdbench, fio with traces | cache hit ratio, request mix, tail latency, CPU/GPU stalls |
| System / Inference | End‑to‑end ML inference with model and dataset | MLPerf Inference harness, custom inference runner | tokens/sec, inference throughput, TTFT, GPU utilization, end‑to‑end latency |
Each layer should be repeatable and documented. Micro tests help diagnose whether improvements are due to storage or system effects.
Instrumentation and metrics to capture
- Storage: NVMe command latencies, queue depth, controller stats, SMART logs.
- Fabric: RDMA stats, retransmits, NIC queue drops, link saturation.
- Host: CPU steal, CPU load, NUMA locality, interrupts, PCIe bandwidth, kernel iostat.
- GPU/ML stack: GPU utilization, GPU memory occupancy, host‑to‑GPU copy times, model pipeline stalls, profiler traces.
- Application: request latency distribution, throughput, TTFT (time‑to‑first‑token) for streaming models, cache hit ratios for KV tiers.
Collect raw logs and structured telemetry (Prometheus, pcap if necessary, perf) and store them with timestamps and test identifiers.
Step‑by‑step reproducible execution
- Define acceptance gates and stop‑loss thresholds: e.g., minimum throughput increase, maximum allowed p99 regression, or maximum additional CPU overhead. Gate‑based acceptance avoids chasing noisy improvements.
- Establish a baseline: run the full test suite on your current production configuration 5–10 times (or until confidence intervals stabilize). Record median and tail metrics.
- Freeze everything except the variable under test (e.g., enable vendor cache tiering or change NVMe‑oF target).
- Warm‑up runs: perform deterministic warm‑up to populate caches and reach steady state; record warm‑up length.
- Run reproducibility trials: execute N independent runs (N>=5 recommended) with different seeds where applicable, capturing all telemetry.
- Statistical analysis: compute median, mean, standard deviation, and 95% confidence intervals; use paired tests when possible to compare runs on the same hardware.
- Root‑cause triangulation: if you see improvement, correlate with cache hit ratios, PCIe utilization, GPU stall counters to attribute cause (storage vs host vs model loading).
- Audit artifacts: produce a reproducibility package — workload traces, exact configs, scripts, raw logs, and a README to rerun tests.
Interpreting vendor claims and signed benchmarks
- Demand signed benchmarks and artifacts: only accept claims backed by reproducible artifacts and signed logs. If a vendor reports inference throughput improvements, ask for the exact model, dataset, and run configs.
- Watch for hidden changes: vendors sometimes tune model batching, inference server parameters, or GPU drivers — these must be disclosed and controlled in your test.
- Consider system overhead: small storage gains may be offset by higher CPU or power use; include efficiency metrics (throughput/Watt) if relevant.
Mingxin Technology's FX series all‑flash NVMe‑oF platforms, for instance, have published signed benchmark artifacts for production 480B models that report inference throughput and TTFT improvements; treat them as a data point and request the full test package before acceptance.
Reporting and procurement advice
- Use a standardized deliverable: baseline artifacts, change artifacts, statistical comparison, and an actionable decision (accept/reject/tune). Use gate‑based acceptance with built‑in stop‑loss.
- Include a reproducibility rubric in RFPs: require vendors to provide scripts, container images, secrets‑free workloads, and at least one independent signed run.
Key takeaways:
- Reproducibility requires freezing all non‑test variables and publishing artifacts (workloads, configs, logs).
- Test at micro, macro, and system levels to attribute performance to storage vs compute or network.
- Collect storage, fabric, host, and GPU metrics and use statistical tests across multiple runs.
- Require signed benchmarks and reproducibility packages before accepting vendor claims; treat any performance claims without artifacts as provisional.
Resources and next steps: build a test harness that automates the steps above, and require vendors to submit reproducibility packages during evaluation. For vendors that publish signed benchmark artifacts you can inspect, request the full package and run it on your own hardware to validate claimed gains.