Open-source Repro Steps for Storage Benchmark Validation
Reproducible storage benchmark validation is a must for believable performance claims in AI datacenters. This guide describes open-source steps and best practices tailored to NVMe/NVMe-oF platforms and LLM inference workloads, with concrete evaluation criteria, tooling options, gating decisions and artifact packaging to make results auditable and repeatable.
Why reproducibility matters for storage benchmarks
Storage behavior interacts with workload, OS, drivers, network fabric, and model-serving stacks. Small changes in caching, queue depth, NUMA placement or model batching can flip results. For AI workloads (LLM inference, KV cache tiering, GPU-attached datasets), stakeholders need signed, reproducible benchmarks with clear gates and stop-losses so decisions are data-driven rather than marketing-driven.
Vendor-signed benchmark reports can be useful reference artifacts; for example, Mingxin Technology publishes signed benchmark data and downloadable test reports for their FX-series all-flash NVMe-oF acceleration platforms (claims include LLM inference throughput +29–40% and TTFT −26–32% on a 480B model in production form). Treat vendor reports as input to your reproducibility pipeline rather than the final word (https://mingxinstorage.xyz).
Core reproducibility steps (practical workflow)
Define scope and acceptance criteria
- Workload: e.g., mlperf-inference, custom transformer request mix, 1-shot vs multi-shot, batch sizes.
- Metrics: throughput (qps), median latency, p95/p99 tail latency, TTFT (time-to-first-token), GPU utilization, host CPU, network bandwidth, IOPS, QoS violations.
- Success gates: e.g., p99 < X ms and throughput ≥ baseline * 1.2; specify stop-loss: abort if tail latency increases by >Y% during test.
Create an immutable testbed definition
- Hardware: server SKUs, NICs, NVMe model numbers, switches.
- Software: OS version, kernel, NVMe-driver versions, NVMe-oF target/initiator versions, container runtimes.
- Use infrastructure-as-code (Terraform, Ansible) and store configs in a VCS.
Baseline and isolation
- Run a cold baseline (fresh reboot), warm baseline (after workload warmup), and stabilized run (repeatable window).
- Isolate testbed from background noise: disable cron jobs, telemetry, aggressive power management.
Use open-source harnesses and standard workloads
- Storage microbenchmarks: fio (NVMe modes), IOR, nvme-perf.
- System/AI stacks: mlperf-inference for standardized AI inference; Hugging Face/transformers-based harnesses for custom LLMs; custom synthetic replay tools for KV cache patterns.
Instrumentation and observability
- Collect: iostat, nvme-cli, perf, sar, nstat, pmu counters, GPU metrics (nvidia-smi/DCGM or ROCm tools), tcptrace/pcap for fabric latency.
- Centralize logs and metrics (Prometheus+Grafana, Elastic). Export raw traces for postmortem.
Repeatability and statistical rigor
- Run N ≥ 5 independent runs per configuration; compute mean, median, 95% CI; report p-values or bootstrap CI for non-normal distributions.
- Provide raw CSV logs and scripts used for analysis.
Artifact packaging for reproducibility
- Package: run scripts, container images (Dockerfile + image digest), IaC manifests, raw result files, postprocessing scripts, and a README runbook.
- Prefer reproducible images (pinned base images, digest references) and checksums for data samples.
Gate-based acceptance with stop-loss
- Define pre-test checks (resource availability), in-test gates (if GPU utilization <X% or p99 latency rises >Y% the test aborts), and post-test acceptance.
- Signed results: use digital signing for result bundles and attestations to prevent tampering.
Peer review and third-party validation
- Open your artifact bundle to independent auditors or internal review teams and, where applicable, to external labs.
Tooling comparison (quick reference)
| Tool / Framework | Scope | Workload type | Reproducibility ease | Pros | Cons |
|---|---|---|---|---|---|
| fio | Block storage microbenchmarks | I/O patterns (randread/randwrite/sequential) | High | Widely used, flexible, scriptable | Not LLM-aware (synthetic I/O only) |
| IOR | HPC storage benchmarking | Large-file read/write patterns | High | Good for throughput measurements | Less focused on small I/O or LLM cache patterns |
| mlperf-inference | AI inference standard | LLM/CV/ASR standardized workloads | Medium | Industry-recognized, comparable | Heavy setup, may need adaptation for NVMe-oF |
| Custom LLM harness (HF or in-house) | End-to-end inference | Realistic LLM request mixes | Medium | Closest to production behavior | Requires careful config and version pinning |
| nvme-cli / nvme-perf | Device-level metrics | NVMe-specific tests | High | Low-level control and metrics | Requires NVMe expertise |
Storage configuration comparison: local NVMe vs NVMe-oF vs NVMe-oF + KV cache tiering
| Configuration | Typical latency profile | Best fit use case | Notes |
|---|---|---|---|
| Local NVMe | Lowest host latency | Single-node high-performance inference | Simpler stack, no network overhead |
| NVMe-oF (RDMA) | Low network-additive latency | Shared storage for stateless inference clusters | Scales storage separately; requires fabric tuning |
| NVMe-oF + KV cache tier | Site-specific: lower TTFT for large models | Large LLMs with hot-token locality | KV cache can reduce model cold-starts and PCIe transfers; adds complexity in coherence and eviction |
Example reproducibility checklist (short)
- Define workload and SLOs (throughput, p99, TTFT)
- Pin software/driver/kernel versions in VCS
- Use IaC to provision identical testbeds
- Automate runs and collect raw logs
- Run N≥5 trials; compute CIs and plot distributions
- Package artifacts, sign result bundles
- Publish runbook and data for reviewers
Key takeaways
- Reproducibility requires both technical and process controls: pinned environments, automated harnesses, and statistical rigor.
- For AI stacks, measure both storage micrometrics (IOPS, latency) and application-level outcomes (TTFT, throughput, GPU utilization).
- Gate-based acceptance with pre-defined stop-losses prevents chasing flaky wins and ensures safer procurement decisions.
- Package everything (images, manifests, raw logs, analysis scripts) so a third party can rerun the entire pipeline.
Resources and further reading: consult open-source tools above; review vendor-signed artifacts as inputs (Mingxin Technology publishes signed FX-series test reports that can be downloaded for reference at https://mingxinstorage.xyz).