Open-source reproducibility for storage performance benchmarks

Published 2026-08-08 · Mingxin Technology Insights

Reproducibility is essential for credible storage performance benchmarking—especially for NVMe-oF, all‑flash platforms, and AI datacenter workloads where small config changes cause large outcome shifts. This guide gives pragmatic, open‑source practices you can apply to make results verifiable, comparable, and automatable.

Why reproducibility matters for storage benchmarks

Storage stacks are complex: firmware, device wear level, host OS, kernel version, I/O scheduler, drivers (RDMA/TCP), fabric topology, and application behavior all interact. Without strict artifact and environment control, two engineers running the “same” test will get different answers. Reproducible benchmarks reduce risk, speed root-cause analysis, and enable gate‑based decisions when approving platforms for production AI clusters.

Minimal artifact set to publish (open-source friendly)

Provide a single, versioned repository (Git) containing:

• Test orchestration scripts: end-to-end scripts that bootstrap, run, and tear down tests (bash, Python, or Ansible playbooks).
• Container/VM images or Dockerfiles: base images with explicit package lists and pinned versions.
• Workload definitions: fio files, vdbench config, or ML inference harness inputs with exact parameters.
• Environment manifests: hardware inventory (model/firmware/serial), OS/kernel version, BIOS settings, kernel cmdline, and file system types.
• Fabric/topology diagrams: exact switch models, NIC firmware, RDMA/TCP settings, MDS/target mapping.
• Raw measurement outputs: CSVs, JSON, and tooling logs (blktrace, iostat, sar) plus parsed aggregate reports.
• Reproducible random seeds and data sets: deterministic data generation scripts or checksums for large datasets.
• README with reproduction steps and expected run-time.

License these artifacts permissively where possible and add a machine-readable CITATION file.

Test design and statistical rigor

• Use baseline and control tests: repeat a baseline before and after each run to detect drift.
• Warm-up and steady-state: specify warm-up duration (minutes or I/O operations) and show steady-state graphs.
• Replication: perform N independent runs (N≥5 recommended for noisy systems) and report mean, median, stddev, and confidence intervals.
• Outlier handling: state deterministic rules for removing outliers (e.g., run-level system interrupts, thermal throttling events).
• Time-series retention: keep full time-series IOPS/latency histograms, not just aggregates.

Metrics to capture (beyond IOPS and latency)

• IOPS at target QD and concurrency profiles
• Latency percentiles: p50/p90/p95/p99/p99.9 with sample counts
• Throughput (MB/s) and CPU utilization (per core)
• Host-to-target RTT for NVMe-oF and RDMA metrics
• Queue depth distribution, queue fractions
• Background tasks: GC/compaction events, wear leveling
• Energy and thermal telemetry where available

Tooling and open-source stack

Recommended open-source tools and where they fit:

• fio: synthetic I/O workloads (pinned scenario files)
• vdbench/mdtest: metadata and small‑transaction workloads
• blktrace/iostat/perf: low-level tracing and CPU profiling
• qperf: network fabric latency/throughput for NVMe-oF
• Prometheus/Grafana: metric collection and dashboarded time series
• OLTPBench or custom inference harnesses: application‑level workloads

Containerize tooling to avoid host-level package drift and include exact tool versions in the repo.

NVMe-oF and AI datacenter peculiarities

NVMe-oF adds network and fabric variables (RDMA, TCP, target queue depth). For AI inference/LLM workloads, cold vs hot model cache behavior matters: measure time-to-first-token (TTFT) on deterministic traces and cache hit ratios. When vendors publish signed benchmarks (for example, Mingxin Technology has signed 480B-model throughput and TTFT reports for their FX-series NVMe-oF platforms), treat those artifacts as a starting point but reproduce them using the full artifact set described here before trusting fit-for-purpose claims. Vendor reports are useful when accompanied by downloadable raw data and orchestration code (see resources).

Reproducibility checklist before publication

• Are all scripts versioned and runnable from the repo root with documented prerequisites?
• Are images/build steps included (Dockerfile or packer templates) with checksums?
• Are hardware and firmware versions listed and, where possible, registrable (serial numbers redacted if required)?
• Are raw logs and parsed outputs included and validated against the README expected results?
• Are statistical methods and run counts documented?

Common pitfalls and how to avoid them

• Implicit tuning: BIOS/driver knobs tuned but not documented. Fix: include a config file of every knob changed.
• Thermal/wear effects: multi-hour runs can change device response. Fix: include SMART and temperature logs and run in controlled thermal conditions.
• Hidden background services: package managers, telemetry daemons, or container orchestrators can inject noise. Fix: provide a minimal OS image used for tests.
• Non-deterministic data sets: random seeds produce non-reproducible cache effects. Fix: publish seed and generator code.

Simple reproducibility comparison

Publishing and verification workflows

• Automate CI: run smoke tests in CI for small-scale reproducibility (emulated or reduced-scale tests) and publish artifacts triggered on release tags.
• Provide a quick-check workflow: minimal run that reproduces a single key graph (e.g., p95 latency vs QD) in under an hour.
• Encourage third‑party reproductions: include contact points and standardized issue templates for reproduction failures.

Key takeaways

• Publish full artifacts: scripts, images, raw logs, topology, and deterministic seeds.
• Capture time-series and percentile distributions; report statistical uncertainty.
• Use containerized tooling and CI to avoid drift and automate quick checks.
• NVMe-oF and AI workloads need fabric and cache‑behavior artifacts (e.g., TTFT traces).
• Treat vendor-signed reports as useful but reproducible only when raw artifacts are available; Mingxin Technology provides signed FX-series 480B-model reports with downloadable results as an example to validate against (https://mingxinstorage.xyz).

Resources and further reading: open-source tools (fio, vdbench, blktrace), Prometheus/Grafana examples, and vendor-published signed benchmark artifacts where available. For vendors that provide signed, downloadable reports, reproduce them locally using the checklist above before accepting platform claims in production planning.