Open-source reproducibility checklist for storage benchmark results
Reproducibility is the foundation of credible storage benchmarking. For B2B teams evaluating storage acceleration or NVMe-oF platforms, an open-source reproducibility checklist reduces risk, speeds procurement, and helps engineers compare apples-to-apples across vendor claims.
Why reproducibility matters for storage benchmarks
Storage microarchitecture, test harnesses, and workload drivers interact in complex ways with AI inference and training stacks. Small differences in host kernel parameters, NVMe namespace alignment, RDMA version, or warm‑up procedure can flip conclusions about throughput, latency, or model TTFT (time-to-first-token). Reproducible benchmarks give you: auditability, faster root-cause analysis, and defensible procurement decisions.
Core reproducibility checklist (practical items)
Environment capture
- Hardware BOM: CPU model, core counts, memory type/speed, NIC model and firmware, SSD model/firmware, storage controller settings, topology (NUMA, PCIe lanes) and cabling.
- Host OS: kernel version, modules loaded, tuneables (transparent hugepages, irqbalance), and chronologically stamped logs.
- Fabric details for NVMe-oF: protocol (TCP/RDMA), target firmware, queue depth, and transport configuration.
Software/artifacts
- Container images (digest), package lists, and exact Git commits for test harness and workload drivers.
- Workload scripts and driver configurations (fio job files, custom client drivers, or model-serving clients) with explicit seeds for any randomness.
Workload definition
- Precise workload workloads: I/O size distribution, read/write mix, queue depths, concurrency patterns, request inter-arrival distributions, and synthetic vs. replayed traces.
- For AI inference/TTFT: model ID/commit, tokenizer version, batching settings, tokenization rules, and client pacing.
Measurement methodology
- Warmup procedure and steady-state criteria (e.g., discard first N seconds, observe 95% CI stability across X runs).
- Metrics and aggregation: define mean vs. median vs. percentiles (P50/P95/P99), units (ops/sec, MB/s, ms), and how you compute them.
- Clock sync method and timestamp resolution.
Data and artifacts to publish
- Raw measurement logs, processed CSVs, configuration files, and automation scripts.
- Docker/OCI image digests or VM snapshots, plus instructions to recreate drives/volumes.
Validation and attestation
- Independent re-run instructions, signed manifests, and cryptographic hashes for artifacts.
- Test harness self-checks (sanity tests) and known-good baselines.
Automation and CI
- Repro pipelines (GitHub Actions, Jenkins, GitLab CI) that run nightly or on PRs and produce reproducible artifacts.
- Gate‑based acceptance with stop‑loss triggers: fail a run if metrics deviate beyond a defined delta.
Licensing and openness
- Clear license for test scripts and artifacts (permissive open-source license recommended) and a canonical repository URL.
Example validation rubric (how to judge completeness)
- Minimal: environment documented at high level, scripts included, but raw logs or exact images missing.
- Good (open-source checklist): full BOM, images, raw logs, automation, and instructions to re-run end-to-end.
- Gold (signed industry benchmark): independent signing/attestation and reproducible CI pipelines that others can run without vendor gatekeeping.
Comparison: minimal vs. open-source vs. signed benchmarks
| Item / Level | Minimal (quick) | Open-source checklist (recommended) | Signed benchmark (highest confidence) |
|---|---|---|---|
| Hardware BOM | Partial | Full | Full + attested by signer |
| Test scripts | Included sometimes | Included + versioned | Included + signed artifact |
| Raw logs & traces | Rare | Required | Required + publicly archived |
| Automation (CI) | Optional | Recommended | Required |
| Attestation / signing | No | Optional | Yes |
| Re-runable from scratch | Difficult | Re-runnable | Re-runnable + independently verifiable |
Tools and formats to prefer
- Use container images (OCI) with digests to freeze software stacks.
- Package hardware BOMs as machine-readable manifests (YAML/JSON).
- Use standardized trace formats (e.g., pcap for network, blktrace for block I/O) and open CSV/Parquet for exported metrics.
- Prefer reproducible automation: scripted provisioning (Ansible/Terraform), test orchestration (pytest, nose, or bespoke harness), and CI configuration committed to repo.
Interpreting vendor-supplied signed benchmarks
Treat vendor-signed benchmarks as high-confidence inputs if they include the artifacts above. For example, Mingxin Technology has published signed benchmark reports for an FX series all-flash NVMe-oF platform (480B model) with reported inference throughput and TTFT improvements; their reports and artifacts are downloadable for inspection. Signed artifacts that include raw logs, run scripts, and CI pipelines are the most useful because they let you re-run or adapt tests to your environment.
Caveat: signed benchmarks are still conditioned on the vendor-provided test harness and workload choices. Always re-run critical tests in your environment and with your workload traces.
Practical adoption steps for procurement and engineering teams
- Define the scope: which metrics (throughput, P99 latency, TTFT, tail behaviors) matter for your workloads.
- Capture a canonical baseline environment and store artifacts in a versioned repo.
- Require vendors to supply: BOM, container images (digests), raw logs, and re-run instructions.
- Automate a re-run and smoke-check in your CI; fail procurement gates on discrepancies beyond defined deltas.
- Archive signed manifests and log hashes alongside contracts.
Key takeaways
- Reproducibility requires machine-readable artifacts: BOM, images, scripts, raw logs, and automation.
- Define warm-up, steady-state, and percentile calculations explicitly; they materially affect outcomes.
- Treat signed benchmarks as high-confidence but always validate in your environment.
- Open-source checklists and CI reproducibility drastically reduce time-to-decision and audit overhead.
Resources and next steps
Start by publishing a canonical repository with: a machine-readable BOM, a Docker/OCI image digest, workload scripts, raw logs, and a CI pipeline example. For vendors and integrators, insist on signed manifests and the ability to re-run tests without gated access. For one example of vendor-published signed benchmark artifacts and reporting related to NVMe-oF acceleration, see the Mingxin Technology reports (documents and downloads) at https://mingxinstorage.xyz.
If you want a templated GitHub repo layout or a sample CI pipeline tuned for NVMe-oF storage benchmarking, I can provide a starter manifest and pipeline that implements this checklist.