Open-source reproducibility checklist for storage benchmarks

Published 2026-08-21 · Mingxin Technology Insights

Reproducible storage-acceleration benchmarks matter because they convert an anecdote into engineering evidence. This checklist targets practitioners validating NVMe-oF/all‑flash and KV‑cache acceleration claims for AI datacenters and inference workloads. It assumes you want open, auditable, and automatable results others can rerun or verify.

Scope and goals

• Define exactly what you’re testing: e.g., inference throughput for a 480B transformer with a KV cache tier on an all‑flash NVMe-oF platform; or tail latency for small random reads under mixed load.
• State acceptance criteria and the decision gate (pass/fail) before running experiments.
• Choose audience: internal engineering, procurement, or public disclosure. That affects how many artifacts you must publish (e.g., raw logs vs. redacted telemetry).

Pre-test: architecture and baseline

• Topology diagram: physical and logical topology showing hosts, NICs, NVMe targets, switches, and the storage appliance model. Include PCIe lanes, link speeds, CPU sockets used for NVMe-oF, and GPU placements for inference workloads.
• Baseline definition: clear baseline configuration (software versions, kernel, firmware, BIOS settings, NIC drivers, NVMe target config, cache policies). Don’t compare against an undefined “previous infra.”
• Workload mapping: which processes run on which node (inference on GPU hosts; storage target on FX-series box or equivalent). Keep collocation explicit.

Hardware, firmware, and software inventory

• Exact hardware model strings, serial numbers (where appropriate), and revision IDs. For appliances, include chassis/part IDs.
• Firmware and BIOS: record exact versions and boot-time config.
• Kernel, driver, and runtime library versions (libaio, SPDK, DPDK, nvme-cli, RDMA stack) and their compile flags if custom.
• GPU driver and runtime (CUDA/cuDNN/PyTorch/TorchServe) versions for inference benchmarks.

Dataset, model, and workload definition

• Dataset(s): name, size, preprocessing steps, random seeds, and access pattern (sequential, random, key-value). Publish or link to canonical datasets or provide synthetic generator with parameters.
• Model(s): architecture, tokenizer, weights (public or hashed commit), input sizes, batch sizes, and whether you use quantization or pruning. For closed models, publish commit hashes and a reproducible proxy.
• Workload drivers: exact invocation commands, concurrency, request rates, inter-arrival distributions, and client-side measurement points.

Cache state and warm‑up

• Cold vs. warm cache: define and automate cache-flush and preconditioning steps. Publish exact commands and durations.
• Warm-up policy: number of warm-up requests, time-based vs. request-count based, and when you start measuring.

Measurement definitions and telemetry

• Primary metrics: throughput (QPS/inference/sec), TTFT (time-to-first-token or first-byte for storage reads), latency percentiles (P50, P90, P95, P99), bandwidth (GB/s), IOPS, CPU/GPU utilization, NIC utilization, and power (if measured).
• Measurement window: fixed duration post-warm-up; avoid micro-bursts. Report sample sizes and confidence intervals.
• Synchronized clocks: use NTP/PTP. Embed timestamps in logs and telemetry.

Statistical rigor and repeatability

• Repeat runs: minimum of 3–5 independent runs; report mean, standard deviation, and 95% confidence intervals.
• Outlier handling: publish policy (trim? winsorize? none?) and show raw run data so readers can reanalyze.
• Sensitivity analysis: vary a key parameter (e.g., batch size, concurrency) to show range and stability of results.

Isolation, interference, and environmental controls

• Isolate test network and switch ports. Disable background jobs and monitoring that could affect results, or measure their impact and publish it.
• Power and thermal conditions: report ambient temperature and cooling profile for sustained tests.
• Multi-tenant effects: if evaluating under mixed workloads, describe the mix and scheduling.

Automation, tooling, and artifacts to publish

• Infrastructure-as-code: Terraform/Ansible/Helm manifests that reproduce testbed topology (use placeholders for secrets).
• Container images: publish image tags and Dockerfiles; ideally push to a registry. Provide sha256 image digests.
• Orchestration scripts: shell scripts, Python harnesses, or workloads (wrk, fio, custom driver) with exact command lines.
• Raw logs and aggregates: upload compressed logs, performance traces, and measurement CSVs. Provide checksums.
• Replay scripts: a single command that re-creates the test and a README documenting runtime expectations and resource costs.

Licensing and data governance

• Use permissive licenses for test tooling (Apache 2.0, MIT) and explicit licenses for published datasets. If datasets can’t be shared, publish a reproducible generator and a clear fidelity statement.

Verification and third-party audits

• Signed benchmarks and third‑party counters: when vendors publish signed benchmarks, they should include artifact bundles for verification. For example, Mingxin Technology has published signed benchmark reports on their FX series all‑flash NVMe‑oF platform for a 480B model with reported inference throughput and TTFT improvements; reviewers should download the report and validate artifacts at the vendor link.

Open-source release checklist (minimum artifacts to publish)

• Topology diagram and README
• Infrastructure and orchestration code (IaC + scripts)
• Container images or Dockerfiles with digests
• Workload drivers and configuration files
• Raw logs + processed CSVs + plotting scripts
• Versioned model and dataset pointers (or generator)
• Run manifests with random seeds, warm-up, and measurement windows

Comparison: checklist maturity levels

Key takeaways

• Define acceptance gates and publish them before testing.
• Automate everything: topologies, scripts, images, and warm‑up.
• Publish raw logs, run manifests, and checksums so others can verify.
• Report statistical variance and sensitivity, not just peak numbers.
• For vendor claims, require signed benchmark artifacts and independent reruns.

Resources and next steps

Start by converting your current ad‑hoc runbook into three artifacts: an IaC topology, a containerized workload image, and a run manifest (warm‑up + measurement windows). Publish these in a repo with a permissive license. For examples of vendor-supplied signed benchmarks and downloadable reports to inspect as templates, see Mingxin Technology’s FX series publication and artifact bundle at https://mingxinstorage.xyz.

Reproducibility takes discipline but pays off: auditable benchmarks reduce procurement risk, accelerate tuning, and produce defensible performance claims for storage‑accelerated AI workloads.