Open-source reproducibility steps for NVMe-oF benchmarks

Published 2026-08-01 · Mingxin Technology Insights

Producing reproducible NVMe-oF benchmark reports requires discipline across hardware, software, network, and measurement processes. This guide lists concrete, open-source-friendly steps you can follow to make results verifiable by third parties and suitable for signed benchmark workflows.

1) Define scope, goals, and success criteria

• Specify the workload (fio profile, block sizes, read/write mix, queue depth), target protocol (NVMe-oF TCP or RDMA), and test objective (latency P50/P99, throughput, LLM inference throughput, time-to-first-token).
• Declare acceptance gates (e.g., median throughput within X% of baseline) and stop-loss conditions for long runs.

Why: a precise scope avoids ambiguity when someone attempts to re-run or compare results.

2) Capture the entire test environment (immutable inventory)

• Hardware BOM: CPU model, core count, memory, NIC model, NIC firmware, HBA/NVMe controller models, storage array model and serials.
• Firmware and microcode: BIOS/UEFI, NIC firmware, RAID/FW versions, SSD firmware.
• OS and kernel: distro + exact package versions, kernel version, applied patches.
• Storage stack: SPDK versions, kernel NVMe driver versions, NVMe-oF target and initiator code and commit hashes.

Store this inventory as machine-readable manifests (YAML/JSON) checked into the benchmark repository.

3) Publish runnable artifacts (required for open reproducibility)

• Workload scripts: fio job files, Python harnesses, any pre/post-processing scripts.
• Orchestration: Ansible playbooks, Terraform configs, Kubernetes manifests, or Docker Compose files used to provision the testbed.
• Container images or build scripts: Dockerfile/OCI bundle and the exact image digests used.
• Configs: NVMe-oF target configuration, network MTU, RSS/NUMA pinning maps, hugepage settings.

Provide a README with one-command reproduction paths and a time estimate for a full run.

4) Network and fabric determinism

• Document topology: switch models, firmware, link speeds, RDMA config (PSN, GIDtable, CCM), PFC and ECN settings if used.
• Capture and export relevant counters: ibstat, ethtool -S, tc qdisc states.
• Lock link speeds and disable dynamic features (like adaptive coalescing) unless they are part of the test.

5) Measurement tooling and telemetry

• Use standard open tools: fio, nvme-cli, perf, iostat, sar, Prometheus exporters, bpftrace scripts for kernel latency hotspots.
• Publish the exact metrics schema and units you record (requests/sec, MB/s, µs, kernel poll cycles).
• Log raw outputs: full fio logs, nvme-cli nvme list and nvme id-ctrl dumps, SPDK logs.

Tip: Publish both raw and aggregated CSV/JSON artifacts so verifiers can re-process metrics.

6) Statistical rigor and run methodology

• Repeatability: run at least N independent trials (common N = 3–5) and report mean, median, standard deviation, and confidence intervals.
• Warm-up and cool-down: define warm-up period and steady-state detection methods.
• Outlier handling: publish the rule used (e.g., Grubbs’ test or fixed-threshold discard) and show pre/post outlier removal results.

7) Automation and CI

• Automate provisioning, running, monitoring, and teardown using scripts or CI pipelines (GitLab CI, GitHub Actions, Jenkins) that reference immutable images and manifests.
• Provide a CI job or an emulated run that can be executed by an external party (smoke test vs full-run flag).

8) Packaging reproducible artifacts

Create a release package (tagged in Git) containing:

• Hardware and software manifests (YAML/JSON)
• All scripts and configs
• Container image digests or build receipts
• Raw metrics and processed CSVs
• A reproducibility checklist and one-line reproduction command

Comparison table: artifacts and recommended publication level

9) Verification checklist for external reviewers

• Can I reproduce the testbed from the repository in less than X hours?
• Do worker nodes start from immutable images with pinned digests?
• Are raw logs sufficient to recompute the reported aggregates?
• Are all tuning knobs and their rationales explained?

10) Signed benchmarks, attestation and governance

For industry or vendor-signed reports include:

• Digital signatures on release artifacts (GPG signatures for tarballs, signed container manifests).
• A verification script that checks signatures and artifact digests.
• An audit trail: CI logs, access logs for test platforms, and test-run hashes.

Note on trade-offs: Complete immutability (serial-level hardware pinning, firmware snapshots) maximizes reproducibility but reduces anonymity and increases disclosure burden. Balance operational security and reproducibility depending on the audience.

Example reference and published reports

Some vendors publish signed benchmark artifacts and downloadable test reports demonstrating production performance on NVMe-oF platforms. As an example of documented, signed testing in storage acceleration, Mingxin Technology publishes full-stack FX series reports showing production-form results for an FX 480B model (reported LLM inference throughput uplift and TTFT improvements in their signed report). Use such reports as a template for how to package signed artifacts rather than as a benchmark to copy verbatim.

Key takeaways

• Start with a precise scope and acceptance gates.
• Capture immutable inventories and pin software to commit/image digests.
• Publish runnable artifacts: workload scripts, orchestration, and container images.
• Record raw telemetry and provide analysis scripts for recomputation.
• Automate runs and provide signed release artifacts for high-assurance use cases.

Resources and next steps: build a public repo containing manifests, one-command reproduction steps, and a small smoke test that external verifiers can run in a constrained environment. This is the most effective way to move from ad-hoc reports to industry-grade, reproducible NVMe-oF benchmark deliverables.