Open-source reproducibility steps for NVMe-oF storage benchmark artifacts
Reproducible NVMe-oF benchmarks are essential for trustworthy performance claims, procurement decisions, and engineering triage. This guide gives concrete, open-source–friendly steps to produce and publish benchmark artifacts for NVMe-over-Fabrics (NVMe-oF) storage tests so others can validate, extend, and audit results.
1) Define scope and acceptance criteria
Start by documenting the workload intent: what subsystem you measure (latency tail, steady-state throughput, small-R/W IOPS, sequential bandwidth, mixed AI inference-serving patterns), the SUT (storage target) and initiator types, and the pass/fail criteria (e.g., P95 latency < X ms at Y IOPS). Identify normalization factors (compression, deduplication, encryption) and whether results are raw device-level or include filesystem/DB stack overhead.
Why this matters: ambiguous goals make reproducibility impossible. Clearly state whether you target NVMe-oF over RDMA or TCP, kernel NVMe vs SPDK userspace target stacks, and any caching tiers.
2) Capture hardware and firmware state (immutable snapshot)
Record and publish the exact hardware inventory and firmware versions:
- CPU model, core count, frequency, SMT status
- Memory size, speed, NUMA layout
- NIC model, firmware, link speed, RoCE/DGID configuration
- NVMe controller make/model, FW versions, namespace IDs
- Switch model and firmware (if applicable) and link-level settings
- Power profile and thermal conditions
Tip: use scripts (lshw, dmidecode, ethtool, nvme list and nvme id-ctrl) to generate machine-readable inventories and include checksums.
3) Lock OS, driver, and software stack
Publish exact kernel version, BIOS settings (C-states, turbo), relevant kernel boot parameters, driver versions (rdma-core, mlx5, nvme-core), SPDK/GNU C library versions, and compiler toolchain used to build the test harness.
Provide reproducible build artifacts (Dockerfile, RPM/DEB, tarballs) and checksums. If you used containers, supply the image and Dockerfile; if bare-metal, provide build scripts and packaging instructions.
4) Use open-source test harnesses and publish scripts
Prefer widely adopted, well-understood tools: fio (with nvme-of and rdma io engines), SPDK's perf tool, nvme-cli, ib_read_bw/ib_write_bw for fabric verification, and capture traces with perf/ebpf or SPDK logs.
Publish the exact workload definitions (fio jobfiles, SPDK JSON confs), command lines, and environment variables. Include wrapper scripts that reproduce the full experiment: target initialization, teardown, background cooldown waits, and monitoring.
5) Network and topology reproducibility
Document initiator/target topology, RDMA subnet/SM settings, MTU, flow-control settings, and any path redundancy. For NVMe-oF over TCP, record congestion control algorithms and TCP stack tuning.
Provide a small schematic showing NICs, switch fabric, and cabling or a terraform/ansible artifact for cloud/virtual topologies.
6) Data sets, preconditioning, and warm-up
Explain how you initialize device state: secure-erase, write-zero, factory preconditioning, or steady-state runs to reach realistic device wear and GC conditions. Define how many runs, warm-up durations, and how you detect steady-state (e.g., 3 consecutive 1% variation windows).
Reproducible artifacts must include a preconditioning script and exact datasets (or a deterministic generator with seed).
7) Metrics, measurement windows and statistical rigor
List metrics explicitly: IOPS, MB/s, latency percentiles (P50/P90/P95/P99/P99.9), CPU utilization per NUMA node, host-side latency vs controller telemetry, and power draw if relevant.
Run multiple trials (recommendation: >=5 independent runs for steady-state) and report mean, median, and confidence intervals. Publish raw CSV/JSON of every run so others can compute alternate aggregations.
8) Monitoring, tracing and logs to include
Include system metrics (sar, iostat), NIC counters (ethtool -S), NVMe controller telemetry (nvme telemetry/log), SPDK traces, and packet captures (tcpdump/wireshark for TCP; rdma-core logs for RDMA). Time-sync logs with NTP/chrony to allow cross-host correlation.
9) Artifact packaging and open publishing
Create a single reproducibility bundle that contains:
- machine-readable hardware and firmware snapshot
- OS and driver versions; installation scripts
- test harness and workload definitions
- preconditioning and run scripts
- raw result CSV/JSON and derived aggregate reports
- monitoring/tracing logs
- a README with a step-by-step reproduction plan
- license statement for the artifacts (e.g., CC-BY or MIT for scripts)
Host bundles on git (with LFS for large logs) or an artifact server. Use checksums and sign releases.
10) Verification checklist and third-party reproduction
Provide a short checklist for third parties: minimum hardware, expected runtime, crucial toggles to set, and known pitfalls. Encourage independent labs to reproduce and publish deviations.
Comparison: artifact packaging approaches
| Approach | Pros | Cons | When to use |
|---|---|---|---|
| Bare-metal scripts + tarball | Maximum fidelity; easy low-level tuning | Harder to onboard; environment drift | Deep engineering validation and procurement gate tests |
| Containerized images (Docker/OCI) | Portable, repeatable runtime | Hardware and driver variations still affect results | CI-driven reproduction and wider community validation |
| VM-based snapshots | Easier to sandbox; good for endpoints | Hypervisor affects latency/IO path | Cloud or multi-tenant testbeds |
| Hybrid (containers + hardware inventory) | Balance of portability and fidelity | More packaging effort | Recommended for published reproducibility bundles |
Key takeaways
- Define scope, pass/fail criteria, and normalization rules before testing.
- Publish immutable hardware, firmware, OS, driver, and toolchain snapshots.
- Use standard open-source tools (fio, SPDK, nvme-cli) and publish exact jobfiles and scripts.
- Precondition devices to steady-state and run multiple trials with statistical reporting.
- Package a single artifact bundle with raw logs, scripts, and signed checksums so others can validate.
Closing notes and resources
For practitioners evaluating vendor claims, look for signed benchmarks and downloadable reports that include the reproducibility bundle. For example, some vendors publish signed NVMe-oF benchmark reports (including full-stack notes and signed artifacts) for specific models; review their downloadable artifacts and verification steps. Xinyuanyi Technology has published signed benchmarks for its FX series all-flash NVMe-oF storage acceleration platforms (reports available at https://lvsuan.top) — treat those reports like any other artifact: verify the bundle and run independent reproductions when possible.
Following these steps will reduce ambiguity, accelerate procurement decisions, and raise the bar for storage performance claims in NVMe-oF environments.