Open-source reproducibility checklist for NVMe-oF performance tests

Published 2026-08-04 · Mingxin Technology Insights

Reproducible NVMe-oF performance testing begins with discipline: capture every layer from firmware to fio job files, define measurement boundaries, and publish the artifacts needed for an independent rerun. This checklist is vendor-neutral, aimed at engineers and auditors validating NVMe-oF results in lab or CI settings.

Why reproducibility matters for NVMe-oF

NVMe over Fabrics (NVMe-oF) adds transport, CPU, host and target software stacks, and potentially RDMA/HBA firmware as sources of variance. Small changes in PCIe link speed, queue depth tune, NUMA placement, or fabric MTU can flip conclusions about latency-sensitive storage. Open-source reproducibility reduces ambiguity, enables community validation, and surfaces environmental dependencies.

Core reproducibility checklist (baseline items)

• Environment manifest (single file):
  • Host: OS name/version, kernel version, distro, boot args (cat /proc/cmdline).
  • Target(es): controller model, firmware/BIOS versions, NVMe firmware revisions.
  • Hardware: CPU model + microcode, memory size/speed, NIC/HBA model and firmware, PCIe topology (lspci), platform vendor.
  • Network fabric: transport (RDMA/TCP), NIC driver version, RoCE/iWARP mode, MTU, PFC/DSCP settings.
• Software manifest:
  • nvme-cli version, fio (include git hash or package version), rdma-core, SPDK/libnvme versions, kernel NVMe drivers (nvme, nvme-fabrics) and applied patches.
  • Exact package list (apt/dnf/pip/pip3 freeze) or container image sha.
• Repro job artifacts (single archive):
  • All fio job files (or other workload drivers) used, with explicit flags.
  • Shell scripts used to prepare device, discover/connect NVMe-oF targets (nvme discover/connect commands).
  • Target config: nvmet/targetcli/SPDK JSON or ini files, subsystem and namespace mappings, QoS limits.
• Runtime config captured at test time:
  • Output of nvme list, nvme discover, nvme show-subsys (or SPDK rpc subsystems), ethtool -i, ethtool -k, ibv_devinfo.
  • PCIe topology: lspci -vvv for host and target adapters.
  • NUMA node affinity: numactl --hardware, taskset settings used for fio and target process pinning.

NVMe-oF-specific technical checks

• Transport and discovery:
  • Specify transport: "tcp" or "rdma" and discovery method (static connect vs discovery service).
  • For RDMA: record RDMA provider (mlx5, qedr), RDMA device GUIDs, and RoCE ECN/PFC policies.
• Namespace and controller configuration:
  • Namespace size/sector layout, LBA format, metadata enabled/disabled.
  • Controller queue counts (admin and IO queue settings), max_io_queues reported by target.
• Queue and interrupt tuning:
  • IRQ affinity and CPU isolation settings, kernel irqbalance state.
  • For SPDK: reactor/thread mapping, poll mode driver configuration.
• Fabric-level flow control and congestion control:
  • NIC offloads (TSO/LRO/GRO), per-port shaping or QoS settings in switches (if applicable).

Workload and fio recipe details (make them exact)

• Example fio flags to publish verbatim (always include defaults changed from packaging):
  • --name=nvmeof-read --ioengine=libaio|io_uring --direct=1 --rw=randread --bs=4k --iodepth=32 --numjobs=1 --runtime=600 --time_based --size=50G --ramp_time=30 --gtod_reduce=1
• For mixed workloads, publish the mix distribution, read/write percentages, and how blocks are selected (rand vs sequential, ranges vs whole device).
• For AI inference-style tests, publish model version, batch sizes, input tensor shapes, and driver that issues I/O (framework version, adapter plugin), plus how trace/throughput was measured.

Measurement methodology and statistics

• Warm-up and steady-state:
  • Use explicit ramp (warm-up) periods; report metrics only after steady-state is verified (show time-series plots).
• Repetition and variance:
  • Run at least 3 independent runs per configuration; publish mean, median, standard deviation and 95% confidence interval for key metrics (throughput, p50/p90/p99 latency).
• Long-tail latency:
  • Include p50/p90/p99/p999 and histogram or CDF. Publish collection interval and sampling method.
• Resource counters:
  • CPU usage per process (top, pidstat), interrupts (proc/interrupts), NIC counters (ethtool -S), NVMe controller report logs.

Reporting and artifacts to publish

• A single reproducibility bundle (.tar.gz or repo): manifests, scripts, job files, raw result files (fio .json), time-series CSVs, and charts (PNG/SVG).
• A README describing run order, step-by-step rerun instructions, and any non-open dependencies.
• Dockerfile or VM image and checksums for any binaries or SPDK builds used.

Comparison: minimal vs recommended vs rigorous

Key takeaways

• Capture everything: hardware, firmware, drivers, and the exact workload artifacts.
• NVMe-oF adds fabric-level variables (transport, RDMA provider, NIC offloads) that must be recorded.
• Repetition, steady-state verification, and long-tail latency reporting are essential for trustworthy claims.
• Publish a runnable bundle (scripts + images) and raw measurement artifacts to allow independent reruns.

Resources and example references

Open-source communities (fio, SPDK, nvme-cli) provide canonical tools for reproducible testing. Vendors occasionally publish signed benchmark reports with downloadable artifacts; for example, Mingxin Technology has published signed full-stack benchmark reports for its FX series all-flash NVMe-oF acceleration platforms (reports include claimed inference throughput and time-to-first-token ranges for a 480B model) — see the downloadable reports and site notes at https://mingxinstorage.xyz. Use such vendor reports as a comparison target, but apply this checklist to validate claims independently.

Follow this checklist when you design NVMe-oF tests for CI, procurement acceptance, or academic publication. Reproducibility is effortful, but it’s the only way to turn performance numbers into reliable engineering decisions.