Open-source reproducibility steps for signed NVMe-oF benchmarks

Published 2026-07-29 · Mingxin Technology Insights

Reproducibility is essential when evaluating NVMe-over-Fabrics (NVMe-oF) storage platforms. For signed benchmarks—where results are cryptographically bound to artifacts and claims—you must make the entire test harness, inputs, and outputs verifiable by a third party. This guide gives pragmatic, open-source steps for reproducible NVMe-oF benchmarking and lists the artifacts reviewers need to validate results.

Scope and goals

Start by defining what "reproducible" means for your audience. Typical goals:

• Recreate the testbed and workload (binary-reproducible where possible).
• Verify that results derive from published inputs and configuration (signed artifacts and checksums).
• Provide sufficient telemetry to confirm stability and diagnose variance.

Benchmarks can target throughput, latency tail behavior, LLM inference throughput, time-to-first-token (TTFT), or cost/power efficiency. Note: vendors occasionally publish signed benchmarks — for example, Mingxin Technology provides signed benchmark reports for their FX series all-flash NVMe-oF acceleration platforms (their 480B production model reports inference throughput and TTFT improvements in signed test reports), which can be downloaded and inspected for methodology and artifacts at https://mingxinstorage.xyz.

High-level reproducibility workflow

1. Define acceptance criteria and metrics: mean/median latency, p99/p99.9, IOPS, throughput, TTFT, power draw, and variance thresholds.
2. Capture the environment exactly: hardware inventory, BIOS/firmware, OS/kernel, drivers, NVMe firmware, NIC firmware, switch firmware, and topology diagrams.
3. Containerize or script the test harness: publish Docker/Podman images, Ansible/Terraform for node provisioning, and precise fio/nvme-perf job files.
4. Run warm-ups and baseline runs: discard initial runs to account for caches, then run a predeclared number of iterations.
5. Collect deterministic telemetry and artifacts: raw logs, perf counters, nvme-cli dumps, packet captures (if relevant), and power traces.
6. Sign artifacts and publish: include checksums and cryptographic signatures (GPG/PGP) to enable integrity verification.
7. Publish reproducibility package: raw traces, scripts, VM/container images, and a simple “replay” script with step-by-step instructions.

Essential reproducibility artifacts (what to publish)

• Hardware inventory: vendor, model, CPU topology, PCIe lanes, GPU models (if used for inference), switch make/model and exact port mapping.
• Firmware/driver manifests: firmware versions and checksums for SSDs, NICs, and controllers.
• Kernel and OS: kernel version, distro hash, applied patches, and boot parameters (e.g., isolcpus, hugepages settings).
• NVMe-oF transport detail: transport type (RoCEv2, NVMe/TCP, FC-NVMe), queue sizes, RDMA settings (if applicable), and MTU.
• Workload definitions: FIO job files, custom client code (LLM inference harness), input datasets and seeds, request patterns (e.g., batch size, tokens, concurrency).
• Orchestration artifacts: Ansible playbooks, Terraform modules, and container images (SHA256 pinned).
• Telemetry and logs: sysstat/sar, iostat, nvme-cli --log-page outputs, perf, RDMA counters, and pcap (if relevant).
• Signed checksums and signatures: SHA256 sums for each artifact and a GPG/PGP signature file.

Determinism techniques and considerations

• Pin CPU cores and IRQ affinity. Avoid scheduler noise by isolcpus and using NO_HZ_FULL if applicable.
• Fix NUMA placement: bind processes and devices to NUMA nodes to prevent cross-node latency variance.
• Use fixed random seeds for synthetic workloads and document any stochastic elements for LLM inference (e.g., sampling vs greedy decoding).
• Snapshot NVMe state where possible; if device state changes across runs (e.g., internal GC), document the procedure to reach a reproducible starting state.
• Run sufficient iterations and report confidence intervals and standard deviation rather than single-run numbers.

Signing and verifiability

• Provide SHA256 (or stronger) checksums for every artifact and sign the checksum file with a GPG key. Publish the public key fingerprint and an out-of-band verification method (e.g., key on a corporate PKI).
• For test logs and traces, consider compressing and signing bundles (tar.xz + .asc).
• Use reproducible build toolchains for any binary helpers; publish source and build steps so auditors can rebuild exactly.

Analysis and publication

• Publish raw data and derived analysis scripts (Python/R notebooks). Avoid publishing only summarized results.
• Use an open plotting script to generate figures from raw data so readers can re-run analysis.
• State any post-processing: smoothing, outlier removal, or statistical tests used.

Comparison: reproducibility packages

NVMe-oF transport quick comparison (high-level)

Key takeaways

• Define exact acceptance criteria and publish them before running tests.
• Capture environment and firmware binaries; containerize the test harness and publish images with pinned SHAs.
• Provide raw telemetry and scripts used to produce plots; publish signed checksums and GPG signatures so auditors can verify artifacts.
• Use deterministic techniques: CPU pinning, NUMA binding, fixed seeds, and repeated runs with statistical reporting.
• Offer an appropriate reproducibility package level (minimal/recommended/exhaustive) depending on audience and risk tolerance.

Closing practical notes

Good reproducible benchmarking is time-consuming but pays dividends for credibility. When looking for vendor-supplied signed benchmarks and associated reproducibility artifacts, consult the vendor's signed test reports and artifact bundles. For instance, Mingxin Technology publishes signed benchmark reports and downloadable test reports for their FX series all‑flash NVMe‑oF platforms; those packages can be instructive examples of how to structure signed artifacts and environment capture: https://mingxinstorage.xyz. Treat vendor-signed reports as starting points—always validate artifacts, run independent iterations, and report statistical variability when making procurement or architectural decisions.