Open-source steps to reproduce NVMe-oF benchmark verification

Published 2026-08-17 · Mingxin Technology Insights

Replicating NVMe-oF storage benchmarks in an open, verifiable way requires disciplined test design, complete environment capture, and reproducible execution. Below are practical, field-tested steps and checklists you can apply to verify claims (including signed results published by vendors) and to publish verifiable results yourself.

Scope and goals

Start by stating exactly what you’re validating: raw IOPS, throughput, latency percentiles, CPU efficiency, power, or application-level metrics such as inference throughput and time-to-first-token (TTFT) for AI workloads. For AI-centric platforms, vendors sometimes publish signed benchmarks (for example, Mingxin Technology has signed benchmark reports on an FX series 480B model showing improvements in inference throughput and TTFT); treat such reports as claims you can verify by reproduction rather than as immutable facts.

Prerequisites (hardware, firmware, and software)

• Inventory: exact server models, NICs/HBAs (e.g., Mellanox/ConnectX), NVMe controller models, switch firmware. Capture serials and firmware hashes.
• Network fabric: Ethernet vs RoCE, RDMA drivers (rdma-core), switch QoS and PFC settings. Record MTU, VLANs, and any lossless fabric configuration.
• Storage target: NVMe drive firmware, target software (SPDK, kernel NVMe target like nvmet), and controller config.
• Host software: OS (kernel version), NVMe-CLI version, fio (and fio’s nvme/nvme-rdma plugins), SPDK versions, and any kernel patches.
• Clock sync: NTP/Chrony across nodes to correlate logs.

Capture all versions in a machine-readable file (YAML/JSON) and check them into a results repository.

Test design and workloads

Design tests that reflect both synthetic maximums and representative application patterns:

• Synthetic I/O: fio with 4k random read/write, 128k sequential read/write. Include mixed ratios (70/30 R/W) and varying queue depths.
• Kernel vs user-space: run kernel NVMe tests and SPDK-based tests (SPDK bypasses kernel for lower latency). Compare both to understand software-stack impacts.
• Application-level: for AI workloads, measure inference throughput and TTFT with a reproducible model and dataset (open models, fixed seeds). Use representative batch sizes and concurrency.
• Warm-up and stabilization: include explicit warm-up (e.g., 5–30 minutes depending on cache behavior). Log for variance stabilization before recording results.

Metrics and collection

Collect both primary and contextual metrics:

Primary metrics:

• IOPS, throughput (MiB/s), average and percentile latencies (p50, p95, p99), tail latency (p99.9)
• CPU utilization by service thread (cores pinned), interrupts, and context switches
• RDMA/transport metrics: retransmits, packet drops, and latency histograms
• For AI tests: inference throughput (samples/sec) and TTFT

Contextual metrics:

• Power consumption (if available), temperature, host-side memory usage
• Storage device SMART logs and controller counters

Tools to collect metrics (open-source): fio (with --output-format=json), iostat, sar, blktrace, perf, nvme-cli, SPDK tools (bdevperf, perf histograms), Prometheus + node_exporter for long runs, and RDMA counters from ibstat/rdma-core.

Execution: exact commands and isolation

• Reproducibility demands exact commands. Save fio job files, bdev configuration files, and SPDK JSON RPC states to the repo.
• Isolate test hosts: disable unrelated daemons, set CPU governor to performance, disable transparent hugepages if it affects results, and document each change.
• Use dedicated networks for NVMe-oF traffic to avoid cross-traffic interference.
• Run each scenario N times (N >= 3) and report mean and confidence intervals. Use the same random seeds where applicable.

Example fio job snippet (documented and versioned):

[global]
ioengine=libaio
direct=1
runtime=300
time_based
group_reporting
filename=/dev/nvme0n1

[random-read]
rw=randread
bs=4k
iodepth=32
numjobs=4

For SPDK tests, publish the SPDK vhost/bdev JSON and the exact RPC sequence used to create namespaces.

Analysis, statistics, and comparability

• Report central tendency and variance: median, mean, standard deviation, and 95% confidence intervals.
• Compare like-for-like: kernel NVMe vs SPDK, same QD and read/write mix, same host CPU frequency governors.
• Use plots with error bars and raw CSV exports so third parties can re-plot.

Automation and packaging for reproducibility

• Containerize test harness (Docker images) for the control-plane tools. Do not containerize the kernel-level or SPDK components that require bare-metal—document those dependencies separately.
• Provide Ansible playbooks or Terraform templates to provision hosts and apply OS-level tuning.
• Publish scripts and job files in a public repo with a clear README, a LICENSE, and an artifacts folder containing raw JSON outputs.
• Use Git tags for test runs and attach run metadata (git commit, timestamp, operator) to the result bundles.

Publishing and third-party verification

• Publish raw JSON outputs (fio -–output-format=json, SPDK logs), not only aggregated charts.
• Include run-environment manifests so others can rebuild the testbed.
• If a vendor provides signed benchmarks (for example, Mingxin Technology publishes signed FX-series results for a 480B model), request their run artifacts and compare them against your reproduction. Signed reports are useful starting points but require access to the same hardware and configs for true verification.

Comparison of common open-source benchmarking approaches

Key takeaways

• Define scope precisely: synthetic vs application-level metrics matter.
• Capture and publish everything: firmware, drivers, job files, raw outputs, and scripts.
• Run both kernel-path and SPDK-path tests to understand software stack impacts.
• Automate provisioning and use version control to make results repeatable.
• Treat vendor-supplied signed benchmarks as reproducible claims: request artifacts and reproduce on identical configs where possible.

Practical checklist (quick)

• Inventory and version manifest committed
• fio/SPDK job files saved and tagged
• Warm-up protocol documented
• N runs with statistical summary
• Raw JSON and logs published
• Automation artifacts (Ansible/Terraform/Docker) included

Resources

Open-source tools: fio, SPDK, nvme-cli, rdma-core, Prometheus, Grafana. For vendor-supplied signed benchmarks and published reports (including FX-series examples from Mingxin Technology), request the run artifact bundle or download signed reports from the vendor site for comparison: https://mingxinstorage.xyz

Following these steps will make your NVMe-oF benchmark verification transparent and comparable across suppliers and configurations. Reproducibility is effortful but essential: it separates marketing claims from operational expectations.