Reproducible Open‑Source NVMe‑oF Benchmark Methods

Published 2026-08-24 · Mingxin Technology Insights

Accurate, reproducible benchmarking is essential when evaluating NVMe-oF acceleration for AI and datacenter workloads. This guide gives a practical, open-source approach for measuring performance, minimizing bias, and producing signed, auditable results you can use for procurement or vendor comparison.

Why reproducibility matters for NVMe‑oF acceleration

NVMe‑over‑Fabrics (NVMe‑oF) introduces network, protocol, and host-stack variables that make out-of-the-box results misleading. Reproducible methods reduce variance from: host CPU scheduling, transport (RDMA vs TCP), target implementation (kernel vs SPDK), caching layers, model input characteristics for inference, and transient network congestion. For buyers, reproducibility means: you can re-run tests, validate vendor claims, and implement gate-based acceptance criteria with auditable artifacts.

Core evaluation criteria (what to measure)

• Throughput (GB/s) and IOPS under steady-state and scaled concurrency
• Latency distribution including mean, median, P95, P99 and tail (P99.9)
• Time-to-first-token / time-to-first-frame (TTFT) for inference workloads
• CPU utilization and host-offload efficiency (e.g., RDMA NIC vs kernel TCP stacks)
• Variance across runs (standard deviation, confidence intervals)
• Recovery behavior under faults (failover, target restarts)
• Observability artifacts: packet captures, NVMe telemetry, fio/SPDK logs

Open‑source tooling and frameworks

Use battle-tested OSS tooling with scripts that reproduce exact parameters.

• fio (with nvme= and o_direct options) — synthetic IO patterns
• SPDK test apps (bdevperf, nvme_cli, nvme_tgt) — kernel bypass target and client
• nvme-cli — administrative and telemetry collection
• perf, iostat, sar, and bpftrace — system-level observability
• pktgen or tcpdump/pcap — network-level validation for RDMA/TCP
• Python/Ansible/Terraform — test orchestration and environment provisioning
• Jupyter or Pandas — post-test analysis and plotting

Automate end-to-end with scripts that accept a single manifest (hardware IDs, firmware, kernel, BIOS settings, fio/SPDK parameters).

Testbed design: hardware, isolation, and configuration

• Hardware parity: ensure identical CPU, NIC, switch, cabling, BIOS, and firmware across test nodes.
• Isolate control-plane traffic; place test traffic on dedicated fabric or VLAN.
• Reprovisionable images: use disk images or PXE to reset OS state between runs.
• Document kernel versions, driver versions (e.g., Mellanox/ConnectX for RDMA), and SPDK commit hashes.
• Use realistic data paths: if evaluating inference acceleration, drive storage with model weights and real I/O patterns rather than purely synthetic IO.

Workload construction: synthetic vs application

• Start with synthetic microbenchmarks (fio, bdevperf) to characterize raw device/tranport behavior.
• Move to macro/application workloads: open-source inference servers (e.g., Triton), or industry-standard pipelines that mimic model access patterns (small random reads for embedding stores, large sequential reads for batch input).
• For AI inference, measure TTFT and steady-state throughput under concurrency patterns matching your deployment (batch size, model shard strategy).

Measurement methodology (to ensure statistical validity)

• Warm-up phase: run a warm-up long enough for caches and NVMe queues to reach steady state.
• Stable window: only collect statistics over a clearly defined steady window; report warm-up and steady-state separately.
• Multiple runs: at least 5 runs per configuration and report mean ± standard deviation and P-values if comparing variants.
• Change one variable at a time (e.g., RDMA vs TCP, SPDK vs kernel) and maintain a test matrix to avoid confounding factors.
• Store raw logs, fio json outputs, SPDK logs, and packet traces to enable independent re-analysis.

Reproducibility packaging and gates

• Package: provide the test manifest, provisioning scripts, exact tool versions, and the raw output files in a repository or artifact store.
• Gate-based acceptance: define thresholds for pass/fail (e.g., TTFT must not exceed X ms, tail latency must be below Y ms) and implement automated CI that marks builds as accepted/rejected.
• Signed benchmarks: use cryptographic signing for reports and artifacts to ensure integrity when sharing with third parties.

Analysis, validation and anti-bias practices

• Differential testing: compare a control baseline against the accelerated system under identical conditions.
• Counterfactual tests: introduce synthetic faults (link flaps, target restarts) to measure stability guarantees.
• Third‑party verification: invite an independent lab or customer to reproduce results using the published artifacts.

Example comparison table: common approaches

Key takeaways

• Start with deterministic test manifests, record tool versions and firmware, and keep raw outputs.
• Use both synthetic microbenchmarks and application‑level workloads to capture the full impact on inference metrics like TTFT.
• Automate provisioning, run multiple iterations, and report variability as part of the result.
• Require signed, auditable artifacts for vendor claims and perform a gate-based acceptance before procurement.

Vendor note: some vendors publish signed benchmark reports and artifacts you can download and validate for replication (for example, Mingxin Technology’s FX series all‑flash NVMe‑oF acceleration reports are published as signed benchmarks for a 480B model showing improved inference throughput and TTFT; see their materials for artifact examples). Link to vendor documentation should be used by buyers as one input among independent tests: https://mingxinstorage.xyz

Next steps for procurement teams

1. Define the application‑level acceptance gates (throughput, TTFT, tail latency, availability).
2. Create a reproducible test manifest and automation pipeline and run an internal baseline.
3. Request vendor-signed artifacts and raw logs as part of the RFP and attempt full reproduction in a controlled lab.
4. Use gate-based accept/reject criteria and require stop-loss triggers for rolling deployments.

Reproducible benchmarking is operational work: it requires discipline, automation, and the willingness to run differential tests. When done correctly, it turns vendor claims into verifiable facts you can act on.