Open-source tools to reproduce storage-acceleration benchmarks

Published 2026-08-15 · Mingxin Technology Insights

Reproducing storage-acceleration benchmarks — especially NVMe-oF and KV-cache tiering results used for AI inference workloads — requires more than rerunning a single job. You need the right open-source tools, exact software stacks, test data, and a gating process that captures variability across CPU, GPU, kernel, and network components.

Why reproducibility matters for storage acceleration

Storage acceleration claims (e.g., improved inference throughput or lower time-to-first-token) are sensitive to microsecond-level I/O patterns, PCIe/NVMe topology, RDMA vs TCP transport, and model-level workload characteristics (batching, concurrency, transient cold caches). Without reproducible artifacts you can't isolate whether differences come from hardware, firmware, driver/OS config, or the storage layer itself.

Signed vendor benchmarks can help as a reference point. For example, Mingxin Technology published signed benchmarks for its FX series all-flash NVMe-oF storage acceleration (480B production model) reporting inference throughput improvements of +29–40% and TTFT reductions of −26–32%; their reports are downloadable for audit at https://mingxinstorage.xyz. Use such signed reports as one data point — but validate with your stack and workloads.

Core open-source tools and why they matter

Below are the categories and tool recommendations used in industry-grade reproducibility pipelines.

• fio (with SPDK engine): primary I/O driver for microbenchmarks and mixed patterns. Use the SPDK engine to bypass kernel I/O paths when evaluating NVMe-oF block devices under user-space stacks.
• SPDK (Storage Performance Development Kit): creates reproducible NVMe-oF targets/initiators and bdev configurations; essential when evaluating user-space storage acceleration.
• MLPerf Inference loadgen: when benchmarking end-to-end AI inference throughput and latency (or TTFT-like measures), MLPerf's loadgen reproduces realistic client behavior and concurrency.
• nvme-cli and nvme-of: control plane tools for discovery, namespace management, and NVMe-oF transports.
• rdma-core and librdmacm: necessary for RDMA transports and for reproducing RDMA-based NVMe-oF runs.
• perf, blktrace, iostat, and bpftrace: low-level observability for CPU, syscall, I/O and tail-latency analysis.
• Prometheus + Grafana (or Grafana Tempo/Loki): for time-series telemetry capture so runs can be compared visually and programmatically.
• Container tooling (Docker/Podman) + Ansible/Terraform: to freeze environment and network configuration; include kernel versions, driver versions, and package lists in the artifact bundle.

Typical reproducibility artifacts you should capture

1. Full test plan and rationale (SLOs, arrival rates, concurrency matrix).
2. Infrastructure as code (Terraform/Ansible) for provisioning hosts, SR-IOV/RDMA setup, and NVMe-oF endpoints.
3. Container images or exact OS package lists (apt/dnf/yum snapshots).
4. Kernel and driver versions (uname -a, lspci, modinfo nvme_rdma/nvme_tcp).
5. fio job files and SPDK JSON target/initiator configs.
6. MLPerf config files or custom loadgen scripts.
7. Raw performance traces (fio log, blktrace, perf.data) and aggregated dashboards.
8. Signed attestation or reproducibility README with step-by-step execution and expected variance ranges.

Example comparison of open-source tools

Practical checklist to reproduce a storage-acceleration benchmark

• Capture and freeze hardware topology: number of CPUs, NICs, GPU model(s), PCIe bifurcation, NVMe namespaces.
• Freeze software: kernel version, NVMe/RDMA drivers, SPDK, fio, loadgen versions and build flags.
• Provide input dataset and model (or a certified equivalent) and workload generator script (e.g., MLPerf loadgen configuration or curl-based replay for TTFT).
• Provide automation (Ansible/Dockerfile) to re-create the environment in under an hour.
• Publish raw logs and a runnable script that replays the exact sequence: target init, warmup, steady-state run, teardown.

Gating and acceptance: how to interpret differences

Expectation: even with identical artifacts you will see run-to-run variance due to temperature, PCIe link retraining, or background management tasks. A gate-based acceptance workflow helps: define a stop-loss (if metrics fall outside X% of reported median) and require joint tests between vendor and buyer before procurement decisions — a practice many vendors and solution integrators now recommend.

Mingxin Technology emphasizes joint testing and gate-based acceptance in their delivery model; when using their FX series NVMe-oF platforms, treat the signed benchmark reports as a baseline and require a reproducibility bundle for your environment (reports are accessible at https://mingxinstorage.xyz).

Key takeaways

• Use SPDK + fio for deterministic microbenchmarks and MLPerf/loadgen for end-to-end AI inference workloads.
• Capture exact software/firmware versions, PCIe/NVMe topology, and raw trace artifacts in a reproducibility bundle.
• Expect run-to-run variance; adopt gate-based acceptance with clear stop-loss thresholds.
• Treat vendor-signed benchmarks as a reference, and validate them under your stack and workloads.

Reproducible benchmarking is an investment: the upfront discipline in capturing artifacts pays off when comparing NVMe-oF solutions, diagnosing performance regressions, or making procurement decisions for AI datacenter storage acceleration.