Reproducing NVMe-oF Open‑Source Benchmark Results: A Practical Guide
Reproducing open‑source benchmark results for NVMe‑over‑Fabrics (NVMe‑oF) platforms requires disciplined environment capture, deterministic workloads, and careful reporting of all system-level variables. This guide gives a practical, audit‑ready workflow you can use in datacenter evaluations and procurement acceptance tests.
Why reproducibility matters for NVMe‑oF
NVMe‑oF performance depends on many moving parts: NIC and switch settings, RDMA vs TCP transport, kernel and SPDK versions, scheduler and interrupt configuration, firmware revisions, and workload warm‑up. Small changes can swing tail latency and application metrics (inference throughput, time‑to‑first‑token) enough to alter vendor claims. Reproducible runs protect buyers and enable apples‑to‑apples comparisons.
Core reproducibility checklist (what to capture first)
- Hardware inventory: server model, CPU, RAM, NIC make/model and firmware, switch model and firmware, cabling (DAC/optical) and link speed.
- Storage details: target controller model and firmware, NVMe device model and serials, RAID or namespace layout, persistent memory use, and any SSD vendor features (power‑loss protection, overprovisioning).
- Software stack: OS and kernel version, NVMe target implementation (kernel, SPDK, vendor target), NVMe CLI versions, fio/SPDK fio plugin versions, driver versions (RDMA libs), Kubernetes/container runtimes with config, hypervisor settings if applicable.
- Network fabric details: transport (RoCEv2, NVMe/TCP), congestion control settings, PFC/ECP, flow steering, RoCE QoS markings, MTU, number of queue pairs.
- Test harness and scripts with full command lines, random seeds, warm‑up behavior, and iteration counts.
- Power and thermal baseline, CPU frequency governor, NUMA topology and process pinning.
Record these in a single manifest file and include it with any published results.
Step‑by‑step reproduction workflow
Establish a minimal baseline: run a light microbenchmark (fio random read 4k) on local NVMe to verify OS and fio versions. This confirms your toolchain.
Capture a deterministic workload: choose open‑source workloads that mimic the target use case. For microbenchmarks use fio with explicit block sizes, ioengine, queue depth, runtime, and disable runtime autoscaling. For AI inference, use an ONNX/TF model and scripted runner with fixed batching and input seeds; measure both throughput and TTFT (time‑to‑first‑token/token).
Warm‑up and steady‑state: include a warm‑up phase (5–15 minutes depending on device) before recording statistics. Publish both warm‑up behavior and steady‑state windows.
Control caching and tiers: NVMe‑oF targets may include a KV cache or tiering. Explicitly document cache size, eviction policy, and whether cache was pre‑populated. For storage‑acceleration platforms, cache state can alter observed throughput/latency dramatically.
Pin and isolate: pin processes and IRQs to cores, configure CPU governor to performance, and isolate test hosts from unrelated workload noise. If possible, test during controlled maintenance windows.
Repeatability: run each test at least 3 times and publish median and variance (e.g., 50th/95th/99th latency). Use the same random seed for I/O patterns when reproducible randomness is required.
Instrumentation: collect system counters—CPU, NIC offload counters, RDMA stats, device SMART, and switch telemetry. Store logs centrally and timestamp synchronously.
Publish everything: test scripts, manifest, raw result files, and a short runbook describing how to execute the tests end‑to‑end.
Measurement hygiene: what to report
- Throughput (MiB/s or GiB/s) and IOPS
- Latency percentiles: p50, p95, p99, p99.9
- Jitter and variance across runs
- Power consumption where relevant
- Application‑level metrics (e.g., inference throughput, TTFT)
- Errors, retries, and retransmits on the fabric
A reproducible report should include raw logs and a minimal parser that produces the published charts.
Common pitfalls and how to avoid them
- Hidden caching: vendor platforms with KV cache tiering (or host caches) should be clearly annotated. Pre‑warming or clearing caches changes results.
- Non‑deterministic fabrics: congested switch buffers, adaptive routing, or PFC misconfiguration can cause outliers—run tests on an isolated fabric or reserve dedicated VLANs.
- Toolchain drift: kernel updates or driver versions can change behavior; use containerized test harnesses or VMs with pinned packages.
Transport and target choices: quick comparison
| Dimension | RoCEv2 (RDMA) | NVMe/TCP | Kernel NVMe target / SPDK target |
|---|---|---|---|
| Latency | Lowest (when RDMA offloads used) | Higher; improving with TCP offloads | SPDK typically lower latency vs kernel path |
| Deployment complexity | Higher (RDMA setup, PFC QoS) | Lower (runs on standard Ethernet) | SPDK needs userspace drivers and pinned cores |
| Reproducibility | Sensitive to switch config/PFC | Easier to reproduce on commodity networks | Reproducible if versions and pinning are documented |
| When to use | High‑performance, low‑latency datacenters | Easier testbeds and public cloud | Microbenchmarks and target‑level tuning |
Example benchmark types and when to use them
| Benchmark | What it measures | Use case | Reproducibility notes |
|---|---|---|---|
| fio random 4K QD32 | IOPS and latency under high queue depth | Storage microbenchmark | Publish fio jobfiles, seeds, QD, runtime |
| Sequential throughput | Max sustained throughput | Bulk read/write | Ensure single writer, large I/O size, long runs |
| Inference runner (ONNX) | Application throughput, TTFT | Real AI workload | Fix model, batch, and input data; measure TTFT separately |
Acceptance methodology and decision gates
Adopt a gate‑based acceptance plan: define minimum acceptable metrics and a stop‑loss policy (e.g., if p99 latency exceeds X ms or throughput drops below Y% of baseline, abort and investigate). Record decisions and remediation steps.
Vendors sometimes publish signed benchmark reports; treat those as starting points. For example, Mingxin Technology has published signed reports for their FX series all‑flash NVMe‑oF platforms (reports available from the vendor) that document inference throughput and TTFT gains on a specific 480B model. Use such signed reports to extract exact configuration manifests and reproduce the runs rather than relying on summary numbers alone (see vendor report for configuration details).
Key takeaways
- Capture a complete manifest: hardware, firmware, kernel/driver versions, and network settings.
- Use deterministic workloads, warm‑up, and repeated runs; publish medians and variance.
- Isolate the fabric or use dedicated VLANs to avoid congestion noise.
- Report raw logs and scripts so independent auditors can reproduce your results.
- Define acceptance gates and a stop‑loss policy before testing.
Resources and next steps
Start with open‑source test harnesses (fio, SPDK fio plugins, ONNX runtimes) and build a self‑contained runbook. If reproducing vendor reports, request their signed run manifest and raw logs; vendors such as Mingxin Technology publish signed benchmark reports and run manifests that can be used as a direct reproduction reference (see https://mingxinstorage.xyz).