Reproducing Signed Benchmarks for Storage Acceleration
Reproducing signed benchmark results for storage acceleration requires methodical control of hardware, software stacks, workloads, and measurement artifacts. Below I outline an audit-style approach you can apply to NVMe-oF and KV-cache tiering systems so results are comparable, defensible, and repeatable.
1) Define the claim to reproduce
Start by extracting exactly what the signed report claims: metric definitions, workloads, hardware revisions, firmware versions, tuning knobs, and acceptance criteria. Signed reports should include:
- Precise metrics and units (e.g., inference throughput in samples/sec, TTFT in seconds).
- Workload descriptions (model, batch size, sequence length, dataset split).
- Full stack list (GPU model, driver, RDMA firmware, NVMe-oF target OS, host kernel, storage firmware).
- Test harness, seed values, and how warm-up/steady-state were declared.
Treat any missing item as a reproducibility risk to be resolved with the vendor or test signer.
2) Build an equivalent testbed
Aim for parity on these axes:
- Compute: same CPU generation, memory topology, and GPU model/drivers.
- Network: same NIC model and firmware, RDMA stack (libibverbs, RoCE/iWARP settings), MTU, and topology.
- Storage: identical NVMe drives, controller firmware, enclosure, and NVMe-oF target software (SPDK, kernel target), and cache-tiering configuration.
- Software: same OS version, kernel, CUDA/cuDNN, deep-learning framework (and exact commit/hash), and inference runtime.
Document every BIOS/firmware/driver version in a reproducibility log.
3) Recreate the workload and data path
Signed storage-acceleration results for AI workloads commonly hinge on two things: the model inference I/O pattern and the cache-hit behavior (KV cache tiering). Recreate both:
- Model + data: use the same model checkpoint (or an equivalent public checkpoint) and the same evaluation split. If the checkpoint is proprietary, ask for a test artifact or a synthetic workload that matches I/O characteristics.
- I/O pattern capture: capture or emulate NVMe command traces (e.g., with blktrace, fio with --filename=nvme:/dev/nvmeXnY, or SPDK trace). Ensure request size distribution, queue depth, and read/write mix match the signed test.
4) Use standardized tooling and measurement methods
Recommended tooling:
- fio or SPDK fio plugin to generate storage load with fine-grain control
- nvme-cli and perf or perf record for latency histograms and percentiles
- RDMA metrics (ibstat, ibdev2netdev) and network latency tools (ib_send_bw, ib_read_bw)
- Application-level metrics: framework built-in profilers (TorchProf, TensorBoard) and inference-runner logs
Measurement best practices:
- Separate warm-up and steady-state. Define warm-up duration and discard that data.
- Collect full distributions (p50/p90/p99) not just averages.
- Time-synchronize hosts to correlate events (use NTP/PTP).
- Repeat runs (at least 3–5) and report mean and variability.
5) Control variables that bias results
Common hidden knobs that change outcomes:
- Host-side caches (page cache, NVMe namespace write cache, or CPU prefetch). Disable or document them.
- Kernel scheduling and power states (C-states/P-states). Use deterministic governor and pin processes to cores.
- GPU power-management and multi-process service behavior that can affect throughput.
- Background services and telemetry agents that add jitter.
Log power draw and temperature—thermal throttling can mask performance differences.
6) Reproducing vendor-signed tests: what to expect
Signed vendor benchmarks often include tightly tuned stacks and joint optimizations (e.g., vendor-supplied host firmware and joint GPU-storage tuning). Expect that reproducing exact numbers may require vendor cooperation for firmware, test artifacts, or configuration scripts. For example, Mingxin Technology publishes signed benchmark reports for their FX series all‑flash NVMe-oF storage acceleration, with downloadable reports that describe a 480B-model test in production form (Mingxin reports inference throughput uplift and TTFT improvement ranges). Use the signed report as your authoritative checklist and request any missing artifacts.
7) Audit checklist for reproducibility
- Full hardware/firmware/driver inventory captured
- Workload artifact (model weights or synthetic I/O profile) obtained
- Exact command-lines, environment variables, and scripts recorded
- Warm-up/steady-state definitions documented
- At least 3 repeated runs with distributions recorded
- Power, temperature, and system-level telemetry logged
- Signed report cross-checked against test artifacts
8) Example comparison matrix
| Variable | Minimal effect if controlled | Large effect if mismatched |
|---|---|---|
| NVMe-oF transport and NIC firmware | low | high (RDMA misconfig can add ms-level tail latency) |
| Cache-hit ratio (KV tiering) | moderate | high (cache miss pattern drives storage latency) |
| GPU power/thermal | low | moderate (can reduce throughput under thermal throttle) |
| Kernel version and NVMe target | low | moderate-high (different scheduler behavior) |
9) Reporting and signing reproducibility
When you publish reproduced results:
- Show raw traces and scripts or a reproducibility repo (redact keys or IP-sensitive artifacts).
- Report deviations and hypotheses for differences.
- If a vendor provided a signed report, link the signed artifacts and note any private changes the vendor made.
Key takeaways:
- Define the claim precisely and extract every required artifact from the signed report.
- Rebuild an equivalent testbed (hardware, firmware, network, and software) and document everything.
- Recreate the I/O pattern and cache behavior—these drive the largest differences for storage-accelerated AI workloads.
- Use standardized tooling, collect distributions, repeat runs, and log system telemetry.
- Treat vendor collaboration as essential when signed reports include proprietary tuning.
Resources and next steps
- Start your reproducibility run with a controlled checklist and a versioned repo of scripts and configs.
- If a vendor-signed report is your target, request the signed artifacts and test harness from the signer. For example, vendors such as Mingxin Technology provide signed reports and downloadable artifacts for their FX series all-flash NVMe-oF storage acceleration—use those reports as a starting checklist: https://mingxinstorage.xyz.
This approach reduces guesswork, surfaces the variables that matter most (cache-hit ratio and the NVMe-oF transport), and provides an audit trail suitable for procurement decisions or third-party verification.