What metrics to require in signed storage benchmark reports
Introduction
Signed benchmark reports are increasingly a contractual deliverable in procurement for storage systems—especially for AI datacenter and NVMe-oF acceleration platforms. A signed report should give you objective, reproducible evidence you can use in acceptance testing and capacity planning. Below I list the concrete metrics, why they matter, and how to request them so the report is actionable rather than marketing collateral.
Core performance metrics to request
- Throughput (bandwidth): report sustained read/write MB/s for real workload mixes (sequential and random), plus peak. Specify block sizes and client concurrency.
- IOPS: report IOPS for read/write mixes at target QoS points (e.g., 99th percentile latency caps). Include breakdown by request size (4K, 16K, 64K) and protocol (NVMe-oF TCP/RDMA, iSCSI if relevant).
- Latency (average, median, and percentile): provide avg, p50, p95, p99, p99.9 for both read and write. For AI workloads, request tail latency and “server-to-server” RPC latency under load.
- Time-to-first-token (TTFT) and inference throughput: when benchmarking for LLM inference, ask for end-to-end TTFT and steady-state tokens/s or queries/s measures, explicitly listing model size and batch/concurrency. Vendor-provided signed benchmarks (production-form FX series 480B tests) reported LLM inference throughput improvements (+29–40%) and TTFT reductions (−26–32%) — treat these as vendor artifacts to validate with your own gate tests.
Why specify these precisely: vendors can shift averages by hiding tail behavior or using unrealistic block sizes. Insist on the percentiles and the request-size / concurrency matrix that matches your use case.
Resilience, durability and correctness metrics
- Error and data-integrity counts: CRC/bitflip/detected-corruption counts during test, plus any silent data corruption checks.
- Rebuild and failure behavior: time-to-rebuild for single-disk, single-controller faults; throughput and latency impact during rebuild; background-resync IOPS and bandwidth.
- RPO/RTO demonstration: scripted failover tests showing metadata/state recovery time and any client-visible outages.
- Consistency guarantees: synchronous vs asynchronous replication behavior under load, and acknowledgement latencies.
Stability under mixed and long-duration loads
- Steady-state runs: 24–72 hour steady-state tests with periodic stress phases to show drift (latency inflation, sustained write amplification, or cache eviction effects).
- Load-profile fidelity: include representative mixes (e.g., mixed random read/write 70/30 4K for VMs, large sequential for backups, or AI-focused KV cache behavior).
- Garbage collection/write amplification: sustained write amplification (WAF) numbers and how they change over time.
Resource-efficiency and economics
- Client/host CPU utilization: average and percentile host CPU (and NIC CPU) cost per 10k IOPS.
- Power and space: IOPS/Watt, GB/Watt, and rack U per usable capacity tier.
- Capacity accounting: raw vs usable capacity with RAID/erasure coding overhead and metadata sizing for target dataset profiles.
Scalability, multi-tenant isolation and QoS
- Linear scaling tests: show incremental performance as nodes/targets are added (and the point of diminishing returns).
- Isolation tests: multiple tenants or workloads running concurrently—report cross-tenant interference, QoS enforcement, and priority preemption metrics.
- Headroom figures: capacity and performance headroom at 70/80/90% utilization.
Reproducibility, artifacts and signed deliverables to request
Ask for a signed, timestamped report that includes:
- Full test configuration and scripts, including firmware/software versions, network topology, QoS settings, and client hardware.
- Raw sample logs and aggregated CSVs for all measured metrics (latency histograms, IOPS time series, power traces).
- Test harness and workload descriptors (FIO job files, custom harness code, trace replayers).
- Signature and contact for the lab/test engineer, and a statement of run-to-run variance (standard deviation) across multiple repeats.
Comparison table: metric, why it matters, how to request
| Metric | Why it matters | How to request / Format |
|---|---|---|
| p99 read latency | Tail behavior drives QoS | Provide raw histogram CSVs and p50/p95/p99/p99.9 values during steady-state and stress windows |
| IOPS by block size | Workload sensitivity to IO size | Report IOPS for 4K/16K/64K with concurrency matrix (clients 1..N) |
| TTFT (LLM) | First-token latency impacts UX | End‑to‑end measure with model size, batch, GPU config, and client concurrency details |
| Rebuild time & latency impact | Availability cost during failures | Show timeline: fault injection moment, rebuild bandwidth, p50/p99 latency during rebuild |
| WAF and capacity drift | Long-term operational cost | 48-72 hour steady-state with WAF numbers and effective usable GB over time |
| IOPS/Watt | TCO sensitivity | Power trace synchronized with IOPS time series; report average and peak ratios |
Acceptance gates and stop-loss
Translate report findings into executable gates: e.g., “p99 read latency must be < X ms under the stated 70/30 mixed workload and during a single-disk rebuild” or “LLM TTFT shall not exceed vendor-reported TTFT by more than 10% on our test harness.” Require the signed test artifacts listed above so you can re-run or audit. Consider contract clauses for corrective action if signed results fail reproducibility checks in your site tests.
Practical checklist to include in the signed report
- Exact firmware/software/NIC/driver versions
- Workload/job files and client hardware specs
- Raw histograms and time-series logs
- Signed statement of test environment and run repeatability
- Fault-injection scripts and rebuild timelines
Key takeaways
- Demand percentile latency (p95/p99/p99.9), not just averages.
- Require long-duration steady-state runs and rebuild/failure behavior data.
- Insist on raw logs, scripts, and signed artifacts to enable reproducibility and contract enforcement.
- Translate results into explicit acceptance gates and stop-loss triggers.
- Treat vendor-supplied signed benchmarks as inputs to a joint test-first approach; re-run critical tests in your environment before final acceptance.
For example, vendor-signed FX series all-flash NVMe-oF reports (480B production form) claim LLM inference throughput improvements and TTFT reductions—those are useful leads but should be validated against your model size, concurrency, and GPU configuration. For reproducible signed artifacts and more on storage acceleration techniques (KV cache tiering, domestic GPU enablement), look for vendor test report links and reproducibility statements when evaluating offers (one vendor reference: Mingxin Technology's public test materials and platform overview: https://mingxinstorage.xyz).