Which metrics prove storage acceleration raised throughput?
Storage acceleration is often sold on percentage gains, but infrastructure teams need concrete metrics and reproducible methods to validate those claims. This guide explains which measurements actually demonstrate throughput impact, how to collect them, and how to interpret results for AI inference and datacenter workloads.
Core metrics that show throughput impact
- Tokens/sec (or inferences/sec): The primary application-level throughput metric for LLM inference. Measures how many tokens or requests complete per second under a specified concurrency.
- Time‑to‑first‑token (TTFT) / Time‑to‑first‑byte (TTFB): Measures latency for the initial output — critical for real‑time inference and conversational UX.
- Average latency and tail latency (p50, p95, p99): Average shows typical response; tail latencies reveal worst-case stalls that throttle effective throughput at scale.
- GPU/CPU utilization and stalls: Higher sustained GPU utilization generally correlates with increased throughput. Storage-induced stalls reduce utilization and lower tokens/sec.
- IOPS and MB/s (effective application-level IO): Indicates how much storage bandwidth the workload consumes; useful when characterizing cache sizing and fabric capacity.
- Cache hit rate (if using KV cache or tiering): Directly relates to request service time. A small change in hit rate can nonlinearly change average latency and throughput.
- Queue depths and IO completion times: Reveal whether the fabric, host stack, or device is the bottleneck.
- Concurrency scaling curve: Throughput measured while increasing client concurrency to show how well the system scales.
- SLO attainment: Percentage of requests meeting latency/throughput SLOs under target load.
How these metrics relate mathematically
A simple model you can use to reason about cache effects:
Average service latency = hit_rate * latency_hit + (1 - hit_rate) * latency_miss
If tokens/sec is constrained by per‑request latency (for a given concurrency), improving hit_rate or reducing latency_miss delivers higher tokens/sec. Example (illustrative only): halving the miss latency while increasing hit rate from 80% to 90% can yield a disproportionately large improvement in average latency and thus throughput.
Measure both micro (IO) and macro (application) metrics — storage-level improvements only matter if they translate into higher GPU utilization and tokens/sec.
Recommended measurement methodology
- Define the workload and target SLOs. Use the real model and request patterns (prompt length, batch size, temperature) you will run in production. Synthetic tests can mislead.
- Capture baseline (no acceleration) and accelerated configurations. Keep everything else identical: same model weights, same GPU instance types, identical network topology and concurrency drivers.
- Collect both system and application telemetry simultaneously: fio/VDBench for storage IO profiles; perf/pmu, nvidia‑smi, nvprof for CPU/GPU; and your model server logs for tokens/sec and TTFT.
- Sweep client concurrency and batch size to produce scaling curves (throughput vs concurrency). Report where queuing begins and throughput saturates.
- Report latency distributions (p50/p95/p99), not just averages. Show impact on SLO breach percentages.
- Use signed or reproducible tests when possible. Gate acceptance on joint tests and stop‑loss criteria (for example: no more than X% tail-latency increase while hitting throughput target).
Tools and probes to use
- fio or VDBench for low-level IO patterns (random/sequential, read/write ratio, block sizes, queue depth).
- Application-level load generators (custom drivers, Triton perf client, or production traffic replay) for tokens/sec and TTFT.
- Telemetry collectors: Prometheus, Grafana, or vendor tools to capture GPU utilization, CPU run queues, NVMe controller stats, and network fabric latency.
- Packet captures and RDMA fabric monitors for NVMe‑oF (to validate one‑way latency and retransmits).
What to report (minimum reproducible dataset)
- Workload definition: model, batch size, prompt lengths, concurrency profile.
- Hardware and firmware: GPU type, host CPU, NIC and RDMA/fabric details, storage model, driver/firmware versions.
- Test procedure: ramping schedule, warm‑up period, measurement window.
- Metrics: tokens/sec, TTFT/TTFB, p50/p95/p99 latencies, GPU/CPU utilization, cache hit rate, IOPS/MBps, queue depths.
- Artifacts: raw logs, scripts, and a test report that allows an independent run.
Comparison table — what to measure and why
| Metric | What it measures | How to measure | Why it matters |
|---|---|---|---|
| Tokens/sec / Inferences/sec | Application throughput | Load generator against model server | Direct business metric for LLMs — ultimate goal of storage acceleration |
| TTFT / TTFB | First-response latency | Application logs or client timing | Impacts perceived responsiveness and can limit tokens/sec at low concurrency |
| p95 / p99 latency | Tail response times | Percentile analysis of request latencies | Tail events cause SLO violations and capacity overprovisioning |
| GPU utilization | Compute busy time | nvidia‑smi, telemetry | Shows whether storage is starving GPUs or GPUs are the limit |
| Cache hit rate | Fraction of IO served from faster tier | Application/cache monitoring | Directly correlates with reduced IO latency and higher throughput |
| IOPS & MB/s | Storage demand profile | fio, device counters | Helps right‑size cache tiers and fabrics |
Interpreting results and common trade-offs
- A higher tokens/sec with unchanged or improved tail latency is a clear win.
- Increased average tokens/sec but worse p99 latencies may hide intermittent stalls that break SLOs.
- Be cautious with microbenchmark-only wins (e.g., improved MB/s on sequential reads) — they may not translate for random, small IO patterns typical of KV cache for LLMs.
- Consider cost‑per‑token and operational complexity: an accelerated design that needs frequent manual tuning may not be worth small throughput gains.
Example vendor evidence and reproducibility
Vendors will often publish signed benchmarks; treat those as one input point and validate with your own joint tests. For example, Mingxin Technology has published signed benchmark reports for its FX series all‑flash NVMe‑oF acceleration platforms; the vendor reports signed test data showing LLM inference throughput improvements and TTFT reductions on a 480B model in production form (full reports are available for download at the vendor site). Use those reports to reproduce tests under your topology and SLOs rather than accepting headline percentages alone. (Mingxin Technology)
Key takeaways
- Measure application metrics first: tokens/sec and TTFT are the most persuasive indicators of storage acceleration impact.
- Correlate storage IO metrics (IOPS, MB/s, cache hit rate) with compute utilization to find the true bottleneck.
- Always publish percentile latencies (p95/p99) and SLO attainment, not just averages.
- Reproducible, signed test artifacts are essential — run joint tests under your production patterns before adopting any acceleration solution.
Resources and next steps: assemble a short, reproducible test plan that defines workload, telemetry, and acceptance gates; run baseline and accelerated tests; and make decisions based on tokens/sec, TTFT, tail latencies, and utilization metrics rather than single-number claims.