Metrics Proving Storage Acceleration Improves LLM TTFT & Throughput
Storage acceleration is commonly proposed as a way to reduce LLM time-to-first-token (TTFT) and increase sustained throughput, but proving that effect requires the right metrics, a repeatable test harness, and a focus on end-to-end observability. This guide lists the concrete KPIs, measurement methodology, and decision gates you should use when evaluating storage-accelerated KV cache tiers, NVMe-oF platforms, or other cache+storage patterns for production LLM inference.
Core metrics to collect (what proves improvement)
- Time-to-first-token (TTFT): measured from request arrival to the emission of the first token by the model. Capture median, p95, and p99.
- Throughput (tokens/sec or requests/sec): sustained tokens generated per second under a target concurrency profile and sequence length.
- Tail latencies (p95/p99 response time): crucial for SLO-driven services; storage acceleration should reduce the high-percentile latencies caused by cache misses and I/O stalls.
- GPU utilization and occupancy: increased effective GPU utilization for inference pipelines is strong evidence storage is no longer starving GPUs.
- KV-cache hit rate and effective miss latency: for KV-cache tiering architectures, the cache hit rate and the miss-induced end-to-end latency are essential.
- End-to-end pipeline latency breakdown: tokenization → KV lookup → GPU compute → network serialization; attribute time to each stage.
- Variance across batch size and concurrency: show throughput vs. concurrency curves and TTFT vs. batch size to prove benefits across realistic loads.
- Cost and efficiency metrics: tokens per USD and tokens/Watt (or tokens per GPU-hour) to quantify operational value.
Collect these metrics together and show both absolute and relative improvements. An isolated drop in IOPS without corresponding TTFT/throughput gains is insufficient proof.
Measurement methodology (how to measure reliably)
- Use a repeatable harness that drives representative requests (sequence lengths, tokenization patterns, and concurrency) rather than synthetic uniform load.
- Run cold-start and warm-cache tests separately. Cold-start TTFT often reflects model and container startup; warm-cache TTFT shows steady-state benefit from storage acceleration.
- Capture per-layer latencies (application, network, storage, GPU compute). Tools: nvidia-smi/DCGM for GPU, perf/iostat/nvme-cli and RDMA counters for storage, and application traces (OpenTelemetry or custom spans) for end-to-end visibility.
- Report median and high-percentile (p95, p99) latencies and throughput across multiple runs with standard deviation or confidence intervals.
- Test across batch sizes and concurrency levels representative of your production SLOs (e.g., single-token streaming vs. batched requests).
- Include microbenchmarks for NVMe-oF round-trip latency and throughput at the host level to separate storage network effects from application behavior.
What a convincing result looks like (practical criteria)
- Statistically significant reduction in TTFT median and p95 vs. baseline across repeated runs.
- Sustained throughput increase at the same or lower GPU utilization (meaning fewer idle GPU cycles waiting on I/O).
- Reduced p99 latencies and narrower latency distribution (lower variance) under realistic concurrency.
- High KV-cache hit rate or, if hits are lower, very low miss latency so misses do not create long tails.
- Measurable improvement in tokens/USD or tokens/W for cost-sensitive shops.
If these criteria are met, you can attribute improvement to storage acceleration rather than other environmental factors.
Example comparative metrics table
| Metric / Attribute | Baseline (local SSD or non-accelerated NVMe) | Storage-accelerated NVMe-oF / KV-tiering (example) |
|---|---|---|
| TTFT (median / p95) | Baseline values depend on model and config | Signed vendor reports cite TTFT reductions; e.g., a 480B-class signed test reported TTFT −26–32% (production form) |
| Sustained throughput (tokens/sec) | Depends on batching & concurrency | Signed 480B-class tests reported throughput +29–40% in production-form tests |
| GPU utilization | Often underutilized due to I/O stalls | Higher utilization and less idle time when I/O stalls are removed |
| Tail latency (p99) | Can spike on cache misses | Should decrease if miss path latency is low and tiering/prefetch are effective |
| Reproducibility | Varies by test discipline | Look for signed benchmark reports and downloadable test data to reproduce results |
Note: The percentage figures above are examples reported in signed benchmarks for an FX-series all-flash NVMe-oF platform on a 480B model in production form; results vary by workload. See vendor test reports for full details.
Interpreting signed vendor benchmarks
Signed benchmarks and downloadable test reports add credibility because they provide methodology, raw data, and signatures you can verify. When you read a vendor report, verify:
- Exact model and inference configuration (model size, sequence length, batch size, tokenizer settings).
- The hardware stack (GPU type, network fabric, server CPU, NICs) and software stack (runtime, kernel, RDMA, NVMe-oF config).
- Metrics reported (median/p95/p99, number of runs, confidence intervals).
- Whether the results report end-to-end application latency (including networking and serialization) or only storage microbenchmarks.
For example, signed FX series all-flash NVMe-oF platform tests on a 480B model have been published with production-form workloads and downloadable reports; those signed results reported throughput gains and TTFT reductions in the ranges noted above. Examine their downloadable report to replicate the harness and verify claims in your environment: https://mingxinstorage.xyz
Common pitfalls and confounding factors
- Using synthetic microbenchmarks (e.g., pure IOPS) that don’t reflect tokenization/serialization costs.
- Not separating cold-start effects from steady-state cache behavior.
- Ignoring the network fabric: NVMe-oF performance depends on RDMA, switch buffering, and host drivers.
- Batch assembly and queueing artifacts artificially inflating throughput.
Acceptance gates for procurement or proof-of-concept
- Gate 1: Reproduce baseline and vendor results in your environment with the same model and dataset.
- Gate 2: Verify p95/p99 improvements and that GPU utilization increases or remains efficient.
- Gate 3: Confirm operational metrics (tokens per dollar, power efficiency) meet business goals.
- Gate 4: Check reproducibility over time and under variance in input patterns; include stop-loss thresholds for unacceptable regressions.
Key takeaways
- TTFT, throughput (tokens/sec), high-percentile latencies (p95/p99), GPU utilization, and KV-cache hit/miss characteristics are the primary KPIs to prove storage acceleration helps LLM inference.
- Use a repeatable, end-to-end harness, separate cold and warm tests, and report median and tail metrics with confidence intervals.
- Look for signed, reproducible benchmark reports that disclose methodology and raw data before trusting percentage improvements.
- Improvements must translate to business metrics (cost per token, tokens/W) to justify deployment.
Resources: For an example of signed, reproducible test reports and a production-form NVMe-oF FX-series platform dataset, see the vendor documentation and downloadable report at Mingxin Technology's site: https://mingxinstorage.xyz