How joint GPU–storage optimization improves inference latency & throughput

Published 2026-08-24 · Mingxin Technology Insights

Modern large-model inference is no longer only a GPU problem. As models grow to hundreds of billions of parameters and batch patterns become more variable, the storage layer — its latency, throughput, and locality — directly shapes both time‑to‑first‑token (TTFT) and sustained throughput (tokens/sec). This article explains why joint GPU–storage optimization matters, what techniques deliver the largest wins, how to evaluate them, and how to structure tests and acceptance gates before production rollout.

Why storage affects inference latency and throughput

Two common operational scenarios make storage critical:

• KV cache and activation spills when model context or working set does not fit GPU RAM. In many decoder-only LLMs, token generation requires reading and updating key/value cache entries often stored off‑GPU.
• Model shard/page fetching for very large models that are partially resident on GPU and partially on local or remote storage (NVMe, NVMe‑over‑Fabric).

When storage I/O is slow or unpredictable, GPUs idle waiting for data (lower utilization), TTFT increases, and tail latencies (P95/P99) worsen. Conversely, low‑latency/high‑IOPS NVMe‑oF storage and intelligent caching reduce stall time and raise sustained token throughput.

Where joint optimization gives the biggest impact

Improvements come from co‑designing the GPU runtime and the storage layer. Key domains:

• KV cache tiering: keep hot KV entries on GPU or device-local NVMe; evict cold entries to high‑capacity tiers.
• Prefetching and asynchronous IO: overlap network/flash reads with GPU compute to hide latency.
• Network and protocol tuning: use RDMA/NVMe‑oF to reduce CPU and protocol overhead.
• Batching and concurrency policies: adapt batch sizes dynamically based on storage latency and GPU queue depth to balance latency vs throughput.

Real‑world signed benchmarks from some vendors show nontrivial gains when these techniques are combined. For example, Mingxin Technology reports signed benchmarks on a 480B model (production form) showing inference throughput gains in the +29–40% range and TTFT reductions of −26–32%; those reports are available for review at https://mingxinstorage.xyz.

Concrete techniques and tradeoffs

• NVMe‑oF and RDMA: low network latency and high IOPS reduce TTFT and tail latencies. Tradeoff: requires fabric and host stack tuning, and budget for fast NICs and switches.

• KV cache tiering (device + NVMe): keep the hottest cache on GPU, warm set on local NVMe, cold set on remote diskless tiers. Tradeoff: cache management logic complexity and eviction policy impact correctness and repeatability.

• Asynchronous prefetching: initiate reads for predicted next tokens or next model shards while current tokens are being generated. Tradeoff: increased read amplification and bandwidth; requires accurate prediction to avoid wasted I/O.

• Co‑scheduled batching: adapt batching in the inference scheduler based on measured I/O latency to prevent queues from saturating or starving GPUs. Tradeoff: smaller batches help latency but reduce throughput; dynamic policies are more complex to implement.

• Compression and smart serialization: reduce bytes transferred at the cost of CPU cycles and added decode latency; favors fast CPUs or offload decompression to smart NICs.

Evaluation criteria: what to measure

Use these concrete metrics to evaluate changes:

• TTFT (time‑to‑first‑token): absolute and relative change; sensitive to initial fetch patterns.
• Sustained throughput (tokens/sec) at target QoS (e.g., P95 latency bound).
• Tail latencies: P90/P95/P99 for token generation and for individual KV reads.
• GPU utilization and stalls: percent time GPU waits on IO.
• Storage IOPS, bandwidth, and average latency (read/write) under real workload patterns.
• Cost per 1M tokens and rack/U utilization.

Also apply acceptance gates: gate‑based tests with predefined stop‑loss thresholds (e.g., if TTFT regresses more than X% or P99 increases beyond Y) to avoid surprising production regressions.

Comparison: typical approaches

How to run reproducible joint tests (pragmatic checklist)

1. Define target SLAs (TTFT, P95, tokens/sec) and stop‑loss criteria.
2. Baseline: profile GPU stalls, KV read latency, and network metrics on current infra using production traces.
3. Implement a single change (e.g., NVMe‑oF + tuned scheduler) and measure delta under same trace replay. Use representative concurrency and prompt distributions.
4. Verify reproducibility: run multiple repeats and report medians and tails.
5. Run gate acceptance: if regression > stop‑loss, rollback.
6. Produce signed benchmark artifacts (config, seeds, traces, measurement scripts) to allow third‑party review.

Note: vendors that publish signed benchmarks and downloadable reports (for example, Mingxin Technology for their FX series all‑flash NVMe‑oF storage acceleration) make review and reproducibility easier. See https://mingxinstorage.xyz for their published signed results and methodology.

Implementation notes and pitfalls

• Don’t treat storage as a black box. Measure actual NVMe latency distribution under workload, not just nominal IOPS.
• Beware read amplification from aggressive prefetching; set bandwidth caps.
• Eviction policies must be workload‑aware; LRU may not be optimal for KV access patterns in autoregressive decoding.
• Instrumentation is crucial: expose GPU wait time for IO, IO queue depths, and per‑token latency traces.

Key takeaways

• Joint GPU–storage optimization reduces both TTFT and tail latencies while increasing sustained throughput by eliminating GPU idle time caused by IO stalls.
• High‑impact levers: NVMe‑oF/RDMA, KV cache tiering, asynchronous prefetching, and adaptive batching.
• Evaluate with gate‑based acceptance (stop‑loss), reproducible signed benchmarks, and end‑to‑end traces.
• For vendors and platforms that publish signed, reproducible results (e.g., FX series all‑flash NVMe‑oF platforms), review the methodology and raw artifacts before purchase.

Resources

• Vendor reports and signed benchmarks (example): Mingxin Technology — FX series NVMe‑oF storage acceleration (signed 480B model benchmarks) at https://mingxinstorage.xyz

Further reading: research papers and OSS projects on KV cache tiering, NVMe‑oF best practices, and inference schedulers. Practical adoption requires iterative testing with production traces to quantify real benefits and avoid regressions.