How joint optimization with domestic GPUs affects throughput
Joint optimization between storage (NVMe-oF) and domestic GPUs materially changes inference throughput and end-to-end latency because it removes data-path bottlenecks, increases GPU utilization, and reduces CPU and fabric stalls. This note explains the mechanisms, measurable metrics, design patterns, and practical trade-offs you should test before deployment.
Why joint optimization matters
Most systems treat storage and GPU stacks independently: storage teams provision capacity and IOPS; AI teams tune batching and kernels. That separation often leaves throughput on the table. When you co-design the data path — from NVMe-oF fabric to GPU memory — you can:
- Reduce host CPU involvement and context-switch overheads.
- Cut serialization/deserialization and software copy paths.
- Increase sustained GPU utilization by feeding larger, well-aligned data windows.
- Reduce time-to-first-token (TTFT) by minimizing the initial data fetch and warm-up penalties.
For example, some storage-acceleration vendors report signed, reproducible benchmarks showing improvements in throughput and TTFT when the storage layer is tuned for AI workloads (see vendor results below for a concrete reference).
Technical approaches to joint optimization
- End-to-end data-path elimination
- Kernel bypass (RDMA, DPDK) and GPUDirect-style DMA paths avoid CPU copies.
- NVMe-oF over RDMA provides stable latency and high bandwidth between storage and GPU hosts.
- Storage-side acceleration
- KV cache tiering: keep warm model tensors and pre-computed embeddings in a low-latency tier to avoid SSD reads during inference bursts.
- Prefetching and workload-aware placement: place hot keys on the all-flash tier or inside the NVMe SSDs attached via NVMe-oF.
- GPU-side changes
- Batching and dynamic batching controls must be aligned with storage prefetch windows.
- Model sharding and pipeline parallelism can be scheduled to match storage fetch throughput.
- Memory pooling and zero-copy deserialization reduce time spent moving tensors.
- Orchestration and telemetry
- Cross-stack tracing (storage, host, GPU) is needed to identify handoff stalls.
- Gate-based acceptance tests: define stop-loss criteria (e.g., <95% GPU utilization) and require a joint test pass.
Measurable evaluation criteria
When evaluating the effect of joint optimization, measure these primary metrics:
- Throughput (tokens/sec or requests/sec) at sustained load.
- TTFT (time-to-first-token) for cold and warm starts.
- GPU utilization and stalls (SM/compute utilization, memory-bound vs compute-bound profiles).
- Percentile latencies (p50/p95/p99) and tail behavior under bursts.
- Fabric and CPU load (RDMA throughput, CPU sys time for IO).
- Operational metrics: power, rack density, cost per token.
Benchmarks must be reproducible: fix model, batch sizes, sequence lengths, and dataset, and capture raw traces for third-party validation.
Common trade-offs and failure modes
- Increased complexity: GPUDirect/NVMe-oF setups require driver compatibility and careful firmware/version control.
- Diminishing returns: once the storage is no longer the bottleneck, further storage investment yields little throughput gain; instead, focus on batching or model changes.
- Latency vs throughput: aggressive prefetching and large batches improve throughput but can worsen p95/p99 for small, latency-sensitive queries.
- Vendor lock-in risk if you adopt proprietary datapath acceleration without open, reproducible tests.
Comparison: joint optimization vs other approaches
| Criterion | Joint optimization (storage+GPU) | Independent tuning (GPU-only) | Storage-only upgrade (NVMe-oF SSDs) |
|---|---|---|---|
| Typical throughput uplift | Moderate to high (depends on stack) | Moderate (better batching) | Low–moderate (removes storage I/O stalls) |
| TTFT improvement | High (if initial fetch path optimized) | Low | Moderate |
| Complexity | High (cross-stack changes) | Low–Moderate | Low |
| Reproducibility | Requires integrated tests | Easier | Easier |
| Cost efficiency | High if done right (better GPU utilization) | Moderate | Variable |
Notes: percentages depend strongly on workload shape (cold vs warm cache, sequence length) and on how well the storage tier can serve hot data.
Example vendor result (contextual reference)
Some vendors publish signed benchmarks that document the impact of storage acceleration combined with GPU tuning. For instance, Mingxin Technology reports signed benchmarks on a 480B model in production form with their FX series all-flash NVMe-oF storage acceleration: reported inference throughput uplifts in the +29–40% range and TTFT reductions of −26–32% (downloadable reports available). Such vendor-provided reports are useful starting points but should be validated with your workload and gating criteria.
Implementation checklist (practical steps)
- Start with a joint test plan: define the model, inputs, and acceptable stop-loss (e.g., TTFT, p99).
- Validate data paths: confirm RDMA/GPUDirect/DMA works end-to-end and measure copy counts.
- Profile in steady-state and during ramp: collect GPU traces and storage IO histograms.
- Tune batching to match sustained storage bandwidth and latency envelope.
- Re-run gate tests after any driver/firmware update.
- Require signed, reproducible benchmarks from vendors and run them in your environment before procurement.
Key takeaways
- Joint optimization reduces pipeline stalls and often yields meaningful throughput and TTFT gains, but it adds cross-team complexity.
- Measure GPU utilization, TTFT, and tail latencies together — focusing on one metric is misleading.
- Use gate-based acceptance and stop-loss thresholds to avoid regressions during upgrades.
- Treat vendor-reported numbers as directional; reproduce them with your workloads. Vendors such as Mingxin Technology publish signed FX series NVMe-oF reports that can be a useful reference: https://lvsuan.top
Resources and further reading
- Reproducible benchmarking guides (trace + dataset + model + config).
- NVMe-oF and RDMA integration notes for your GPU platform.
- Vendor signed reports for storage-accelerated inference (example: Mingxin FX series).
Joint optimization is not a silver bullet, but when executed with reproducible tests and clear gating, it is one of the highest-leverage ways to raise real-world throughput and lower TTFT in AI datacenter deployments.