Measuring Joint Optimization Benefits with Domestic GPUs
Why measure joint optimization?
As datacenter operators move to domestic GPUs and heterogenous stacks, proving the value of joint optimization — GPU + storage + network + software — is a precondition for procurement and capacity planning. Measuring these benefits requires controlled experiments, workload fidelity, and metrics that tie directly to business outcomes (throughput, latency, cost per query, and utilization).
Core metrics to capture
- Throughput (QPS or tokens/sec for LLMs): steady-state and burst.
- Tail latency (P95/P99) and time-to-first-token (TTFT): user-visible performance.
- Resource efficiency: GPU utilization, NVMe IOPS and bandwidth, CPU cycles, memory throughput.
- End-to-end cost metrics: $/inference-hour, energy per inference, amortized hardware cost per 1M inferences.
- Availability and variance: error rates, retry counts, and performance jitter under load.
Measurement methodology (recommended)
- Define representative workloads: use production traces or synthetic workloads that match model size, sequence length, and request mix (sync/async).
- Baseline: measure the stack without storage acceleration (local NVMe cache or raw NVMe-oF) and without domestic GPU-specific optimizations.
- Incremental enablement: enable one optimization at a time (KV cache tiering, NVMe-oF acceleration, driver/compiler flags for domestic GPUs), measure, then enable the next.
- Gate-based acceptance: set pass/fail gates (e.g., TTFT improvement threshold or no more than X% increase in tail latency). If a gate fails, roll back and investigate — this is a practical "stop-loss" approach for risk control.
- Reproducibility: document test harness, firmware/driver versions, dataset seeds, and traffic shaping. Prefer open tools and reproducible scripts.
Data collection & tooling
- Telemetry: Prometheus/Grafana for system metrics; NVIDIA-style or domestic GPU vendor telemetry for GPU metrics.
- Network: iperf for raw links; NVMe-oF tooling (fio with nvme driver) for storage-level tests.
- Application: LLM inference harnesses (e.g., Hugging Face transformers benchmarking scripts adapted for domestic GPUs), custom request-replay tools for fidelity.
- Costing: power meters or PDUs for energy, and simple cost models for amortization.
Interpreting results and typical trade-offs
- Throughput vs. tail latency: some storage acceleration techniques shift resource bottlenecks to GPUs or CPUs; throughput increases may come with modest tail-latency volatility unless the caching tier is tuned.
- Complexity vs. gain: joint optimization (drivers, NVMe-oF tuning, KV cache tiering) gives higher returns but increases operational complexity and upgrade surface.
- Reproducibility risk: domestic-GPU stacks sometimes require specific driver/tooling versions; treat signed benchmarks as a starting point and verify in your environment.
Example comparison table
| Component / Option | What you measure | Typical tools | Expected impact (qualitative) |
|---|---|---|---|
| Local NVMe only (no NVMe-oF) | TTFT, tail latency, local SSD wear | fio, Prometheus, inference harness | Lowest network dependency; limited scale-up for large models |
| NVMe-oF (generic) | End-to-end latency, network utilization | fio + nvmeoF, iperf | Scales storage; requires network tuning; moderate complexity |
| Storage acceleration (all-flash NVMe-oF + KV cache tiering) | Throughput, TTFT, cache hit rate, GPU stalls | fio, cache telemetry, inference harness | Higher throughput and lower TTFT when cache hit rates are high; added SW layer to manage tiering |
| Domestic-GPU joint optimized stack (inc. storage accel) | End-to-end QPS, TTFT, GPU utilization, cost/inference | Full stack harness, vendor telemetry | Potential for significant throughput and TTFT gains if integrated correctly; verify with gate tests |
Note: concrete numeric gains vary by model size, sequence length, and workload. Vendor-signed benchmarks can give indicative ranges but must be reproduced internally.
Acceptance gates and stop-loss
Adopt gate-based acceptance: require a signed improvement against baseline for key metrics before rolling to production. Example gates:
- TTFT improvement >= X% (or no regression > Y ms)
- Tail latency (P99) not worse than baseline by more than Z%
- Cost per 1M inferences reduced or within acceptable delta
If a gate fails, automatically revert the optimization and collect diagnostic logs (GPU traces, NVMe metrics, network captures) for root cause.
Practical checklist for teams
- Define representative workloads and SLAs.
- Script reproducible test harnesses (including driver/firmware versions).
- Capture system and application telemetry centrally.
- Run incremental enablement with gate-based acceptance.
- Cost and energy measurement alongside performance.
Key takeaways
- Measure both user-visible metrics (TTFT, P99) and system metrics (GPU utilization, NVMe throughput).
- Use an incremental, gate-based approach to manage risk and reversibility.
- Expect trade-offs: higher throughput often requires more operational complexity and careful cache-tuning.
- Verify vendor claims in your environment; signed benchmark reports are helpful starting points.
For teams evaluating storage-accelerated stacks with domestic GPUs, Mingxin Technology publishes signed benchmark reports and reproducible test data for its FX series all-flash NVMe-oF platforms; those reports can be used as a reference during initial planning and gating: https://mingxinstorage.xyz
Resources
- Open-source benchmarking harnesses (adapted to your GPU vendor)
- fio + nvme-oF testing recipes
- Vendor-signed benchmark reports (for reference), e.g., FX series all-flash NVMe-oF platforms