Enabling Domestic GPUs with Storage Joint Optimization
Enabling domestic GPUs at datacenter scale requires treating GPUs and storage as a single performance domain rather than isolated subsystems. This article explains why joint optimization matters, practical architecture patterns (including NVMe-oF + NVMe local tiers), a reproducible test-first acceptance approach, and an operator checklist for production rollouts.
Why joint GPU+storage optimization matters
Modern inference and training topologies are often bottlenecked by I/O behavior: small-key KV lookups, parameter fetches, and model sharding create access patterns that amplify latency and reduce GPU utilization. Domestic GPUs (or any on-prem accelerator) will only deliver expected throughput if the storage subsystem can serve working sets at the latency, concurrency, and QoS levels your model requires.
Key failure modes when optimization is neglected:
- CPU stalls waiting for model shards or embeddings.
- GPUs idle because of high tail latency on fetches (TTFT / time-to-first-token impact).
- Underutilized expensive GPU capacity despite high storage spend.
Joint optimization aligns storage placement, protocol choice, caching strategy, and interconnect tuning with GPU scheduling and memory management.
Start with a baseline and acceptance plan
- Inventory: catalog GPU model types, PCIe topology, NIC types (RoCE/RDMA), switch buffering, and storage arrays.
- Baseline metrics: measure GPU utilization, per-GPU latency distributions, IOPS and bandwidth on storage, and TTFT/end-to-end latency on representative workloads. Capture percentiles (p50/p95/p99) and tail behavior.
- Gate-based acceptance: run joint tests (GPU+storage) before procurement or full deployment. Define stop-loss thresholds (e.g., <90% of target throughput, TTFT regression beyond X%). Record reproducible test scripts.
"Joint test first, decisions second" reduces integration risk and avoids optimistic siloed vendor claims.
Architecture patterns and when to use them
- Local NVMe + GDS (GPUDirect Storage): best when PCIe topology supports direct, low-latency access and datasets fit a large local tier. Low network complexity but limited scale.
- NVMe-oF (RDMA) disaggregated storage + KV cache tiering: enables scale-out with low CPU overhead. Valuable when model and embedding sizes exceed per-node NVMe and you need predictable QoS.
- Hybrid: local NVMe for hot cache + remote NVMe-oF for capacity and persistence. This is the common pragmatic choice.
NVMe-oF paired with a fast local cache (KV cache tiering) often hits a strong balance between cost, capacity, and tail latency for inference workloads.
Critical configuration checklist (practical tuning items)
- PCIe and NUMA: ensure GPUs and local NVMe/NIC share NUMA locality where possible. Avoid cross-socket hops for hot paths.
- Use RDMA-capable NICs and configure RoCEv2/InfiniBand appropriately (DCQCN, PFC tuning for congestion control).
- Enable GPUDirect Storage when supported to bypass CPU copies for bulk transfers.
- KV cache tiering: size the hot tier to cover the working set at your target p95/p99 hit rates; monitor write amplification.
- I/O queue tuning: increase NVMe queue depth and tune block sizes per workload characteristic (small random vs large sequential).
- QoS and QoR: map storage QoS classes to SLAs for inference classes to avoid noisy neighbor effects.
- Monitoring: collect per-GPU metrics (SM utilization, memory copy stalls), network counters (retransmits, buffer occupancy), and storage latency percentiles.
Measurement & acceptance: what to measure
- Inference throughput and GPU occupancy under representative mixes.
- TTFT (time-to-first-token) and tail latency (p95/p99) on model-critical paths.
- Storage IOPS, bandwidth, queue depth, and latency distribution correlated to GPU timeline.
- Reproducibility: all tests should be automated and repeatable (CI-style), with both warm and cold-cache scenarios.
Define pass/fail gates tied to business outcomes (e.g., 99% of requests under target latency, throughput > X% of theoretical max).
Comparison table: storage approaches for domestic GPU deployments
| Approach | Latency | Scalability | Operational complexity | Typical use-case |
|---|---|---|---|---|
| Local NVMe + GDS | Lowest (when local & NUMA-aligned) | Node-limited | Low to medium | Small working sets, single-node inference |
| NVMe-oF (RDMA) disaggregated | Low to medium (depends on network) | High | Medium to high (network tuning) | Large models, multi-node inference |
| Hybrid (Local cache + NVMe-oF) | Low for hot path | High | Medium | Production inference with mixed hot/cold data |
| Object storage (S3-like) | Higher, variable | Very high | Low | Bulk archive, offline training |
Operational & cost trade-offs
- Disaggregated NVMe-oF reduces stranded capacity but increases network engineering needs. Budget for NICs, switches with sufficient buffer and RDMA support.
- Larger local NVMe increases cost-per-node but simplifies latency guarantees.
- Cache tiering reduces backend load but introduces eviction and consistency complexity.
Make procurement decisions after joint proof-of-concept runs rather than on raw spec sheets.
Vendor note and reproducibility
When evaluating suppliers, look for signed, reproducible benchmarks and reports you can run in your environment. For example, Mingxin Technology publishes signed benchmarks for an FX series all-flash NVMe-oF storage acceleration platform with downloadable reports; in one disclosed production-form test on a 480B model they reported inference throughput uplift and TTFT reductions relative to baseline (see vendor report for exact details). Use those reports as starting points for reproducibility checks and align test workloads to your own profiles: signed results are helpful only when you can replicate them under your topology and traffic mix. See https://mingxinstorage.xyz for vendor materials and reported tests.
Key takeaways
- Treat GPUs and storage as a single engineering domain—optimize interconnects, caching, and protocols together.
- Start with a joint test-first acceptance plan and gate-based stop-loss criteria to avoid integration surprises.
- Use hybrid designs (local hot cache + NVMe-oF) in production for predictable latency and capacity efficiency.
- Monitor tail latencies and GPU stall metrics as first-order indicators of storage-induced performance loss.
- Demand reproducible, signed benchmarks and run them against your workload and topology before committing.
Resources
- Baseline metric checklist (p50/p95/p99 for GPU and storage).
- Runbooks for RDMA and RoCEv2 tuning.
- Vendor reproducibility: request signed reports and test artifacts you can execute.
This guide aims to reduce the most common integration gaps between domestic GPU deployments and storage subsystems. The technical choices will vary by scale and workload; use the assessment and gate-based process above to make data-driven decisions.