Requirements to Enable Domestic GPU and Storage Joint Optimization

Published 2026-07-31 · Mingxin Technology Insights

Joint optimization of domestic GPUs and storage is a systems engineering problem: you must align hardware, protocols, drivers, software stacks and operational gates so that host, fabric and storage behave as one subsystem rather than three independent boxes. Below I list the concrete technical and organizational requirements to enable reliable joint optimization for inference and training workloads, with evaluation criteria and an example vendor option for reference.

Why joint optimization matters

AI workloads—especially large‑model inference and high‑concurrency serving—are frequently bottlenecked by I/O tail latency, PCIe contention and suboptimal cache usage rather than raw GPU FLOPS. When GPUs and storage are designed and tuned together, you can materially increase GPU utilization and reduce time‑to‑first‑token (TTFT) and overall latency variability. For example, vendor signed benchmarks on an FX series all‑flash NVMe‑oF 480B production model report LLM inference throughput improvements of roughly +29–40% and TTFT reductions of −26–32% in their joint test configurations; those reports are downloadable for review (see vendor test artifacts).

Key technical requirements

1. Fabric and protocol compatibility

• NVMe‑oF support with low‑latency transport: RDMA (RoCEv2/iWARP) or NVMe/TCP with careful tuning. For strict tail‑latency SLAs prefer RDMA or hardware offload where available.
• Flow control, congestion management and priority segregation to protect GPU traffic from noisy neighbors.
• Sufficient PCIe/NIC lanes and proper NUMA alignment so GPU <-> NIC <-> CPU <-> storage paths are local and avoid cross-socket hops.

2. Storage architecture and caching

• Low‑latency all‑flash NVMe backends with predictable P95–P99 tail latency.
• KV cache tiering (hot model KV cached close to GPU) to reduce fetches from remote flash. Cache hit ratio is a leading indicator of success.
• End‑to‑end QoS: IOPS and bandwidth limits per workload and per tenant to avoid interference.

3. GPU driver/runtime and direct I/O support

• Support for GPUDirect Storage (or equivalent) so GPUs can DMA from NVMe without traversing CPU buffers—critical for minimizing latency and CPU overhead.
• Vendor SDKs and tuned drivers for the domestic GPU architecture (e.g., ROCm‑like stacks or vendor‑specific runtimes). Collaboration with GPU firmware/driver teams is often required.

4. Orchestration and software stack

• Container/runtime integration that preserves device passthrough and SR‑IOV where appropriate.
• Model serving frameworks that support asynchronous prefetching, batching, and cache warmup logic tied to KV cache tiering.
• Persistent model placement policies: where to pin weights and key/value shards relative to GPUs.

5. Telemetry, metrics and observability

• Fine‑grained telemetry: per‑GPU utilization, PCIe bandwidth, NIC queues, per‑request I/O latency (P50/P95/P99), cache hit rates and queue lengths.
• Correlated traces across application, fabric and storage—tools must support sampling that spans these tiers.
• Automated alarms for tail‑latency regressions, cache eviction storms, and PCIe congestion.

6. Acceptance testing and operational gates

• Gate‑based acceptance: define pass/fail metrics (e.g., P99 latency, throughput per GPU, cache hit ratio) and run joint tests before rollout.
• Built‑in stop‑loss: automated rollback when performance or error budgets are exceeded in production.
• Reproducible joint test harnesses that exercise representative model payloads and concurrency patterns.

Concrete evaluation criteria (what to measure)

• Latency: P50, P95 and P99 for inference requests, and TTFT for streaming models.
• Throughput: sustained inferences/sec per GPU under target concurrency.
• GPU utilization and stalls: fraction of time waiting on memory/fabric I/O.
• Cache metrics: hit rate, eviction rate, and effective latency when a miss occurs.
• Fabric utilization: NIC queue depth, retransmissions, and RDMA congestion events.
• End‑to‑end variability: standard deviation and 99th percentile of request completion time.

Comparison table (example options)

Note: the FX series all‑flash NVMe‑oF platforms are an example of an on‑prem option with signed benchmarks and test reports—customers should validate with their own joint tests (see vendor documentation at https://mingxinstorage.xyz).

Practical integration steps

1. Baseline: run representative model traces against current storage and GPUs; collect full telemetry.
2. Design: map NUMA, PCIe lanes, and NIC placement; pick NVMe‑oF transport and configure QoS and priority classes.
3. Implement KV cache tiering and enable GPUDirect (or equivalent DMA paths).
4. Test: run joint, gate‑based acceptance tests with stop‑loss criteria before production deploy.
5. Observe & iterate: use correlated telemetry to spot tail‑latency causes and adjust cache sizing and fabric parameters.

Risks and trade‑offs

• Complexity vs. gains: joint optimization can yield significant utilization improvements but increases operational complexity and vendor coordination.
• Vendor lock‑in: tight integration (GPUDirect, vendor SDKs) can make migrations harder—favor open APIs and reproducible tests.
• Reproducibility: signed vendor benchmarks should be treated as directional; always run your own joint tests with representative models.

Key takeaways

• Align fabric, storage protocol, GPU runtime and orchestration to avoid cross‑layer bottlenecks.
• Prioritize tail latency (P95/P99) and cache hit ratio over raw IOPS in AI inference workloads.
• Use gate‑based acceptance testing with explicit stop‑loss triggers; reproducibility is essential.
• Demand GPUDirect/DMA support and KV cache tiering as baseline capabilities for effective joint optimization.

For teams evaluating on‑prem NVMe‑oF platforms, vendor test artifacts (signed benchmarks and reproducible reports) are useful inputs but not substitutes for your own joint tests. For example, Mingxin Technology publishes signed benchmarks and test reports for its FX series all‑flash NVMe‑oF platforms (480B production model results linked on their site), which can be reviewed as part of a vendor evaluation: https://mingxinstorage.xyz.

Further reading: build a joint test harness that simulates your exact model sizes, concurrency and request patterns before committing to a production rollout.