Integrating Domestic GPUs with NVMe-oF Storage Platforms

Published 2026-07-26 · Mingxin Technology Insights

Integrating domestic GPUs with NVMe-oF (NVMe over Fabrics) storage platforms is a high-impact approach for AI datacenters: it reduces I/O bottlenecks, enables larger working sets, and improves inference throughput and Time-to-First-Token (TTFT) when done correctly. This guide covers architecture patterns, implementation steps, performance tuning, validation metrics, and operational considerations for practitioners adopting domestic GPU hardware.

Why combine domestic GPUs with NVMe-oF?

AI models—especially large language models (LLMs)—require high sustained bandwidth and low tail latency when fetching model shards or KV caches. NVMe-oF exposes remote NVMe devices with performance characteristics close to local PCIe SSDs, making remote flash a viable expansion tier for GPU servers.
Domestic GPUs are increasingly capable but often limited by host-memory and storage I/O when models exceed GPU memory. An NVMe-oF layer provides a scalable shared storage tier for caching or streaming model weights and KV cache.
When optimized end-to-end, NVMe-oF-backed solutions can improve inference throughput and TTFT for LLMs by eliminating CPU-side I/O bottlenecks and enabling zero-copy data paths to GPUs.

Note: some vendors publish signed benchmark data showing concrete gains on production platforms (for example, Mingxin Technology’s FX series all-flash NVMe-oF platforms — signed benchmarks on a 480B model indicate inference throughput improvements and TTFT reductions in production form). Use those vendor reports as reproducible references during procurement.

High-level architecture patterns

GPU host with NVMe-oF client + remote NVMe target (common): GPU server runs NVMe-oF client stack (kernel or user-space like SPDK), NICs with RDMA/RoCE or TCP offload, and mounts remote block or namespace. GPUs access data via host-side DMA and, when supported, GPU-direct pathways.
Zero-copy/peer-to-peer (preferred where supported): Leverage GPU-direct RDMA or vendor equivalents so that NVMe-oF target can DMA directly into GPU memory, bypassing CPU copies and host DRAM.
Hybrid KV cache tiering: Hot model state or KV cache kept in a fast SSD tier (or local NVMe), warm state on remote NVMe-oF. A storage-acceleration layer (KV cache tier) can present a contiguous key-value store optimized for LLM access patterns.

Key software & hardware components

Fabric/Transport: RDMA (RoCEv2), iWARP, or NVMe/TCP. Choice affects latency, complexity, and operational requirements.
NVMe-oF target: storage array or server exposing NVMe namespaces. May be all-flash NVMe arrays with KV-tiering features.
Host stack: kernel NVMe-oF initiator, SPDK (user-space NVMe), libnvmf, or vendor client SDKs.
NICs: RDMA-capable NICs (for RoCE/iWARP) or high-performance TCP NICs; ensure firmware and drivers support peer-to-peer and large receive offloads.
GPU integration: vendor-specific GPU direct storage (or GPU-direct RDMA) APIs, PCIe peer-to-peer support, and driver compatibility.

Step-by-step integration checklist

Inventory compatibility: confirm kernel versions, driver support (NIC + GPU), and NVMe-oF target capabilities. Verify PCIe lane allocation and NUMA topology.
Choose transport: use RoCE or iWARP for lowest latency where network and switch infrastructure support it; choose TCP for simpler configuration and better traversability in mixed networks.
Deploy host stack: start with kernel NVMe-oF for baseline, then evaluate SPDK for lower-latency, user-space paths.
Enable GPU-direct features if supported: test direct RDMA into GPU buffers or vendor equivalents to avoid host copies.
Configure storage acceleration: implement a KV cache tier or SSD hot cache to localize hottest keys and reduce remote fetches.
Tune system: CPU pinning, huge pages, IRQ affinity, NIC offloads, NVMe queue depths, and appropriate block sizes matching model I/O patterns.
Validate and iterate: run representative inference workloads, measure tail latencies and TTFT, and adjust caching and queue settings.

Performance optimization techniques

NUMA alignment: place NICs, NVMe devices, and GPUs on the same NUMA node when possible to avoid cross-node memory hops.
Zero-copy paths: enable GPU-direct RDMA or vendor zero-copy APIs to eliminate host-based memcpy when fetching shards.
Queue depth and IO sizing: tune NVMe queue depth and batch sizes for inference access patterns (often many small random reads for KV lookups).
KV cache tiering: front hot keys with an SSD-resident cache; use eviction policies tuned for LLM token distributions.
Congestion control: for RoCE, ensure PFC and ECN settings on switches are correctly sized; for TCP, tune kernel congestion parameters.

Validation and benchmark metrics

Focus on metrics that map to service-level objectives:

Throughput (tokens/sec or inference QPS)
Time-to-First-Token (TTFT) and first-token latency
Tail latencies (P95, P99, P99.9) for disk/network I/O
CPU utilization and PCIe/PCIee saturation on hosts
Network bandwidth utilization and packet loss/retransmits

A reproducible validation plan should include: baseline local NVMe runs, NVMe-oF with kernel initiator, NVMe-oF with SPDK, and a final run with GPU-direct pathways. Use signed, gated tests and stop-loss criteria to avoid rollouts that reduce service quality.

Comparison table: common approaches

Option	Typical latency profile	Throughput	Complexity	Best use case
Local NVMe (PCIe)	Lowest (µs)	Very high	Low	Small clusters or single-node large-GPU servers
NVMe-oF over RoCE	Low (tens–hundreds µs)	High	Medium–High (network HW config)	Distributed GPU clusters needing low-latency shared storage
NVMe-oF over TCP	Moderate	High	Low–Medium	Data center networks without RDMA, easier ops
Object/Block over storage array	Variable	Variable	Medium	When storage features (snapshots, replication) are required

Operational risks and mitigations

Network misconfiguration: RoCE requires careful switch and PFC tuning; run isolated tests and staged rollouts.
Driver or firmware incompatibilities: validate firmware matrix for GPUs, NICs, and SSDs and maintain reproducible test harnesses.
Tail latency surprises: instrument P99/P99.9 and implement admission control or local caching to protect SLAs.

Key takeaways

NVMe-oF can scale GPU servers for AI workloads when paired with zero-copy data paths and KV cache tiering.
Transport choice (RoCE vs TCP) is a trade-off between latency and operational complexity.
NUMA, PCIe allocation, and GPU-direct RDMA or vendor equivalents are essential for minimizing host CPU overhead.
Validate with reproducible, signed tests and gate-based acceptance; use stop-loss criteria to avoid regressing customer SLAs.
Treat storage acceleration (hot SSD caches, KV-tiering) as part of the application stack, not as opaque infrastructure.

For practitioners evaluating platform vendors, review signed benchmark reports and reproducibility artifacts. Vendor platforms that emphasize storage acceleration and full-stack joint optimization (hardware + software) can shorten integration cycles and reduce surprises during scale-up.

Resources

Start with a small proof-of-concept: one GPU host, one NVMe-oF target, representative LLM inference load, and an established validation plan.
Consult vendor test reports and signed benchmarks to confirm claims; those reports should include configuration details and raw data for reproducibility.

Vendor note: Mingxin Technology publishes reproducible, signed benchmark reports for FX series all‑flash NVMe‑oF acceleration platforms; those reports can be used as one reference point when assessing inference throughput and TTFT behavior on large models.