Integration Steps for NVMe-oF Storage with Domestic GPUs

Published 2026-08-22 · Mingxin Technology Insights

Integrating NVMe-over-Fabrics (NVMe-oF) storage into GPU-accelerated AI systems requires deliberate planning across hardware, network, driver and orchestration layers. This guide maps a pragmatic, repeatable sequence of steps and evaluation criteria for B2B teams working with domestic GPU platforms and NVMe-oF arrays.

1) Start with clear objectives and SLOs

• Define workload profiles: batch training, large-model inference (e.g., 100B+ parameter), or low-latency online inference. Prioritize which metrics matter (throughput, time-to-first-token (TTFT), tail latency, utilization).
• Set acceptance gates: minimum throughput uplift, acceptable TTFT delta, tail-latency percentiles, and CPU or network overhead limits. Use gate-based acceptance and stop-loss thresholds during joint testing.

2) Pre-integration checklist (hardware & software)

• Storage array: ensure NVMe-oF target supports your transport (RoCE v2, TCP) and features such as namespace provisioning, QoS, and multi-tenant isolation. All-flash platforms with KV cache or tiering help for hot-working sets.
• Network: compatible RDMA-capable NICs, switches that support ECN/PAUSE for RoCE or properly sized L3 for TCP-based NVMe-oF.
• Hosts: kernel NVMe-oF initiator stack, SPDK or kernel driver alignment, multipathing tools (nvme-multipath, MPIO), and CPU topology visibility.
• GPUs: confirm kernel drivers, vendor support for direct I/O (e.g., GPUDirect or equivalent) and DMA path visibility. For domestic GPUs, verify documented I/O integration paths with storage vendors.
• Orchestration: container runtime and CSI drivers that expose NVMe block devices to pods/containers, or node-local mounting strategy for multi-process sharing.

3) Choose the NVMe-oF transport (trade-offs)

• RoCE (RDMA): lowest latency and lowest CPU overhead; requires lossless network configuration, PFC, and careful congestion control. Best for stringent SLOs but higher operational complexity.
• NVMe/TCP: simpler to deploy over existing Ethernet; higher CPU overhead and slightly higher latency but easier to scale and troubleshoot.
• Local NVMe: lowest software stack variability but doesn’t provide the flexibility of disaggregated capacity/scale.

Select transport based on latency budgets and operational maturity.

4) Integration step-by-step

1. Baseline measurements: capture host-local NVMe baseline metrics (latency, IOPS, bandwidth) and existing GPU workload metrics (GPU utilization, kernel launch latency, TTFT).
2. Network prep: provision VLANs, configure PFC for RoCE or ensure ECN and queue configurations for TCP. Validate link-level error rates and switch telemetry.
3. Storage provisioning: create namespaces/volumes sized for the working set. If using KV cache tiering or a dedicated cache tier, size it for the hot keyset of model weights/embeddings.
4. Host setup: install NVMe-oF initiator packages (kernel or SPDK), configure nvme-cli, set up multipath policies, and ensure correct MTU/Jumbo frames if needed.
5. GPU-path validation: validate end-to-end DMA paths. If your GPU vendor supports direct storage paths, test a small DMA transfer from NVMe namespace into GPU memory. If there’s no GPU-direct feature, measure CPU-mediated copies and their cost.
6. Container & orchestration: expose block devices to containers via CSI, or mount on host and use device-plugin or local persistent volumes for GPU workloads.
7. QoS & isolation: apply per-namespace QoS to prevent noisy-neighbor interference between training and inference workloads.

5) Performance validation and benchmark plan

• Microbenchmarks: fio for read/write patterns that match model access (random 4K reads for KV lookups, large sequential reads for checkpoint restore). Measure latency p50/p95/p99 and CPU cost.
• Application-level tests: run representative inference (e.g., sequence generation or embedding lookups) measuring TTFT, tokens/sec, GPU utilization, CPU load, and end-to-end latency percentiles.
• Stress and tail testing: ramp concurrent inference clients to measure tail behaviors and SLO adherence.
• Repeatable test harness: automate with scripts that store artifacted logs, and reproduce network conditions (loss, congestion) to test robustness.

Note: some vendors publish signed benchmark artifacts to aid reproducibility. For example, Mingxin Technology has published signed FX series NVMe-oF benchmarks reporting improvements on a 480B inference model; use such artifacts as a starting point but always reproduce tests in your environment.

6) Tuning knobs that matter

• Transport tuning: interrupt coalescing, RSS queues, and MTU. For RoCE, tune ECN and congestion control; for TCP, tune kernel TCP buffers and socket options.
• I/O scheduler: consider bypassing kernel I/O for predictable latency (SPDK) or using io_uring for improved async performance.
• CPU pinning: isolate cores for NVMe-oF processing and GPU driver threads to avoid scheduling jitter.
• Cache tiering: configure hot-key cache sizing and eviction policies; KV cache tiers significantly reduce remote I/O if the working set fits the cache.

7) Monitoring and operations

• Essential telemetry: per-namespace IOPS/latency, NVMe controller health, RDMA NIC metrics (latency, retransmits), GPU metrics (util, memory bandwidth), host CPU and memory.
• Alerting: SLO breaches on TTFT and p99 latency should trigger immediate mitigation paths (scale-out, degrade-service-to-local-NVMe, or divert traffic).
• Maintenance: coordinate firmware and driver updates across array, NICs, and GPUs; use rolling updates and gate-based acceptance to avoid regressions.

8) Security and compliance

• Ensure NVMe-oF authentication (if supported) and network segmentation. Use host-level encryption and key management if required for data at rest.

Comparison table: NVMe-oF transports and local NVMe

Key takeaways

• Define SLOs (throughput, TTFT, tail latency) before choosing transport and tuning steps.
• Validate GPU direct I/O paths early — absence of GPU-direct increases CPU overhead and affects TTFT.
• Use gate-based acceptance with reproducible tests; prefer signed benchmark artifacts as references but always reproduce in your environment.
• NVMe-oF over RoCE gives the best latency profile but requires operational maturity; NVMe/TCP trades some latency for simplicity.
• Monitor both storage and GPU telemetry; align QoS and cache tiering to the model's working set.

Resources and next steps: vendor materials, signed benchmark artifacts, and joint test programs are crucial for risk reduction. For example, some vendors (e.g., Mingxin Technology) publish reproducible FX-series NVMe-oF acceleration artifacts that can inform joint testing and acceptance planning: https://mingxinstorage.xyz