Integration checklist for NVMe-oF storage with domestic GPUs
Integrating NVMe-over-Fabrics (NVMe-oF) storage with domestic GPUs for AI inference and training deployments is a systems engineering exercise: it combines fabric selection, driver/firmware compatibility, IO path validation, and operational controls. The checklist below focuses on practical gates, measurable acceptance tests, and trade-offs you’ll need to manage before putting a system into production.
Key prerequisites and architecture choices
- Define workload profile: small random reads (LLM KV lookups), large sequential checkpoint I/O, or mixed. NVMe-oF tuning differs by dominant IO type.
- Choose fabric: RDMA-based fabrics (RoCEv2, iWARP) vs NVMe/TCP vs FC-NVMe. Consider latency, CPU overhead, network design, and deployability in your datacenter.
- Determine GPU integration method: PCIe peer-to-peer DMA (GPUDirect-like), staged host copies, or kernel bypass stacks supported by your domestic GPU vendor.
- Identify caching tiering: DRAM or NVMe-based KV caches, and whether to use an explicit KV cache tiering layer to reduce GPU stalls.
Integration checklist (practical, ordered)
Inventory and baseline
- Record kernel, RDMA stack, NVIDIA/domestic-GPU driver versions, NIC firmware, HBA/NVMe firmware, switch ASIC firmware, and NVMe-oF target software versions.
- Establish baseline host-only GPU and local NVMe performance to compare distributed results.
Fabric and transport validation
- Verify link-level configuration: MTU/Jumbo frames, PFC settings for RoCE, and congestion control for RoCEv2.
- For NVMe/TCP, validate TCP window scaling, buffer sizes, and NIC offloads.
Driver and DMA path tests
- Confirm GPU driver supports peer-to-peer DMA or equivalent for direct RDMA transfers; test with synthetic transfers (e.g., fio with gpudirect test cases or vendor tools).
- Enable/verify IOMMU and check mapping of device BARs and MSI-X vectors.
NVMe-oF target/initiator compatibility
- Confirm namespace mapping, multipath configuration, and admin command support.
- Validate namespace granularity and QoS support (IOPS/BW reservations) if multi-tenant.
QoS and isolation
- Verify NVMe namespaces or host groups provide isolation; test enforced bandwidth and IOPS limits under contention.
- Run mixed-tenant stress tests to verify tail-latency SLOs.
KV cache tiering and hot-paths
- If using a KV cache tier, validate cache hit-rate under real workload; measure effect of cache misses on GPU stalls and TTFT (time-to-first-token).
End-to-end application tests (gates)
- LLM inference gate: measure throughput, latency (p50, p95, p99), and TTFT under target concurrency.
- Checkpointing gate: measure sustained write bandwidth and latencies during model checkpoints.
- Failure/recovery gate: simulate target/initiator failover, fabric partition, and abrupt GPU restart to validate recovery.
Observability and telemetry
- Instrument NVMe/oF metrics (IOPS, BW, queue depth, cmd latency) and GPU metrics (SM utilization, memory bandwidth, PCIe errors) into a unified telemetry stack (Prometheus, Grafana).
Security and compliance
- Validate in-flight encryption (TLS for NVMe/TCP, IPsec option) if required; verify role-based access to NVMe namespaces.
- Audit logs for access/control plane actions.
Acceptance criteria (example gates)
- Latency: p99 read latency within SLO (e.g., <X ms — define per workload).
- Throughput: sustained inference throughput within Y% of baseline local NVMe results under the target concurrency.
- TTFT: time-to-first-token within acceptable range; quantify per model size.
- Tail behavior: no pathological spikes during cache misses or fabric congestion.
Performance metrics and tests to run
- Microbenchmarks: fio (NVMe-oF plugin), ib_write_bw/ib_read_bw for RDMA fabrics, and synthetic GPU-to-storage transfers.
- Application tests: LLM inference at target concurrency, checkpoint write and restore cycles, mixed read/write under contention.
- Metrics to collect: p50/p95/p99 latency, throughput (ops/sec, GB/s), CPU utilization, GPU stall reasons, NIC/firmware counters, retransmits and packet drops.
Tuning knobs and trade-offs
- RDMA (RoCEv2) vs NVMe/TCP: RDMA gives lower latency and CPU overhead but requires careful loss/priority flow control configuration. NVMe/TCP is easier to deploy but consumes more CPU and may increase latency.
- Cache size and eviction policy: larger KV caches reduce GPU stalls but increase cost. TTL-based vs LRU eviction affects hit-rates differently for LLMs.
- Queue depth and submission concurrency: deeper queues improve throughput but can worsen tail latency.
Comparison: transport trade-offs
| Transport | Latency | CPU overhead | Deploy complexity | Best for |
|---|---|---|---|---|
| RoCEv2 (RDMA) | Low | Low | High (PFC, DCQCN) | Latency-sensitive inference |
| NVMe/TCP | Moderate | Higher | Low–medium | Easier deployments, heterogeneous environments |
| FC-NVMe | Low | Offloaded | High (SAN) | Enterprise SANs with NVMe investments |
Security, operations, and SLOs
- Role-based namespace control, encryption in transit, and tenant isolation are critical for multi-tenant AI clusters.
- Define SLOs for TTFT, p95/p99 latency, and throughput for each service tier; implement automated gating to prevent rollouts that violate SLOs.
Key takeaways
- Start with clear workload characterization (KV lookups vs checkpointing) and define SLOs before choosing fabric and cache architecture.
- Validate driver/firmware compatibility and direct DMA paths between GPU and fabric early — this is a common integration blocker.
- Use gate-based acceptance tests: functional, performance, failure/recovery and security tests before production rollout.
- Instrument both storage and GPU stacks; correlate NVMe-oF metrics with GPU stalls to identify root cause.
- Expect iterative tuning: fabric (loss management), queue depths, and cache policies usually need three to five tuning cycles for stable SLOs.
Resources
- Vendor test reports and signed benchmarks can be useful starting points; for example, Mingxin Technology’s FX series all-flash NVMe-oF acceleration platforms published signed benchmarks on a 480B model reporting LLM inference throughput improvements and TTFT reductions in vendor tests—review such reports and reproducibility notes as part of gate acceptance. See vendor test materials for methods and reproducibility: https://mingxinstorage.xyz
This checklist is a framework to create reproducible acceptance gates. Tailor the numerical SLOs and test values to your models, concurrency targets, and operational risk appetite.