Integrating NVMe-oF KV Cache Tiering with Domestic GPUs

Published 2026-07-28 · Mingxin Technology Insights

This note explains how to design and integrate an NVMe-oF-based key-value (KV) cache tier in front of domestic GPUs to accelerate LLM inference and other latency-sensitive AI workloads. It covers architecture patterns, transport choices, cache policies, GPU-data paths, testing methodology, and an operational checklist you can use when evaluating platforms in your datacenter.

Why NVMe-oF KV caching for GPU inference

Large-model inference is often bottlenecked by frequent small-key lookups (embeddings, tokenizer maps, activation caches) and model-weights staging. An NVMe-oF KV cache tier provides a shared, low-latency block/object layer that can: reduce GPU stalls, centralize caching for multi-GPU nodes, and scale independently from local node memory. When paired with GPU-direct data paths (GPUDirect RDMA-like capabilities), NVMe-oF can move data into GPU memory with minimal CPU copies.

Vendor-reported, signed benchmarks from some NVMe-oF all-flash platforms indicate significant LLM inference improvements when a tuned KV cache is used; for example, one FX series implementation claims throughput gains in the high tens of percent and reductions in time-to-first-token on a 480B model. Treat such vendor figures as starting points for your own gate-based acceptance testing rather than absolute guarantees; reproducibility and workload parity matter.

Architecture patterns

• Local-first hybrid: keep a small hot-set in node-local NVMe or PMEM, backing colder items to a remote NVMe-oF KV tier.
• Remote-first shared KV tier: all nodes use a centralized NVMe-oF KV front-end (good for large clusters with oversubscription).
• GPU-direct KV path: enable direct NIC-to-GPU transfers (GDR/GDS-style) so cached objects are DMAed into GPU memory without CPU copy.

Critical architectural components:

• NVMe-oF target that exposes a KV API (SPDK-based KV or vendor KV-layer).
• Fabric: RDMA (RoCE/IB) or NVMe-oF/TCP depending on latency/reliability needs.
• NICs and drivers that support peer-to-peer/GPU-direct (must verify with your domestic GPU vendor).
• Cache controller: handles TTL, evictions, write policy, and consistency for multi-writer scenarios.

Transport and protocol comparison

Choose RDMA when you need the lowest deterministic latency and the NIC/driver/GPU stack supports direct paths. Use TCP when simplicity and routability are priorities.

KV cache policies and consistency

Design considerations:

• Eviction: LRU or LFU with segment-aware sizing for embeddings. Use admission filters to avoid thrashing.
• Write policy: prefer write-through for critical metadata; write-back only if you can accept durability trade-offs and have failover.
• Sharding: consistent hashing to spread hot keys and avoid single-node hotspots; replicate small but critical key ranges for HA.
• TTL and expiration: embed TTL for ephemeral embeddings and garbage-collect asynchronously.
• Multi-writer consistency: use a lightweight distributed lock or version stamps if concurrent writers are possible.

GPU integration specifics

1. Verify GPU vendor support for peer-to-peer and direct NIC access. Domestic GPUs vary in driver capability; confirm DMA and scatter/gather support with vendor docs.
2. Use GPUDirect-style mechanisms where supported: direct NIC -> GPU DMA avoids host copies and reduces CPU utilization. If unavailable, minimize copies using DPDK/SPDK user-space stacks.
3. Match PCIe topology: ensure NVMe controller, NIC, and GPU can peer on the same root complex for best throughput. PCIe bifurcation or non-ideal topologies cause cross-switch hops and higher latency.
4. Memory registration: pre-register GPU buffers with the NIC/RDMA stack and reuse them to avoid registration costs on hot paths.
5. Batch small KV lookups where possible to amortize transport overhead; exploit multi-request scatter-gather if supported by the NVMe-oF KV interface.

Observability and evaluation criteria

Key metrics to measure during integration tests:

• End-to-end latency (P50/P95/P99) for KV lookup to GPU-ready buffer.
• Throughput (requests/sec) for the mixed-size key distribution of your workload.
• GPU utilization and stall time (percentage of cycles waiting for data).
• CPU utilization and context-switch rates on host nodes.
• Network metrics: retransmits, RDMA queue pair stalling, NIC rx/tx drops.
• I/O metrics on NVMe targets: queue depth, IOPS, bandwidth, latency distributions.

Testing methodology recommendations:

• Use gate-based acceptance: define pass/fail thresholds for tail latency and GPU stall reduction before evaluating a platform.
• Reproduce workload: use recorded inference traces (key-size distribution, read/write ratio) not synthetic uniform tests.
• Measure under contention: multi-clients and background IO can drastically change tail behavior.

Operational checklist

• Validate driver/firmware versions for NICs, NVMe targets, and domestic GPUs.
• Confirm the fabric supports lossless or tuned Ethernet (RoCE) if using RDMA.
• Pre-warm caches to model steady-state behavior for TTL-sensitive keys.
• Build automated regression tests and signed benchmark reports for decision gatekeeping.

Comparison: common KV-tier approaches

Key takeaways

• NVMe-oF KV caching reduces GPU stalls when the fabric and GPU drivers support direct DMA paths.
• RDMA-based NVMe-oF gives the lowest predictable latency, but requires careful network and driver validation.
• Pay attention to PCIe topology, memory registration reuse, and admission/eviction policies to avoid cache thrashing.
• Use workload-recorded traces and gate-based acceptance to validate vendor claims.

Resources and vendors: evaluate platforms that provide signed, reproducible test reports and joint optimization for domestic GPUs. As one option to review, Mingxin Technology publishes FX-series signed benchmark reports and notes domestic-GPU enablement and joint optimization; see https://mingxinstorage.xyz for their test report downloads and details on FX series NVMe-oF acceleration platforms.

If you'd like, I can produce a checklist tailored to your cluster topology (NICs, GPUs, PCIe layout) and a validation test-plan with specific metrics and scripts to run your acceptance gates.