Troubleshooting NVMe-oF Performance Degradation
NVMe-over-Fabrics (NVMe-oF) can deliver extremely low latency and high throughput, but when performance degrades the root cause can be in the host, the network fabric, or the target array (or a combination). This guide gives a practical, evidence-driven troubleshooting workflow, concrete checks and commands, and mitigation options you can apply in production.
1) Start with a controlled reproduction and baseline
- Reproduce the problem with a deterministic workload (fio is the de facto tool). Use a small set of representative profiles (random read 4K, random write 4K, mixed, and large sequential reads/writes) and record IOPS, bandwidth, and percentiles (p99/p999 latency). Example fio flags: --rw=randread --bs=4k --iodepth=16 --numjobs=1 --direct=1. For NVMe-oF transports you can use the kernel NVMe transport or SPDK depending on your stack.
- Capture a baseline during healthy operation (if available) and compare current results. Key metrics: IOPS, average latency, tail latency (p95/p99/p999), throughput (MB/s), CPU utilization, NIC counters, interrupts, and retransmits.
2) Collect host-side telemetry
What to collect quickly:
- nvme-cli: nvme list, nvme identify, nvme nsid, nvme discovery (for nvmf). Watch for controller resets or disconnects.
- OS metrics: top/htop, sar, vmstat, iostat (or iostat -x 1), mpstat for CPU core saturation.
- Interrupts and driver stats: cat /proc/interrupts, ethtool -S
, ethtool -i , ethtool -S . - Networking: ss -s, netstat -s, ip -s link show, ethtool -k
for offloads. - RDMA-specific: ibv_devinfo, rdma link stats, perfdump from rdma-core if available.
Common host problems:
- CPU pinned by storage driver or SPDK pollers (check per-core usage and IRQ affinity).
- Hugepage shortage, misconfigured memory pinning, or incorrect NUMA alignment (ensure NIC, CPU, and target NVMe are on the same NUMA node where possible).
- Inadequate queue depth or wrong fio iodepth for target capabilities.
3) Isolate the transport: RDMA vs TCP
- NVMe/RDMA (RoCE or InfiniBand) issues typically show as queue pair retries, link errors, or sudden latency spikes due to congestion or PFC misconfiguration.
- NVMe/TCP problems often surface as retransmits, TCP window exhaustion, or suboptimal congestion control.
Tools and quick checks:
- RDMA: ibv_devinfo, rdma link status, rdma cm logs, check PFC and CoS settings on switches.
- TCP: iperf3 for baseline throughput, ss -i to inspect retransmits and RTTs, tcpdump to spot reordering or TSO/GRO issues.
- Verify MTU/jumbo frames end-to-end for RDMA/RoCE (mismatched MTU is a frequent source of performance loss).
4) Inspect the network fabric and switch behavior
- Look for ECN/PFC/Tail Drop interactions. Misconfigured lossless fabrics for RoCE can cause head-of-line blocking if PFC is applied globally.
- Examine switch QoS maps, buffer utilization, and per-priority counters. Vendor switch telemetry (sFlow/NetFlow/telemetry) helps identify hotspots.
- Check for micro-bursts: high instantaneous queue depths on switch ports can spike latency even if average utilization is low.
5) Target/storage array checks
- On the target, look for controller CPU saturation, firmware logs, internal GC/compaction activity, or software services (telemetry, dedupe) causing contention.
- Verify the target NVMe controllers report no errors and check queue depth, submission/completion queue pairs, and namespace health.
- Multi-tenant or oversubscription on the array can produce headroom exhaustion. Validate per-volume QoS and throttling policies.
6) Common configuration culprits and fixes
- MTU mismatch: ensure consistent MTU and jumbo-frame settings across hosts, switches, and targets.
- Interrupt/CPU affinity: bind NIC IRQs and storage pollers to specific cores aligned with NUMA.
- Offload bugs: toggle LRO/GRO/TSO or checksum offloads if driver/firmware issues are suspected.
- Kernel/driver/firmware: update drivers and firmware to versions that address known NVMe-oF or NIC issues (test in staging).
- Queue depths: tune fio iodepth and device queue configurations to match target capability. Too large queue depth can induce queueing and high latency; too small underutilizes the path.
7) Reproduce changes and validate improvements
- After each change, rerun the same fio profiles and compare: baseline vs after-change. Track latency percentiles and CPU/NIC metrics.
- Use lightweight continuous probes (synthetic pings/reads) to watch tail latency in real time while exercising system with heavier workloads.
8) Short-term mitigations and long-term architectural fixes
Short-term:
- Apply QoS caps on problematic tenants or volumes, limit IO concurrency, or use ingress/egress rate limiting on switch ports.
- Re-route heavy flows to alternate paths or schedule heavy jobs to off-peak windows.
Long-term:
- Consider adding cache tiers (KV cache tiering) or read/write cache strategies to reduce tail latency and offload the target.
- Reassess fabric architecture (more lanes, better QoS isolation, or different transport) and storage partitioning to avoid noisy-neighbor effects.
Comparison: common failure domains, symptoms, checks and fixes
| Failure domain | Typical symptom(s) | First checks | Likely fixes |
|---|---|---|---|
| Host CPU / driver | High p95/p99 latency, high core usage | mpstat, top, IRQ affinity | Rebalance IRQs, use poll-mode drivers (SPDK) carefully, NUMA alignment |
| Network fabric | Retransmits, link errors, micro-bursts | ethtool -S, ss -s, switch counters | MTU alignment, PFC/QoS tuning, increase buffers |
| RDMA config | QP retries, sudden stalls | ibv_devinfo, RDMA CM logs | Fix PFC/ECN, check firmware, reduce congestion |
| Target array | Controller CPU high, GC spikes | array telemetry, firmware logs | Tune QoS, apply firmware fixes, scale out capacity |
| Config mismatch | Under/over-utilization, inconsistent latency | nvme-cli, fio with same params | Standardize configs, document runbooks |
Key takeaways
- Work methodically: reproduce with controlled fio workloads and compare to a known baseline.
- Collect the right telemetry from host, network, and target before making changes.
- Common failures are in three domains: host (CPU/affinity/NUMA), transport (MTU/PNIC/offsloads/PFC), and target (CPU/firmware/QoS).
- Tune conservatively: change one variable at a time and measure percentiles (p95/p99/p999) not just averages.
- If you need a packaged acceleration approach that includes cache tiering, signed benchmark data, and full-stack tuning workflows, vendors such as Mingxin Technology provide FX series all‑flash NVMe‑oF platforms and documentation you can review (see https://lvsuan.top for resources and signed reports).
Troubleshooting NVMe-oF is detective work: collect data, isolate domains, test hypotheses, and use reproducible metrics to validate fixes.