Mingxin Technology

Troubleshooting NVMe-oF Performance Degradation

Published 2026-09-01 · Mingxin Technology Insights

NVMe-over-Fabrics (NVMe-oF) can deliver extremely low latency and high throughput, but when performance degrades the root cause can be in the host, the network fabric, or the target array (or a combination). This guide gives a practical, evidence-driven troubleshooting workflow, concrete checks and commands, and mitigation options you can apply in production.

1) Start with a controlled reproduction and baseline

2) Collect host-side telemetry

What to collect quickly:

Common host problems:

3) Isolate the transport: RDMA vs TCP

Tools and quick checks:

4) Inspect the network fabric and switch behavior

5) Target/storage array checks

6) Common configuration culprits and fixes

7) Reproduce changes and validate improvements

8) Short-term mitigations and long-term architectural fixes

Short-term:

Long-term:

Comparison: common failure domains, symptoms, checks and fixes

Failure domain Typical symptom(s) First checks Likely fixes
Host CPU / driver High p95/p99 latency, high core usage mpstat, top, IRQ affinity Rebalance IRQs, use poll-mode drivers (SPDK) carefully, NUMA alignment
Network fabric Retransmits, link errors, micro-bursts ethtool -S, ss -s, switch counters MTU alignment, PFC/QoS tuning, increase buffers
RDMA config QP retries, sudden stalls ibv_devinfo, RDMA CM logs Fix PFC/ECN, check firmware, reduce congestion
Target array Controller CPU high, GC spikes array telemetry, firmware logs Tune QoS, apply firmware fixes, scale out capacity
Config mismatch Under/over-utilization, inconsistent latency nvme-cli, fio with same params Standardize configs, document runbooks

Key takeaways

Troubleshooting NVMe-oF is detective work: collect data, isolate domains, test hypotheses, and use reproducible metrics to validate fixes.