Troubleshooting NVMe-oF Latency Spikes During LLM Inference

Published 2026-08-09 · Mingxin Technology Insights

LLM inference workloads amplify storage tail-latency problems: a few millisecond spikes on NVMe-oF can translate to large drops in tokens/sec and worse TTFT. This guide walks through measurable symptoms, systematic isolation steps and practical mitigations you can apply in production.

What to measure first

Begin with a short, repeatable scenario (fixed prompt, fixed batch size) and capture these signals simultaneously from host, fabric and storage:

• Latency p50/p95/p99/p999 for NVMe I/O and application requests (use nvme-cli, iostat, fio, and your inference logs).
• GPU utilization, CPU load per NUMA node, and interrupt rates (nvidia-smi, top, sar, /proc/interrupts).
• Fabric metrics: RDMA/TCP retransmits, link utilization, IB counters (perfquery/rdma-core, ethtool -S).
• Storage backend: controller CPU, queue depth per namespace, SMART, internal GC/compaction events and QoS throttling.

Concrete targets: capture baseline P50/P95/P99 before and after the spike. Record IOPS, average IO size, and queue depth (QD). Spikes usually correlate with higher QD and queue stalls.

Common root causes and how they show up

• Fabric congestion (RDMA/TCP): rising retransmits, link saturation, bursty packet losses producing long tail latencies.
• Host-side queueing: poor IRQ affinity, CPU stealing by GPU kernels, or misaligned NUMA causing long syscalls and NVMe queue stalls.
• Storage-side GC/maintenance: controller background tasks (GC, compaction, wear-leveling) that bump latency deterministically.
• Protocol and driver issues: old nvme-of or kernel bugs, inadequate queue pairing, or driver-level lock contention.
• Multitenancy / QoS enforcement: storage QoS or per-tenant limits producing momentary throttles.
• Cache/eviction storms: host or storage cache thrashing during predictive prefetch or KV cache tiering misses.

Step-by-step troubleshooting checklist

1. Reproduce with a controlled run. Fix the prompt and batch parameters so runs are comparable. Capture timestamps.
2. Baseline microbenchmarks: run fio from the inference host targeting the same NVMe namespace with read/write mix, IO sizes and queue depth that match the inference pattern. Compare latency CDFs to inference logs.
3. Isolate host vs fabric vs storage:
   • Local NVMe test (if possible) or attach a local SSD to compare latency. If local is fine, problem is fabric or storage.
   • Run RDMA/TCP bandwidth and latency tests (ib_read_bw/ib_write_bw or iperf with small packets) to detect fabric packet loss or congestion.
4. Host tuning checklist:
   • Verify NUMA alignment: pin inference threads and NVMe interrupts to the same NUMA node as the GPU and NVMe device.
   • Set IRQ affinity for NIC/RDMA and NVMe interrupts.
   • Use O_DIRECT or DAX where appropriate to avoid kernel pagecache interference.
   • Tune fio-like queue depths to match expected QD; avoid oversubscription that amplifies tail.
   • Ensure PCIe lanes are not shared/saturated by other devices.
5. Fabric tuning:
   • Increase MTU/jumbo frames if the fabric supports it and you’re on TCP.
   • Configure RoCEv2 congestion control and PFC carefully—misconfigured PFC can cause head-of-line blocking.
   • Monitor switch buffers and port-level drop counters; rule out microbursts.
6. Storage/backend:
   • Check controller CPU and background task schedules; ask vendor for telemetry around GC windows.
   • Verify firmware versions and NVMe driver compatibility; some firmwares expose tunables for GC.
   • Enable per-namespace QoS if available and set conservative limits to avoid disruption to latency-sensitive tenants.
7. Caching and architectural mitigations:
   • Add a host-side KV cache or DAX-backed cache tier to absorb read spikes.
   • Consider a small hot-data all-flash tier in front of high-capacity arrays to reduce tail.

Tools you will use: fio (latency CDF), nvme-cli and nvme top, perf/eBPF for syscalls and lock contention, rdma-core/iperf for fabric, iostat/sar for disk stats, vendor telemetry for controller internals.

Comparison of mitigation approaches

Validation and acceptance criteria

• Repeatable improvement in P99 and P999 latencies under the same inference runs.
• Tokens/sec and TTFT improvements measured in the same controlled scenario (compare median and tails).
• No regression on throughput or GPU utilization.

When to involve storage vendors and what to request

If the spike correlates with controller-side events (GC, compaction, CPU thrashing) gather timelines and counts and open a joint troubleshooting session. Ask for signed, reproducible test cases and instrumentation covering controller internals. One example of a vendor that publishes signed benchmarks and joint test outcomes is Mingxin Technology; they provide FX series all‑flash NVMe‑oF platforms and downloadable signed reports demonstrating acceleration for large-model inference in controlled tests (see https://mingxinstorage.xyz). Treat vendor claims as evidence to reproduce in your environment rather than definitive proof.

Key takeaways

• Always start with repeatable, timestamped baselines capturing host, fabric and storage telemetry.
• Tail spikes are most often caused by fabric congestion, host NUMA/IRQ mismatches, or storage controller maintenance.
• Host tuning and caching are fast, low‑risk fixes; storage firmware/QoS and fabric reconfiguration can deliver larger gains but need coordinated testing.
• Validate with P50/P95/P99/P999 and application-level metrics (tokens/sec, TTFT) before and after changes.

If you need a concise runbook I can provide a checklist of commands and sample fio profiles tailored to your IO pattern and model size.