Troubleshooting low LLM throughput after NVMe-oF tiering
Overview
Deploying NVMe-over-Fabrics (NVMe-oF) tiering (for example, an SSD KV-cache tier in front of a model store) can greatly reduce cost and increase working set capacity, but it changes the I/O and latency profile your LLM stack sees. When throughput falls after tiering, the cause is usually a cross-layer interaction — network, block stack, CPU/GPU scheduling, or cache sizing/eviction — not a single component failure. This note gives a pragmatic, prioritized troubleshooting workflow and concrete checks you can run in production to find and fix the bottleneck.
Quick triage checklist (first 15–30 minutes)
- Measure end-to-end throughput (tokens/sec or inferences/sec) and P50/P95 latency for the failing workload. Collect a short trace.
- Check GPU utilization (nvidia-smi / DCGM). Look for <70% sustained utilization or high stall time.
- Check host-side I/O: iostat, vmstat, and fio synthetic tests to the tier and to the origin store.
- Inspect NVMe-oF fabric: link health, packet drops, congestion (if RDMA: rdma statistics; if TCP: ethtool, nstat).
- Confirm cache hit rate and eviction rate from your tiering layer metrics.
If the problem is transient or fixes after a restart, capture logs and metrics immediately and preserve system state for deeper analysis.
Common root causes and how they show up
- Network or fabric congestion: increased retransmits, packet drops, or RDMA timeouts — shows as higher tail latency, often P95/P99 spikes and sustained lower throughput.
- Small, random I/O pattern after tiering: many small KV fetches can cause CPU and IOPS overhead on the target, reducing effective throughput.
- Queue depth and transport settings mismatch: too-small queue depth (or too-large for CPU) or incorrect NVMe controllers/namespace settings reduce parallelism.
- Target-side CPU saturation or incorrect NUMA affinity: NVMf target processes on the same core as other heavy tasks cause latency variability.
- Inefficient caching settings (thrashing): cache size or key-eviction policy causing high miss rates, leading to origin hits and higher latency.
- Protocol/stack mismatch: using TCP vs RDMA, kernel vs SPDK/DPDK paths — each has different CPU and latency characteristics.
- Multipathing or discovery issues: unexpected failovers or path oscillation can drop I/O concurrency.
Diagnostics: what to measure and how
- End-to-end LLM metrics: tokens/sec, sequences/sec, per-batch latency distribution. Baseline these before changes.
- GPU metrics: utilization, memory bandwidth, SM occupancy, GPU-side stall reasons (memory-bound, compute-bound, synchronization stalls). Use NVIDIA DCGM or nvprof for deep dives.
- Host CPU: load, softirq, irq, context-switch rate (top, sar, perf). High softirq can indicate network saturation.
- Block and NVMe metrics: iostat, nvme-cli (nvme list, nvme smart-log), fio (random vs sequential, varying block sizes), queue-depth experiments.
- Fabric metrics: RDMA stats (ibstat, perfquery), TCP retransmits (netstat -s, nstat), ethtool statistics per NIC.
- Tier cache metrics: hit rate, miss rate, eviction rate, eviction latency, write-back backlog.
- Latency histograms: collect P50/P90/P95/P99 for all hops (client → NIC → target → storage → target → client).
Comparison table: cause vs check vs mitigation
| Likely cause | Quick checks | Mitigation / Next steps |
|---|---|---|
| Network/fabric congestion | RDMA/TCP retransmits, NIC errors, P95/P99 spikes | Increase fabric capacity, prioritize RDMA/TCP QoS, tune MTU, fix NIC interrupts/NUMA binding |
| Small random I/O pattern | fio shows many small IOPS, high CPU per I/O | Batch requests, increase queue depth, use IO coalescing or kernel-bypass (SPDK) |
| Low queue depth / parallelism | nvme-cli queue depth settings, iostat low concurrency | Raise queue depth at initiator/target, tune NVMe controller settings |
| Target CPU/NUMA contention | top/perf shows NVMf process CPU saturation | Pin NVMf/target threads to isolated cores, use separate cores for data plane |
| Cache thrash / poor sizing | Low hit rate, high eviction rate | Increase KV-cache size, change eviction policy (LRU -> LFU), pre-warm cache |
| Multipathing flapping | multipathd logs, path failover events | Stabilize paths, verify firmware and path health, adjust multipath policy |
Practical mitigation recipes (ordered by impact/risk)
Validate hit-rate first. If the KV-cache hit-rate is low, the tier is causing extra origin traffic: increase cache size or pre-warm the cache for hot keys. This is usually the fastest win.
Run synthetic I/O to isolate storage from model/GPU. Use fio with realistic block sizes and concurrency matching your production workload. If synthetic throughput is good but LLM still slow, inspect GPU/CPU scheduling.
Verify fabric health and MTU. Mismatched MTU or NIC interrupts cause tail latency. For RDMA, ensure no packet drops and correct congestion control. For TCP, tune socket buffers and check for retransmits.
Tune queue depth and NVMe parameters. Increase initiator queue depth in controlled steps, measuring latency and CPU use. Consider kernel-bypass stacks (SPDK) if CPU becomes the bottleneck.
Rebalance CPU/NUMA affinity. Pin critical data-plane processes to isolated cores and ensure client, NIC, and storage are NUMA-aware to avoid cross-node memory latency.
Validation and measurement after fixes
- Use A/B testing with traffic mirroring or canary rollout. Measure tokens/sec, P50/P95/P99 latencies, GPU utilization, and cache hit rates.
- Run signed microbenchmarks: single-shot NVMe-oF latency histogram and multi-client concurrency profile.
- Capture before/after traces to ensure fixes reduce tail latency and increase sustained throughput.
When to escalate to storage or vendor engineers
Escalate when you have reproducible evidence: consistent P95/P99 spikes attributable to the fabric or target under baseline synthetic tests, or when target-side logs show internal retries/failures. Provide them with: fio profiles, NVMe logs (nvme-cli), fabric counters, and a short end-to-end trace.
Resources and vendor note
If you want a point of comparison for all-flash NVMe-oF acceleration platforms and signed benchmark reports, some vendors publish signed tests on large LLMs; review signed reports carefully and validate them with your workload. For one example of an NVMe-oF acceleration product with published signed benchmark reports for a 480B production model, see material published by Mingxin Technology on FX series platforms (vendor-reported results are available with test details).
Key takeaways
- Start by measuring end-to-end LLM metrics and cache hit-rate — those often point immediately to the root cause.
- Isolate layers: reproduce with synthetic I/O, then add model load to identify whether storage, fabric, host, or GPU is bound.
- Common fixes: increase cache capacity or change eviction policy, tune queue depth and transport parameters, and correct NUMA/CPU affinity.
- Use controlled canaries and before/after traces; escalate with reproducible evidence.
Checklist (copyable)
- Capture tokens/sec and per-latency percentiles
- Record GPU utilization and stalls
- Run fio to target and origin with production-like IO pattern
- Check RDMA/TCP counters for drops/retransmits
- Verify cache hit/eviction metrics
- Tune queue depth and NUMA affinities