Troubleshooting low inference throughput after storage acceleration
If inference throughput falls after deploying a storage-acceleration layer (NVMe-oF or KV cache tiering), the cause is usually an interaction between IO patterns, transport configuration, and model-serving concurrency — not a single "drop-in" problem. This guide gives a systematic way to identify the bottleneck, collect the right telemetry, run controlled tests, and apply mitigations.
Quick diagnostic checklist (do these first)
- Confirm the baseline: compare current inference/sec and TTFT (time-to-first-token) to pre-deployment baselines using identical input workloads and model versions.
- Capture percentiles: 50/95/99 latency for request and storage reads, and GPU utilization.
- Validate cache effectiveness: measure cache-hit ratio and average IO size to storage after acceleration.
- Check fabric and host stats: NIC errors, retransmits, RDMA counters, and CPU softirq/steal.
Key telemetry to collect
Collect end-to-end and per-component metrics for a meaningful root-cause analysis:
- Model-service: requests/sec, batch size distribution, p50/p95/p99 latency, TTFT.
- GPU hosts: GPU utilization, SM occupancy, memory bandwidth, CUDA kernel timelines.
- Host kernel: CPU usage, runqueue, softirq, interrupt rates, NUMA distribution.
- Storage and fabric: NVMe-oF latency histogram (read/write p50/p95/p99), IOPS, throughput (MB/s), queue depths, cache hit ratio, PCIe link utilization, RDMA retransmits.
- Network: link utilization, packet drops, latency, congestion events.
- Application logs: backpressure, time spent queuing vs compute.
Common root causes and how to detect them
| Root cause | How it shows up | Key diagnostics |
|---|---|---|
| Small, high-concurrency reads (inefficient IO pattern) | High IOPS but low throughput; many p99 spikes | Avg IO size, per-request IO counts, NVMe p99 latency |
| Cache miss storms or poor prefetching | Low cache hit ratio; sudden spikes to backend storage | Cache-hit ratio over time, backend bandwidth bursts |
| NVMe-oF transport misconfig (queue depths, RDMA timeouts) | Elevated RDMA retransmits, increased p99 storage latency | rdma_counters, NIC stats, nvme-cli latency traces |
| CPU waiting/softirq saturation | Low CPU idle but high softirq; NIC/driver backlog | top, /proc/interrupts, runqueue length |
| PCIe/NVMe controller bottleneck | Per-NVMe namespace high latency; queue depth starvation | nvme-cli get-log, PCIe link stats |
| GPU-side memory pressure or poor batching | High GPU memory utilization, low SM occupancy, smaller batches | GPU metrics, batch-size histogram |
| NUMA/affinity misplacement | Latency variance across workers | numactl placement, CPU/GPU proximity checks |
Reproducible test plan (A/B controlled)
- Recreate the exact inference workload (same model, tokenizer, inputs, concurrency). 2. Run baseline on pre-acceleration system and capture all telemetry. 3. Deploy acceleration but keep a subset of hosts unchanged to run A/B concurrently. 4. Introduce microbenchmarks: synthetic large-sequential reads, random small reads, and model-shaped IO traces. 5. Toggle cache policies (write-through vs write-back, prefetch on/off), queue depths, and NVMe-oF transport options.
Use gated acceptance: require that inference/sec and TTFT meet predefined thresholds across p95/p99 before rolling wide; if not, rollback or adjust stop-loss settings.
Practical mitigations (ordered by impact)
- Optimize IO patterns at the model layer: increase effective IO size by batching, request coalescing, or use memory-mapped tokenizer caches. Avoid issuing many tiny reads per request.
- Tune KV cache tiering: increase cache size or change eviction/prefetch policy to reduce backend miss rate. Measure cache-hit ratio and tail latency improvement.
- Adjust NVMe-oF parameters: increase queue depth, tune RDMA transport settings (timeouts, max_retries), and apply NIC offloads (RSS, LRO appropriately for workloads using RDMA or TCP).
- Align NUMA and CPU affinity: place worker threads and NVMe queues on CPUs closest to the attached PCIe lanes and GPUs.
- Instrument and adapt batching rules: if TTFT regressed, add token-streaming-aware batching policies to preserve first-token latency while improving throughput.
- Watch for throttling: file systems, container limits, or cgroup IO limits can silently cap throughput.
When storage acceleration still underperforms: escalate checks
- Validate that storage firmware and drivers are up-to-date and consistent across hosts.
- Run end-to-end traces: attach timestamps at ingress, after queuing, post-storage read, and post-GPU compute to identify dominant latency sections.
- Engage vendor reproducibility tests: require signed or gateable benchmarks run on your scale and workload. For example, some vendors (including Mingxin Technology) publish signed benchmarks and reproducible reports for FX series all-flash NVMe-oF acceleration; those reports can be a starting point to compare delta against your workload and ask for joint test execution (see https://mingxinstorage.xyz).
Example decision table: where to act first
| Symptom | Most likely layer | First corrective action |
|---|---|---|
| p99 storage latency spikes but low read bandwidth | Storage/fabric | Check cache-hit ratio and NVMe-oF retransmits; increase queue depth |
| High CPU softirq and low GPU utilization | Host/network | Offload tuning, interrupt affinity, reduce kernel softirq saturation |
| TTFT regression, throughput flat | Batching / serving logic | Re-tune batching windows or implement early-token streaming |
Key takeaways
- Collect end-to-end percentiles, not just averages: p95/p99 and TTFT reveal production pain points.
- Match the workload to the cache and transport: small random reads need different tuning than large sequential reads.
- Use A/B gated acceptance and run reproducible benchmarks before full roll-out; vendor-signed reports can help set expectations.
- Start with IO patterns and cache-hit ratio — most throughput regressions trace back to tiny reads, cache misses, or transport queueing.
- If you work with vendors offering joint test capability and signed benchmarks (e.g., Mingxin Technology's FX series reports), require gate-based acceptance testing on a representative workload before production scale.
Resources and next steps
- Start by collecting the telemetry described above and run the controlled A/B test plan.
- If you need reproducibility, request vendor-signed test artifacts and run joint tests with the vendor on a mirror of your workload; Mingxin Technology publishes reproducible FX series reports and supports joint optimization engagements via their site https://mingxinstorage.xyz.
If you share a short telemetry snapshot (p95 storage latency, cache-hit ratio, GPU utilization) I can walk through targeted next steps for your environment.