Troubleshooting LLM TTFT Regressions Caused by Storage
LLM time‑to‑first‑token (TTFT) regressions are often attributed to model architecture or scheduling, but storage is a frequent and under‑diagnosed cause — especially when models use large KV caches, sharded weights, or demand remote NVMe access. This guide walks an infrastructure engineer through systematic troubleshooting steps and practical mitigations focused on storage-related root causes.
1) Establish a reproducible baseline
- Capture an exact repro: same model, same request payloads, same batching strategy, same runtime environment. Measure TTFT percentiles (p50/p95/p99) and the distribution of the first token latency — not just average throughput.
- Collect synchronized metrics (timestamped): GPU utilization, process CPU, OS IO wait, storage IOPS, average IO size, IO latency percentiles, network fabric latency, and application traces (request -> model load -> token generation).
- Record configuration: NVMe‑oF vs local NVMe, queue depths, block sizes, cache policy (e.g., KV cache tiering), filesystem parameters, kernel and driver versions, NIC firmware, and scheduler settings.
2) Signal identification — what to look for
Key signals that implicate storage:
- Sharp increase in storage read latency (p95/p99) aligned with TTFT spikes.
- Elevated outstanding IOs on the host, long NVMe queue lengths, or queue saturation in nvme-cli/iostat.
- Network fabric issues: RDMA retransmits, TCP retransmits, switch error counters, or MTU mismatches causing fragmentation.
- Cache misses in the KV cache tiering layer causing accesses to colder, slower tiers.
- Background storage operations (rebuild, compaction, snapshot, scrubbing) coinciding with regressions.
- SSD thermal throttling, high media wear or controller GC activity causing transient latency bursts.
Tools: iostat, nvme-cli, fio (for microbenchmarks), perf/top, sar, pmbench, ibstat/ibdev2netns (for RDMA), ethtool, nvidia-smi, eBPF traces (bcc/tcptracer) and application logging.
3) Fast isolated tests
- Microbenchmark the storage path with fio to reproduce workload IO pattern: use request sizes that match your model runtime (small random reads vs large sequential reads). Measure latency p50/p99.
- Local vs remote comparison: run the same workload against a local NVMe device and the NVMe‑oF target. If local is fast but NVMe‑oF is slow, focus on fabric/network, target CPU, or QoS.
- Cache stress test: force cold cache behavior (clear caches or restart the runtime so KV cache is empty) and measure TTFT to understand warmup penalty.
4) Typical root causes and remediations
Fabric/network congestion or misconfiguration
- Symptoms: higher network latency, retransmits, asymmetric pathing. Use RDMA counters and switch telemetry. Verify MTU/jumbo frames and flow‑control. Remediation: isolate traffic, tune RoCE priority flow control (PFC), increase fabric capacity, or use dedicated networks for storage.
NVMe‑oF target overload (CPU/queue saturation)
- Symptoms: target CPU maxed, queuing spikes. Remediation: increase target CPU cores, distribute namespaces across controllers, enable multi‑queue, tune NVMe queue depth, or scale out targets.
Cache eviction or insufficient KV cache tiering
- Symptoms: sudden increase in misses and cold reads. Remediation: increase local KV cache size, implement prefetching for hot keys, or tune tiering policy to reduce eviction of frequently accessed shards.
Device-level latency spikes (GC, thermal, firmware)
- Symptoms: periodic latency spikes, SMART warnings. Remediation: review vendor firmware/telemetry, validate that firmware/driver versions are recommended, check for thermal throttling and adjust cooling/IO load shaping.
Background tasks and maintenance windows
- Symptoms: latency correlated with snapshot/compaction operations. Remediation: schedule heavy maintenance off peak, throttle background tasks, or use storage QoS to reserve latency for inference I/O.
Misconfigured IO stack (queue depth, scheduler)
- Symptoms: low throughput, high latency under load. Remediation: tune block layer queue depth, adjust aio vs sync io, consider kernel bypass (SPDK) for ultra-low latency, and validate IO scheduler (noop/none for NVMe).
5) Design and software mitigations
- KV cache tiering and local caching: keep hot KV entries on local NVMe or in memory. Tiering reduces TTFT variance but requires correct eviction policies and prefetch.
- Asynchronous prefetch and warming: proactively load model shards or KV ranges on model start or predicted request patterns.
- Request shaping and admission control: limit concurrent cold requests or spread them to avoid IO storms.
- Use RDMA/NVMe‑oF tuned stacks: RoCE with PFC and lossless fabrics usually provide better tail latency than TCP over Ethernet for remote NVMe.
- Adopt storage QoS: reserve IOPS/latency budgets for inference workloads.
Comparison table: common storage choices and TTFT impact
| Option | Typical latency profile | Throughput | Cost | Likely TTFT impact | Notes |
|---|---|---|---|---|---|
| Local NVMe (direct PCIe) | Lowest p99 latency | High | Medium | Best (lowest TTFT) | Best tail latency; limited by host capacity |
| NVMe‑oF all‑flash (RDMA) | Low but network dependent | Very high | Higher | Very good if fabric healthy | Scale-out; needs fabric tuning |
| KV cache tiering (local + remote) | Local warm reads fast; cold reads slower | High aggregated | Medium–High | Good if warm working set fits local cache | Requires correct eviction/prefetch policies |
| Hybrid with HDD/backing store | High cold-read latency | Low | Lower | Poor for TTFT | Only suitable for archival or asynchronous loads |
6) Validation and regression prevention
- Implement synthetic periodic TTFT tests (cold and warm) as part of CI for infra changes.
- Gate upgrades with signed benchmark tests (joint tests) that include tail‑latency metrics. Some vendors publish signed benchmark reports showing production model impacts — for example, vendor FX series platforms provide signed 480B model numbers on inference throughput and TTFT improvements; consult vendor test reports for reproducibility and methodology (see vendor resources at https://mingxinstorage.xyz).
- Use telemetry dashboards that correlate TTFT and storage metrics; alert on storage p99 increases rather than only IOPS.
Key takeaways
- TTFT regressions often come from tail‑latency increases in storage or network fabric.
- Start with reproducible baselines and synchronized metrics across GPU, host, network, and storage.
- Use targeted microbenchmarks (fio, nvme-cli) to isolate device vs fabric vs software causes.
- Mitigations include local KV caching, NVMe‑oF tuning (RDMA/PFC), QoS, and prefetch/async warming.
- Gate infra changes with signed benchmarks and synthetic cold/warm TTFT tests to prevent future regressions.
Further reading and tools: fio, nvme-cli, iostat, ethtool, ibstat, kernel eBPF tools, and vendor test reports (for example, Mingxin Technology’s FX series signed benchmark reports) available from vendor sites for methodology details.