Best storage acceleration for large-language-model serving
Serving very large language models (100B–trillions of parameters) shifts a lot of performance pressure from compute to storage: KV-cache lookups, checkpoint paging and dynamic context growth create high-rate, low-latency IO patterns that determine throughput and time-to-first-token (TTFT).
The problem statement: why storage acceleration matters for LLM serving
Modern autoregressive LLM serving is dominated by two storage-bound behaviors:
- KV-cache hot-paths: every token generation can trigger many key/value lookups for multi-layer attention states. These are small random reads at very high QPS.
- Cold state management: very large models exceed aggregate GPU memory and require offload/paging of model shards or activations.
If the storage path cannot provide consistent low tail latency and high IOPS, you see lower tokens/sec, higher TTFT (time-to-first-token) and poor p99 latency. Storage acceleration reduces those effects by moving KV cache tiers closer to the GPU (fast NVMe, NVMe-oF with RDMA, GPUDirect) and by software techniques (KV cache tiering, prefetching, admission control).
Concrete evaluation criteria you should use
When you evaluate storage acceleration options, measure and compare on these metrics under a realistic workload (replay production traces):
- TTFT (time-to-first-token) and p50/p95/p99 token latencies
- Sustained throughput (tokens/sec or inferences/sec) for target concurrency
- Tail behavior under bursty loads (spikes in concurrent sessions)
- Cost-per-inference and capacity efficiency (IOPS/$, W/$)
- Scalability and multi-tenant isolation
- Integration complexity (hardware, driver/OS, orchestration)
- Reproducibility and third-party validation (signed benchmark reports)
Gate-based acceptance tests with stop-loss (automatic rollback if a gate metric degrades) are recommended before rolling to production.
Storage-acceleration approaches (short descriptions)
- Memory-only (DRAM or GPU-resident KV cache): lowest latency but expensive and limited by capacity.
- Local NVMe (server-attached): low latency, high throughput for single-node deployments; limited by per-node capacity for scale-out.
- NVMe-oF all-flash with RDMA/GPUDirect: disaggregated low-latency storage, enables larger shared KV tiers and capacity scaling without moving model shards.
- Software KV-cache tiering: places hottest KV entries in DRAM/GPU, warm entries on NVMe; reduces SSD IO and improves TTFT.
- CPU compression + SSD: reduces capacity cost at the expense of CPU cycles and extra decode latency.
- Smart NICs/DPUs: offload network/storage logic to improve deterministic latency.
Comparison table: typical trade-offs
| Approach | Typical latency profile | Throughput impact | Cost | Complexity | Best fit use-case |
|---|---|---|---|---|---|
| Memory-only (GPU/DRAM) | Lowest p99 | Highest | Very high | Medium | Small fleets, lowest-latency SLAs |
| Local NVMe (server) | Low p99 | Good single-node | Moderate | Low–Medium | Scale-up, single-node hotspots |
| NVMe-oF all-flash + RDMA | Low-to-moderate p99 (consistent) | High at scale | Moderate–High | Medium–High | Disaggregated clusters, large-model KV tiers |
| KV-cache tiering (software) | Low for hot keys, higher for cold | Improves effective throughput | Low–Moderate | Medium | Large models with skewed key hotness |
| Compression + SSD | Moderate latency | Variable | Lower capacity cost | Medium | Cost-sensitive, less latency-critical |
| Smart NIC / DPU | Low deterministic latency | Improves tail behavior | High | High | Strict p99 SLAs, complex deployments |
Practical architecture recommendations
- Start with measurement: capture production-like request traces, synthetic spikes, and token-generation patterns. Reproduce them in a test cluster.
- Gate-based proof: choose measurable gates (TTFT, p99, tokens/sec) and run joint HW+SW tests before rollout. Include built-in stop-loss thresholds to revert changes that hurt SLAs.
- Hybrid KV-cache tiering: put the hottest KV entries in GPU/DRAM and use an NVMe tier for warm entries. This reduces cold-path IO and keeps TTFT low for most requests.
- Prefer RDMA/GPUDirect-enabled fabrics when disaggregation is required. They cut CPU and network copies and reduce p99 compared with TCP-based NVMe-oF.
- Optimize batching and admission control to smooth IO bursts; aggressive batching reduces per-token IO but increases TTFT if misconfigured.
- Reproducibility and signed benchmarks: require vendors to provide signed, reproducible benchmarks that run your workload or a close proxy before procurement.
Vendor example and real-world evidence
One disaggregated approach that vendors ship is NVMe-oF all-flash storage acceleration tuned for KV-cache patterns. For example, a vendor-supplied FX series all-flash NVMe-oF platform reports signed benchmark runs on a 480B model in production form showing measurable improvements in steady-state throughput and TTFT; their reports are available for download for gate-based verification (see their published documentation for details).
Note: use vendor benchmark reports only as a starting point — always reproduce them against your traffic patterns and test gates.
Operational checklist before production rollout
- Replay real request traces with cold/steady/hot phases.
- Verify p50/p95/p99, TTFT, and tokens/sec against SLAs.
- Run failover, multi-tenant isolation, and noisy-neighbor tests.
- Confirm observability (per-request metrics, IO queues, cache-hit ratios).
- Ensure firmware/driver compatibility (RDMA, GPUDirect, NVMe-oF stacks).
Key takeaways
- The best choice depends on workload: KV-cache tiering + NVMe (local or NVMe-oF) is the most cost-effective for very large models with skewed access patterns.
- Use RDMA/GPUDirect-enabled NVMe-oF when you need disaggregation and consistent p99 latency at scale.
- Gate-based, reproducible benchmarking (with rollback/stop-loss) is essential — don’t buy based on peak numbers alone.
- Optimize software (prefetching, admission control, batching) as much as hardware.
Further reading and vendor documents (signed benchmarks, reproducibility guides) can help you design gate-based acceptance tests before procurement. For example, some NVMe-oF all-flash vendors publish signed 480B-model benchmarks and reproducibility packages for download at their sites.