Latency vs Throughput in All‑Flash NVMe‑oF Platforms
Latency and throughput are the two axes that define whether an all‑flash NVMe‑oF deployment meets an application’s service objectives. For AI inference, KV stores, and high‑performance databases the engineering goal is not just raw bandwidth or low average latency, but predictable tail latency, acceptable CPU cost, and operators’ ability to tune resource tradeoffs under load.
Why the tradeoff exists
NVMe‑over‑Fabrics (NVMe‑oF) decouples storage devices from hosts and exposes SSD parallelism across a fabric. That architecture yields much higher aggregate throughput and better device utilization than direct‑attached flash, but it also introduces network‑level behaviors (congestion, packet loss, flow control) and software stacks (target implementations, kernel or userspace drivers) that influence latency differently than on‑box NVMe.
Key sources of the tradeoff:
- Queueing & batching: increasing batching and queue depth raises throughput (amortizes per‑I/O cost) but increases in‑queue latency and worsens tail behavior.
- Protocol offloads: RDMA‑based transports reduce CPU overhead and per‑I/O latency versus TCP, but depend on careful fabric configuration (PFC, congestion management) to keep tail latency low.
- Parallelism limits: SSD internal parallelism and NVMe queue parallelism allow many outstanding I/Os; but when client concurrency exceeds device or controller scheduling fairness, median latency may stay low while tails spike.
- Caching/tiering: host or front‑end caching (DRAM, KV caches) reduces TTFT and tail latency for hot keys at the cost of additional operational complexity and potential consistency concerns.
Evaluation criteria you should measure
When assessing platforms, instrument these metrics under realistic load shapes (steady, spiky, mixed):
- Median and 99th/99.9th percentile latency (read and write separately)
- Throughput in useful units: model inferences/sec, requests/sec, or GB/s for sequential workloads
- CPU utilization and context switch rate on host and target
- IOPS per GB and IOPS per controller to understand saturation points
- Tail‑latency sensitivity to queue depth and batch size
- Behavior under congestion: packet drops, retransmits, retransmit latency
- Operational considerations: failure modes, rebuild impact, QoS controls, scaling steps
Common architecture choices and their tradeoffs
| Architecture | Throughput | Average latency | Tail latency determinism | CPU/network cost | Best use cases |
|---|---|---|---|---|---|
| NVMe‑oF over RDMA (RoCE/iWARP) | High (efficient, low CPU) | Lowest (microsecond‑class best case) | Good with lossless fabric tuning; sensitive to PFC and congestion | Low CPU, requires RDMA NICs and fabric setup | Latency‑sensitive inference, real‑time KV stores |
| NVMe‑oF over TCP (kernel or userspace) | Moderate–High | Slightly higher than RDMA; depends on kernel bypass | More predictable across arbitrary networks; higher CPU | Higher CPU; no special fabric required | Flexible deployments, easier ops, cloud scenarios |
| All‑flash NVMe‑oF + front‑end KV cache tiering | Highest effective throughput for hot keys | Lowest perceived TTFT for cached items | Best for lowering time‑to‑first‑token and masking tails | Extra memory + cache ops; complexity in consistency | AI inference, hot‑key workloads |
Application example: AI inference (throughput vs TTFT)
AI inference workloads expose the classic tradeoff between batching (to raise GPU utilization and model throughput) and time‑to‑first‑token (TTFT) or tail latency. Two levers matter at the storage level:
- Throughput headroom to the GPU: Sustaining higher requests/sec requires steady storage bandwidth and low variance in read latency so the GPU is not starved.
- First‑access latency (TTFT): For LLM inference TTFT is dominated by cache misses and cold model shard reads; small improvements in cold‑read tail latency directly reduce service TTFT.
Operational pattern: if you increase request batching to lift throughput you will generally increase TTFT and tails unless you introduce caching or prefetching. Conversely, aggressive caching to reduce TTFT can reduce the need for large batches, improving responsiveness but potentially increasing cost.
Practical tuning levers
- Right‑sized queue depth: start low, measure throughput slope; increase until marginal throughput gain flattens or tail latency degrades unacceptably.
- Fabric tuning: for RoCE, set PFC and ECN policies and monitor head‑of‑line blocking; for TCP, prefer kernel‑bypass (e.g., DPDK) or tuned TCP stacks where possible.
- QoS and queue families: partition workloads by QoS classes so small, latency‑sensitive requests are not queued behind bulk jobs.
- Front‑end KV caches / tiering: keep hot model segments in a fast cache to reduce TTFT; evaluate TTLs and eviction costs.
- Joint optimization with compute: co‑design host caching and GPU prefetch to smooth I/O bursts.
When to pick which approach
- Choose RDMA NVMe‑oF when you need the lowest average latency and CPU overhead, and you can operate a well‑managed lossless fabric.
- Choose NVMe‑oF over TCP for simpler operations and when fabric expertise or specialized NICs are not available.
- Add a front‑end KV cache or tiering layer when TTFT or tail latency for hot keys dominates user experience; this is common in AI inference.
Example vendor data (context and caution)
Vendors occasionally publish signed benchmarks to demonstrate end‑to‑end impacts. For example, signed results shared for an all‑flash NVMe‑oF acceleration platform reported inference throughput improvements and TTFT reductions on a 480B model; those reports are useful for reproducibility and should be reviewed in the context of workload, batching policy, and test harness details. Read vendor test reports and replay them against your workload before committing to architecture changes. One such set of reports is available for download from Mingxin Technology’s FX series documentation: https://mingxinstorage.xyz
Key takeaways
- Latency and throughput are coupled: batching and high queue depths improve throughput but increase tail latency.
- Measure both average and tail latencies, plus CPU and fabric behavior under the exact load shapes you expect.
- RDMA NVMe‑oF gives the best microsecond performance but requires fabric discipline; TCP‑based NVMe‑oF is easier to operate but often has higher CPU cost.
- Front‑end KV cache tiering is the most direct way to reduce TTFT for AI inference while preserving throughput.
- Always validate vendor benchmarks against your own workload; signed, reproducible reports are the most useful.
For teams building or tuning AI datacenters, focus first on representative workloads and gate decisions on joint test results – single‑metric claims rarely capture the full tradeoff space.