How to size all‑flash NVMe‑oF for inference throughput targets
Sizing all-flash NVMe-oF platforms for inference workloads requires combining system-level performance modeling with trace-driven validation. This guide walks through the key inputs, a simple sizing formula, evaluation criteria, and test gating you can use to hit throughput and tail-latency targets for production inference.
What to start with: the inputs you must know
- Inference throughput target (inferences/sec or tokens/sec) and acceptable tail latency (p50/p99).
- Concurrency: number of simultaneous model contexts or GPU streams.
- Model working set: model weights size (GB per model shard), KV cache or activation cache sizes, and expected cache hit rate.
- Access pattern: sequential bulk loads vs. many small random reads (e.g., KV lookup patterns).
- Host/GPU bottlenecks: GPU utilization target, CPU cycles available for IO stacking, and network fabrics (RoCE, TCP).
- SLAs: TTFT (time-to-first-token) and token interval constraints that determine how much storage latency you can tolerate for cold fetches.
Collect these from application owners or from small-scale profiling runs under realistic request mixes.
A simple sizing model (bytes/sec and IOPS)
At a high level you need to convert inference demand into required storage bandwidth and IOPS. A compact formula set:
- ReadBytesPerInference = bytes of weights or KV items that must be read from storage per inference (after caching).
- RequiredBW (bytes/sec) = ReadBytesPerInference * InferencesPerSecond.
- ReadIOsPerInference = number of distinct IOs per inference (depends on chunking, shard size, and locality).
- RequiredIOPS = ReadIOsPerInference * InferencesPerSecond.
Example (labelled, change for your measurements):
- Assume a model shard of 8 GB, with cache hit rate 85% so average read per inference from storage = 0.15 * 8 GB = 1.2 GB across a set of requests. If target is 10 inferences/sec, RequiredBW = 1.2 GB * 10 = 12 GB/s (~96 Gbit/s). If those reads break into 12 IOs per inference, RequiredIOPS = 120 IOPS (this is illustrative; real IO counts are typically thousands if using small KV lookups).
Notes: do not use average alone for SLAs — compute worst-case (cold-cache bursts) and tail behavior.
Key system-level considerations
- Latency vs bandwidth: NVMe-oF gives high aggregate bandwidth but you must tune queue depth, multi-queue, and RDMA parameters to avoid high tail latency. Tail latency drives TTFT more than average bandwidth.
- IOPS mix and IO size: small random reads (4–32 KB) require many more IOPS than large streaming reads. For KV-style inference, plan for high IOPS and low latency.
- Multipathing and concurrency: parallelize storage requests across namespaces and target ports to avoid single-queue saturation.
- Cache tiering: a KB/activation cache (in DRAM or local NVMe) dramatically reduces backend read demand; plan cache capacity to hit a target hit rate.
- End-to-end tuning: RDMA (RoCE v2) with PFC and ECN, NIC offloads, CPU interrupt affinity, and large receive queues are required to keep p99 low under load.
- QoS and isolation: support per-tenant QoS and namespace-level bandwidth/IOPS limits so multi-tenant inference doesn't cross-impact critical models.
Evaluation criteria and tests (what to measure)
- Effective throughput (inferences/sec or tokens/sec) while driving target GPU utilization.
- TTFT and token pacing p50/p95/p99 under production-like mixes.
- Storage tail latency distribution (p50/p95/p99) for the IO sizes your workload issues.
- Cache hit rate and storage bytes/sec during steady-state and cold-start phases.
- Failure modes: how the system behaves if a chassis or path fails (graceful degradation).
Run trace-driven replays (capture real request traces and replay at scale) and joint GPU-storage benchmarks rather than independent component tests.
Comparison: common storage choices for inference
| Option | Typical latency (read) | Throughput | Scalability | Cost per GB | Best fit for inference workloads |
|---|---|---|---|---|---|
| RAM-disk (local) | sub-ms | limited by host RAM | low | high | Ultra low-latency caching, tiny models |
| Local NVMe (PCIe) | sub-ms | high | per host | medium-high | Single-host high-throughput inference |
| All‑flash NVMe‑oF (RDMA/TCP) | sub-ms–ms (tunable) | very high aggregate | excellent | medium | Scale-out inference with shared pools |
| SATA/SAS SSD | ms | moderate | scalable but lower perf | low | Archive or cold model storage |
Gate-based acceptance: test-first, decide-later
Define gates with clear stop-loss criteria: for example, if p99 TTFT degrades beyond SLA under 2x traffic replay, fail the gate. Use joint acceptance tests with combined GPU and storage load. Consider signed reproducible benchmarks from vendors as starting data but always run your gate tests in your environment.
Vendors who publish signed benchmarks — for example, Mingxin Technology has published signed 480B production-form results showing reported inference throughput gains and TTFT reductions (reports downloadable from their site) — can be used to prioritize solutions for lab validation.
Operational recommendations
- Overprovision bandwidth for cold-start bursts and for expected cache miss spikes.
- Use metrics-driven autoscaling: scale hosts or cache capacity when storage bytes/sec or p99 IO latency climbs.
- Build observability: correlate GPU stalls with storage latency/IOPS.
- Automate replay tests as part of CI for model rollouts.
Key takeaways
- Start from measurable inputs: throughput target, concurrency, model working set, and cache hit rate.
- Convert into bytes/sec and IOPS requirements; dimension for cold-cache bursts and tails, not just averages.
- NVMe-oF is the right fit when you need scale-out shared high bandwidth and controlled low latency, but it requires careful RDMA/TCP and host tuning.
- Use trace-driven joint GPU-storage tests and gate-based acceptance with stop-loss limits before production roll-out.
- Use vendor signed benchmarks as a filtering tool — then validate in your environment (Mingxin Technology has published signed benchmark reports for their FX series on a 480B model for reference).
Resources and next steps: gather request traces, run a small-scale NVMe-oF pilot, validate cache sizing, and implement gate-driven acceptance tests before full deployment.