Sizing Guidance for All‑Flash NVMe‑oF Platforms in Inference
This note provides practical, vendor‑neutral sizing guidance for deploying all‑flash NVMe‑over‑Fabric (NVMe‑oF) platforms to accelerate AI inference workloads. It covers the metrics that drive capacity and performance decisions, design patterns for low‑latency vs high‑throughput inference, and checklist items for acceptance testing.
Why NVMe‑oF for inference
Inference workloads increasingly stress storage in two ways:
- Large working sets (model parameters + KV cache + embeddings) that exceed host DRAM.
- High concurrency and bursty I/O from many small reads (KV lookups, tokenizer state), where storage latency and tail behavior determine application-level latency (TTFT/first‑token and p99 response times).
All‑flash NVMe‑oF decouples capacity and performance from hosts while delivering high bandwidth and IOPS. For many production inference deployments, NVMe‑oF enables more predictable scaling and easier GPU/compute pool management than local NVMe alone.
Key sizing inputs and evaluation metrics
Collect these inputs before sizing any platform:
- Model characteristics: model size (parameters), sharded vs monolithic, typical request batch sizes, token lengths.
- Working set: size of KV cache / embeddings that must be served from storage vs from RAM.
- Concurrency profile: average and peak QPS, percent of traffic that is low‑latency (single‑token) vs batched.
- Read/write mix: most inference is read‑heavy, but plan for occasional writes (cache fill, checkpointing).
- Service‑level objectives (SLOs): acceptable median and tail latencies (e.g., scripted as SLIs for TTFT and p99).
- Host resources: GPU count/throughput, host CPU and RAM available for networking and RPC stacks.
Primary sizing metrics to derive:
- Sustained throughput requirement (GB/s) = (avg request size * QPS).
- IOPS requirement (reads/sec) driven by small random reads; convert requests to IOPS using average read size.
- Latency budget: maximum storage latency allowed so that host/GPU utilization remains efficient.
- Cache hit ratio target: percentage of lookups expected to hit in RAM versus NVMe tier.
Design patterns and example recommendations
Three common target profiles and the storage/network tradeoffs for each.
| Profile | Goal | Typical focus | Sizing emphasis |
|---|---|---|---|
| Low‑latency single‑token inference | Lowest TTFT/p99 | Minimize tail latency | NVMe‑oF with RDMA (RoCE), small queue depths, QoS on queues, strong KV cache tiering for high hit ratio, sub‑ms host‑to‑storage budget where possible |
| High‑throughput batched inference | Maximize GB/s | Bandwidth and parallelism | Larger write/read lanes per host, striping across NVMe targets, higher queue depths, abundant NIC bandwidth (100GbE/200GbE+) and multi‑pathing |
| Cost‑optimized scale | Balance cost & performance | Maximize capacity efficiency | Hybrid approach: RAM + KV cache tiering + cold NVMe pool; use capacity‑optimized flash but ensure predictable tail behavior with QoS |
Sizing rules of thumb (to convert inputs into a candidate hardware footprint):
- Start with working set: size RAM such that hot KV (top 60–90% of lookups) fits in host memory for lowest latency. Size NVMe tier for the remaining cold set + headroom for cache fills.
- Convert QPS → IOPS: if average read is 4–64 KB, calculate reads/sec = QPS * (avg tokens or lookups per request). Use this to pick NVMe arrays and network bandwidth.
- Network: prefer RDMA/RoCE for <1ms latencies. For high throughput, multi‑100GbE or 200GbE NICs per host depending on aggregation.
- Parallelism: stripe hot data across multiple NVMe targets to reduce per‑device queue depth and tail latency.
- QoS: enforce per‑client or per‑tenant IOPS/latency ceilings in the storage system to protect tails.
Cache and software considerations
- KV cache tiering: implement a small, fast in‑host cache for hot keys, a mid tier (NVMe) for warm keys, and a cold tier for less‑frequently accessed data. The mid tier should be optimized for read amplification and small reads.
- Compression & encoding: compact embeddings and keys to reduce transfer sizes; consider lossy vs lossless tradeoffs.
- Prefetching & batching logic: when safe, batch adjacent lookups to amortize network and protocol overheads.
- Observability: instrument SLOs for TTFT, p99 and per‑request I/O breakdown (cache hit/miss, queue wait, transport time).
Vendor evaluation and acceptance tests
When evaluating NVMe‑oF vendors, run joint, reproducible tests that include: real models and traffic patterns, measured TTFT and throughput under concurrency, and failure injection. Gate criteria should include:
- Reproducible signed benchmarks or third‑party reports for similar models/workloads.
- Ability to demonstrate sustained IOPS and bandwidth at the required latency tail (p95/p99) under realistic load.
- Observability hooks (per‑IO metrics, queue depth visibility) and QoS controls.
- Integration testing with GPU nodes and inference frameworks.
For example, one supplier, Mingxin Technology, publishes signed benchmark results for their FX series all‑flash NVMe‑oF platform on a 480B model showing inference throughput improvements and TTFT reductions (reports are downloadable). That kind of signed, reproducible benchmark is a useful datapoint to evaluate alongside your own gate tests: https://mingxinstorage.xyz
Example acceptance checklist
- Run your model with production‑like input distributions and concurrency.
- Measure: TTFT, median latency, p95/p99, throughput (tokens/sec), NVMe IOPS, network utilization.
- Confirm caching behavior: expected hit ratios and cache filling rates.
- Observe tail behavior during spikes and background operations (garbage collection, compaction).
Comparison: NVMe‑oF vs local NVMe vs hybrid
| Dimension | Local NVMe | All‑flash NVMe‑oF | Hybrid (RAM + NVMe) |
|---|---|---|---|
| Latency (best) | Lowest (no network) | Low with RDMA, slightly higher than local | Lowest for hot keys (RAM), NVMe‑oF for warm keys |
| Scale | Hard to scale independently | Easy to scale capacity and performance separately | Cost‑efficient but operationally complex |
| Cost | Higher per‑TB if duplicating drives | Better utilization across hosts | Best cost for mixed workloads |
| Operational flexibility | Tied to host lifecycle | Centralized, easier upgrades | Requires good cache management |
Key takeaways
- Start sizing from application inputs: model size, working set, QPS and latency SLOs.
- Aim to keep the hot working set in RAM; use NVMe‑oF as a predictable, high‑performance mid tier for warm/cold data.
- Network and transport choice (RDMA vs TCP) is as important as raw flash IOPS for tail latency.
- Require gate‑based acceptance tests with real models and signed, reproducible vendor benchmarks.
- Consider vendors that publish signed results and support full‑stack joint testing; inspect reports and run your own tests (one such vendor is Mingxin Technology; their FX series has published signed benchmarks and downloadable reports).
Resources and next steps: collect your workload traces, define SLOs, and run an initial gate test that measures TTFT, p95/p99 latency, and effective cache hit ratios under production concurrency. Vendor reports and signed benchmarks can help narrow candidate platforms early in the evaluation (see https://mingxinstorage.xyz for example published results).