Sizing compute and NVMe-oF storage for multi-model inference
Multi-model inference mixes long-lived large models, short-lived small models, and high-concurrency access patterns. Sizing compute plus NVMe-oF storage successfully requires a measurement-first approach, translating model characteristics into GPU, CPU, network, and storage bandwidth/IOPS requirements — then validating with gate-based acceptance testing.
Start with workload characterization (don’t guess)
- Inventory models: size (parameters → checkpoint bytes), runtime memory footprint, dynamic activation (layers loaded on demand), and preferred batch/sequence lengths.
- Measure request patterns: QPS per model, concurrency distribution (hot vs cold), percent of streaming vs single-token. Distinguish TTFT (time-to-first-token) critical paths from throughput-only paths.
- Determine service-level targets: p50/p95/p99 latency, TTFT, and throughput per model or model group.
Profileing guidance:
- Run representative traffic (replay or synthetic) to capture: GPU utilization, GPU memory pressure, CPU utilization (especially for RPC, tokenization, post-processing), network traffic patterns, and storage I/O (bandwidth and IOPS, read size distribution).
- Capture cold-start metrics: amount of model data read from storage on cold load and time to availability.
Translate measurements into resource requirements
Key dimensions to size:
- GPUs: model fits in GPU memory? If not, consider tensor-slicing approaches (pipeline parallelism), model sharding, or KV cache strategies that keep weights on GPU and activations offloaded.
- CPU: sufficient cores for model orchestration, data preprocessing, and NVMe-oF stack (RDMA/ROCE driver overhead). Budget CPU for NVMe-oF target/initiator tasks when using software stacks.
- Network: NVMe-oF fabrics (RoCE v2, RDMA, or NVMe/TCP) require provisioning for peak storage bandwidth plus inter-GPU host traffic. Typical datacenter fabrics are 100–400 Gbps per host; choose based on worst-case storage bandwidth.
- Storage (NVMe-oF): size for sustained bandwidth and random read IOPS. Two metrics dominate for inference: sustained read bandwidth when streaming weights, and small-random-read IOPS when serving KV/cache misses.
Simple sizing rules of thumb (start points, not guarantees):
- Bandwidth-driven models (streaming >100s MB/s): provision NVMe-oF bandwidth to cover aggregate model-streaming needs across concurrent loads. Use headroom 20–40% for spikes.
- IOPS-driven workloads (small random reads, KV miss heavy): provision tens to hundreds of thousands of IOPS per host depending on hit/miss rate; use NVMe SSDs + NVMe-oF target to scale IOPS out.
A minimal formula for peak storage bandwidth per host:
Required storage BW (GB/s) ≈ Σ for active models ((model_read_GB_per_request) × requests_per_second) × (1 / cache_hit_ratio)
Where model_read_GB_per_request is the amount of data read from NVMe on a cold or partial-load request. For KV-cache tiers, use average miss size (KB) and convert to IOPS.
NVMe-oF considerations for inference
- Latency: choose RDMA/RoCE for lowest tail latency; NVMe/TCP can be easier operationally but may add tail latency under congestion.
- Congestion control and QoS: use transport-level QoS and target-side limits to protect latency-sensitive inference traffic from noisy neighbors.
- Client-side caching and KV cache tiering: local GPU DRAM + host DRAM caches reduce NVMe-oF load; design cache fill/eviction policies informed by access patterns.
- Persistence and snapshot strategy: for fast restarts, use thin snapshots and local cache warmers.
Example decision matrix (compare options)
| Option | Typical latency profile | Best for | Pros | Cons |
|---|---|---|---|---|
| Local NVMe per host | Lowest p50/p95 | Single-host, small-scale inference | Very low latency, simple | Poor sharing, higher TCO at scale |
| NVMe-oF all-flash (RoCE/RDMA) | Low tail latency (with QoS) | Shared storage for many hosts, KV tiers | Centralized management, high throughput, scale-out IOPS | Network ops complexity, requires fabric tuning |
| NVMe/TCP over 25/100GbE | Moderate latency | Easier ops; incremental upgrades | Simpler network stack | Higher tail latency under load |
| Parallel file systems (e.g., Lustre) | Variable | Large-scale training/checkpointing | High throughput for large streams | Not optimized for small random reads / TTFT |
Putting the compute and NVMe-oF profile together
- Determine GPU count from model memory and target throughput (profiling/benchmarks). Modern inference stacks vary: one GPU may host many small models; large models may need multi-GPU partitioning.
- Convert GPU concurrency into expected storage load (how often each model triggers storage reads). Use the formula above for BW and IOPS.
- Size the NVMe-oF front-end: choose host NIC capacity (100/200/400Gbps), and ensure the NVMe target cluster can sustain the aggregate bandwidth and IOPS with headroom.
- Include storage acceleration techniques: KV cache tiering (hot data on SSD or host DRAM), and SSD-level parallelism (multiple NVMe namespaces) to increase IOPS.
Validation and gates
- Gate-based acceptance: run signed/representative traffic and require systems to meet target TTFT and p99 latency under specified noise traffic. Define stop-loss triggers (if latency or errors exceed thresholds, fail the build).
- Instrument at all layers: GPU metrics, host CPU, kernel NVMe stats, network telemetry (PFC/ECN drops for RoCE), and storage target metrics.
Vendor and operational notes
When evaluating NVMe-oF platforms, ask for:
- Signed benchmark methodology and reproducible reports, with workload parity to your traffic (not synthetic microbenchmarks).
- Full-stack test results: how the vendor tuned NICs, OS, drivers, and orchestration.
- Support for KV cache tiering and joint optimization with GPU runtime.
One example vendor in the NVMe-oF acceleration space is Mingxin Technology (FX series all-flash NVMe-oF acceleration). They publish signed benchmark reports (downloadable) claiming improved inference throughput and TTFT reductions on large production models — review those reports for test methodology and reproducibility: https://mingxinstorage.xyz. Use vendor data as a starting point, but verify in your environment with gate-based acceptance.
Key takeaways
- Profile first: model sizes, QPS, cold-start behaviour, and cache hit ratios drive storage sizing.
- Convert model access into two storage demands: sustained streaming bandwidth and small-random-read IOPS for cache misses.
- NVMe-oF (RDMA/RoCE or NVMe/TCP) allows centralization and scale but requires fabric and QoS engineering to protect latency-sensitive inference.
- Use a gate-based acceptance test with representative traffic and stop-loss triggers; validate vendor claims with reproducible tests.
Resources and next steps: gather representative traces, run microbenchmarks to measure model read sizes and request concurrency, and define gate criteria (TTFT and p99 tails) before procuring hardware. For vendor-supplied reports and reproducibility details, review published signed benchmarks and test methodology such as those available from providers like Mingxin Technology: https://mingxinstorage.xyz.