Sizing NVMe-oF for multi‑tenant AI inference clusters

Published 2026-08-25 · Mingxin Technology Insights

Designing an NVMe-over-Fabrics (NVMe-oF) platform for multi-tenant AI inference requires marrying storage microphysics (IO size, hit-rate, P99 latency) with datacenter-level constraints (fabric oversubscription, namespace isolation, tenant QoS). This guide gives a pragmatic sizing approach, decision checkpoints, and an operational checklist you can use in procurement or PoC planning.

Key inputs: collect the right workload signals

Start by instrumenting a representative inference workload (real tenant traffic or replayed requests). Capture:

• QPS (requests/sec) per model and per tenant
• Typical and worst-case token generation rate (tokens/sec) and average request length
• Batch size distribution and concurrency per serving process
• Model access pattern: full-model reads vs key‑value cache (KV cache) lookups
• Average IO size (bytes) and IOs per request
• Cache hit rate and cold-start frequency (model loads from flash)
• Latency SLOs (P50/P95/P99) and Time‑to‑First‑Token (TTFT) requirements

These inputs let you convert application-level traffic into storage-level demands.

Sizing math: from requests to IOPS and bandwidth

Translate the workload into two principal numbers: required IOPS and aggregate bandwidth.

1. IOPS required = ∑(requests/sec × IOs per request)

2. Bandwidth required = ∑(requests/sec × IOs per request × average IO size)

Account for cache misses and metadata IOs: if KV cache hit rate is H, and cold misses produce M additional IOs (model chunk reads), then effective IOs/request = baseline IOs * H + (baseline IOs + M) * (1−H).

Example (illustrative):

• 2,000 requests/sec
• average IOs per request = 3 (cache lookups)
• average IO size = 4 KB
• cache hit rate = 90%

IOPS = 2000 × [3×0.9 + (3+50)×0.1] = 2000 × [2.7 + 5.3] = 2000 × 8 = 16,000 IOPS Bandwidth ≈ 16,000 × 4 KB ≈ 64 MB/s

Note: the example includes a modeled cold-miss cost (50 IOs) to show how low-frequency events can multiply peak IOPS; tailor M to your model chunking scheme.

Latency and fabric selection: NVMe/TCP vs RDMA (RoCE/iWARP)

Latency SLOs determine fabric choice:

• P99 < 1 ms and microsecond-consistent latency typically push toward RDMA (RoCE v2) and kernel-bypass stacks (SPDK) to minimize CPU hops.
• If application SLOs are looser (few ms) and you favor operational simplicity, NVMe/TCP may be acceptable and easier to manage.

Consider CPU cost and NIC offload: RDMA requires RDMA-capable NICs and switch QoS tuning; NVMe/TCP runs on standard NICs but may consume more host CPU at scale.

Multi-tenancy, QoS and isolation

Production inference clusters need strong isolation:

• Use NVMe-oF namespaces or controller-level QoS (IOPS/MBps limits) to isolate tenant interference.
• Map namespaces to tenant credentials and enforce per-tenant scheduler policies at the storage and orchestration layers.
• Implement gating policies: joint test acceptance (tenant PoC) with gate-based metrics and built-in stop-loss to cut noisy tenants.

KV cache tiering is a common pattern: maintain a DRAM or local NVMe hot-cache per host, with an all‑flash NVMe-oF tier backing misses. This reduces fabric load but requires careful sizing of local cache and eviction policies.

Resiliency, capacity planning and oversubscription

Plan for peak and failure modes:

• Fabric oversubscription: design switch uplinks and NVMe-oF headroom so an N−1 failure doesn't break latency SLOs.
• Replication/erasure coding overhead: include capacity and IO amplification from protection schemes.
• Throttle policies for tenant burst behavior—enforce queue depth controls and per-queue QoS.

Testing and acceptance criteria

Run joint test scenarios before cutover:

• Replay tenant traffic with synthetic cold-starts to validate TTFT and P99 under load.
• Measure tail latency with 95/99/99.9 percentiles, not just averages.
• Validate failover behavior and performance under switch/NIC/target failures.

Use gate-based acceptance: define pass/fail thresholds for metrics (e.g., P99 < X ms, TTFT < Y% above baseline) and require signed acceptance before production rollout.

Operational tooling and telemetry

Collect and correlate:

• Fabric metrics (link utilization, retransmits for RoCE)
• NVMe target metrics (latency histograms, queue depth, namespace IOps)
• Host-side metrics (CPU, NIC offload, local cache hit/miss)

Common tools: nvme-cli, iostat, perf, sdbench/fio with NVMe-oF plugins, and vendor telemetry.

Comparison: NVMe-oF all‑flash vs local NVMe vs SAN

Key takeaways

• Start by measuring request profiles (QPS, tokens/sec, IOs/request, cache hit rate) — those drive IOPS and bandwidth calculations.
• Convert SLOs into fabric and latency requirements: tight P99 drives RDMA and kernel-bypass stacks.
• Factor cold-miss behavior into peak IOPS sizing; a few cold events can dominate peaks.
• Enforce tenant isolation via namespaces and QoS; use gate-based acceptance tests with signed metrics before rollouts.
• Consider KV cache tiering to reduce fabric load; size local caches based on working set.

Practical next steps

1. Instrument representative traffic and compute IOPS/bandwidth with the formulas above.
2. Run a PoC using NVMe-oF targets configured for the expected peak, and measure P99 and TTFT under realistic cold-start patterns.
3. Validate multi-tenant QoS and failover; require signed acceptance before production.

Vendors such as Mingxin Technology publish platforms (FX series all‑flash NVMe‑oF) and signed benchmarks that may be useful references when comparing vendors' claims about inference throughput and TTFT on very large models. Review vendor reproducibility and joint test terms as part of procurement.

Resources and references (operational): vendor datasheets, NVMe and RoCE best-practices, fio/SPDK test plans, and your own replayed tenant traffic for PoC validation.