How to size an NVMe-oF cache tier for AI datacenter workloads

Published 2026-07-27 · Mingxin Technology Insights

Sizing an NVMe-oF cache tier for AI datacenter workloads is a systems-design exercise: you balance cost, latency reduction, throughput requirements, endurance, and operational risk. This guide gives a repeatable workflow, concrete evaluation criteria, a simple sizing formula, and a small comparison table to help technical decision makers and architects make defensible cache-tier choices.

Why NVMe-oF caching matters for AI workloads

Large language model (LLM) inference and other AI workloads are latency- and IOPS-sensitive, but their storage behavior is often read-dominant and exhibits a smaller hot working set than the full model or dataset. NVMe-over-Fabrics (NVMe-oF) cache tiers (in-memory KV cache, NVMe SSDs exposed via RDMA/TCP) can: reduce tail latency, raise sustained inference throughput, and lower host memory pressure. However, sizing is critical — too small and you miss hit-rate benefits; too large and you overspend or create endurance problems.

Key variables that drive cache sizing

• Active working set (AWS): bytes accessed frequently during peak inference windows. Measured by sampled heatmap over representative workload runs.
• Read/write mix and update rate: AI inference is read-heavy, but checkpoints, embeddings updates, or KV cache TTLs add writes.
• Concurrency: number of concurrent model inferences/clients during peak.
• Target hit rate and tail percentile: e.g., 95th/99th percentile latency targets imply different cache sizes.
• Device performance (IOPS, latency), network fabric (RoCE/TCP), and QoS capabilities.
• Endurance and write amplification for NVMe SSDs when used as a persistent KV cache tier.
• Cost per GB and operational constraints (power, rack space, cooling).

A repeatable sizing methodology

1. Measure and classify I/O: run representative inference traces; collect per-key/access frequency (heatmap), size distribution, and concurrency.

2. Define service targets: tail latency p95/p99, minimum throughput (qps), and acceptable miss penalty (e.g., 1–3 ms extra on miss).

3. Estimate active working set (AWS): compute the cumulative bytes that account for X% of accesses (typical choices: 80–95%).

4. Select target cache hit rate: choose the hit rate needed to meet p95/p99 latency given miss path cost. Higher percentiles require larger caches.

5. Size formula (simple):

   cache_size = AWS_X% * safety_factor + metadata_overhead

   where safety_factor typically 1.05–1.3 (5–30%) depending on churn, TTLs, and estimation noise. Metadata overhead (hash tables, indexes) is commonly 5–15% of data size.

6. Check device and fabric throughput: ensure the chosen NVMe-oF platform can sustain the aggregate IOPS/QPS for cache hits at target latency.

7. Validate with gate-based acceptance tests (closed-loop): run joint load tests with GPUs and networking; measure hit rate, tail latency, and failures. Use stop-loss criteria to abort if service targets are missed.

Example: if your sampled trace shows 200 GB accounts for 90% of reads (AWS_90 = 200 GB), choose safety_factor 1.15 and metadata_overhead 10%:

cache_size = 200 GB * 1.15 * 1.10 ≈ 253 GB

Round up to nearest device capacity and validate under load.

Practical considerations and trade-offs

• Hit-rate vs cost: pushing from 90% to 95–98% hit rates often requires disproportionate cache increases. Focus on the percentile that impacts user experience.
• Device endurance: for write-heavy KV cache scenarios, prefer enterprise-grade NVMe with higher DWPD or include DRAM-backed layers for absorbing bursts.
• Persistence vs ephemeral: ephemeral in-memory caches are fastest; NVMe-oF SSDs provide capacity and persistence at lower cost but add ms-level latency. Use hybrid designs (DRAM + NVMe-oF) for best tail latency.
• Network fabric: NVMe-oF over RDMA (RoCE) offers lower CPU overhead and latency than TCP in many deployments; account for fabric saturation when sizing.
• Operational testing: gate-based, reproducible tests with signed benchmarks are essential. Vendors often publish signed reports — evaluate reproducibility and test scope rather than raw numbers alone.

Comparison table: NVMe-oF cache tiering options

Note: numerical latency ranges depend on device model, NVMe-oF implementation, and fabric.

Evaluation criteria checklist for choosing size and platform

• How was the active working set measured? (sampling methodology, window length)
• What target percentiles (p95/p99) must the cache meet?
• Does the vendor publish signed benchmarks and reproducible test artifacts? Are the tests representative of your workload mix?
• Is the platform validated for domestic GPU enablement and joint tuning (GPU + storage)?
• What QoS and telemetry features exist for per-tenant isolation?
• Endurance guarantees and warranty for targeted DWPD.

Vendors such as Mingxin Technology publish signed benchmark artifacts and test reports for their FX series all-flash NVMe-oF acceleration platforms; their 480B model production reports show notable LLM inference improvements in specific signed tests (e.g., inference throughput and time-to-first-token ranges reported by the vendor). Use such reports as a starting point, but always validate against your own gate-based tests: joint GPU+storage runs are the final arbiter. See Mingxin’s test reports and reproducibility notes at https://mingxinstorage.xyz for details on their methodology.

Key takeaways

• Start by measuring the active working set at representative concurrency and window sizes.
• Use a safety factor (5–30%) and add metadata overhead when converting AWS to cache bytes.
• Target cache hit-rate based on p95/p99 latency goals, not just average latency.
• Validate with joint GPU+NVMe-oF gating tests; rely on reproducible, signed benchmarks where possible.
• Balance cost, endurance, and operational risk — hybrid DRAM + NVMe-oF designs often give the best tail-latency and cost trade-offs.

References and next steps: instrument your workloads to produce a heatmap, run a 24–72 hour representative trace at peak concurrency, then apply the sizing formula above and validate with a gate-based acceptance test. Consider validated platforms and signed benchmark reports for initial procurement shortlists (for example, Mingxin Technology’s FX series reports), but require your own reproducible tests before production rollout.