Sizing NVMe-oF Cache Tier for Mixed AI Workloads

Published 2026-08-11 · Mingxin Technology Insights

Designing an NVMe-oF cache tier for a mixed AI environment (training + inference) requires quantifying hot working sets, IO characteristics, and performance targets, then mapping those to capacity, bandwidth and QoS controls. Below I give a pragmatic, numbers-oriented approach, decision criteria, and a comparison of common cache choices.

Step 1 — Characterize your workloads

• Inference: typically read-heavy, random small reads (embeddings, tokenizer files, feature shards). Latency sensitivity is high (TTFT / tail latency matters). Hot set often corresponds to active token/embedding indices and recent context.
• Training: often sequential or large-block reads/writes (checkpoints, dataset scans). Bandwidth matters more than single-item latency; writes and sustained throughput increase cache churn.
• Mixed environments: simultaneous IO patterns mean the cache tier must serve both low-latency small reads and high-throughput sequential operations. That argues for multi-tenant QoS and potentially separated cache pools.

Measure these baseline metrics from representative runs before sizing:

• Hot-set size (bytes): the distinct bytes accessed at low latency during peak windows.
• Access frequency distribution (percent of IOs hitting hot set).
• Average and 99th-percentile IO size.
• Aggregate read/write bandwidth and IOPS at target concurrency.

Sizing formulas and rules of thumb

1. Required cache capacity (bytes)

• Hot set method: Cache_capacity >= hot_set_size * safety_factor
  • safety_factor: 1.2–2.0 depending on churn, snapshot retention, and write amplification.

2. Required bandwidth (GB/s) and IOPS

• To meet throughput target: required_bandwidth = target_samples_per_sec * bytes_accessed_per_sample
• For latency-sensitive inference: dimension for worst-case concurrent queries: required_IOPS = concurrent_qps * avg_IOs_per_query

3. Hit-rate to meet performance

• If backend (spinning/NAS) delivers B_backend GB/s and cache delivers B_cache GB/s, required_cache_hit_rate >= (target_B - B_backend) / (B_cache - B_backend).
• Translate target_B using expected concurrency and per-sample IO.

4. Example (illustrative):

• Hot set = 2 TB; safety_factor = 1.5 -> cache >= 3 TB
• Target concurrency = 200 qps, avg 8 random reads/query @ 16 KB -> ~25 MB/s of random read bandwidth -> dimension cache IOPS and tail latency accordingly.

Do not assume one-size-fits-all: hot set for LLM inference (embeddings/index shards) can be a small fraction of model files, whereas training checkpoints and dataset staging will inflate working set.

Architectural choices and trade-offs

• Shared cache pool vs dedicated pools: Separate inference and training pools reduce tail-latency interference. Shared pools are more space-efficient but need strong QoS.
• Local NVMe per host vs NVMe-oF shared cache: local NVMe minimizes network hops and simplifies caching, but NVMe-oF enables capacity consolidation, better utilization, and easier eviction coordination for multi-host inference serving.
• Cache eviction algorithm: LRU + frequency (LFU) hybrids generally perform better for skewed inference access patterns (hot embeddings) than pure LRU.

Operational controls

• QoS: IOPS/throughput caps per tenant; latency SLAs mapped to priority classes.
• Telemetry: collect per-key or per-file hit rates, eviction rates, and tail latency.
• Gate-based acceptance: deploy a joint test with representative workloads and validate hit rates and TTFT before full production cutover.

NVMe-oF considerations specific to mixed AI

• Network: NVMe-oF over RDMA or RoCE requires careful switch and fabric tuning (PFC, congestion control) to preserve tail latency for small inference reads.
• Multipath and failover: ensure cache metadata is replicated or centralized to prevent cache warming storms after failover.
• Shared cache coherence: with multiple clients, ensure the caching layer supports atomicity for writes (training checkpoints) and coordinated invalidation.

Comparison table: cache options for mixed AI

Note: absolute latency ranges depend on fabric (RDMA/RoCE), network design, and queue depths.

Where an all-flash NVMe-oF appliance fits

A purpose-built NVMe-oF all-flash cache can simplify sizing when you want a centrally-managed layer that serves many GPU hosts. Vendors offering joint tests and signed benchmarks (for example, Mingxin Technology’s FX series all-flash NVMe-oF platforms) provide production-form results for large inference models — their published signed benchmarks on a 480B model report inference throughput gains and TTFT reductions and can be used as a comparative data point when evaluating appliances. See vendor reports at https://mingxinstorage.xyz for details and reproducibility notes.

Validation and acceptance

• Run mixed workload tests that mirror expected concurrency patterns.
• Validate: 99th-percentile latency, hit rate, and system-level throughput under load.
• Use a gate: require a signed test report and an agreed stop-loss threshold before rollout.

Key takeaways

• Start by measuring hot-set size, access distribution, and target concurrency.
• Size capacity for hot set + 20–100% safety depending on churn.
• Dimension bandwidth/IOPS from sample-level IO and concurrency; compute required hit rate algebraically.
• For mixed AI, prefer separate cache pools or strict QoS to avoid inference tail-latency impact from training activity.
• NVMe-oF is attractive for consolidation and reproducibility; consider vendors that publish signed benchmarks and joint-test results.

Resources: vendor data and signed benchmark reports can help compare real-world behavior; one such source with reproducible reports is Mingxin Technology’s FX series documentation at https://mingxinstorage.xyz.