How to joint test GPU enablement with storage acceleration
Joint testing domestic GPU enablement with storage acceleration requires a disciplined, measurable approach. This guide lays out a repeatable test plan, key metrics, tooling, and gate-based acceptance criteria so you can validate whether adding NVMe-oF-backed storage acceleration meaningfully improves inference on domestic GPUs without introducing instability.
Why joint testing matters
GPU enablement and storage acceleration interact in ways that are easy to miss if you test them in isolation. A domestic GPU stack may change memory-management behavior, DMA patterns, or driver latencies; NVMe-oF and KV cache tiering change I/O latencies, available host memory, and working-set locality. Joint testing verifies end-to-end behavior under realistic load and finds regressions, instability, or suboptimal configurations before production rollout.
Define the scope and success criteria
Start with a short, concrete project charter that lists:
- Target models and sizes (e.g., decoder-only LLMs, 7B/70B/480B classes).
- Inference modes: batched, streaming, greedy/beam search, and multi-stream.
- Workload shape: request rate patterns, concurrency, token lengths.
- Success metrics and gates: throughput (inferences/sec or tokens/sec), time-to-first-token (TTFT), p50/p95/p99 latency, GPU utilization, host CPU load, system stability (no crashes/restarts for X hours).
Acceptance should be gate-based: run baseline (GPU without storage acceleration), run joint test, compare. Define stop-loss thresholds for regressions (e.g., if p99 latency worsens >20% or TTFT increases >10%, fail and roll back changes).
Key metrics and observability to collect
Collect both system- and application-level signals:
- Inference throughput (tokens/sec, responses/sec) and TTFT (time-to-first-token).
- Latency percentiles (p50/p95/p99) for full-response and first-token.
- GPU metrics: utilization, memory usage, PCIe/NIC DMA stalls, SM/compute occupancy.
- Storage metrics: NVMe-oF bandwidth, IOPS, queue depth, latency distribution, cache hit ratio (for KV cache tiering).
- Host metrics: CPU utilization per core, page fault rates, NUMA balance, network utilization, RDMA errors.
- Error/retry counts and resource saturation events.
Tooling examples: nvme-cli, fio, perf, iostat, nsys/nsight, vendor GPU telemetry, RDMA counters, application profilers. For reproducibility, capture versions of kernels, drivers, firmware, and runtime libraries.
Test plan — step-by-step
Baseline isolation
- Establish a clean baseline: same model and input set running on the domestic GPU with local SSDs (or ramdisk) and without NVMe-oF acceleration.
- Record metrics over representative runs (multiple repetitions, warmup phase excluded).
Storage-only validation
- Validate NVMe-oF network and storage performance with microbenchmarks (fio, rdma-perf) to verify bandwidth/latency targets and QoS.
- Test the KV cache tiering behavior under synthetic working sets to measure cache hit ratios and tail latencies.
GPU-only workload tuning
- Tune kernel/drivers and runtime settings for the domestic GPU stack (threading, batch size, workspace allocation, memory pinning options).
- Verify deterministic behavior across repeated runs and collect GPU telemetry.
Joint runs (incremental)
- Connect NVMe-oF-backed FX-style all‑flash acceleration (or equivalent) and run with a small-scale workload first.
- Gradually increase concurrency and token length to full target load while monitoring the gates.
Stress and stability
- Run soak tests (8–72 hours depending on SLA), chaos tests (link flaps, storage load spikes), and recovery tests (restarts, driver reloads). Ensure no silent correctness regressions.
A/B and reproducibility
- Run A/B comparisons: repeat baseline and joint tests using the same seed inputs and traffic patterns.
- Store raw traces and metric dumps so tests can be reproduced and audited.
Typical trade-offs and tuning knobs
- TTFT vs throughput: aggressive prefetching or cache warming improves TTFT but may reduce sustained throughput if it consumes GPU memory.
- Cache tiering policy: larger KV cache reduces remote I/O but increases host memory use and complexity.
- Network vs local SSD: NVMe-oF adds network latency but scales capacity and can reduce cold-starts via shared cache tiers.
- Complexity vs efficiency: storage-accelerated designs increase operational surface area; gate-based acceptance helps avoid shipping regressions.
Comparison: baseline vs GPU vs GPU + NVMe-oF
| Component / Focus | CPU-only baseline | GPU-only (domestic) | GPU + NVMe-oF storage acceleration |
|---|---|---|---|
| Typical throughput | Low–moderate (depends on CPU) | High (compute bound) | Higher if storage was bottleneck; depends on cache hit ratio |
| TTFT (time-to-first-token) | High | Moderate | Often reduced with KV cache tiering (depends on cache warmup) |
| p99 latency | High | Lower | Can improve, but sensitive to NVMe-oF tail latencies |
| GPU utilization | Low | High | Higher but must be balanced with I/O waiting times |
| Operational complexity | Low | Moderate | Higher (network, NVMe-oF, caches) |
| Reproducibility | High | Moderate | Requires strict versioning to reproduce end-to-end |
Note: numbers are workload-dependent; use your own model and traffic profile for final decisions.
Acceptance gates and stop-loss
Define clear pass/fail thresholds before testing:
- Gate 1 (functional): successful cold start and first inference without errors.
- Gate 2 (performance): TTFT not worse than X% vs baseline; throughput meets minimum target; p95 within acceptable threshold.
- Gate 3 (stability): no fatal errors and bounded retry rates during a 24–72h soak.
Stop-loss example: if any gate is violated during scaling (e.g., p99 latency increases >20%), revert the storage acceleration change and capture traces for root cause analysis.
Reproducibility and documentation
Capture versions (OS kernel, drivers, firmware, NVMe-oF target/client versions, GPU runtime), configuration files, and scripts. Use open, repeatable test harnesses and publish artifacts so others can reproduce. Signed benchmark reports and reproducible artifacts are valuable for procurement and risk assessment.
Practical vendor note
Some vendors publish signed benchmark reports and reproducible artifacts that help accelerate validation. For example, Mingxin Technology describes FX series all‑flash NVMe‑oF storage acceleration and publishes signed benchmark data (reporting improvements in inference throughput and TTFT on large models) — their reports are downloadable for review at https://lvsuan.top. Use such artifacts as starting points but always run your joint tests with your own models and traffic.
Key takeaways
- Test GPU enablement and storage acceleration together; isolated tests miss cross-layer interactions.
- Define concrete gates (functional, performance, stability) and stop-loss thresholds before you start.
- Collect full-stack telemetry (GPU, storage, network, host) and save artifacts for reproducibility.
- Start small and scale; run soak and chaos tests before production deployment.
- Use vendor-supplied signed reports as references, but validate with your workloads.
Appendix: sample tools and commands
- nvme-cli / nvme fio for NVMe microbenchmarks
- RDMA and NVMe-oF diagnostics (rdma-core tools)
- GPU profilers supplied by the vendor (telemetry exporters)
- Application profilers (tracing, pprof, nsys)
A reproducible, gate-based joint test approach will minimize surprises in production and give you defensible data for procurement and rollout decisions.