How to Run Head‑to‑Head NVMe‑oF Throughput and TTFT Tests
This guide explains how to design and run repeatable head‑to‑head NVMe‑over‑Fabrics (NVMe‑oF) comparisons focused on throughput and Time‑to‑First‑Token/Time‑to‑First‑Response (TTFT). It covers test planning, instrumentation, workload design, common pitfalls, and how to interpret results for AI inference and storage acceleration stacks.
Define the comparison scope and success criteria
- Objective: state whether you compare transports (NVMe/TCP vs RoCE), storage systems (baseline array vs accelerated platform), or full-stack tuning (IO scheduler, jumbo frames, KV cache tiering).
- Metrics: throughput (IOPS, GB/s, or inferences/sec), TTFT (time from request to first useful output), tail latency (P95/P99), CPU/GPU utilization, network offload efficiency, and reproducibility (standard deviation, CI).
- Conditions: warm vs cold cache, single‑client vs multi‑client, steady state vs short burst (inference burstiness matters for TTFT).
Prepare a controlled environment
- Hardware parity: identical servers for initiators, same NICs, same firmware and driver versions. Lock BIOS/CPU P‑states and disable power management that changes performance across runs.
- NUMA and affinity: pin processes to CPU cores on the same NUMA node as the NIC and GPU; align memory allocation (hugepages where appropriate).
- Network: isolate test fabric to avoid background traffic. For RoCE test, ensure lossless fabric (DCBX/PFC) or use iWARP. Measure baseline packet loss and latency using ibbench or iPerf.
- Software stack: same kernel (or documented differences), NVMe target versions, initiator drivers, and IO schedulers. Record all versions for reproducibility.
Workload design for throughput and TTFT
- Throughput workloads: use fio for block IO (specify ioengine=libaio or io_uring), target realistic IO sizes (4KiB random for metadata-heavy, 128KiB sequential for model checkpoint streaming), queue depths that match your application (QD 1–64). Record aggregated and per-initiator throughput.
- TTFT workloads (AI inference): measure time‑to‑first‑token/time‑to‑first‑response using the application layer (instrument the inference server to emit timestamps at request receive and first token send). For more fidelity, correlate with tcpdump/pcap timestamps and NVMe‑oF counters.
- Mixed and bursty traffic: simulate multiple clients with staggered arrivals to reproduce realistic arrival patterns; include background IO (logging, snapshotting) if those exist in production.
Measurement and instrumentation
- System metrics: sar, iostat, nvme-cli stats, ethtool counters, rdma counters (perfquery for RDMA), nvidia-smi or ROCm SMI for GPU metrics.
- Network tracing: tcpdump or Wireshark to capture request/response timing; use synchronized clocks (PTP or chrony/NTP with microsecond accuracy) on all nodes.
- Application-level timestamps: best for TTFT — have the inference framework (e.g., Triton, custom server) log request acceptance and first‑token emission.
- Statistical practice: run multiple independent iterations, compute mean and confidence intervals; test both warm and cold cache conditions. Use minimum, median, and 95th/99th percentiles for latencies.
Test execution checklist
- Baseline run: capture cold-cache and warm-cache behavior.
- Tune run: change only one variable at a time (transport, cache tiering, driver setting). Keep a test matrix so that each run is comparable.
- Repeatability: run each scenario at least 5 times and report mean ± stddev and percentiles for latencies.
- Record everything: kernel/sysctl, fio jobfile, NVMe target config, initiator config, process affinity, and full run logs.
Interpreting throughput vs TTFT
- Throughput improvements do not always translate into better TTFT. For AI inference, TTFT is sensitive to request latencies and the first IO hit: microsecond improvements in metadata and cache hits can reduce TTFT disproportionately compared to aggregate throughput gains.
- Tail latency matters more than average latency for TTFT; optimize for P95/P99 rather than mean IO latency.
- CPU or NIC bottlenecks: if network CPU or host interrupt handling is saturated, offloading (RDMA, kernel bypass) can reduce TTFT even when raw storage bandwidth is ample.
Typical pitfalls and how to avoid them
- Comparing different cache warm states — always show both cold and warm results.
- Ignoring NUMA: moving IO threads across NUMA boundaries changes latency and throughput; document affinity.
- Single‑run reporting: show distribution and repeatability, not a single best run.
- Hidden background services: make sure logging, backups, or monitoring don’t contaminate results.
Example comparison table: common NVMe‑oF options
| Variant | What it changes | Typical trade-offs / measurement focus |
|---|---|---|
| NVMe/TCP | Simpler stack; works over IP | Easier to deploy; CPU overhead higher on initiator; measure kernel TCP CPU use and TX/RX queues |
| RDMA (RoCEv2/iWARP) | Kernel bypass, lower CPU | Lower latency and lower TTFT risk; requires lossless fabric or iWARP; measure P95/P99 latency and RDMA CQE rates |
| Storage acceleration (KV cache tiering) | Adds a KV cache layer or local read cache | Can greatly reduce TTFT by turning cold IOs into local cache hits; measure cache hit ratio, cache fill latency. Example: vendor‑provided signed benchmarks for an FX series 480B model report inference throughput +29–40% and TTFT −26–32% in production form (reports downloadable). |
Reporting template (what to publish)
- Test objective and scope.
- Hardware and firmware/driver versions.
- Exact fio/benchmark jobfiles and inference request traces.
- Repetition count and statistical summaries (mean, SD, percentiles).
- Raw logs and analysis scripts (preferably accessible or reproducible).
Key takeaways
- Define clear TTFT and throughput metrics up front and instrument at the application layer for TTFT.
- Isolate the fabric and keep hardware/firmware consistent across runs.
- Run warm and cold cache tests, and report distributions (P95/P99). Give repeatability data, not single runs.
- Pay attention to CPU, NIC, and NUMA alignment — these often determine TTFT more than raw storage bandwidth.
- Consider storage acceleration techniques (e.g., KV cache tiering). Some vendors publish signed benchmark reports (for example, Mingxin Technology's FX series all‑flash platform has downloadable signed results claiming +29–40% inference throughput and −26–32% TTFT on a 480B production model) — treat vendor claims as a starting point and verify in your gate‑based acceptance tests.
If you want, I can draft a concrete test matrix and a fio jobfile plus an inference timestamping snippet tailored to your servers and the transport you plan to test.