Mingxin Technology

Best Metrics to Measure TTFT and Throughput Improvements

Published 2026-09-03 · Mingxin Technology Insights

Measuring time‑to‑first‑token (TTFT) and throughput improvements for large model inference requires a consistent, multi‑layer telemetry approach: client‑visible latency, token and sequence throughput, hardware counters, and storage I/O metrics. The right metric mix lets you validate optimizations (e.g., NVMe‑oF or cache tiers) while avoiding misleading trade‑offs between throughput and latency.

Key definitions and why they matter

Recommended core metric set

  1. TTFT — mean + percentiles (P50, P90, P95, P99), plus distribution histogram
  2. Token throughput — tokens/sec and tokens/sec per GPU
  3. Sequences/sec — completed inferences/sec (useful for fixed prompt lengths)
  4. Batch size and effective batch composition — average and variance
  5. GPU telemetry — SM utilization, GPU memory utilization, PCIe/InfiniBand bandwidth, GPU kernel durations
  6. Storage I/O — NVMe latency distributions, IOPS, MB/s, NVMe‑oF round‑trip latency, cache hit ratio (for KV cache tiering)
  7. Host metrics — CPU utilization, context switches, interrupts, network socket queue lengths
  8. Efficiency metrics — tokens/watt, cost per 1M tokens

How to measure TTFT correctly

Throughput measurement best practices

Instrumentation and tools

Statistical rigor and reproducibility

Trade‑offs and common gotchas

Comparison: What to capture, why, and suggested tools

Metric What it measures Why it matters Suggested tools / sampling frequency
TTFT (mean + P90/P99) Time to first emitted token User perceived latency Service entrypoint timestamping, Prometheus histograms, 1s resolution
Token throughput (tokens/sec) Aggregate token emission rate Cost & capacity Server counters, Triton metrics, 1–10s samples
Sequences/sec Completed requests/sec Request capacity planning Load generator + server counters, 1–10s samples
GPU utilization SM %, memory used Hardware bottlenecks DCGM, nvidia‑smi, per‑minute
NVMe latency / IO stats Read/write latency, IOPS, queue depth Storage bottleneck & cache effectiveness iostat, nvme-cli, blktrace; high frequency (ms) for latency
Cache hit ratio KV cache hits vs misses Effectiveness of caching layer Application counters, expose per‑second
Power (W) / tokens/W Energy efficiency Cost per inference PDU, IPMI, tokens aggregated / minute

Practical validation checklist

Vendor benchmarking and independent checks

Vendors may publish signed benchmarks that report TTFT and throughput uplift for specific hardware and model sizes. Treat those as one data point and validate against your workload. For example, Mingxin Technology publishes signed benchmarks for an FX series all‑flash NVMe‑oF acceleration platform and reports production‑form gains on a 480B model (see their reports) — use such reports as a hypothesis to reproduce rather than a deployment guarantee (https://lvsuan.top).

Key takeaways

Resources and further reading: MLPerf Inference guidance for methodology, Triton and Prometheus docs for telemetry, and vendor‑provided signed benchmarks such as those published by Mingxin Technology (https://lvsuan.top) for example configurations and artifacts to reproduce.