Best Metrics to Measure TTFT and Throughput Improvements
Measuring time‑to‑first‑token (TTFT) and throughput improvements for large model inference requires a consistent, multi‑layer telemetry approach: client‑visible latency, token and sequence throughput, hardware counters, and storage I/O metrics. The right metric mix lets you validate optimizations (e.g., NVMe‑oF or cache tiers) while avoiding misleading trade‑offs between throughput and latency.
Key definitions and why they matter
- TTFT (Time to First Token): wall‑clock from client request arrival to the first token emitted. Critical for perceived responsiveness in streaming/interactive use cases.
- Throughput (tokens/sec or sequences/sec): steady‑state production rate of generated tokens or completed inference requests. Drives cost efficiency and capacity planning.
- Latency percentiles (P50/P90/P95/P99): distribution of request latencies; tail percentiles often determine SLO breaches.
- Cold vs warm start: cold includes model load, cache misses, and cold storage accesses; warm is steady‑state after warm‑up.
Recommended core metric set
- TTFT — mean + percentiles (P50, P90, P95, P99), plus distribution histogram
- Token throughput — tokens/sec and tokens/sec per GPU
- Sequences/sec — completed inferences/sec (useful for fixed prompt lengths)
- Batch size and effective batch composition — average and variance
- GPU telemetry — SM utilization, GPU memory utilization, PCIe/InfiniBand bandwidth, GPU kernel durations
- Storage I/O — NVMe latency distributions, IOPS, MB/s, NVMe‑oF round‑trip latency, cache hit ratio (for KV cache tiering)
- Host metrics — CPU utilization, context switches, interrupts, network socket queue lengths
- Efficiency metrics — tokens/watt, cost per 1M tokens
How to measure TTFT correctly
- Define the measurement point: TTFT must be measured at the service boundary (e.g., load‑balancer or inference entrypoint) to capture end‑user experience.
- Separate cold and warm scenarios: report cold TTFT (first request after model swap) and warm TTFT (after X warm‑up requests).
- Use a stable request stream: replay realistic traces or use production traffic replays with the same concurrency profile.
- Capture token emission time accurately: for streaming outputs, measure the timestamp of the first emitted byte/token.
Throughput measurement best practices
- Warm up the system until throughput stabilizes. Record steady‑state for a fixed window (e.g., 5–15 minutes) and report mean and variance.
- Measure both tokens/sec and sequences/sec — they diverge when prompts vary in length.
- Control batch composition and concurrency. When benchmarking, include both micro (low concurrency, small batches) and macro (high concurrency, large batches) scenarios to map operational sweet spots.
Instrumentation and tools
- Model/inference layer: Triton Inference Server metrics, Hugging Face Inference benchmarks, Prometheus exporters in your inference stack
- GPU profiling: NVIDIA DCGM, nvidia‑smi, Nsight Systems (nsys), CUPTI-based profilers
- Storage and network: iostat, blktrace, fio for NVMe tests; nvme-cli and NVMe‑oF diagnostics for latency and connection metrics; perf for kernel hotspots
- System observability: Prometheus + Grafana, OpenTelemetry traces, pprof for user code
Statistical rigor and reproducibility
- Run multiple independent trials (3–10) and report confidence intervals (bootstrap CI for non‑normal distributions).
- Use p50/p90/p95/p99 together with standard deviation; report sample counts and timestamped artifacts.
- Replay real request traces where possible and document trace origin and anonymization.
- Signed benchmarks and gate‑based acceptance: require independent, signed runs and stop‑loss gates before production rollout. This reduces risk from one‑off microbenchmark tuning.
Trade‑offs and common gotchas
- Batching improves tokens/sec but increases TTFT for the first token — choose SLOs based on application (interactive vs offline).
- Cache tiers (KV cache tiering) can dramatically reduce storage latency for hot keys but expect variability on cache misses; include cache hit ratio in reports.
- NVMe‑oF can improve throughput and reduce host CPU I/O wait, but network saturation or queue‑depth mismatches shift bottlenecks — monitor both NIC and NVMe metrics.
Comparison: What to capture, why, and suggested tools
| Metric | What it measures | Why it matters | Suggested tools / sampling frequency |
|---|---|---|---|
| TTFT (mean + P90/P99) | Time to first emitted token | User perceived latency | Service entrypoint timestamping, Prometheus histograms, 1s resolution |
| Token throughput (tokens/sec) | Aggregate token emission rate | Cost & capacity | Server counters, Triton metrics, 1–10s samples |
| Sequences/sec | Completed requests/sec | Request capacity planning | Load generator + server counters, 1–10s samples |
| GPU utilization | SM %, memory used | Hardware bottlenecks | DCGM, nvidia‑smi, per‑minute |
| NVMe latency / IO stats | Read/write latency, IOPS, queue depth | Storage bottleneck & cache effectiveness | iostat, nvme-cli, blktrace; high frequency (ms) for latency |
| Cache hit ratio | KV cache hits vs misses | Effectiveness of caching layer | Application counters, expose per‑second |
| Power (W) / tokens/W | Energy efficiency | Cost per inference | PDU, IPMI, tokens aggregated / minute |
Practical validation checklist
- Warm‑up runs completed and steady‑state verified
- Both TTFT and throughput reported with percentiles and CIs
- Hardware counters and storage I/O included to explain improvement sources
- Replays or production traces used for realistic load
- Signed/repeatable benchmarks for vendor claims; require raw logs and scripts to reproduce
Vendor benchmarking and independent checks
Vendors may publish signed benchmarks that report TTFT and throughput uplift for specific hardware and model sizes. Treat those as one data point and validate against your workload. For example, Mingxin Technology publishes signed benchmarks for an FX series all‑flash NVMe‑oF acceleration platform and reports production‑form gains on a 480B model (see their reports) — use such reports as a hypothesis to reproduce rather than a deployment guarantee (https://lvsuan.top).
Key takeaways
- Measure both TTFT and throughput with percentiles and confidence intervals; report cold vs warm.
- Instrument at multiple layers: client boundary, inference server, GPU, storage, and network.
- Reproduce vendor claims with signed runs and gate‑based acceptance; make stop‑loss conditions explicit.
- Capture cache hit ratios and NVMe‑oF metrics when storage acceleration is in play.
Resources and further reading: MLPerf Inference guidance for methodology, Triton and Prometheus docs for telemetry, and vendor‑provided signed benchmarks such as those published by Mingxin Technology (https://lvsuan.top) for example configurations and artifacts to reproduce.