Tokens Per Second: Measuring LLM Generation Speed
What tokens per second (TPS) measures, how it affects real-world AI applications, which models are fastest, and how to interpret speed vs. quality tradeoffs.
What Tokens Per Second Measures
Tokens per second (TPS) measures how quickly a model generates new tokens after it begins producing output. It's the 'streaming speed', how fast text appears after the initial delay (TTFT). A model generating 100 TPS produces approximately 75 words per second, which feels fast in a chat interface. Sampling strategy (Temperature, Top-P, Top-K) affects TPS because higher-temperature Sampling with nucleus filtering adds compute to each token selection step, though the effect is small compared to model size and hardware.
TPS is a function of model size, hardware, batch size, and infrastructure efficiency. Larger models are generally slower; specialized inference hardware (Groq LPU, Cerebras WSE) can dramatically outperform standard GPU inference. TPS is typically measured per concurrent request, throughput metrics (tokens per second across all users) are different.
Typical Speeds Across Providers
Groq achieves 500-800 TPS on Llama 3.1 70B using its LPU hardware, far faster than any GPU-based inference. Cerebras offers similar performance. Among standard GPU providers: Gemini 2.5 Flash typically achieves 100-150 TPS; Claude Haiku 4.6 achieves 80-120 TPS; GPT-4o achieves 40-80 TPS; frontier reasoning models (o3, Claude Opus) often fall below 40 TPS.
These numbers vary significantly based on load, region, prompt length, and current infrastructure. What matters for applications is the 95th-percentile TPS under realistic load, not peak performance under ideal conditions. Building on top of a fast model during low traffic that slows dramatically under load creates poor user experiences.
When TPS Matters vs. When It Doesn't
TPS matters most for: voice AI (must match speech pace, ~150 words/minute), real-time chat (perceived naturalness improves with smoother streaming), code autocomplete (must respond before the developer continues typing), and interactive data analysis. For these applications, anything below 50-80 TPS will feel sluggish.
TPS matters less for: batch processing (where you care about total throughput, not per-request speed), long-form generation that users read later, document analysis where the full response is processed programmatically, and tasks where quality dramatically outweighs speed. For essay generation, a 20 TPS model and a 100 TPS model produce the same result, the user just waits longer.
Read next
Time to First Token (TTFT): The Most Important Latency Metric
Why time to first token defines perceived AI responsiveness, what drives TTFT differences between models and providers, and how to optimize for low-latency applications.
Fastest LLMs: Latency Benchmarks Compared
Time to first token and tokens per second compared across major AI providers. Which models are fast enough for real-time applications?
LLM Inference: How Models Generate Text
Inference is what happens when an LLM produces a response. Understanding it helps you optimize for speed, cost, and quality.