The Illusion of Speed: Why tok/s Varies Wildly
When you interact with a local Large Language Model (LLM), the most immediate feedback on performance is the time it takes to generate a response. This perceived speed is universally quantified as tokens per second (tok/s), a metric that appears in every benchmark and hardware review. Yet, this single number is deeply misleading. The same LLM can report vastly different tok/s figures—ranging from tens to hundreds or even thousands—depending on the testing methodology. This variability stems from a fundamental misunderstanding of what tok/s truly represents and the conditions under which it is measured.
At its core, tok/s measures how many discrete pieces of text (tokens) a model can generate within one second. Tokens are the fundamental units of text processing for LLMs, often corresponding to words or parts of words. A higher tok/s generally indicates a faster model. However, the context in which this generation occurs dramatically alters the outcome. The critical factor often omitted in simple benchmarks is concurrency—how many requests the system is handling simultaneously.
Understanding Concurrency and Its Impact
A benchmark that tests a single, isolated request will show the theoretical maximum throughput of the model under ideal, unloaded conditions. This is often referred to as the 'peak' or 'single-user' performance. In this scenario, the LLM can dedicate all available computational resources—CPU, GPU, and memory—to processing that one request. The model is essentially running at full throttle, pushing out tokens as fast as its architecture and hardware allow.
However, real-world applications rarely operate with a single user. A common use case for local LLMs might involve a chatbot interface, a coding assistant, or even a backend service handling multiple user sessions. When concurrency increases, the system must divide its resources among multiple requests. Each request then competes for processing time and memory. This competition inevitably slows down the generation rate for each individual request. A model that achieves 45 tok/s in a single-user test might drop to 15 tok/s or lower when handling 5 concurrent requests, and even further as concurrency climbs.
The type of tokens generated also matters. LLMs typically operate in two phases: prompt processing and token generation. Prompt processing involves the model understanding the input query, which is usually a one-time cost per request. Token generation is the iterative process of producing the output, one token at a time. Benchmarks that focus solely on the generation phase, especially after an optimized prompt processing stage, can inflate the perceived performance. A more realistic benchmark needs to account for the entire request lifecycle, including prompt processing, especially for shorter outputs where prompt processing can be a significant portion of the total time.
The Prompt Processing Bottleneck
The initial phase of processing the user's prompt is often overlooked in simple tok/s benchmarks. This phase involves the LLM encoding the input text into a format it can understand and perform calculations on. For longer prompts, this can be computationally intensive and time-consuming. The time taken here is a fixed cost per request, regardless of how many tokens are eventually generated. Consequently, for short outputs, the prompt processing time can dominate the total response time, making the token generation speed less relevant to the user's experience.
Consider a scenario where prompt processing takes 2 seconds, and the model then generates 20 tokens at a rate of 20 tok/s (meaning it takes 1 second to generate those 20 tokens). The total time for the response is 3 seconds. In this case, the effective tok/s, averaged over the entire response, is 20 tokens / 3 seconds = 6.67 tok/s. A benchmark that only measures the generation phase would report 20 tok/s, which is three times faster than the user actually experiences.
This highlights why simple tok/s numbers are insufficient. They fail to capture the user experience of waiting for the first token to appear and the overall latency of a complete response. For interactive applications, the time-to-first-token (TTFT) is often more critical than the peak generation speed. A fast TTFT means the user sees output quickly, even if the subsequent generation is slower. This can make the application feel much more responsive.
What Constitutes a Meaningful Benchmark?
To gain a true understanding of local LLM performance, benchmarks must move beyond single-user, peak-generation metrics. A more comprehensive approach would involve:
- Concurrent Load Testing: Simulating realistic user loads with multiple simultaneous requests to measure performance degradation under pressure. This reveals how the system scales.
- Time-to-First-Token (TTFT): Measuring the latency from request submission to the first token being generated. This is crucial for interactive applications.
- End-to-End Latency: Measuring the total time from request submission to the complete response being generated, including prompt processing.
- Throughput Under Load: Measuring the total number of tokens generated per second across all concurrent requests. This provides a measure of system capacity.
- Varying Prompt and Output Lengths: Testing with a range of prompt sizes and desired output lengths to understand performance across different use cases.
The surprising detail here is not that tok/s can vary, but how dramatically it can change. A single-digit increase in concurrency can sometimes halve the reported tok/s. Users and developers need to understand these nuances to accurately assess hardware capabilities and choose models that best fit their specific deployment scenarios. What remains unaddressed by most benchmarks is the trade-off between model size, quality, and achievable concurrency for a given hardware configuration.
Moving Beyond Raw Numbers
As local LLMs become more accessible and integrated into daily workflows, the importance of accurate performance metrics grows. Relying solely on raw tok/s figures is akin to judging a car's performance solely on its top speed without considering acceleration, braking, or how many passengers it can comfortably carry. For developers building applications, understanding the interplay between concurrency, prompt processing, and generation speed is paramount. For users evaluating hardware, looking at benchmarks that simulate real-world usage patterns—including concurrent requests and TTFT—will provide a far more accurate picture of what to expect.
The quest for faster LLMs is ongoing, but so too must be the quest for more meaningful ways to measure that speed. Without a deeper understanding of the factors influencing tok/s, users will continue to be misled by numbers that don't reflect their actual experience.