Ollama Performance: 5 Key Settings for Serious Local LLM Deployment
Many users install Ollama, expecting blazing-fast responses from their local large language models, only to hit a wall. Slowdowns, even on powerful hardware, are common. The culprit is rarely the hardware itself; more often, it’s Ollama’s default configurations, which are not optimized for sustained, heavy usage. Tuning specific settings can unlock significant performance gains.
Flash Attention: Speeding Up Computation
Flash Attention is a technique that significantly accelerates the attention computation within LLMs. It achieves this by intelligently managing data flow, prioritizing the GPU’s cache over slower main memory. A common misunderstanding is that Flash Attention is enabled by default. While Ollama attempts to enable it automatically when system hardware supports it, this auto-detection isn't foolproof. Forcing it on via an environment variable guarantees its activation, especially in scenarios where auto-detection might miss a supported configuration or when explicit control is desired. The benefits are tangible: faster inference speeds and improved output quality, particularly for models handling long contexts. This setting essentially makes the LLM’s “working memory” much more efficient.
Context Length: Balancing Detail and Performance
The context length parameter determines how much previous conversation or input the model can consider when generating its next output. A larger context length allows for more nuanced and coherent responses over extended interactions, but it also demands more VRAM and computational resources. Ollama allows explicit control over this via the --context flag. Setting it too high can lead to out-of-memory errors or drastically reduced performance, while setting it too low can cause the model to “forget” earlier parts of the conversation, leading to repetitive or irrelevant outputs. Finding the right balance is crucial and depends heavily on the specific model being used and the available hardware. For instance, models like Llama 3 70B benefit greatly from longer contexts, but require substantial VRAM. Experimentation is key here; start with a reasonable default (often 2048 or 4096 tokens) and adjust based on your model’s capabilities and your hardware’s VRAM capacity.
GPU Layers: Optimizing VRAM Usage
When running LLMs on GPUs, a critical decision is how many of the model’s layers to offload to the GPU. Ollama’s --num-gpu-layers setting controls this. The GPU offers much faster computation than the CPU, so offloading as many layers as possible to the GPU is generally beneficial for speed. However, each layer consumes VRAM. If you try to offload more layers than your GPU can accommodate, Ollama will either fail to start or resort to CPU fallback for the excess layers, negating the performance benefits and potentially causing instability. The optimal number of layers is a direct function of your GPU’s VRAM. Smaller models might fit entirely onto modest GPUs, while larger models may only allow a partial offload. Monitoring VRAM usage during inference is essential. A common strategy is to start with a high number (e.g., 99, which attempts to offload all layers) and then decrease it incrementally if you encounter VRAM-related errors or performance degradation. The sweet spot is the highest number of layers that fits comfortably within your VRAM without causing excessive swapping or errors.
Batch Size: Improving Throughput for Multiple Requests
The batch size dictates how many input sequences (prompts or parts of prompts) are processed simultaneously by the model. A larger batch size can increase throughput – the number of requests processed per unit of time – especially when dealing with multiple parallel requests. This is because processing several sequences together can better utilize the parallel processing capabilities of the GPU. However, increasing the batch size also increases VRAM consumption. Ollama’s --batch-size parameter allows you to tune this. For single-user, interactive use, a smaller batch size might be sufficient and conserve VRAM. For serving multiple users or running batch inference jobs, a larger batch size can be more efficient. Similar to GPU layers, the optimal batch size is constrained by VRAM. If you observe high VRAM utilization and stable performance with a moderate batch size, you might be able to increase it to improve throughput. Conversely, if you experience VRAM errors or slowdowns, reducing the batch size is the first step to take.
Temperature: Controlling Randomness and Creativity
While not strictly a performance setting in terms of speed or resource utilization, the --temperature parameter is crucial for controlling the output quality and behavior of the LLM. Temperature influences the randomness of the model’s predictions. A lower temperature (closer to 0) makes the output more deterministic and focused, favoring the most probable words. This is ideal for tasks requiring factual accuracy or predictable responses, like summarization or question answering. A higher temperature (closer to 1 or above) increases randomness, allowing the model to explore less probable word choices and generate more creative, diverse, or surprising outputs. This is useful for brainstorming, creative writing, or generating varied responses. The default temperature in Ollama is often around 1.0, which can lead to highly creative but sometimes nonsensical outputs. For serious applications, especially those requiring reliability and consistency, tuning the temperature down is essential. Values between 0.2 and 0.7 are common starting points for more controlled generation. If your model’s responses feel too generic or too wild, adjusting the temperature is the primary lever to pull.
Conclusion: Beyond Defaults
Running LLMs locally with Ollama offers incredible flexibility, but relying on default settings is a disservice to both your hardware and your expectations. Flash Attention, context length, GPU layer offloading, batch size, and temperature are not mere knobs to twiddle; they are critical parameters that profoundly impact performance, VRAM usage, and output quality. By understanding and tuning these five settings, you can move beyond the initial performance bottlenecks and unlock the true potential of local LLMs for your specific use cases.