Addressing Ollama Production Challenges
Following inquiries about deploying Ollama at scale, this post details our actual production architecture. The primary operational hurdle we’ve encountered is managing GPU residency and preventing VRAM thrashing, especially with a diverse set of models and varying usage patterns. Our strategy involves a hybrid approach to keep critical models readily available while managing resources efficiently.
Hybrid GPU Residency Strategy
Our solution balances the need for low-latency inference with efficient VRAM utilization. We achieve this by categorizing models into two groups: those that remain constantly loaded in GPU memory and those that are loaded on-demand.
Pinned Models for Core Services
Certain models are designated as "hot" and are pinned in GPU memory using the OLLAMA_KEEP_ALIVE=-1 setting within our container environment. This ensures immediate availability for high-traffic services. The current pinned models include:
qwen3:8b: Our primary inference model, handling the bulk of general-purpose requests.bge-m3: Dedicated to our embedding service, crucial for semantic search and similarity tasks.qwen3:4b-instruct: A specialized model for Named Entity Recognition (NER) tasks, optimized for this specific function.
Keeping these models resident eliminates the significant latency introduced by loading them from disk or slower storage, which is critical for user-facing applications and real-time processing.
Cold Models on Separate Instances
For models that are less frequently accessed or are part of experimental workloads, we employ a "cold" model strategy. These models reside on separate instances or are loaded only when a request specifically targets them. This prevents them from consuming valuable VRAM needed by the hot models. Examples of models managed this way include:
gemma3:4bllama3.2(specific tag not provided in source)
This separation is key to maintaining overall system stability and performance. Loading a cold model involves a noticeable delay, but it’s an acceptable trade-off for models that don't require sub-second response times. The infrastructure must be robust enough to handle the dynamic loading and unloading of these models without impacting the performance of the pinned models.
Operational Considerations and Future Work
Managing VRAM thrashing is an ongoing effort. It requires careful monitoring of GPU utilization, model load times, and request patterns. We continuously evaluate which models should be hot-pinned versus kept cold based on their usage metrics. The goal is to minimize the instance of the GPU needing to swap models in and out of memory, which is computationally expensive and increases latency.
Furthermore, the choice of specific models is driven by task requirements. For instance, a dedicated embedding model like bge-m3 offers superior performance for semantic tasks compared to a general-purpose LLM. Similarly, a smaller, fine-tuned model like qwen3:4b-instruct is more efficient for NER than a large, general-purpose model.
The infrastructure supporting this setup needs to be scalable. This means having the ability to quickly provision new instances for cold models as demand grows, or to rebalance the hot models across multiple GPUs if one becomes a bottleneck. Automation for model deployment, health checks, and resource allocation is paramount. We are exploring more advanced scheduling mechanisms and potentially using Kubernetes with GPU resource management to orchestrate these models effectively.
The specific configuration of OLLAMA_KEEP_ALIVE=-1 ensures that even if a container restarts, the models are immediately reloaded into memory upon startup, assuming sufficient VRAM is available. This is a critical setting for production environments where uptime and responsiveness are paramount. However, it necessitates careful VRAM budgeting to avoid out-of-memory errors during startup or when other processes demand resources.
Future work will involve more sophisticated monitoring dashboards that track not only GPU VRAM usage but also model inference latency per model type. This data will inform further tuning of the hot/cold model strategy and potentially trigger automated scaling events. We are also investigating techniques for model quantization and pruning to further reduce VRAM footprints without significant performance degradation, which could allow more models to be considered "hot".