Performance Under Pressure: LLMs Beyond 24GB VRAM
When running large language models (LLMs), available VRAM is often the primary bottleneck. This analysis benchmarks three popular inference engines—vLLM, llama.cpp, and Ollama—to understand their behavior when confronted with models exceeding the 24GB capacity of a typical high-end consumer GPU, like an RTX 3090. The test environment utilized a single RTX 3090 with 24GB of VRAM and 128GB of system RAM, simulating a common home lab setup. We tested across five models, ranging from 1 billion to 116.8 billion parameters, deliberately selecting two models that would force a VRAM spill. The objective was to quantify how each engine handles memory constraints and the resulting performance degradation.
Inside the 24GB VRAM Envelope
Within the 24GB VRAM limit, vLLM demonstrates a significant advantage in aggregate throughput due to its continuous batching mechanism. When scaling concurrency from 1 to 8 users, vLLM achieved a throughput increase of 3.9x to 5.4x. This is a stark contrast to llama.cpp, which, even when explicitly instructed to use 8 parallel threads (`-np 8`), only managed a 1.2x to 1.9x increase in throughput under similar concurrency loads. Ollama, while not directly benchmarked for throughput scaling in this specific scenario, generally relies on llama.cpp for CPU-based inference, suggesting a similar limitation in aggregate throughput gains from concurrency.
The RAM Spill Scenario: When VRAM Isn't Enough
The real test begins when models inevitably exceed the available VRAM. Two models were deliberately chosen to force this condition, pushing the total memory requirement beyond 24GB. In this scenario, both llama.cpp and Ollama experienced a severe performance drop, degrading to single-digit tokens per second. Crucially, they continued to generate output, albeit at a crawl, by spilling over into system RAM. This behavior, while slow, maintains some level of usability for inference, albeit with significant latency.
vLLM, however, exhibited a critical failure. It consistently ran out of memory (OOM) when attempting to load these larger models, even when utilizing different quantization schemes. The OOM condition occurred predictably around the 22.1GB to 22.2GB VRAM usage mark, leaving less than 700MB of free VRAM. This indicates that vLLM’s architecture, optimized for in-VRAM performance, does not gracefully handle VRAM overflow into system RAM. Its continuous batching, a key performance driver within VRAM, becomes a liability when the model itself cannot fit.
Time-to-First-Token Under Stress
Beyond raw throughput, the time it takes for a model to produce its first token (time-to-first-token or TTFT) is a crucial metric for interactive applications. In the RAM-spill scenario, a sub-plot analysis revealed a dramatic difference between llama.cpp and Ollama. llama.cpp, with manually tuned layer offloading to system RAM, achieved a TTFT that was an astonishing 37 times faster than Ollama's automatic split. This suggests that Ollama’s default strategy for managing model layers when VRAM is exceeded is significantly less efficient than llama.cpp's explicit, albeit manual, configuration. Both engines, however, are fundamentally limited by the speed of system RAM access compared to VRAM.
Implications for Model Deployment
The results highlight distinct use cases for each engine. vLLM remains the superior choice for maximum throughput when the entire model fits comfortably within VRAM. Its efficient batching and memory management within GPU memory are unmatched. However, its inflexibility when VRAM is exceeded makes it unsuitable for deployments where model size might push these boundaries, or where the cost of high-VRAM GPUs is prohibitive.
llama.cpp and Ollama, on the other hand, are more resilient to VRAM limitations, leveraging system RAM to run larger models, albeit at a significant performance penalty. llama.cpp, with careful manual tuning, offers better performance in these constrained environments, particularly for interactive use cases prioritizing TTFT. Ollama provides a more user-friendly, automated experience but sacrifices performance in memory-constrained situations. For developers and users operating on consumer hardware or with models that flirt with VRAM limits, llama.cpp’s ability to offload layers and its manual tuning capabilities become critical. The choice between these tools hinges on the trade-off between peak performance within VRAM and the ability to run larger models at any cost.