Self-Hosting AI Stack: 7 Months, 4 Bugs, 2,100 Commits

The Setup: A Hybrid AI Infrastructure

Running a team's AI stack on proprietary hardware is an ambitious undertaking. For OpenMake, this journey began with the creation of a repository on February 3, 2026, following an initial thought that setup would take a single weekend. Seven months later, the project boasts approximately 2,100 commits under an MIT license, a testament to the iterative and often challenging nature of building and maintaining custom AI infrastructure. The changelog, meticulously maintained, serves as a public record of the hurdles overcome, detailing four specific bugs that significantly impacted development.

The core of their setup relies on a dual-machine architecture. A Mac mini serves as the application host, running the API and web interfaces managed by PM2. Supporting services like PostgreSQL, Redis, and the sandboxed agent, MCP, and artifact processes are containerized within Docker. Adjacent to this, an NVIDIA DGX Spark GB10 is dedicated to high-performance AI tasks, hosting vLLM for model inference, BGE-m3 for embeddings, and FLUX for image generation. Connectivity between these machines is secured via a private Tailscale link. This configuration is not presented as a rigid requirement, but rather as a functional example, emphasizing that any OpenAI-compatible endpoint can theoretically be integrated. The default local model chosen is qwen3.8-27b, served through vLLM and proxied by LiteLLM, offering a substantial 262K context window. For flexibility, external providers such as OpenRouter, NVIDIA NIM, and Ollama are integrated, but their role is secondary to the self-hosted core.

Bug 1: The Unexpected Token Limit on Model Inference

The first significant bug encountered revolved around an unexpected limitation in the token handling during model inference. While the system was configured with a large 262K context window, the underlying vLLM setup, when combined with specific model configurations, began to truncate inputs prematurely. This wasn't a hard error but a silent data loss, where prompts exceeding a certain, undocumented threshold were effectively being cut short without any explicit warning or error message. The impact was subtle yet critical: response quality degraded noticeably for longer, more complex queries, leading to incomplete or irrelevant outputs that masked the true cause.

Debugging this required a deep dive into the vLLM source code and extensive experimentation with different prompt structures and batching strategies. The team discovered that certain internal buffer sizes and attention mechanisms within vLLM, particularly when handling extremely long sequences, were implicitly capping the effective context window, irrespective of the proxy configuration. This issue was eventually resolved by adjusting specific vLLM kernel parameters and implementing a more robust input validation layer within their custom API wrapper. The fix involved a combination of fine-tuning vLLM's internal memory management and adding a pre-processing step that would either chunk longer inputs or flag them for specialized handling, ensuring that the full context was always utilized.

Bug 2: Redis Connection Pooling Under High Load

The second major issue surfaced under sustained high-load conditions, specifically concerning Redis connection pooling. As the AI stack processed a high volume of concurrent requests, the Redis client instances, managed by a pooling mechanism, began exhibiting erratic behavior. Instead of gracefully managing connections, the pool would occasionally fail to release connections back to the pool, leading to a gradual depletion of available connections. This manifested as intermittent request failures and timeouts, particularly affecting the caching and session management layers that relied heavily on Redis.

The root cause was traced to a race condition within the connection management logic. Under heavy concurrent access, the atomic operations responsible for acquiring and releasing connections were not always behaving as expected, particularly when dealing with connection errors or timeouts. When a connection would error out, the release mechanism might not trigger correctly, leaving the connection in a stale state and unavailable for reuse. The solution involved implementing a more resilient connection validation strategy within the pooling library. This included periodic health checks for pooled connections and a more aggressive cleanup mechanism that would forcibly remove and re-establish connections that showed signs of being unresponsive or stale. Additionally, they introduced jitter to the connection acquisition process to mitigate the impact of thundering herd problems.

This problem is akin to a busy restaurant where the waiters forget to clear tables. Eventually, there are no tables left to seat new customers, even though the restaurant isn't full, just poorly managed.