The Problem: Scaling LLM APIs with Multiple Tenants

As applications scale to serve multiple distinct customers (tenants), managing shared resources like API keys and rate limits for third-party services becomes a critical challenge. CitizenApp, a company serving 15 tenants, faced this exact issue with their integration of Anthropic's Claude LLM. A single, global Claude API key meant that one tenant's heavy usage could inadvertently throttle or exhaust the quota for all other tenants. This not only impacted user experience but also created an opaque billing and usage monitoring nightmare. Debugging issues related to API access or rate limiting became a complex task, akin to untangling a ball of yarn where each strand represented a different tenant's request.

The naive approach of using middleware to parse tenant IDs, look up keys, and manage state in request contexts or global variables quickly devolved into what the author describes as "middleware spaghetti." This approach is brittle, difficult to test, and prone to race conditions in concurrent environments. The core problem is the lack of a clean, isolated way to provide tenant-specific configurations to the Anthropic Claude client for each incoming request.

The Solution: FastAPI's Dependency Injection System

FastAPI's powerful dependency injection (DI) system offers an elegant solution. Instead of relying on global state or complex middleware logic, the application can dynamically provide tenant-specific instances of the Anthropic Claude client. This means each request, based on its originating tenant, can be automatically handed a Claude client configured with that tenant's unique API key and associated rate-limiting parameters.

The core idea is to define a dependency that resolves to a configured Claude client. This dependency function can inspect the incoming request to determine the tenant context. Once the tenant is identified, the dependency can fetch the correct API key and any tenant-specific rate-limiting configurations. This tenant-specific client is then injected directly into the route handler that needs to interact with Claude, ensuring isolation and correctness.

Consider a FastAPI route like this:

from fastapi import FastAPI, Depends
from anthropic import Anthropic

app = FastAPI()

# Assume get_tenant_claude_client is a dependency function
def get_tenant_claude_client(tenant_id: str = Depends(get_tenant_id)) -> Anthropic:
    # Logic to fetch API key and rate limits based on tenant_id
    api_key = get_api_key_for_tenant(tenant_id)
    rate_limit_config = get_rate_limit_config_for_tenant(tenant_id)
    
    # Initialize and configure the client
    client = Anthropic(api_key=api_key)
    # Potentially apply rate limiting logic here or via a wrapper
    return client

@app.post("/chat/{tenant_id}")
def chat_with_claude(tenant_id: str, client: Anthropic = Depends(get_tenant_claude_client)):
    # Use the tenant-specific client
    response = client.messages.create(
        model="claude-3-opus-20240229",
        max_tokens=1000,
        messages=[
            {
                "role": "user",
                "content": "Hello, Claude"
            }
        ]
    )
    return {"response": response.content}

Here, get_tenant_id would be another dependency responsible for extracting the tenant identifier from the request (e.g., from a subdomain, JWT token, or header). The get_tenant_claude_client dependency then uses this tenant_id to fetch the appropriate API key and configuration, returning a fully initialized Anthropic client instance. This client is then automatically passed to the chat_with_claude endpoint.

Implementing Tenant-Specific Rate Limiting

Beyond just API keys, isolating rate limits is crucial for fair usage and cost control. The same dependency injection pattern can be extended to manage rate-limiting buckets per tenant. Libraries like slowapi or custom middleware integrated within the dependency resolution can enforce these limits.

Imagine the get_tenant_claude_client dependency also returns a rate-limited client wrapper. This wrapper would intercept outgoing requests, check against the tenant's allocated rate limits (e.g., requests per minute, tokens per hour), and either allow the request, delay it, or return an error if limits are exceeded. This keeps the rate-limiting logic tightly coupled with the client instance it governs, rather than scattered across the application.

The benefit is a clear separation of concerns. The core business logic in the route handlers remains clean, focused on orchestrating LLM interactions. The infrastructure concerns—authentication (API keys) and resource management (rate limits)—are handled declaratively through FastAPI's DI system. This makes the system far more maintainable, testable, and scalable.

Benefits of This Approach

This DI-centric approach offers several key advantages:

  • Isolation: Each tenant operates with its own isolated API key and rate-limiting context. One tenant's actions do not affect others.
  • Scalability: As the number of tenants grows, the system scales linearly without introducing performance bottlenecks or complex state management.
  • Maintainability: The code is cleaner, more modular, and easier to understand. Debugging is simplified as tenant-specific configurations are readily available.
  • Testability: Dependencies can be easily mocked, allowing for robust unit and integration tests of the LLM interaction logic without needing actual API keys or complex setup.
  • Flexibility: Tenant-specific configurations (e.g., different Claude models, custom prompt templates) can be managed and injected dynamically.

By leveraging FastAPI's dependency injection, developers can build robust, multi-tenant applications that integrate with external LLM APIs like Anthropic Claude in a clean, scalable, and maintainable manner. This pattern moves away from global state and complex middleware towards a more declarative and robust system design.