The Hidden Markup in LLM Gateways

You wanted to experiment with multiple Large Language Models (LLMs) from different vendors. The sensible approach: point a single client at an LLM gateway, configure one API key, and move on. This gateway service speaks the OpenAI API protocol and routes requests to providers like Anthropic, Google, OpenAI, and xAI. It simplifies switching models, turning it into a simple configuration change rather than rewriting SDK integrations.

What few developers check, however, is the price the gateway charges for the exact same tokens that the underlying vendor would have sold you directly. This markup can significantly inflate your operational costs, especially for high-volume applications.

Diagram illustrating LLM gateway architecture and token flow

What You Actually Pay For

When you use an LLM gateway, you are essentially paying for three distinct benefits, each with a different value proposition:

Consolidated Billing and Key Management: The most straightforward benefit is having a single API key and a single invoice. This saves you the administrative overhead of signing up for multiple services and managing several billing portals. For small projects or initial experimentation, this convenience can be worthwhile.

Routing and Fallback Capabilities: Gateways offer resilience. If a specific LLM provider experiences an outage (returning a 529 error, for instance) or a region becomes unavailable, the gateway can automatically retry the request with another provider or in a different region. This feature has significant value for production systems where uptime is critical, but it offers little benefit for personal projects or weekend experiments where occasional downtime is acceptable.

The Token Price Markup: This is the least transparent and often the most costly aspect. The gateway adds its own price per token on top of the vendor's direct pricing. This markup isn't always clearly advertised and can be substantial, turning a simple convenience into a significant expense.

Quantifying the Markup with Python

To understand the actual cost, you need to compare the price you pay through the gateway with the price charged directly by the LLM vendor. This requires a small piece of code that can query both endpoints and calculate the difference. The following seventeen lines of Python demonstrate how to do this, assuming you have configured your gateway and the direct vendor endpoints, and have the necessary API keys.

import os
import requests

# --- Configuration ---
GATEWAY_API_URL = os.environ.get("GATEWAY_API_URL", "http://localhost:8000/v1")
GATEWAY_API_KEY = os.environ.get("GATEWAY_API_KEY", "sk-xxxxxxxx")
DIRECT_VENDOR_API_URL = os.environ.get("DIRECT_VENDOR_API_URL", "https://api.vendor.com/v1")
DIRECT_VENDOR_API_KEY = os.environ.get("DIRECT_VENDOR_API_KEY", "sk-yyyyyyyy")

MODEL_NAME = "gpt-3.5-turbo" # Example model
PROMPT_TEXT = "Tell me a short story about a robot."

# --- Function to call an API ---
def call_llm(api_url, api_key, model, prompt):
    headers = {
        "Authorization": f"Bearer {api_key}",
        "Content-Type": "application/json"
    }
    payload = {
        "model": model,
        "messages": [{"role": "user", "content": prompt}],
        "max_tokens": 50 # Small number to reduce cost for testing
    }
    try:
        response = requests.post(f"{api_url}/chat/completions", headers=headers, json=payload, timeout=30)
        response.raise_for_status() # Raise HTTPError for bad responses (4xx or 5xx)
        data = response.json()
        # Extract usage data - this might vary by API
        usage = data.get("usage", {})
        return usage.get("total_tokens", 0), usage.get("prompt_tokens", 0), usage.get("completion_tokens", 0)
    except requests.exceptions.RequestException as e:
        print(f"API call failed: {e}")
        return None, None, None

# --- Main execution ---
print("Testing gateway...")
gateway_total_tokens, gateway_prompt_tokens, gateway_completion_tokens = call_llm(GATEWAY_API_URL, GATEWAY_API_KEY, MODEL_NAME, PROMPT_TEXT)

if gateway_total_tokens is not None:
    print(f"Gateway Usage: Total={gateway_total_tokens}, Prompt={gateway_prompt_tokens}, Completion={gateway_completion_tokens}")

print("\nTesting direct vendor...")
direct_total_tokens, direct_prompt_tokens, direct_completion_tokens = call_llm(DIRECT_VENDOR_API_URL, DIRECT_VENDOR_API_KEY, MODEL_NAME, PROMPT_TEXT)

if direct_total_tokens is not None:
    print(f"Direct Vendor Usage: Total={direct_total_tokens}, Prompt={direct_prompt_tokens}, Completion={direct_completion_tokens}")

# --- Calculate Markup ---
if gateway_total_tokens is not None and direct_total_tokens is not None and direct_total_tokens > 0:
    # This is a simplified calculation. Actual pricing can be complex.
    # We assume prompt tokens are the primary driver of markup for this example.
    markup_percentage = ((gateway_prompt_tokens - direct_prompt_tokens) / direct_prompt_tokens) * 100
    print(f"\nEstimated Prompt Token Markup: {markup_percentage:.2f}%")
elif gateway_total_tokens is not None and direct_total_tokens is not None:
    print("\nCould not calculate markup: Direct vendor prompt tokens are zero or unavailable.")
else:
    print("\nCould not calculate markup due to API call failures.")

This script makes a single call to your configured gateway and a single call to the direct vendor API using the same prompt and model. It then compares the reported token usage. The core of the calculation lies in the difference between gateway_prompt_tokens and direct_prompt_tokens. If the gateway reports significantly more prompt tokens for the same input, it indicates that the gateway is either inefficiently processing the request or, more likely, inflating the token count to justify a higher price. The completion_tokens can also be a factor, but prompt tokens are often where gateways introduce their primary markup.

The Real Cost of Convenience

The surprise for many developers is not just that a markup exists, but the magnitude of it. For identical requests, some gateways can charge 2x, 3x, or even more than the direct vendor pricing. This is particularly true for prompt tokens, which are often the most expensive component of an LLM API call. The routing and fallback features, while valuable for production, do not inherently justify such a significant price increase on every single token processed.

Consider a scenario where a startup is processing millions of requests per day through a gateway. A 50% markup on prompt tokens, which might be the most expensive part of the call, could translate into tens or hundreds of thousands of dollars in unnecessary expenses each month. This is money that could be reinvested in R&D, marketing, or simply extending runway.

The convenience of a single endpoint and consolidated billing starts to look expensive when you realize you are paying a premium for features that might not be fully utilized, or worse, are being used to mask a substantial price hike. Developers must actively audit their LLM spending, just as they would for cloud infrastructure. Regularly comparing gateway costs against direct vendor pricing is essential for efficient resource management.

What nobody has addressed yet is what happens to the thousands of developers who built their applications assuming gateway pricing was competitive. Migrating away from a gateway can be a significant engineering effort, especially if the gateway offers complex routing logic or specific integrations that are not easily replicated.

Mitigating Gateway Costs

For developers and founders using LLM gateways, several strategies can help mitigate these hidden costs:

  • Direct Vendor Integration: For critical, high-volume applications, integrate directly with LLM vendors. This offers the best pricing and the most control. Use a simple abstraction layer in your code rather than a full gateway if you need some flexibility.
  • Cost Auditing: Regularly run scripts similar to the one above to monitor token usage and compare costs between your gateway and direct vendor APIs. Integrate these checks into your CI/CD pipeline or run them as scheduled jobs.
  • Hybrid Approach: Use a gateway for experimentation or low-volume tasks where convenience outweighs cost. For production workloads, switch to direct API calls. You can even build a dynamic routing system that directs traffic to the cheapest provider based on real-time pricing data.
  • Negotiate with Gateway Providers: If you have significant spend, approach your gateway provider to negotiate custom pricing. They may be willing to reduce their markup for large volumes.

The LLM landscape is evolving rapidly. While gateways offer immediate convenience, their opaque pricing models can become a significant financial burden. Developers must remain vigilant and understand the true cost of their LLM infrastructure.