The Fallacy of Inference as a Gatekeeper
Admission control in production services is a critical function. It acts as a gate, deciding whether a request is legitimate and should proceed or if it's malicious and should be blocked. Traditionally, methods like token buckets or Generic Cell Rate Algorithms (GCRA) have been employed for this purpose. These systems are designed to be cheap, deterministic, fail-closed, and significantly less resource-intensive than the protected service itself. They are the digital equivalent of a bouncer checking IDs at the door – a quick, reliable check.
The temptation to use sophisticated methods for admission control is strong, especially in the burgeoning field of AI. Language models, capable of understanding nuance and intent, seem like a perfect tool to analyze incoming requests. Imagine a system where an incoming support ticket, a user-agent string, or any arbitrary text is first sent to a smaller, cheaper language model. This model then provides a verdict: is this user abusive? Only if the verdict is positive does the request proceed to the main, expensive production service. This approach is often termed a "smart shedder."
The fundamental flaw in this design is treating a control-plane verdict as if it were a simple data point or a deterministic gate. It is not. A token bucket, a GCRA check, or a local rate limit decision must be fast and predictable. They must operate with minimal overhead. The decision to admit or deny a request should not involve a complex computational process that itself can be attacked. When you delegate the admission control decision to an inference engine, you fundamentally change the nature of the defense. Instead of a cheap fence, you've built a conversation. And in an adversarial environment, conversations are expensive and can be gamed.
Why Inference Fails as a Gatekeeper
The core problem lies in the nature of inference itself. Unlike traditional rate-limiting algorithms, inference is inherently variable in latency. The time it takes for a model to process a prompt can fluctuate significantly based on the prompt's complexity, the model's current load, and the underlying hardware. This variability is antithetical to the requirements of a robust admission control system, which demands predictability and speed. A gate that might take milliseconds one moment and seconds the next is not a reliable gate.
Furthermore, inference is computationally expensive. The very work you are trying to protect is often cheaper than the work required to decide whether to allow a request. In a denial-of-service (DoS) attack scenario, an attacker doesn't care about the cost per request if they can flood the system with enough requests to overwhelm it. If the defense mechanism itself involves running an inference model for every incoming request, the attacker has effectively turned the defender's own system against them. Each "attack packet" or malicious request, when processed by the inference gatekeeper, still incurs a cost on the defender's side. The defender, in their attempt to be smart, ends up paying for every single attack packet they process, amplifying the damage rather than mitigating it.
The prompt surface is also untrusted. Users can craft prompts designed to probe, confuse, or overload the inference model. A seemingly innocuous request can be disguised with complex phrasing or malicious payloads that force the inference engine into lengthy or erroneous processing. This means that the decision to allow or deny access to your production service is based on a guess, a variable-latency computation performed on potentially adversarial input. This is not a secure or efficient way to manage access.
The Cost of "Smart Shedding"
Consider a scenario where a free tier of a language model service is offered. The temptation is to use a small, quick inference model to filter out "abusive" users before they hit the main, costly model. The logic is: if the filtering model is cheaper than the main model, we save money. However, this logic breaks down under load. An attacker can simply send a massive volume of requests, each designed to trigger a moderate-to-high processing cost on the filtering inference model. The aggregate cost of running these inference checks on a flood of malicious requests can easily exceed the cost of letting a small fraction of them through to the main model, or even the cost of serving legitimate users.
This is akin to a security guard at a venue asking every single person who walks in to write a 500-word essay on their intentions, only to then decide if they can enter. The guard's task itself becomes the bottleneck and the exploitable resource. The attacker's goal isn't to get in; it's to occupy the guard's time and resources indefinitely. In the context of free inference tiers, the "free" aspect exacerbates this problem. There is no direct per-request cost to the attacker, making it trivial for them to bombard the system with requests designed to maximize the inference cost for the defender.
What Should Be Used Instead?
For true admission control, especially for services that are free or have strict cost constraints, deterministic, low-overhead mechanisms are essential. This means returning to the tried-and-true methods: traditional rate limiters like token buckets or GCRA, implemented at the edge of your network or in a dedicated, lightweight proxy. These systems make decisions based on simple counters, timestamps, and predefined limits – operations that are orders of magnitude cheaper and faster than any inference task.
If you need to detect abuse, do it asynchronously. Log suspicious requests and analyze them offline using your powerful inference models. This allows you to build robust abuse detection profiles without compromising the performance and cost-efficiency of your real-time request handling. The decision to block a user permanently can be made after careful, deliberate analysis, not in the heat of a potential DoS attack via a complex inference pipeline.
The principle is simple: the gatekeeper must be significantly cheaper and faster than the system it protects. Inference, by its very nature, violates this principle. Using it for real-time admission control on free tiers is not just inefficient; it's an open invitation for abuse, turning your defense mechanism into the very vector of attack.