The Problem: Large Datasets and Expensive LLM Calls

Teams building agentic AI pipelines often encounter a common efficiency challenge: they process large datasets that are periodically reloaded. To save on expensive LLM calls, they cache the output for inputs that repeat across these reloads. The naive approach is to store this cache within the same database table as the primary data. While this seems safe, it creates a critical operational bottleneck. When the dataset is reloaded, the old data is wiped, and the entire table is repopulated. If the cache resides within this table, it gets wiped too, negating the cost savings.

A slightly more robust method involves keeping the cache on disk separately from the main dataset. This works until storage becomes a constraint. In environments where disk space is limited, you cannot afford to keep both the old dataset and the new one, along with the cache, simultaneously. This forces a difficult choice: either you risk losing the cache when you wipe the old data to load the new, or you find a way to manage this transition without sufficient storage.

This scenario is particularly acute for teams running AI agents that rely on a consistent memory of past interactions or processed information. Imagine an AI customer support agent that needs to recall past conversations to provide contextually relevant answers. If the training data or knowledge base is refreshed weekly, and the cache of past interactions is lost with each refresh, the agent effectively suffers from amnesia, degrading its performance and utility.

The Discovery: Input Signatures Are Tiny

The breakthrough in architecting a lean LLM caching solution lies in a simple yet powerful observation: the inputs to an LLM call are almost always significantly smaller than the output or the surrounding data context. Consider an LLM query that asks for a summary of a lengthy document, or an analysis of a complex data point. The prompt itself, which includes the specific question and any relevant context, is orders of magnitude smaller than the document or the full dataset record it refers to.

This realization is the cornerstone of an efficient caching strategy. Instead of caching the entire input-output pair, or embedding the cache within the large data rows, we can create a separate, lightweight index. This index maps a stable, unique identifier for the input to the cached LLM output. The key insight is that this identifier can be a hash of the input prompt, or a deterministic key derived from the core components of the input that are unlikely to change unless the input itself fundamentally changes.

Diagram illustrating the concept of input hashing for LLM cache keys

The Two-Table Architecture: Orthogonal Cache Storage

The optimal pattern that emerges from this discovery is a two-table structure. The primary table, which we'll call the Data Table, stores the main dataset. This table is subject to periodic wipes and reloads. The secondary table, the Cache Table, is entirely separate and stores only the cached LLM outputs.

The crucial element connecting these two tables is a stable, unique identifier. This identifier acts as the key in the Cache Table and is also present in the Data Table, or can be deterministically generated from data within the Data Table. When an agent needs to process a piece of data:

  1. A unique identifier (e.g., a hash of the input prompt or a stable ID from the data record) is generated for the current input.
  2. The system checks the Cache Table using this identifier.
  3. If a cache hit occurs, the stored LLM output is retrieved.
  4. If it's a cache miss, the LLM is called, the output is generated, and then it is stored in the Cache Table, keyed by the identifier.

This architecture offers several advantages:

  • Cost Savings: Expensive LLM calls are avoided for repeated inputs.
  • Storage Efficiency: The Cache Table is significantly smaller than the Data Table, as it only stores the stable input identifiers and the LLM outputs, not the entire input context or auxiliary data. A 20 million row Data Table might only require a few hundred thousand or a million entries in the Cache Table, depending on input variability.
  • Data Freshness: The Data Table can be wiped and reloaded without affecting the cache. The Cache Table persists independently. When the new data loads, the system can still look up cached results using the stable identifiers.

The Operational Gotcha: Identifier Stability

While this two-table approach is robust, there is one critical operational gotcha that can undermine the entire system: the stability of the identifier used as the cache key. If the method used to generate the identifier changes between data reloads, or if the identifier itself is not truly stable for identical inputs, cache misses will occur even when the underlying input is the same. This leads to redundant LLM calls and defeats the purpose of caching.

For example, if you hash a complex JSON input string, but the order of keys within the JSON can change between reloads (even if the data is logically identical), the hash will differ. Similarly, if you rely on a timestamp within the input data to generate the identifier, and that timestamp is updated on reload, the identifier will change. The identifier must be a deterministic function of the actual input that the LLM processes. This means using stable IDs from the source data, or robust hashing mechanisms that are insensitive to non-data-altering variations like key order in JSON.

When architecting this, consider the following for your identifier generation:

  • Deterministic Hashing: Use libraries that provide stable JSON serialization (e.g., sorting keys before hashing) or hash specific, immutable fields of your input.
  • Source of Truth IDs: If your data has a true, immutable primary key that uniquely identifies a record and its relevant input parameters, use that.
  • Input Canonicalization: Before hashing, ensure the input is in a canonical form. Remove extraneous whitespace, normalize case where appropriate, and sort any list or map elements that don't have inherent order.

The surprise here is not the complexity of LLM caching itself, but how a simple architectural shift, enabled by understanding the size disparity between LLM inputs and outputs, can yield such significant cost and storage benefits. It transforms the problem from managing massive, volatile datasets to maintaining a small, stable lookup index.

Conclusion: Lean Caching for Scalable AI

By decoupling the LLM cache from the main dataset and storing it in a small, orthogonal lookup table keyed by stable identifiers, teams can effectively manage expensive LLM computations. This architecture allows for frequent data refreshes without losing valuable cached LLM outputs, drastically cutting operational costs and storage requirements. The key to success lies in ensuring the absolute stability of the identifiers used for cache lookups. This lean caching strategy is essential for building scalable and cost-effective AI agentic pipelines that can operate on dynamic data without suffering from performance degradation or memory loss.