Track AI Prompt Changes with Run Manifests for Reproducible Results

The Problem: Lost Context in AI Development

Imagine this: an AI model evaluation shows marked improvement after a prompt edit. A week later, a teammate cannot reproduce that exact comparison. The prompt file exists, but the specific version of the retrieved note has changed, a model alias might now point to a different underlying model, or an adapter revision is undocumented. The result is an answer on disk, but with insufficient context to explain its origin or how it was achieved. This scenario highlights a critical gap in the current AI development lifecycle: the lack of robust traceability for evaluation runs.

Prompt engineering, model selection, and configuration all play a crucial role in AI output. When these variables are not meticulously recorded, reproducibility becomes impossible. This isn't just an inconvenience; it's a fundamental roadblock to scientific rigor, iterative improvement, and reliable deployment of AI systems. Developers and researchers need a way to anchor their findings to the exact conditions under which they were produced.

Introducing the Run Manifest

The solution lies in creating a run manifest. This structured record acts as a comprehensive log, detailing every critical input and configuration parameter for a specific AI evaluation attempt. Think of it less like a simple log file and more like a detailed recipe for an AI experiment, ensuring that anyone can recreate the exact conditions later.

A run manifest should identify:

• The actual request inputs (e.g., the precise prompt text, user queries, or data samples used).
• The complete configuration parameters (e.g., model hyperparameters, temperature settings, sampling methods).
• The source code snapshot or version (e.g., Git commit hash for the application logic).
• The evaluator revision (e.g., version of the evaluation script or framework).
• Details about any specific model versions or aliases used.
• Information about any fine-tuning adapters or weights applied.

By capturing these elements, a run manifest provides the necessary context to understand the conditions that led to a particular AI output. This is essential for debugging, for validating results, and for building confidence in AI system performance over time.

Why a Filename Is Not Enough

Relying solely on a prompt filename is fundamentally insufficient for tracking AI evaluation runs. A filename might indicate the intended purpose of a prompt, but it fails to capture the dynamic and often complex environment in which AI models operate. Consider these points:

• Prompt Templating: Prompts are rarely static strings. They often involve templates with placeholders that are filled dynamically at runtime. The filename might point to the template, but not the specific data used to populate it for a given run.
• Model Aliases and Versions: A model alias (e.g., `gpt-3.5-turbo`) can point to different underlying model versions over time. Without recording the specific alias or the exact model ID used, reproducibility is lost as the alias is updated.
• Configuration Drift: Hyperparameters, sampling strategies, and other configuration settings can easily change between runs. A filename offers no insight into these crucial parameters.
• External Dependencies: The AI system might depend on external libraries, data sources, or even specific hardware configurations. These are typically not reflected in a simple filename.
• Adapter and Fine-tuning Data: If a model has been fine-tuned or uses specific adapters, the filename offers no clue as to which weights or data were applied.

The prompt filename is a useful organizational tool, but it is not a sufficient record for scientific or engineering rigor. It's like having the name of a recipe but not the ingredients or cooking instructions.

Assigning a Stable Digest

Once a run manifest is created, it should be assigned a stable digest. This digest, typically a cryptographic hash (like SHA-256) of the manifest's content, serves as a unique identifier for that specific set of conditions. If any part of the manifest changes—even a single character in a configuration setting—the digest will change.

The digest is powerful for several reasons:

• Change Detection: It immediately signals if the configuration for a recorded run has been altered.
• Integrity Check: It provides a verifiable way to ensure that the recorded manifest accurately reflects the conditions of the original run.
• Referencing: It allows for unambiguous referencing of specific experimental setups.

It is critical to understand that the digest helps detect changes to the recorded configuration. It does not guarantee identical model output. AI models, especially large language models, can exhibit stochastic behavior, meaning that even with identical inputs and configurations, the output might vary slightly due to factors like sampling temperature or internal random processes. The manifest captures the intent and conditions of the run, not necessarily the exact bit-for-bit output.