The Genesis of a Dataset: A Failed Startup's Legacy

The AI security landscape is a constant arms race. Developers build defenses, and users, whether malicious or merely curious, find ways around them. For nearly a year, one individual invested heavily in this dynamic, building a startup centered on an AI guard designed to protect Large Language Models (LLMs). The product’s core pitch was simple: challenge the guard, try to break it, and earn rewards. This bug-bounty-style game, complete with leaderboards and tiered prizes, aimed to crowdsource the discovery of vulnerabilities in the AI’s multimodal prompt firewall.

However, the business model never materialized. The core asset wasn't the product itself, but the data generated by motivated red-teamers relentlessly probing its defenses. Instead of letting this year's worth of logged, adversarial interactions languish in a database, the founder decided to open-source the entire endeavor. This includes the code for the AI guard, the model itself, and a staggering 13,000 real-world bypass attempts. The goal is to provide the AI security and research community with a unique, battle-tested dataset for future development.

Diagram illustrating a two-stage AI prompt firewall architecture

Inside the Prompt Firewall: Architecture and Defenses

The AI guard, now open-sourced, functions as a two-stage filter positioned between user input and the underlying LLM. This layered approach is designed to catch a wide array of adversarial prompts before they can reach the model and elicit unintended responses.

Stage One: The Regex Gate

The initial defense is a sophisticated regex gate. This component employs 119 distinct patterns engineered to identify and block common jailbreaking techniques. These patterns cover a broad spectrum of attack vectors, including:

  • Direct Override Phrasing: Attempts to directly instruct the LLM to ignore its safety protocols, with patterns designed to catch variations in approximately eleven languages.
  • DAN/Persona Jailbreaks: Phrasing that attempts to frame the LLM as a different persona (e.g., “Do Anything Now” or “DAN”) to bypass its ethical guidelines.
  • Template Injection: Detection of common code injection sigils like {{ and ${, which could be used in template manipulation attacks.
  • Agentic Chain-of-Thought Injection: Techniques aimed at manipulating the LLM’s reasoning process by injecting specific phrases that trigger unintended chains of thought.
  • Obfuscation Techniques: Stripping of homoglyphs (characters that look similar but are different, like Latin 'a' vs. Cyrillic 'а') and zero-width characters, often used to hide malicious instructions within seemingly innocuous text.

This first stage acts as a broad net, filtering out a significant volume of easily identifiable malicious prompts. The effectiveness of these regex patterns lies in their specificity and their ability to adapt to linguistic variations and common obfuscation tactics.

Stage Two: Advanced Filtering and Model Integration

Following the regex gate, a more advanced filtering mechanism comes into play. This stage likely involves a smaller, more specialized model or a set of heuristics designed to catch more nuanced or novel bypass attempts that slip through the initial regex checks. The exact nature of this second stage, beyond its role as a secondary defense, is detailed within the open-sourced code.

The entire system was built with the understanding that it was under attack. This adversarial context is crucial; unlike standard security testing, the attackers in this scenario were actively motivated by rewards and the challenge itself, leading to a more robust and diverse set of bypass attempts than might be found in passive datasets.

Snapshot of the leaderboard interface from the prompt firewall game

The Dataset: 13,000 Attempts and What They Reveal

The dataset comprises approximately 13,000 logged interactions. Each entry represents a user's attempt to bypass the prompt firewall, categorized by whether the attempt was successful or not. This massive collection offers invaluable insights into the current state of prompt engineering for adversarial purposes.

Key characteristics of the dataset include:

  • Real-World Adversarial Input: These are not synthetic prompts generated by researchers but actual inputs submitted by users participating in the game. This means the data reflects genuine human ingenuity and the evolving tactics of prompt injection.
  • Motivated Attackers: Participants were actively trying to break the system, incentivized by the potential for rewards. This motivation likely led to more persistent and creative attack vectors than those found in datasets generated by non-incentivized users.
  • Bypass Success Rates: The dataset includes information on which attempts succeeded and which failed. This allows researchers to analyze the specific characteristics of successful bypasses and understand the weaknesses of the firewall.
  • Multimodal Considerations: While the excerpt focuses heavily on text-based prompt injection, the mention of a "multimodal prompt firewall" suggests that some attacks may have involved or attempted to manipulate image or other non-textual inputs, adding another layer of complexity to the security challenge.

The sheer volume and the nature of this dataset make it a significant resource. It allows for the training of new detection models, the evaluation of existing defenses, and a deeper understanding of the adversarial surface area of LLMs. The fact that the game didn't make money is, in a way, a blessing for the AI security community, as it has liberated this valuable research asset.

Why This Matters: The Future of LLM Security

The open-sourcing of this prompt firewall code and dataset arrives at a critical juncture for AI development. As LLMs become more integrated into applications and services, securing them against malicious inputs is paramount. Current defenses often rely on static rule sets or basic content moderation, which are frequently circumvented by sophisticated prompt engineering.

This dataset provides a concrete, empirical basis for developing more resilient AI systems. Researchers can use it to:

  • Train more robust prompt injection detection models.
  • Benchmark the effectiveness of new defense strategies.
  • Identify emergent adversarial techniques.
  • Understand the limitations of current prompt engineering best practices.

The challenge of securing LLMs is not just a technical one; it's a continuous process of adaptation. By sharing these tools and data, the founder is fostering a more collaborative and transparent approach to AI security, allowing the community to build better, safer AI models for everyone.