CodeQL 2.26.0 Introduces AI Prompt Injection Detection

GitHub announced on July 10, 2026, that CodeQL version 2.26.0 now includes built-in AI prompt-injection detection capabilities. This is a significant step towards securing applications that leverage large language models (LLMs). While enabling a query is straightforward, building a robust regression test suite for prompt injection vulnerabilities is a more nuanced task that requires a deep understanding of static analysis and data flow.

The primary source for this update is the GitHub Changelog for July 10, 2026.

Structuring a Prompt Injection Regression Fixture

Effective regression testing for prompt injection vulnerabilities goes beyond simply searching for specific, tell-tale phrases like ignore previous instructions. Static analysis tools like CodeQL operate by analyzing code paths and data flow. Therefore, a useful fixture must model an untrusted source, a prompt construction mechanism, and a model sink where the LLM's output is consumed.

A typical fixture structure, as demonstrated in sample implementations, would involve organizing test cases into positive and negative directories. The positive directory would contain examples that are expected to trigger an alert, showcasing direct data flow from an untrusted source to the LLM prompt. The negative directory would house examples designed to be safe, such as when instructions are properly sanitized or when the prompt is constructed using only trusted data.

Directory structure for prompt injection regression tests.

The directory structure might look like this:

security-fixtures/
└── prompt-injection/
    ├── positive/
    │   ├── direct-flow.ts
    │   └── helper-flow.ts
    ├── negative/
    │   └── trusted-instruction.ts
    └── expected-alerts.json

Each TypeScript file within the positive and negative directories would represent a specific test scenario. For instance, direct-flow.ts might demonstrate a direct path where user input is concatenated into an LLM prompt without sanitization. helper-flow.ts could illustrate a more complex scenario where the untrusted data passes through intermediate functions before reaching the prompt.

Conversely, trusted-instruction.ts in the negative directory would serve as a control, ensuring that the query does not generate false positives for legitimate prompt constructions that do not involve untrusted input. An expected-alerts.json file would accompany these tests, detailing the specific alerts and their expected locations within the code for each positive test case. This allows for automated verification of the query's accuracy.

Modeling the Data Flow Path

The core principle for building effective fixtures is to model the data flow path. This means tracing how untrusted data, originating from an external source such as user input, HTTP requests, or file uploads, travels through the application's code. The analysis must follow this data as it is potentially transformed, sanitized, or combined with other data before ultimately being used to construct a prompt for an LLM.

A prompt injection vulnerability arises when an attacker can manipulate the LLM's behavior by injecting malicious instructions into the prompt. This often involves tricking the LLM into ignoring its original instructions or executing unintended commands. Static analysis tools like CodeQL are well-suited for this because they can identify these data flow paths across function calls and module boundaries.

Consider a scenario where user input is read from a web request and then appended to a system prompt for a chatbot. The CodeQL query needs to identify:

  1. The source of the untrusted data (e.g., the request parameter).
  2. Any transformations applied to the data (e.g., string manipulation, encoding/decoding).
  3. The sink where the prompt is constructed and sent to the LLM.

A fixture should be designed to specifically exercise this path. For example, a positive/direct-flow.ts test case might look conceptually like this:

// security-fixtures/prompt-injection/positive/direct-flow.ts

import { untrustedInput } from "../untrusted-source"; // Simulates external input
import { constructPrompt } from "../llm-utils";

const userInput = untrustedInput();
const systemPrompt = "You are a helpful assistant. Summarize the following text:";

// Potential injection point: Directly concatenating user input
const finalPrompt = systemPrompt + "\n" + userInput;

// Assume constructPrompt sends this to an LLM sink
constructPrompt(finalPrompt);

The corresponding expected-alerts.json would specify that an alert is expected at the line where finalPrompt is constructed, indicating a potential prompt injection vulnerability due to the flow from untrustedInput.

The Importance of Negative Test Cases

False positives are a significant concern in static analysis. To mitigate this, regression fixtures must include comprehensive negative test cases. These tests demonstrate scenarios where the same components might be present, but the data flow does not constitute a vulnerability. This could involve:

  • Sanitizing user input before incorporating it into the prompt.
  • Using allowlists or denylists to filter input.
  • Constructing prompts solely from trusted, hardcoded strings.
  • Ensuring that the prompt is used in a context where injected instructions would not be interpreted as commands by the LLM.

For instance, a negative test case might involve a sanitization function:

// security-fixtures/prompt-injection/negative/trusted-instruction.ts

import { untrustedInput } from "../untrusted-source";
import { constructPrompt } from "../llm-utils";

function sanitizeInput(input: string): string {
  // Simple sanitization: remove potentially harmful keywords
  return input.replace(/ignore previous instructions/gi, '').trim();
}

const userInput = untrustedInput();
const systemPrompt = "You are a helpful assistant. Summarize the following text:";

const sanitizedUserInput = sanitizeInput(userInput);
const finalPrompt = systemPrompt + "\n" + sanitizedUserInput;

constructPrompt(finalPrompt);

This test would be expected to produce no alerts, confirming that the sanitization logic is correctly identified as a mitigation.

What's Next for LLM Security in CodeQL?

The addition of AI prompt-injection detection in CodeQL 2.26.0 is a promising development for securing AI-powered applications. However, the effectiveness of any security tool hinges on the quality of its test coverage. Developers and security professionals must actively build and maintain comprehensive regression test suites that accurately model real-world vulnerabilities.

The challenge lies in the evolving nature of LLM interactions and potential attack vectors. As new techniques for prompt injection emerge, the corresponding CodeQL queries and test fixtures will need to be updated. This necessitates a continuous effort to stay ahead of emerging threats and adapt security testing methodologies accordingly. The ability to model data flow paths is crucial; relying solely on pattern matching for specific phrases will inevitably lead to missed vulnerabilities and excessive false positives.