Self-Healing Playwright Tests Achieved for Free: A Developer's Journey

The Quest for Natural Language E2E Tests

The goal was simple: automated health checks for an e-commerce storefront. This meant verifying core functionalities like search, product page rendering, cart additions, and a successful checkout process. The twist? The tests had to be written in plain natural language, not code, not a Domain Specific Language (DSL), and not via record-and-replay. The desired input was a text file stating, "search for a drill, open the first result, add it to the cart, verify the cart shows 1 item." This requirement stemmed from two key desires: to decouple test writing from the need to maintain Playwright selectors, and to circumvent the notorious fragility of selectors, which often make end-to-end (E2E) testing maintenance more costly than the bugs themselves.

This journey involved five distinct iterations, not all of them linear. The path wasn't a straight shot; there were moments of deliberate regression to explore alternative approaches.

Version 1: Basic Keyword Matching

The initial approach focused on simple keyword matching. The idea was to parse the natural language test, identify keywords (like "search," "add to cart," "verify"), and map them to predefined Playwright actions. For instance, "search for X" would trigger a Playwright `page.fill()` on a search input and then `page.click()` on a search button. "Add it to the cart" would look for an "Add to Cart" button and click it. Verifications involved checking for the presence of specific text or elements.

This version quickly revealed its limitations. Ambiguity in natural language was a major hurdle. "Open the first result" could be interpreted in several ways if multiple elements on a page looked like results. Furthermore, variations in phrasing like "put it in the basket" versus "add to cart" required an extensive, unmanageable list of synonyms. The system lacked any understanding of context or flow, treating each command as an isolated instruction.

Version 2: Introducing Basic State Management

Recognizing the failure of stateless command mapping, Version 2 introduced rudimentary state management. The system now tried to keep track of the last interacted element or the current page context. For example, after "search for a drill," the system would remember that the last action was a search and the target was "drill." When the next instruction was "open the first result," it could more reliably target the first search result element. Similarly, after adding an item to the cart, the system could remember the item that was added.

This improved the reliability for simple, sequential tests. However, it still struggled with complex scenarios, branching logic, and error recovery. If a "click" action failed because the element wasn't found, the state management didn't have a mechanism to try an alternative selector or to report the failure intelligently. It was like having a slightly more organized to-do list, but still prone to getting stuck on the first difficult item.

Version 3: Rule-Based Parsing with Playwright Selectors as Fallback

Version 3 attempted to bring more structure by using a rule-based parsing engine. This involved defining patterns and grammars for common testing actions. For example, a rule might capture the pattern `ACTION_VERB + ITEM + PREPOSITION + TARGET`. "Add [item] to [cart]" would be a pattern. This allowed for more flexible phrasing. Crucially, this version also introduced a fallback mechanism: if the natural language parser couldn't confidently map an instruction, it would attempt to use a predefined, human-written Playwright selector as a last resort.

This brought a significant increase in robustness. When the natural language interpretation failed, the test didn't necessarily halt. The inclusion of Playwright selectors meant that even less common UI elements or more complex interactions could be handled. However, this version started to creep back towards requiring developer intervention. Maintaining the rule-based grammar and, more importantly, the fallback selectors, reintroduced some of the maintenance burden we aimed to avoid.

Version 4: Leveraging LLMs for Intent Recognition

The turning point came with Version 4, which integrated a Large Language Model (LLM). Instead of rigid rules, the LLM was tasked with understanding the *intent* behind the natural language instruction. The prompt engineering was key: the LLM received the natural language instruction, the current page state (or a simplified representation), and a list of potential actions or elements it could interact with. It would then output a structured command, like `{