DoorDash Leverages LLM Juries for Hyper-Accurate Food Metadata

The Challenge of Food Metadata at Scale

DoorDash, a leading food delivery platform, faces a monumental challenge in accurately cataloging the vast and ever-changing world of food items. Millions of dishes, with countless variations in ingredients, preparation methods, and dietary attributes, need precise identification to power effective search, personalized recommendations, and dietary filtering. Traditional methods often rely on manual tagging, which is slow, expensive, and prone to human error and inconsistency, especially across a diverse global menu. This lack of granular, reliable metadata directly impacts user experience: a diner searching for 'vegan lasagna' might be frustrated by results that include meat-based versions, or miss out on hidden gems due to poor categorization.

The complexity is immense. Consider the simple query for 'chicken sandwich.' Does this include fried, grilled, or pulled chicken? Are there gluten-free buns available? Is it spicy? What specific regional variations exist? Each of these questions requires detailed, structured data about the food item. Without it, the platform operates with a fuzzy understanding of its own catalog, hindering its ability to serve users effectively.

Introducing LLM Juries for Context Optimization

To address this, DoorDash developed a sophisticated system employing Large Language Models (LLMs) in a novel 'jury' configuration. This approach moves beyond single-model predictions by leveraging multiple LLM instances, akin to a jury of experts, to reach a consensus on the most accurate metadata for each food item. The system focuses on 'context optimization,' ensuring that the metadata is not just correct in isolation but also relevant to the specific context of a restaurant, cuisine, and user expectations.

The process begins with an initial LLM generating candidate metadata tags for a food item based on its name and description. This is where the 'jury' aspect becomes critical. Instead of relying on a single LLM's output, DoorDash feeds the initial predictions and the original text into a second set of LLMs, tasked with evaluating the accuracy and completeness of the first LLM's suggestions. These 'juror' LLMs act as validators, cross-referencing information and identifying potential discrepancies or omissions. They are trained to consider factors like common ingredient substitutions, regional culinary norms, and typical preparation styles associated with a given dish name and restaurant type.

For instance, if an initial LLM tags a 'Tuna Melt' as potentially containing dairy (due to cheese), a juror LLM might flag this as redundant or obvious, or conversely, it might flag if a specific restaurant's 'Tuna Melt' *doesn't* traditionally include cheese, or if a vegan cheese option is implied. The jury process allows for a more nuanced understanding, moving beyond simple keyword matching to inferring characteristics based on broader culinary knowledge.

Multimodal AI and Data Augmentation

The system further enhances accuracy by incorporating multimodal AI capabilities. Where available, images of the food items are fed into specialized models alongside text descriptions. This allows the system to 'see' the dish, providing visual cues that can disambiguate text-based descriptions. For example, an image can confirm if a 'spicy chicken sandwich' is indeed red with chili peppers or simply uses a spicy sauce, or if a 'salad' contains meat, which would be crucial for vegetarian filtering.

Data augmentation plays a vital role in training these LLMs. By generating synthetic data variations and employing techniques that encourage robustness, DoorDash ensures the models can handle the wide spectrum of language and descriptions found in restaurant menus. This includes variations in spelling, slang, abbreviations, and culturally specific dish names. The LLM jury system, trained on this augmented data, becomes adept at understanding the subtle nuances that differentiate one 'burger' from another, or one 'curry' from a different regional variant.