The Problem: Misidentified Heritage
Kahve Tabela, an open atlas cataloging over 32,000 registered heritage sites in Türkiye, faced a critical data integrity issue. After a reader reported a misidentified photo on their platform, the team embarked on an ambitious audit: verifying if 14,512 photos accurately depicted the heritage sites they were supposed to represent. This wasn't just a matter of aesthetic accuracy; it was about the trustworthiness of their open atlas.
The challenge of building a comprehensive dataset on a budget often forces reliance on readily available open-source information. Kahve Tabela, like many, sourced images from platforms such as Wikipedia, Wikimedia Commons, Google Places, Mapillary, and national heritage inventories. These are typically considered authoritative or at least reliable sources for such information. However, the audit revealed a stark reality: the very sources considered most dependable were the ones contributing the most errors.
Methodology: Vision Model Audit at Scale
To tackle the massive task of verifying each image, Kahve Tabela employed a local, open-source vision model, specifically Qwen3-VL 30B, running on a Mac Studio. This approach avoided API costs and maintained data privacy, but it required significant computational resources. The process involved feeding each of the 14,512 photos into the model to assess its visual content against the purported identity of the heritage site.
The audit aimed to answer a fundamental question: how many images in their database were actually showing the correct location? The scale of the undertaking, coupled with the reliance on automated verification, set the stage for potentially surprising insights into data provenance and the reliability of widely used online resources.
Finding 1: Error Rate is a Property of the Source
The first major finding was counterintuitive. Instead of a uniform distribution of errors across all image sources, the audit demonstrated that the error rate was strongly correlated with the origin of the photograph. Sources typically perceived as highly trustworthy and authoritative, such as national heritage inventories and well-curated Wikipedia collections, exhibited the highest rates of image misidentification. Conversely, sources that might be considered less formal or more user-generated, like Mapillary or even Google Places, showed comparatively lower error rates.
This finding directly challenges the conventional wisdom that more official or established sources are inherently more accurate regarding visual data. It suggests that the mechanisms for data curation, user contribution, and error correction vary significantly between platforms, leading to disparate levels of visual accuracy. For Kahve Tabela, this meant that a substantial portion of their verified data, sourced from seemingly reliable places, might be incorrect.
Finding 2: The Audit Itself as a Source of Truth
The second, and perhaps more significant, finding relates to the audit process and its implications. While the initial goal was to identify and correct errors in the existing dataset, the audit process itself became a new, highly accurate source of truth. The vision model, when properly applied and calibrated, proved to be an exceptionally reliable tool for visual verification. The act of performing the audit generated a dataset of verified images, which is arguably more accurate than the sum of its potentially flawed original sources.
This highlights a meta-level insight: the process of rigorous verification, even if initiated by a dataset cleanup effort, can yield a more valuable and trustworthy dataset than the original collection. The errors found were not random; they were concentrated in specific sources, allowing for targeted improvements and a deeper understanding of data quality. The implication is that such audits, while resource-intensive, can fundamentally improve data integrity and build a more reliable foundation for future use.
Implications for Data Curation and Open Atlases
The audit by Kahve Tabela offers critical lessons for anyone building or maintaining large-scale datasets, particularly those relying on visual information. The assumption that