Unlocking Catawiki's Hidden Valuation Data
Catawiki, the popular online auction platform, has long employed category experts to provide estimated value ranges for every item listed. This expert valuation is arguably the most critical piece of data for discerning buyers, offering a benchmark against which current bids can be measured. Until recently, accessing this information programmatically required workarounds or manual extraction. However, a keen observation by a developer has revealed that Catawiki embeds this crucial data directly within the page's JSON payload, making it readily available for automated analysis and deal-finding bots.
The discovery stems from the fact that Catawiki is built using Next.js, a popular React framework for building server-rendered applications. A common pattern in Next.js applications is to embed the initial state and data required for the page to render within a script tag, typically identified by the `id="__NEXT_DATA__"`. This script contains a JSON object that holds all the necessary props and data for the page. By parsing this JSON, developers can access information that might otherwise be hidden behind API calls or complex DOM scraping.
The author of the discovery, using the handle jdpg23 on Dev.to, found that the search results pages on Catawiki contain a wealth of information within this `__NEXT_DATA__` object. Specifically, the `pageProps.searchLots.lots` array provides details for approximately 24 lots per page. While this array is invaluable for discovering items, it initially presented a challenge: it did not include the expert estimates or current bid prices. This meant that while you could find lots, you couldn't immediately assess their value relative to the expert's opinion directly from the search results themselves. The primary value, the expert estimate, was seemingly absent from this particular data slice.
Building a Deal Finder Bot
The real breakthrough came when realizing that while the search results themselves might omit the estimates, the individual lot pages provided them. The strategy then became clear: use the search results to identify potential lots of interest based on available metadata, and then programmatically visit each lot's page to extract the expert estimate from its respective `__NEXT_DATA__` JSON. This two-step process allows for a comprehensive scan of categories to identify potentially undervalued items.
The process involves several key steps for anyone looking to replicate this functionality. First, one must fetch the search results page for a given category. This page's HTML will contain the `