Observations and versions¶

Databrew stores two kinds of row about each record, and they answer different questions.

Observations¶

An observation is one card as one response showed it. Every card on every listing page that a run fetches becomes an observation, including a card that looks exactly as it did an hour ago, and including two cards with the same key on one page. Observations are the raw sighting history: they tell you when a record was listed, at what position, and with what values.

Each observation is keyed fetch_id:N, where the fetch ID is unique to the response and N counts the cards it recorded, from zero. fetched_at is when the response arrived. A field the card did not show stays missing, and a record that a traversal did not see is not treated as removed.

Versions¶

A version is the content of one record at one time. An item page's item is stored as a version when its content differs from the latest stored version of that record, so unchanged pages add nothing. A price that goes 10, 20, then back to 10 stores three versions.

Versions are keyed item_key:N, where N counts the versions of that record from 1. A version's fetched_at is when databrew stored it, moments after the response arrived. The site may have changed the record any time since the fetch before, up to a revisit interval earlier. Read in order of fetched_at, the versions of one record form a Type 2 slowly changing dimension, with fetched_at as the time databrew saw each version become current.

The comparison covers the item exactly as extracted. A field that changes on every fetch, such as a view counter or a "listed 3 hours ago" label, makes every fetch a new version. Leave such a field out of the config, normalize it in a parser, or drop it in a middleware.

The row schema¶

Both stores share one schema:

Query your data reads them with DuckDB.

Keys¶

The item key ties observations and versions of the same record together, and decides whether an item link is already stored. Databrew picks it in this order:

For a card:

1. the value at [listing.items].key, when the config names one;
2. otherwise a digest of the card's item link;
3. otherwise a digest of the card's own url field;
4. otherwise a digest of the listing page's URL.

For an item page:

1. the key of the card that linked to it;
2. when that card had no key, the value at [item.items].key;
3. otherwise a digest of the page's URL.

A card whose configured key is missing, or is neither a string nor an integer, is dropped and counted in the run's items_dropped. An item page whose item has no usable key fails as an extraction error, which charges the URL a failed run.

A site without [item] needs [listing.items].key, because the card is the only thing that identifies the record across pages and runs. When one storage path combines several sources, make the key unique across them, for example marketplace:country:listing-id.