Skip to content

Databrew

Databrew turns a TOML file into a crawl that keeps a site's records current. You say which pages list the records and which fields to read; databrew fetches the pages, stores every new record and every later change to it, and picks up where it stopped when you run it again.

Install it with uv, on Python 3.14 or later:

uv tool install "databrew @ git+https://github.com/eyayaw/databrew"

Write a config for a practice bookstore:

books.toml
name = "books"
start_urls = ["https://books.toscrape.com/catalogue/page-1.html"]

# Listing pages show 20 books, each in an article.product_pod card.
[listing.items]
selector = "article.product_pod"

[listing.items.fields]
url = { selector = "h3 a", attribute = "href" }
title = { selector = "h3 a", attribute = "title" }
price = { selector = ".price_color", parser = "parse_float" }

[listing.links]
items = "url"               # the card field that links to the book's page
pagination = ["li.next a"]  # the link to the next listing page

# Item pages show one book.
[item.items.fields]
title = ".product_main h1"
price = { selector = ".product_main .price_color", parser = "parse_float" }
availability = ".product_main .availability"

Run it, here capped at 20 requests:

$ databrew run books.toml -n 20
books: success, +19 new items, 20 observations, 0 failed URLs, traversal paused (Reached max requests limit (20))
total: 1 run, 0 failed, +19 new items, 20 observations, 0 failed URLs

The records are in data/books/, and the next run continues where this one stopped. The tutorial walks through the same crawl step by step.

What you get

  • A history, not a snapshot

    Each record is stored once, and again only when its content changes, so you can ask what a price was last month. Every card a listing page shows is kept too, as an observation.

  • Runs that resume

    Stop a crawl with a request budget, a deadline, or Ctrl+C, and the next run continues the same queue without fetching completed pages again.

  • Cheap updates

    After the first traversal, a run stops following pagination where it already has the records, and fetches known item pages again by age or when a watched field changes.

  • One verdict per run

    Every run ends with a stop and an outcome that the exit code, the report, and the library call agree on. A crawl that a site blocks from the start fails, so a schedule notices.

Records land in Parquet or JSONL parts that DuckDB, pandas, and Polars read directly.

Where to go next

  • Tutorial: crawl the bookstore and query the result.
  • How-to guides: write a config, schedule runs, recover from failures, authenticate.
  • Concepts: how a crawl walks a site, what it stores, and how a run ends.
  • Reference: every config key, command, and Python call.

What databrew builds on

Databrew owns what happens between fetching and storing: the queue, retries, pacing, stop rules, revisits, and the run report. Three packages do the rest, and their docs are the reference for their part.

Part Package Docs
Fetching, with HTTP clients or a browser fetchkit README
Selectors, JSON paths, and parsers extractkit Config, parsers
The record files itemstore README, format