Your first crawl¶

This tutorial crawls Books to Scrape, a bookstore built for practicing scrapers. In about ten minutes you will write a config, check what it extracts, run a crawl in two short sessions, and query the result. Every output below comes from a real run.

You need Python 3.14 or later and uv.

1. Install databrew¶

Databrew installs from its repository:

uv tool install "databrew @ git+https://github.com/eyayaw/databrew"
databrew --version

2. Start a config¶

Make a directory for the crawl and ask databrew for a starter file:

$ mkdir bookstore && cd bookstore
$ databrew init books --url https://books.toscrape.com/catalogue/page-1.html
Created config: books.toml

Next steps:
  1. Edit books.toml to customize extraction rules
  2. Run: databrew check books.toml
  3. Run: databrew run books.toml -n 5

The file it writes has placeholder selectors and commented-out options. Replace its contents with this config, which reads the bookstore. The same file is examples/configs/books.toml in the repository.

name = "books"
start_urls = ["https://books.toscrape.com/catalogue/page-1.html"]

# Listing pages show 20 books, each in an article.product_pod card.
[listing.items]
selector = "article.product_pod"

[listing.items.fields]
url = { selector = "h3 a", attribute = "href" }
title = { selector = "h3 a", attribute = "title" }
price = { selector = ".price_color", parser = "parse_float" }

[listing.links]
items = "url"               # the card field that links to the book's page
pagination = ["li.next a"]  # the link to the next listing page

# Item pages show one book.
[item.items.fields]
title = ".product_main h1"
price = { selector = ".product_main .price_color", parser = "parse_float" }
availability = ".product_main .availability"

A crawl has two kinds of page. Listing pages show many records as cards and link to more listing pages; the start URLs are listing pages. Item pages show one record each. [listing] tells databrew how to read the first kind and [item] the second:

• [listing.items].selector picks the cards: each element that matches it is one card.
• [listing.items.fields] reads each card. A field written as a selector string reads the text; one written as a table, such as { selector = "h3 a", attribute = "href" }, reads an attribute, and parser = "parse_float" turns £51.77 into 51.77.
• [listing.links] says what to queue: the card field that links to each book's page, and the link to the next listing page.
• [item.items.fields] reads the book's own page.

The selectors and parsers are extractkit's.

3. Check what it extracts¶

databrew check loads the config, builds everything a run needs, and prints the plan:

$ databrew check books.toml
Valid config: /home/you/bookstore/books.toml
  Name: books
  Listing type: html
  Start URLs: 1
  Fetch backend: httpx
  Concurrency: 5
  Storage: data/books (no state yet)

databrew parse fetches one page and prints what the config reads from it, as JSON, without storing anything. Try the start URL:

$ databrew parse books.toml https://books.toscrape.com/catalogue/page-1.html

The output lists 20 cards, the next-page link, and 20 item links. The first card reads:

{"url": "a-light-in-the-attic_1000/index.html", "title": "A Light in the Attic", "price": 51.77}

Now an item page. --page-type item reads it with [item]:

$ databrew parse books.toml --page-type item \
    https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html

Its one item reads:

{
  "url": "https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html",
  "title": "A Light in the Attic",
  "price": 51.77,
  "availability": "In stock (22 available)"
}

If a field comes back empty, fix its selector and parse again. --save page.html keeps the fetched page, and --file page.html reads it back, so you can iterate without fetching.

4. Run a short crawl¶

-n 20 caps the run at 20 requests, which keeps this first run polite and quick:

$ databrew run books.toml -n 20
books: success, +19 new items, 20 observations, 0 failed URLs, traversal paused (Reached max requests limit (20))
total: 1 run, 0 failed, +19 new items, 20 observations, 0 failed URLs

Databrew fetched the first listing page, which gave 20 observations, one per card, and queued the next listing page and 20 book pages. New book pages go first, so the other 19 requests stored 19 books. The budget ran out before the rest, so the walk through the listing pages, the traversal, is paused.

databrew status shows what the storage path holds:

$ databrew status books.toml
Status for: books
  Storage: data/books
  Item files: 1
  Items: 19
  URLs pending: 2
  URLs completed: 20
  URLs failed: 0
  Traversal: 1, exhaustive, paused
  Listing pages to go: 1

The two pending URLs are the 20th book and the second listing page.

5. Resume¶

Run the same command again:

$ databrew run books.toml -n 20
books: success, +19 new items, 20 observations, 0 failed URLs, traversal paused (Reached max requests limit (20))
total: 1 run, 0 failed, +19 new items, 20 observations, 0 failed URLs

The run continued from the queue: it fetched the waiting book, the second listing page, and the books that page linked to. databrew status now counts 38 items and 40 completed URLs. Without -n, a run continues until the queue is empty, which completes the traversal. The next run then starts an incremental one: it stops following pagination at a page it has read in full before whose books are all stored.

6. Query the data¶

The records are Parquet files under data/books/:

data/books/
├── items/part_000001.parquet           # one row per book version
├── items/part_000002.parquet
├── observations/items/part_000001.parquet   # one row per card seen
├── observations/items/part_000002.parquet
├── .state.db                           # the queue and the traversal
├── .last_run.json                      # the last run's report
└── …                                   # itemstore's catalogs

Each row holds key, item_key, url, fetched_at, and the extracted fields as a JSON column, data. DuckDB reads them directly:

SELECT data->>'$.title' AS title,
       CAST(data->>'$.price' AS DOUBLE) AS price,
       data->>'$.availability' AS availability
FROM read_parquet('data/books/items/part_*.parquet', union_by_name = true)
ORDER BY price DESC
LIMIT 3;

DuckDB prints, with the long title cut for width:

┌───────────────────────────────────────────────────────────────────────┬────────┬─────────────────────────┐
│ title                                                                 │ price  │ availability            │
├───────────────────────────────────────────────────────────────────────┼────────┼─────────────────────────┤
│ Our Band Could Be Your Life: Scenes from the American Indie Undergro… │  57.25 │ In stock (19 available) │
│ Birdsong: A Story in Pictures                                         │  54.64 │ In stock (19 available) │
│ Sapiens: A Brief History of Humankind                                 │  54.23 │ In stock (20 available) │
└───────────────────────────────────────────────────────────────────────┴────────┴─────────────────────────┘

What you learned¶

• A config names the start URLs and says how to read listing pages and item pages.
• check and parse show what a config does before any crawl state exists.
• A run can stop at any point, and the next run resumes it.
• Records are plain files that any Parquet reader can query.

Next, write a config for your own site, or read how a crawl works to see what databrew does between runs. To drive the same crawl from Python, see databrew.run.