Write a config for an HTML site¶

Most sites show their records as HTML: listing pages of cards, each linking to one page per record. The tutorial shows a finished config for such a site; this guide builds one for a site you do not know yet. The selectors, field options, and parsers are extractkit's, and its config reference lists every option left out here.

Start from a template¶

databrew init mysite --url "https://example.com/listings?page=1"

init writes mysite.toml with placeholder selectors. Start from the first listing page, sorted newest first if the site allows it, so an incremental traversal finds new records on the first pages and stops early.

Find the cards¶

In the browser, right-click a record in the listing and choose Inspect. Look for the element that wraps one whole record and repeats once per record, often an article, li, or div with a telling class. Its CSS selector picks the cards, and each field is a selector relative to the card:

[listing.items]
selector = "article.listing-card"

[listing.items.fields]
url = { selector = "a.title", attribute = "href" }  # an attribute
title = "a.title"                                    # the element's text
price = { selector = ".price", parser = "parse_float" }

Check it against the live page with parse:

databrew parse mysite.toml "https://example.com/listings?page=1"

The JSON shows each card under items. An empty items means the selector matched nothing, and diagnostics says so as items_unmatched. A page that builds its listing with JavaScript shows nothing to a plain HTTP fetch: save it with --save page.html, and if the cards are missing from the file, switch to the browser backend.

Follow the item links and the next page¶

[listing.links] says what to queue from each listing page:

[listing.links]
items = "url"                 # the card field that holds the item page's URL
pagination = ["a[rel=next]"]  # selectors for the next listing page

Relative links resolve against the page's URL, or against base_url when the table sets one. parse shows the result as item_links and pagination_links; confirm both hold absolute URLs that lead where you expect. When the next page has no link to select, a pagination builder computes it in Python, as the JSON API guide shows.

Read the item page¶

[item] reads the page of one record. Without items.selector, the whole page is one item:

[item.items.fields]
title = "h1"
price = { selector = ".product-price", parser = "parse_float" }
description = { selector = ".description", required = true }

required = true drops an item that lacks the field, so a broken page is not stored as a version. Check an item page with databrew parse mysite.toml --page-type item URL.

If the cards already hold everything you need, leave out [item] and [listing.links].items, which queues item pages. Each card is then stored as a version, and the config needs [listing.items].key to tell records apart.

Give records a stable key¶

Without a key, databrew identifies a record by a digest of its item page's URL, which works until the site changes its URLs. When the cards show the site's own ID, use it:

[listing.items]
selector = "article.listing-card"
key = "listing_id"

[listing.items.fields]
listing_id = { selector = "a.title", attribute = "data-id" }

An item page takes the key of the card that linked to it; keys gives the full order.

Leave out fields that change on every fetch, such as a view counter or a "posted 3 hours ago" label, or normalize them in a parser: each change stores a new version.

Write your own parser¶

A parser is a Python function that receives the raw value and returns the value to store, or None for absent. Put it in a file beside the config and name the file, without .py, in parsers:

def listing_id(href: str) -> str:
    """Return the record ID from a link such as /listing/123-blue-bike."""
    return href.rsplit("/", 1)[-1].split("-", 1)[0]

parsers = ["mysite_parsers"]

[listing.items.fields]
listing_id = { selector = "a.title", attribute = "href", parser = "listing_id" }

extractkit's built-in parsers cover numbers, booleans, whitespace, and embedded JSON. A parser that raises leaves the value absent and adds a parser_failed diagnostic, which parse shows.

Share settings between configs¶

Configs for related sites can share a base with extends, a path relative to the config:

[policy]
concurrency = 4
max_requests_per_minute = 60

[storage]
compression = "zstd"

extends = "_shared.toml"
name = "site-a"
start_urls = ["https://a.example.com/list"]

extends gives the merge rules. Start a fragment's name with _ so that databrew list leaves it out.

Check the whole config¶

databrew check mysite.toml
databrew run mysite.toml -n 20

check builds the extractors and the fetcher, so a wrong parser name or option fails here rather than mid-crawl. A short run with -n then shows the crawl working end to end.