Config¶

A config is one TOML file per site. Unknown keys fail when the file loads, with a one-line ConfigError that names the file, before any state is opened. [fetch] options are checked when check or run builds the fetcher.

name = "books"                    # default: the file's stem
extends = "_shared.toml"          # a config to merge over, relative to this file
start_urls = ["https://..."]      # or { file = "urls.txt" }, one URL per line
parsers = ["site_parsers"]        # site_parsers.py beside this file

[listing]                         # how to read listing pages
[item]                            # how to read item pages
[fetch]                           # the fetch backend and its options
[storage]                         # where and how records are stored
[policy]                          # pacing, retries, and stop rules
[revisit]                         # when to fetch known item pages again
[[middleware]]                    # extension classes, in order

Top-level keys¶

name: The config's name, used in logs, reports, and the default storage path. Defaults to the file's stem.
extends: A path, relative to this file, to a config this one builds on. Tables merge key by key, a list replaces the parent's list, and [[middleware]] entries follow the parent's in the list, whose order sets the hook order. A chain of extends is allowed; a cycle is an error.
start_urls: The listing pages a traversal starts from: a list, or { file = "path" } with one URL per line, relative to this file. databrew run --start-urls FILE adds start URLs to the traversal that run begins, and every run must name the same file until that traversal completes, or begin a new one with --rescan, because a paused traversal keeps its start URLs.
parsers: Names of Python files beside this config, without .py. The public functions each file defines join extractkit's built-in parsers. Unlike a middleware module, a parser module is never looked up among installed packages.

`[listing]` and `[item]`¶

Each holds one extractkit config: type, which is html (the default) or json, plus items, links, and the other options of extractkit's config reference. The two page types may have different types.

[listing]
type = "json"

[listing.items]
path = "data.items"
key = "id"
fields.id = "id"
fields.url = { path = "id", parser = "item_url" }

[listing.links]
items = "url"
pagination = ["links.next"]

[listing.items].key: The field that identifies a record. Required when there is no [item].
[listing.links].items: The card field that holds the item page's URL. Needs [item].
[listing.links].pagination: Selectors or paths of the next listing pages.
[item.items].key: The field that identifies a record on an item page whose card had no key.

Without [item], the listing cards are the versions, identified by their keys.

`[fetch]`¶

backend is httpx (the default), curl_cffi, or zendriver. Every other key goes to that backend's constructor, as fetchkit documents for the HTTP backends and the browser backend.

[fetch]
backend = "curl_cffi"
timeout = 20.0
headers = { "Accept" = "application/json" }

zendriver takes max_tabs from [policy].concurrency unless [fetch] sets it, and accepts no headers; use a browser or your own transport compares the backends.

`[storage]`¶

Key	Default	Means
`path`	`data/<name>`	The storage path, relative to the working directory
`format`	`parquet`	`parquet` or `jsonl`
`compression`	itemstore's	A codec that itemstore accepts for the format
`part_size_mb`	itemstore's	The size a part file aims for

An existing store checks format and compression against its manifest, and keeps the compression and part size recorded there when the table names none.

`[policy]`¶

Key	Default	Means
`concurrency`	5	Requests in flight
`max_requests_per_minute`	none	Upper bound on the fetch rate
`jitter`	0.1	Upper bound, in seconds, on a random wait before each request
`warmup_seconds`	0.0	Seconds over which concurrency ramps up from 1
`adaptive_concurrency`	false	Halve the cap on failures and grow it back
`max_retries`	3	Retries per URL within a run
`retry_delay`	1.0	Seconds before the first retry
`backoff_factor`	2.0	Multiplies the delay after each retry
`max_retry_after`	600.0	Cap, in seconds, on every retry delay, computed or asked for by `Retry-After`
`max_requests`	none	Fetch attempts a run may make, retries included
`max_consecutive_failures`	50	Failures in a row, after retries, that stop the run
`max_error_rate`	0.5	Share of failures, after retries, above which the run stops
`error_rate_window`	none	The most recent outcomes that `max_error_rate` reads
`max_failed_runs`	3	Runs in a row a URL may fail before it is permanently failed

Pace requests and avoid blocks shows how to choose values.

Retries¶

These failures are retried within a run, up to max_retries. A middleware that sets CrawlError.retryable decides for its own error. The built-in backends never report challenge; a middleware or a fetcher of your own does.

Category	Statuses
`network`	No response
`server`	500, 502, 503, 504, 520, 521, 522, 523, 524
`rate_limited`	429
`client`	408
`challenge`	Any

`[revisit]`¶

Key	Means
`after_days`	Queue item pages last fetched longer ago than this many days
`watch`	`[listing]` fields whose change queues the item page

[revisit] needs [item]; revisit known records shows how to use it.

`[[middleware]]`¶

[[middleware]]
use = "site_auth:RefreshToken"
token_url = "https://example.com/oauth/token"

use names a Middleware subclass as module:Class. A module beside the config is imported from there, and any other from the installed packages. The other keys go to the constructor, and the config fails to load when the constructor does not accept them. The order of the entries sets the hook order.

Config¶

Top-level keys¶

[listing] and [item]¶

[fetch]¶

[storage]¶

[policy]¶