CLI¶

Each crawl command has a library call that does the same thing. init and list work on files only.

Command	Does
`databrew run CONFIG...`	Crawl each config
`databrew check CONFIG...`	Validate configs, build their components, and print their plans
`databrew parse CONFIG URL`	Show what one page extracts, as JSON
`databrew status CONFIG`	Show what a storage path holds
`databrew report PATH...`	Summarize the last run of each storage path
`databrew reactivate CONFIG`	Return permanently failed URLs to the queue
`databrew init [NAME]`	Write a starter config
`databrew list DIR`	List the configs in a directory

databrew --version prints the installed version. Exit codes are at the end.

run¶

databrew run CONFIG... [-q | -v] [-n N] [-c N] [--start-urls FILE] [--rescan | --retry-failed]
             [--passes N|auto] [--parallel N] [--deadline SECONDS] [--fail-when any|all]

Option	Effect
`-q`, `--quiet`	Print only warnings and the summary
`-v`, `--verbose`	Add a line per page and databrew's debug detail
`-n`, `--max-requests N`	Fetch attempts, retries included, overriding `[policy]`
`-c`, `--concurrency N`	Requests in flight, overriding `[policy]`
`--start-urls FILE`	Add one start URL per line of the file
`--rescan`	Begin an exhaustive traversal in place of the active one, keeping item work
`--retry-failed`	Fetch only failed URLs, in one pass, and queue what they link to for the next run
`--passes N\\|auto`	Retry failed URLs in later passes, with what recovered pages link to, up to N passes in all, or until one recovers no URL
`--parallel N`	Run N configs at once
`--deadline SECONDS`	Stop every run this long after the process started
`--fail-when any\\|all`	Exit 1 when any run, or every run, failed

--rescan and --retry-failed exclude each other, and so do -q and -v. --retry-failed also excludes --start-urls and a --passes other than 1, and charges a failed run as a normal run does, so a URL can become permanently failed in it. --start-urls joins the traversal the run begins, and every run must name the same file until that traversal completes.

By default a run prints, on stderr, one progress line per config per minute and any warnings. With more than one config, it also logs each config's result, prefixed Result:, when that config ends. The summary goes to stdout after the last config ends:

2026-11-07 10:12:00,123 - INFO - [ke] 12m: 3412 requests (284/min), +120 new, 5 changed, 340 observations, 2 failed, 1204 pending
ke: success, +120 new items, 5 changed, 340 observations, 2 failed URLs, traversal paused
total: 1 run, 0 failed, +120 new items, 5 changed, 340 observations, 2 failed URLs

A summary line says traversal paused when listing pages wait for the next run, and counts coverage gaps and empty runs in a row when there are any. A config that cannot load prints its error and counts as failed, and the others run; so do configs that never started, as how a run ends describes.

check¶

databrew check CONFIG... [-v]

Loads each config, builds its components, and prints its plan beside what its storage path holds. -v adds the start URLs and the fields each page type keeps. Every config is checked, whatever the one before it did, and the exit code is the worst of them.

parse¶

databrew parse CONFIG URL [--page-type listing|item] [--file PAGE] [--save PAGE]

Fetches one page, with no retry, and prints what the config extracts from it as JSON, writing no crawl state. The JSON holds url, page_type, status, items, pagination_links, item_links, diagnostics, and error, and the exit code is 1 when error is set.

Option	Effect
`--page-type`	Read with `[listing]` (the default) or `[item]`
`--file PAGE`	Read the page from a file instead of fetching it
`--save PAGE`	Keep the fetched body in a file

The middleware's pre_fetch, post_fetch, and post_extract run; on_error and on_challenge do not. --file skips the fetch and pre_fetch.

status¶

databrew status CONFIG

Prints what the config's storage path holds: item files and items, URLs pending, completed, and failed, and the traversal with the listing pages it has to go. Exits 1 when the path holds no state.

report¶

databrew report PATH... [--fail-when any|all] [--stale-after N]

Prints the last run of each storage path, read from its .last_run.json, in the summary's form. Each path is a storage path or a directory that holds storage paths. Writes nothing.

Option	Effect
`--fail-when any\\|all`	Exit 1 when any last run, or every one, failed
`--stale-after N`	Exit 1 when a storage path has N runs in a row that stored no new item and no change

A path without .last_run.json, or with one that cannot be read, is a failed line named by its path.

reactivate¶

databrew reactivate CONFIG [--url PATTERN] [--dry-run]

Returns permanently failed URLs to the queue with no failed runs on record. --url keeps the URLs that contain the text, or that match it entirely as a glob when it holds *, ?, or [. --dry-run lists them and changes nothing. A reactivated listing page competes by queue age with other listing pages; a reactivated item page follows fresh work.

init¶

databrew init [NAME] [--url URL] [--type html|json] [-o FILE]

Writes a starter config to NAME.toml, or to -o FILE, asking for a name or a start URL that the command line leaves out. --type json writes a JSON listing with path, key, a few fields, and a pagination path, and no [item]. An existing file is left as it is.

list¶

databrew list DIR

Lists the configs in a directory and whether each loads, leaving out files whose names start with _, which are fragments for extends. Exits 1 when none loads.

Exit codes¶

Code	Means
0	Every run passed `--fail-when`; `success` and `partial` both pass.
1	A run failed by `--fail-when`, a storage path could not be read or written or held no state, or another crawl held it.
2	An invalid option, or a config that could not load.
130	Ctrl+C outside a crawl, such as while configs load.

--fail-when any, the default, exits 1 when any run failed; --fail-when all only when every run failed. An interrupt during a crawl ends that run as interrupted, and the exit code follows --fail-when. A config that loads but cannot be built, such as one that names an unknown parser, exits 2 from check. run reports it as a failed run with the error stop, so it exits 1 and writes .last_run.json.