How a run ends¶

Every run ends with one verdict: a stop, which says why the run ended, and an outcome, which says how it went. The exit code, .last_run.json, databrew report, and the RunReport that the library returns all read the same verdict, so a scheduled crawl and a person at a terminal never disagree about a run.

Why it stopped¶

The stop is a Stop.

Stop	Means
`queue_empty`	No URL is left to fetch, including the retries the run scheduled and waited for. Coverage gaps can remain. A retry pass that trips a stop rule also reports `queue_empty`, because the first pass emptied the queue.
`max_requests`	The request budget, `-n` or `[policy].max_requests`, ran out.
`consecutive_failures`	`max_consecutive_failures` requests in a row failed after their retries.
`error_rate`	The share of requests that failed after their retries passed `max_error_rate`.
`blocked`	A middleware set `ctx.stop_reason`, for example after a refresh failed.
`interrupted`	A signal, such as Ctrl+C, or `--deadline` stopped the run.
`error`	Setup or the crawl raised an error.

A stop ends the run without losing work: what was stored stays stored, and every unfinished URL waits in the queue for the next run. The detail field adds the specifics, such as Reached max requests limit (20) or the middleware's reason.

The two stop rules, consecutive_failures and error_rate, exist to end a run that a site is blocking before it charges failures to every URL in the queue. A 404, a 410, and a page that extraction cannot use are the site's answer about one URL, so they never count toward either rule.

How it went¶

A run is productive when it stored a new item or a change. The outcome, an Outcome, is set by the first of these rules that matches:

failed when no request succeeded while a failure or a coverage gap remains, or when the run fetched a start URL and no listing page yielded an item or an item link.
success when the run stopped on queue_empty or max_requests.
partial when the run was productive, or was interrupted after a request succeeded.
failed otherwise.

The first rule makes silent failure loud. A crawl whose start URLs have all failed for good would otherwise stop on queue_empty and succeed on every schedule while fetching nothing. The empty-page part applies only to runs that fetched a start URL, so a run that resumes a traversal and reaches only an empty last page still succeeds.

A run can succeed with coverage gaps when it fetched healthy pages. The summary line and the report count the gaps, so they stay visible. The report also keeps the empty-run streak: the number of runs in a row that were not productive, which databrew report --stale-after reads.

A config that never started¶

A config that a --deadline or a signal stops before it starts gets a failed report with the detail "Not started". A config whose storage path another process holds gets a failed report with the error stop and a detail naming the path; run raises BlockingIOError instead. Either way the config writes nothing to its storage path, so .last_run.json and databrew report keep describing the last run that did start.

The exit code of databrew run follows the outcomes: success and partial exit 0, and with --fail-when any, the default, any failed run exits 1. Exit codes lists the rest.