Recover from failures¶

Databrew retries temporary failures and keeps a record of the rest, so most failed URLs need nothing from you. The sections below cover those that do: URLs that keep failing, a blocked run, coverage gaps, and a storage path databrew refuses to open.

See what failed¶

databrew report data/mysite     # how the last run went, and why it stopped
databrew status mysite.toml     # the queue now: pending, completed, failed, the traversal

_failed_urls.json in the storage path lists every URL with a failure on record and its last error, written when a run closes. For the story of a run, run it again with -v, which logs every page.

Failed URLs that will retry on their own¶

A URL that fails after its retries within a run comes back in later runs, until it has failed max_failed_runs runs in a row (3 by default). Usually that is all it takes: a page that timed out today loads tomorrow.

When fresh work fills every run's budget, failed item pages wait behind it for a long time. Give them a run of their own:

databrew run mysite.toml --retry-failed

--retry-failed fetches only URLs with a failure on record, in one pass, including any that a stop left queued or in flight. What the recovered pages link to is queued for the next normal run. To retry within a normal run instead, --passes auto makes further passes over the failed URLs, and what recovered pages link to, until a pass recovers none.

Permanently failed URLs¶

After max_failed_runs failed runs in a row, a URL is permanently failed and stays out of the queue. A 410 Gone gets there at once. When the cause is fixed, return them to the queue:

databrew reactivate mysite.toml --dry-run          # list what would return
databrew reactivate mysite.toml --url "/category/" # only URLs that contain the text
databrew reactivate mysite.toml                    # all of them

import databrew

urls = databrew.reactivate("mysite.toml", dry_run=True)  # list what would return
urls = databrew.reactivate("mysite.toml", url="/category/")
urls = databrew.reactivate("mysite.toml")

--url, or url= in reactivate, keeps the URLs that contain the text, or that match it entirely as a glob when it holds *, ?, or [. Reactivated URLs start over with no failed runs on record.

A blocked run¶

A run that ends with the stop consecutive_failures or error_rate hit a wall: too many requests in a row, or too large a share of them, failed after their retries. That usually means the site is rate-limiting or blocking the crawler. The stop rules end the run early so that the rest of the queue is not charged with failures that say nothing about the pages themselves.

Item pages that failed before the run had a success, with a failure that could be a block, are marked maybe blocked rather than charged. Then:

1. Wait before the next run; many blocks lift by themselves.
2. Slow the crawl down, as pace requests and avoid blocks describes.
3. If the site asks for something the crawler does not send, such as a fresh token or a cookie, add a middleware.

Coverage gaps¶

A coverage gap is a listing page whose failure is not resolved yet. The pages behind it may be missing from the crawl until it loads. The summary line counts gaps, for example traversal paused, 2 coverage gaps. A gap closes by itself when its page succeeds on a later run.

When a listing page fails for good, nothing behind it is reached. If that happens to every start URL, for example after a long outage, a run fetches nothing and fails, which makes the problem visible. Two ways out:

databrew reactivate mysite.toml --url "/listing"   # retry the failed listing pages
databrew run mysite.toml --rescan                  # or begin a fresh exhaustive traversal

A storage error¶

If the disk fills or the records cannot be written, the run stops with the error stop and charges no URL. Requests in flight finish, and their work is kept. Fix the cause and run again; the unfinished requests return to the queue.

A refused storage path¶

databrew run refuses to open a storage path whose .state.db is missing beside earlier records, unreadable, or written by an incompatible databrew version. It refuses, rather than starting over, because starting over would quietly re-fetch the whole site and lose every failure on record. Restore .state.db from the same backup as the records, or run with the databrew version that wrote it; state and storage has the rules.