How a crawl works¶

A crawl is a queue of URLs that databrew works through, one run at a time. The queue outlives each run, so the crawl as a whole decides what to fetch next, when a walk through the listing pages is finished, and what happens to a URL that fails.

Two kinds of page¶

Every URL in the queue is a listing page or an item page, and each kind has its own extraction config in the TOML file.

• A listing page shows many records as cards. The start URLs are listing pages, and so is every pagination link found on one.
• An item page shows one record. A listing page's item links queue item pages. An item page that yields more than one item fails as an extraction error.

A listing page yields an observation per card, pagination links, which queue more listing pages, and item links, which queue item pages; the overview draws the flow.

A site without [item] has no item pages. Each card is then stored as a version too, so the listing pages carry the whole record, as observations and versions describes.

The queue¶

The queue lives in .state.db beside the records, so it survives the end of a run. A run takes URLs in this order, oldest first within each group:

1. New item URLs.
2. Listing pages, in the order they were queued. A listing page that failed earlier competes here by age with fresh ones.
3. Revisits of known item pages, the longest unfetched first.
4. Item URLs that failed in an earlier run.

Failed item URLs wait behind all fresh work, so an item page that keeps failing cannot take the budget of a run from pages that have not been tried.

Before queueing an item link, databrew checks whether that record is already stored, by its key or by its URL. A stored record is not fetched again unless a revisit asks for it.

Traversals¶

A traversal is one walk through the listing pages, from the start URLs to the last page it reaches. It can take as many runs as you like: a run that stops early leaves the traversal paused, and the next run resumes it without fetching any completed page again. Once the only listing pages left are ones that failed, the traversal is completed, and the next run begins a new one from the start URLs.

The first traversal of a crawl is exhaustive: it follows every pagination link on every page that has cards, so it reaches the whole site. Later traversals are incremental. They stop following a branch at a page that is caught up:

• an earlier fetch already expanded the page, recording its pagination links and items, and
• every card and item link on it belongs to an item that was stored before this traversal began.

On a site that lists the newest records first, that means an incremental traversal reads the new pages at the front and stops where the known records begin.

A few cases keep a branch going even in an incremental traversal, because they might hide something new:

• a card without a key, or a card that extraction discarded;
• a keyed card without an item link whose record is not stored;
• a page that has never been expanded, or whose last attempt failed or was cut off. Such a page follows its pagination in any traversal, so the pages behind it are still reached.

Items first stored during the current traversal count as new for this decision, even when another start URL shows them or the pages shift between runs. A watched change or a changed card does not keep a branch going; it queues a revisit or stores a version, and the traversal still stops.

A page with no cards, no item links, and nothing discarded ends its branch in any traversal.

databrew run --rescan replaces the active traversal with a new exhaustive one. It forgets every listing page, with its progress and failures, and keeps the item pages, their failures, and the stored records.

Failures and retries¶

A request that fails is retried within the run when the retry table says the failure is temporary, up to max_retries times with growing delays. A URL that runs out of retries is charged one failed run and returns in later runs. After max_failed_runs failed runs in a row, it is permanently failed and stays out of the queue until databrew reactivate or, for a listing page, --rescan.

Some failures are the site's answer about one URL rather than a sign of trouble: a 404, a 410, and a page that extraction cannot use. These count as failed runs in any run and never trip the stop rules. A 410 makes its URL permanently failed at once.

Until a run has a success, a failure cannot be told apart from a block. An item page that fails then, with a failure that could be a block, is marked maybe blocked and is not charged, so an outage does not use up item pages' budgets. If it has no failed run on record, it waits for the next normal run; otherwise --retry-failed and later passes fetch it too. A listing page is charged in any run.

An unresolved failure of a listing page is a coverage gap: pages behind it may be missing until it succeeds. A new traversal can begin while gaps remain, and the report counts them until they resolve.

Recover from failures shows what to do about each case.

Passes¶

A run normally makes one pass over the queue. With --passes N or --passes auto, a run whose queue empties with failed URLs left makes further passes over them, up to N passes in all, or until a pass recovers none. A later pass also fetches what its recovered pages link to. --retry-failed makes a single pass over the failed URLs only and leaves what they link to for the next run.