Schedule and resume crawls¶

A crawl rarely finishes in one sitting, and it does not need to: each run continues the same queue, so a schedule of short runs covers a site as well as one long run, and recovers from a crash or a blocked afternoon on its own.

Resume by running again¶

Run the same command. The queue lives in .state.db beside the records, and a URL that was in flight when a run stopped returns to the queue. Nothing needs to be cleaned up first, whether the last run ended on its budget, on Ctrl+C, or with a crash.

Bound each run¶

Two options end a run early with its queue intact:

databrew run site.toml -n 500            # at most 500 requests, retries included
databrew run site.toml --deadline 3000   # stop 50 minutes after the process started

-n overrides [policy].max_requests. --deadline counts from process start and suits a scheduler with a hard time limit: set it a few minutes below the limit so the run can close its stores cleanly.

Run several sites¶

One command can run many configs, and --parallel runs several at once. Each config needs its own storage path.

databrew run configs/*.toml --parallel 3 --deadline 3000

import asyncio
from pathlib import Path

import databrew

configs = sorted(Path("configs").glob("*.toml"))
reports = asyncio.run(databrew.run_many(configs, parallel=3, deadline=3000))

While the batch runs, stderr shows a progress line per config each minute and a Result: line as each one ends. stdout prints the summary, one line per config and a total, after the last one ends:

ke: success, +120 new items, 5 changed, 340 observations, 2 failed URLs, traversal paused
ng: success, +88 new items, 340 observations, 0 failed URLs
total: 2 runs, 0 failed, +208 new items, 5 changed, 680 observations, 2 failed URLs

The summary is plain text meant for logs and commit messages. A config that fails to load is reported as failed, and the others still run. run_many returns the same results as RunReport objects, in the order of the configs.

Decide when a scheduled run should fail¶

The exit code tells the scheduler whether to raise an alarm. run exits 1 when a run failed, as how a run ends defines it; --fail-when all relaxes that to when every run failed. A config that cannot load exits 2 after the others have run.

databrew report judges stores after the fact, without running anything, which suits a separate check step:

databrew report data/ --fail-when any --stale-after 6

It prints the last run of every store under data/ and exits 1 when one failed or when one has gone 6 runs in a row without storing a new item or a change. Runs can succeed and still be stale, for example when the site answers normally but its listing stopped updating, so the streak catches what the outcome cannot.

Keep state between machines¶

A scheduled crawl on a fresh machine, such as a CI runner, needs the previous run's storage path. Keep the records and .state.db together, and restore both before a run:

• items/, itemstore.json, and observations/, the records and their manifests;
• .state.db, the queue, traversals, and failures;
• .last_run.json, which carries the empty-run streak.

.itemstore.db is itemstore's catalog and journal; itemstore's format says when it can be left behind.

A sketch of a GitHub Actions job that crawls on a schedule and commits the result:

on:
  schedule:
    - cron: "0 */6 * * *"
permissions:
  contents: write
concurrency: crawl  # one crawl at a time, so two runs never share a storage path
jobs:
  crawl:
    runs-on: ubuntu-latest
    timeout-minutes: 60
    steps:
      - uses: actions/checkout@v7
      - uses: astral-sh/setup-uv@v10.3.3  # setup-uv publishes no major-version tag
      - name: Crawl
        run: >
          uvx --from "databrew @ git+https://github.com/eyayaw/databrew"
          databrew run configs/*.toml --parallel 3 --deadline 3300 > summary.txt
      - name: Commit the records
        if: always()
        run: |
          git config user.name "crawler"
          git config user.email "crawler@users.noreply.github.com"
          git add data
          git diff --cached --quiet || git commit -m "crawl: $(date -u +%F)" -m "$(cat summary.txt)"
          git push

--deadline 3300 leaves five minutes of the hour for closing the stores and committing. The commit step runs even when the crawl step fails, so the records and the queue of a failed run survive. Its body carries the summary, which puts each run's result in the history, and a push that fails fails the job.

Stop a run by hand¶

Ctrl+C, or SIGTERM from a scheduler, stops the crawl cleanly: requests in flight finish, unfinished URLs return to the queue, and the report records the run as interrupted. The next run resumes.