Python API¶

Each crawl command has a call that does the same work, so a script or a scheduler can drive databrew without a shell. databrew.__all__ lists the supported names. This script runs a crawl and fails the way databrew run does:

import asyncio
import sys

import databrew


async def main() -> int:
    report = await databrew.run("books.toml", options=databrew.RunOptions(max_requests=100))
    print(report.name, report.stop, report.outcome, report.stats.new_items)
    return 1 if report.outcome == databrew.Outcome.FAILED else 0


sys.exit(asyncio.run(main()))

run, run_many, parse, and check are coroutines. Where a call takes config, it takes a path or a loaded WebsiteConfig. databrew run goes through run_many even for one config, so a config that cannot load, or whose storage path another process holds, becomes a failed report; run raises ConfigError or BlockingIOError instead.

Logging¶

Databrew logs to the databrew logger and its children and configures no handler; the command does that. The progress line the command prints each minute, and each config's Result: line in a batch, come from databrew.progress at INFO. RunNameFilter on a handler sets record.run to the name of the config that logged each record, as the command's [ke] prefix shows:

import logging

import databrew

handler = logging.StreamHandler()
handler.addFilter(databrew.RunNameFilter())
logging.basicConfig(
    level=logging.WARNING, format="%(levelname)s [%(run)s] %(message)s", handlers=[handler]
)
logging.getLogger("databrew.progress").setLevel(logging.INFO)