Configs¶

load_config reads a TOML file into a WebsiteConfig, and check goes on to build what a run needs from it. Every call that takes config accepts a path or a loaded WebsiteConfig. The config reference describes the file.

A WebsiteConfig is a pydantic model whose fields follow the file: name, start_urls, listing, item, parsers, policy, revisit, and middleware, with [fetch] and [storage] as fetch_config and storage_config and the resolved storage_path. The databrew run flags that override a config for one run are RunOptions.

databrew.load_config ¶

load_config(path: Path | str) -> WebsiteConfig

Load and validate a TOML config file.

extends, start URL files, and parser modules resolve relative to the file, and a middleware module beside it imports from there. A file merges over the one its extends names: tables merge, other values replace, and its [[middleware]] entries follow the parent's. Loading builds no fetcher or extractor, so an option those reject, such as an unknown parser name, passes here and fails in check or run.

Raises:

Type	Description
`ConfigError`	The file cannot be read or does not validate. The message names the file and fits on one line.

Source code in src/databrew/config.py

def load_config(path: Path | str) -> WebsiteConfig:
    """
    Load and validate a TOML config file.

    ``extends``, start URL files, and parser modules resolve relative to the file, and a
    middleware module beside it imports from there. A file merges over the one its ``extends``
    names: tables merge, other values replace, and its ``[[middleware]]`` entries follow the
    parent's. Loading builds no fetcher or extractor, so an option those reject, such as an
    unknown parser name, passes here and fails in ``check`` or ``run``.

    Raises:
        ConfigError: The file cannot be read or does not validate. The message
            names the file and fits on one line.
    """
    # Do not resolve symlinks here: config-relative extends, URL files, and
    # parser modules belong to the user-facing config location.
    path = Path(path).expanduser().absolute()
    try:
        raw = _load_raw_config(path)
        unknown = raw.keys() - _TOP_LEVEL_FIELDS
        if unknown:
            names = ", ".join(sorted(unknown))
            raise TypeError(f"Unknown top-level config field(s): {names}")
        normalized = _raw_to_websiteconfig_dict(raw, path.parent, default_name=path.stem)
        return WebsiteConfig.model_validate({**normalized, "path": path})
    except Exception as error:
        raise ConfigError(f"{path}: {_describe(error)}") from error

databrew.WebsiteConfig ¶

Bases: BaseModel

A validated config, with its extends chain merged.

databrew.ConfigError ¶

Bases: ValueError

A config that cannot be loaded, or whose components cannot be built.

databrew.check ¶

check(config: str | Path | WebsiteConfig) -> Plan

Build the components of config, close them, and return the crawl it describes.

Raises:

Type	Description
`ConfigError`	The config cannot be loaded, or a component rejects it.
`StoreError`	The crawl state cannot be read with this version.

Source code in src/databrew/commands.py

async def check(config: str | Path | WebsiteConfig) -> Plan:
    """
    Build the components of ``config``, close them, and return the crawl it describes.

    Raises:
        ConfigError: The config cannot be loaded, or a component rejects it.
        StoreError: The crawl state cannot be read with this version.
    """
    loaded = _loaded(config)
    components = build_components(loaded)
    await close_fetcher(components.fetcher)
    await close_middleware(components.middleware)
    return Plan(
        name=loaded.name,
        path=loaded.path,
        listing_type=page_format(loaded.listing),
        start_urls=[str(url) for url in loaded.start_urls],
        fetch_backend=loaded.fetch_config.backend,
        concurrency=loaded.policy.concurrency,
        max_requests=loaded.policy.max_requests,
        max_requests_per_minute=loaded.policy.max_requests_per_minute,
        after_days=loaded.revisit.after_days,
        watch=loaded.revisit.watch,
        status=status(loaded),
        listing_fields=item_field_names(loaded.listing),
        item_fields=item_field_names(loaded.item) if loaded.item else [],
    )

databrew.Plan ¶

Plan(
    name: str,
    path: Path | None,
    listing_type: str,
    start_urls: list[str],
    fetch_backend: str,
    concurrency: int,
    max_requests: int | None,
    max_requests_per_minute: float | None,
    after_days: float | None,
    watch: list[str],
    status: Status,
    listing_fields: list[str] | None = list(),
    item_fields: list[str] | None = list(),
)

The crawl a valid config describes, beside what its storage path holds already.

name ¶

name: str

path ¶

path: Path | None

listing_type ¶

listing_type: str

start_urls ¶

start_urls: list[str]

fetch_backend ¶

fetch_backend: str

concurrency ¶

concurrency: int

max_requests ¶

max_requests: int | None

max_requests_per_minute ¶

max_requests_per_minute: float | None

after_days ¶

after_days: float | None

watch ¶

watch: list[str]

status ¶

status: Status

listing_fields ¶

listing_fields: list[str] | None = field(
    default_factory=list
)

The fields a listing item can hold, or None when a JSON item is stored as extracted.

item_fields ¶

item_fields: list[str] | None = field(default_factory=list)

The same for an item page's item, and empty without [item].