Middleware¶

A middleware changes how databrew handles each URL: it adds headers, recovers from errors, edits what extraction found, or ends the run. It subclasses Middleware and overrides the hooks it needs; each hook receives the MiddlewareContext of one URL and returns it:

site_middleware.pysite.toml

from databrew import Middleware, MiddlewareContext


class SkipArchived(Middleware):
    async def pre_fetch(self, ctx: MiddlewareContext) -> MiddlewareContext:
        if "/archive/" in ctx.url:
            ctx.skip = True
        return ctx

[[middleware]]
use = "site_middleware:SkipArchived"

[[middleware]] says how a config names a class and passes it arguments. Instances passed to run follow the configured entries in the list.

Hook order¶

Hook	Runs	Can
`reset()`	At the start of each run	Clear state kept from an earlier run
`pre_fetch(ctx)`	Before the fetch, in list order	Change `ctx.url` or `ctx.headers`, set `ctx.skip`
`post_fetch(ctx)`	After a successful fetch, in list order	Replace `ctx.response`, set `ctx.error`
`post_extract(ctx)`	After extraction, in reverse order	Replace `ctx.extract_result` with `dataclasses.replace`
`on_challenge(ctx)`	On a `challenge` error, in reverse order	Refresh an identity, clear `ctx.error`
`on_error(ctx)`	On any error, in reverse order	Clear `ctx.error`, or change it

The fetch-side hooks run in list order and the result-side hooks in reverse, so the first middleware in the list wraps the others.

Setting ctx.skip in pre_fetch marks the URL done without fetching it.
Clearing ctx.error in on_challenge or on_error retries the URL within max_retries.
Setting ctx.stop_reason in any hook ends the run with the blocked stop and that reason, once the requests in flight finish.
A pre_fetch, post_fetch, or post_extract that raises fails its URL.
A 304 Not Modified answer to a revisit skips post_fetch and post_extract.
The built-in backends report a CDN block, such as a 403 challenge page, as an HTTP error that only on_error sees. on_challenge runs for a challenge error, which a middleware sets with CrawlError.challenge or a fetcher of your own returns.
ctx.headers needs an HTTP backend; see use a browser.

When a run ends, databrew calls aclose() or close() on each [[middleware]] entry that has one. A caller that passes instances to run closes them itself. Authenticate requests is a worked example.

databrew.Middleware ¶

The base of a middleware, whose hooks each take a context and return it.

An instance passed to several runs serves each of them, so the state it keeps lasts from one to the next until reset clears it.

reset ¶

reset() -> None

Clear per-run state; a run calls this first and logs what it raises.

Source code in src/databrew/middleware.py

def reset(self) -> None:
    """Clear per-run state; a run calls this first and logs what it raises."""

pre_fetch ¶

pre_fetch(ctx: MiddlewareContext) -> MiddlewareContext

Change ctx.url or ctx.headers before the fetch, or skip it with ctx.skip.

Source code in src/databrew/middleware.py

async def pre_fetch(self, ctx: MiddlewareContext) -> MiddlewareContext:
    """Change ``ctx.url`` or ``ctx.headers`` before the fetch, or skip it with ``ctx.skip``."""
    return ctx

post_fetch ¶

post_fetch(ctx: MiddlewareContext) -> MiddlewareContext

Read or replace ctx.response after a successful fetch, before extraction.

Source code in src/databrew/middleware.py

async def post_fetch(self, ctx: MiddlewareContext) -> MiddlewareContext:
    """Read or replace ``ctx.response`` after a successful fetch, before extraction."""
    return ctx

post_extract ¶

post_extract(ctx: MiddlewareContext) -> MiddlewareContext

Change ctx.extract_result before its items are stored and its links queued.

Source code in src/databrew/middleware.py

async def post_extract(self, ctx: MiddlewareContext) -> MiddlewareContext:
    """Change ``ctx.extract_result`` before its items are stored and its links queued."""
    return ctx

on_error ¶

on_error(ctx: MiddlewareContext) -> MiddlewareContext

See a failure in ctx.error, and clear it to fetch again within max_retries.

Source code in src/databrew/middleware.py

async def on_error(self, ctx: MiddlewareContext) -> MiddlewareContext:
    """See a failure in ``ctx.error``, and clear it to fetch again within ``max_retries``."""
    return ctx

on_challenge ¶

on_challenge(ctx: MiddlewareContext) -> MiddlewareContext

See a CDN challenge in ctx.error before on_error, and clear it to fetch again.

A hook here can refresh a cookie or an identity, or end the run with ctx.stop_reason. The built-in backends report a CDN block as an HTTP error, which only on_error sees; a challenge comes from a middleware or a fetcher that returns CrawlError.challenge.

Source code in src/databrew/middleware.py

async def on_challenge(self, ctx: MiddlewareContext) -> MiddlewareContext:
    """
    See a CDN challenge in ``ctx.error`` before ``on_error``, and clear it to fetch again.

    A hook here can refresh a cookie or an identity, or end the run with ``ctx.stop_reason``.
    The built-in backends report a CDN block as an HTTP error, which only ``on_error`` sees;
    a challenge comes from a middleware or a fetcher that returns ``CrawlError.challenge``.
    """
    return ctx

databrew.MiddlewareContext ¶

MiddlewareContext(
    url: str,
    url_type: str = "item",
    headers: dict[str, str] = dict(),
    response: FetchResult | None = None,
    extract_result: ExtractResult | None = None,
    error: CrawlError | None = None,
    skip: bool = False,
    stop_reason: str | None = None,
    data: dict[str, Any] = dict(),
)

The state of one URL as it passes through the hooks, each of which may change it.

url ¶

url: str

The URL to fetch, which pre_fetch may change.

url_type ¶

url_type: str = 'item'

Page type: listing or item.

headers ¶

headers: dict[str, str] = field(default_factory=dict)

Headers to send with the request, which pre_fetch may set.

response ¶

response: FetchResult | None = None

Fetch response, including status and headers, when a request completed.

extract_result ¶

extract_result: ExtractResult | None = None

Extraction result, from post_extract on.

error ¶

error: CrawlError | None = None

The failure that on_challenge and on_error see.

skip ¶

skip: bool = False

Set to True in pre_fetch to mark the URL done without a fetch.

stop_reason ¶

stop_reason: str | None = None

Set in any hook to end the run with the blocked stop and this detail once in-flight URLs finish.

data ¶

data: dict[str, Any] = field(default_factory=dict)

Values that one hook leaves for a later one.

databrew.CrawlError ¶

CrawlError(
    category: str,
    message: str,
    retryable: bool = False,
    http_status: int | None = None,
    retry_after: float | None = None,
)

A failure of one URL, carried on MiddlewareContext.error.

A fetch failure keeps fetchkit's category, and the retry table of category and status decides whether it retries. Extractkit failures are non-retryable and use the "extraction" category.

category ¶

category: str

message ¶

message: str

retryable ¶

retryable: bool = False

http_status ¶

http_status: int | None = None

retry_after ¶

retry_after: float | None = None

from_fetch ¶

from_fetch(
    error: FetchError, status: int | None
) -> CrawlError

Convert a fetchkit failure, judging it by the retry table.

Source code in src/databrew/middleware.py

@classmethod
def from_fetch(cls, error: FetchError, status: int | None) -> CrawlError:
    """Convert a fetchkit failure, judging it by the retry table."""
    return cls(
        category=error.category.value,
        message=error.message,
        retryable=is_retryable(error.category, status),
        http_status=status,
        retry_after=error.retry_after,
    )

challenge ¶

challenge(
    message: str,
    *,
    retryable: bool = True,
    retry_after: float | None = None,
) -> CrawlError

Return a CDN-challenge error, which retries unless retryable is False.

A retry suits a challenge that a fresh cookie or identity may pass. retryable=False fails the URL at once.

Source code in src/databrew/middleware.py

@classmethod
def challenge(
    cls,
    message: str,
    *,
    retryable: bool = True,
    retry_after: float | None = None,
) -> CrawlError:
    """
    Return a CDN-challenge error, which retries unless ``retryable`` is False.

    A retry suits a challenge that a fresh cookie or identity may pass. ``retryable=False``
    fails the URL at once.
    """
    return cls(
        category=ErrorCategory.CHALLENGE,
        message=message,
        retryable=retryable,
        retry_after=retry_after,
    )