Use a browser or your own transport¶

Databrew fetches with plain HTTP by default, through fetchkit. This guide covers the cases that need more: a proxy, a site that checks the TLS fingerprint, a page that needs JavaScript, or a transport of your own. [fetch].backend picks one of fetchkit's three backends, and the table's other keys go to that backend's constructor.

Install an extra with the package:

uv tool install "databrew[zendriver] @ git+https://github.com/eyayaw/databrew"

A proxy or a browser's TLS fingerprint¶

The HTTP backends take a proxy, and curl_cffi impersonates a browser's TLS fingerprint:

[fetch]
backend = "curl_cffi"
proxy = "http://proxy.example:8080"

fetchkit's HTTP backends lists their options, among them timeout, headers, and http_version.

A browser for JavaScript pages¶

[fetch]
backend = "zendriver"
headless = true  # the default opens a window, which a server or CI runner lacks
timeout = 30.0

databrew parse CONFIG URL --save page.html shows whether a page needs it: if the saved HTML from the default backend lacks the cards you see in the browser, the page builds them with JavaScript.

The browser backend works in up to max_tabs tabs, which databrew sets from concurrency unless [fetch] sets it. A browser is slower and heavier than an HTTP client, so keep concurrency low. Two features of the HTTP backends are missing:

• No per-request headers. A middleware that sets ctx.headers makes every request fail. Static headers in [fetch] are not accepted either.
• No conditional revisits. A revisit fetches the whole page instead of asking whether it changed.

The browser's other options are in fetchkit's browser backend docs.

Sites behind a challenge¶

Some sites answer a crawler with a challenge page, such as Cloudflare's, instead of content. The built-in backends report it as the HTTP error it arrives with, often a 403, which is not retried and which only a middleware's on_error sees. Databrew does not solve challenges: how to pass one depends on the site, and that knowledge belongs in your code. fetchkit's browser offers ZendriverFetcher.page(url) and its solve_challenge(), so a fetcher of your own, described next, can wrap the browser and try a solve. A fetcher that returns the challenge category gets the URL retried and lets a middleware's on_challenge react.

Your own fetcher¶

For any transport fetchkit does not offer, write a class with two async methods, fetch and close, and pass an instance to run. This one fetches with aiohttp, which you install yourself:

import asyncio

import aiohttp
from fetchkit import ErrorCategory, FetchError, FetchResult

import databrew


class AiohttpFetcher:
    """Fetch with aiohttp, returning failures as results the way fetchkit's backends do."""

    def __init__(self) -> None:
        self._session = aiohttp.ClientSession()

    async def fetch(self, url: str, headers: dict[str, str] | None = None) -> FetchResult:
        try:
            async with self._session.get(url, headers=headers) as response:
                text = await response.text()
        except (aiohttp.ClientError, TimeoutError) as error:
            return FetchResult(url=url, error=FetchError(ErrorCategory.NETWORK, str(error)))
        result = FetchResult(url=str(response.url), status_code=response.status, text=text)
        if response.status == 429:
            result.error = FetchError(ErrorCategory.RATE_LIMITED, "HTTP error: 429")
        elif response.status >= 400:
            category = ErrorCategory.SERVER if response.status >= 500 else ErrorCategory.CLIENT
            result.error = FetchError(category, f"HTTP error: {response.status}")
        return result

    async def close(self) -> None:
        await self._session.close()


async def main() -> None:
    report = await databrew.run("site.toml", fetcher=AiohttpFetcher())
    print(report.stop, report.outcome)


if __name__ == "__main__":
    asyncio.run(main())

The fetcher returns failures as a FetchResult with an error rather than raising, and its category and status decide whether databrew retries, by the retry table. A fetch that raises fails its URL without a retry. Databrew awaits fetch and close without a timeout of its own, so both must finish in bounded time; run closes the fetcher when the run ends.

A supplied fetcher replaces the one [fetch] describes, for that run only. databrew check and the run command still build the configured backend, so a custom fetcher is for runs started from Python. Revisits through it fetch the whole page, because databrew cannot know whether it sends per-request headers.