Pace requests and avoid blocks¶

A crawler that hits a site too hard gets blocked, and one that is too timid never finishes. The [policy] keys below set how fast a crawl sends requests, how it retries, and when it gives up on a site that refuses it.

[policy]
concurrency = 4                  # requests in flight at once
max_requests_per_minute = 60     # an upper bound on the rate, across all of them
jitter = 0.5                     # up to this many seconds of random wait before each request
warmup_seconds = 30              # start at 1 request in flight and ramp up over this long

Choose a speed¶

concurrency bounds how many requests are in flight, and max_requests_per_minute bounds how many start each minute. For a site you do not know, start slow, with concurrency at 2 to 4 and a rate of 30 to 60 requests a minute, and raise them only while runs stay clean. jitter spreads requests out so they do not arrive in lockstep, and warmup_seconds avoids a burst at the start of every run.

adaptive_concurrency = true lets databrew back off on its own: it halves the cap when more than 30% of the last 200 outcomes failed, retryable failures included. While failures stay below that, it grows the cap by one after each round of requests that used at least 90% of it.

When a site answers 429 Too Many Requests with a Retry-After header, databrew holds back every fetch for that long, capped by max_retry_after (600 seconds by default).

Retries¶

[policy]
max_retries = 3          # retries per URL within a run
retry_delay = 1.0        # seconds before the first retry
backoff_factor = 2.0     # each retry waits this much longer than the one before

With these defaults, a failing URL is tried at once, then after 1, 2, and 4 seconds. Only failures that may be temporary are retried, such as timeouts, server errors, and rate limits; the retry table lists them. A URL that still fails returns in the next run.

Give up when the site says no¶

Two stop rules end a run that is being blocked, before it charges failures to the whole queue:

[policy]
max_consecutive_failures = 50   # failures in a row, after retries
max_error_rate = 0.5            # share of failed requests, after retries
error_rate_window = 200         # read only the most recent 200 outcomes

The error rate waits for 20 outcomes before it can stop a run. error_rate_window makes it judge the recent past only, so a burst of failures early in a long run cannot stop it hours later. Answers about one URL, such as a 404, count toward neither rule, as how a run ends describes.

Bound the work¶

max_requests caps the fetch attempts of a run; schedule and resume crawls shows it beside --deadline. Transport options such as timeout and proxy go in [fetch], as use a browser or your own transport shows.