Revisit known records¶

By default databrew fetches each item page once. To catch later changes, such as a new price, a sold-out flag, or an edited description, [revisit], which needs [item], says when to fetch known pages again:

[revisit]
after_days = 30
watch = ["price", "status"]

A revisit fetches a known item page again. When the content differs from the latest version, a new version is stored; when it does not, nothing is stored and only the time of the last fetch moves.

Revisit by age¶

after_days queues, at the start of each run, every item page last fetched longer ago than that many days. Use it for records that change quietly, without a sign on the listing pages. Pick the interval from how often the records change and how many requests you can spend: with 30 days and 30,000 items, revisits add about 1,000 requests a day.

Revisit on a change in the listing¶

watch names fields of [listing] whose change should trigger a revisit. When a card shows a value that differs from the one seen at the item page's last fetch, the item page is queued. Use it for the fields a listing already shows, like price or availability: the listing tells you which records changed, and you fetch only those.

[listing.items.fields]
url = { selector = "a", attribute = "href" }
price = { selector = ".price", parser = "parse_float" }

[revisit]
watch = ["price"]

watch needs keyed cards when the item pages have their own key: set [listing.items].key whenever [item.items].key is set, so a card can find its stored record. Field names are checked when the config loads, except on a JSON listing without fields, whose cards are stored as they arrive.

A watched change queues the page but does not count as new for the traversal, so an incremental traversal still stops at caught-up pages. A change on a page that a run does not reach is noticed when a later traversal reads that page. To make sure every card is seen, combine watch with after_days, or run a --rescan now and then.

When revisits run¶

Revisits wait behind new item pages and listing pages, and the longest unfetched go first. A request budget or a deadline therefore cuts revisits before it cuts discovery. --retry-failed queues the revisits it finds for the next run and fetches none of them.

What a revisit costs¶

On the httpx and curl_cffi backends, a revisit of a page that sent an ETag or Last-Modified header asks the server whether the page changed, with If-None-Match or If-Modified-Since. A 304 Not Modified answer costs one request with no body: nothing is extracted or stored, post_fetch and post_extract middleware do not run, and the request counts as a success. The browser backend and a fetcher you pass to run always fetch the full page.

A 404 or a 410 on a revisit usually means the record is gone. The URL fails like any other: a 410 makes it permanently failed at once, a 404 counts toward max_failed_runs, and the stored versions stay. To track removals, read the failed URLs from _failed_urls.json, which each run writes when it closes.