How-to guides¶

Each guide solves one task and assumes you have run the tutorial or a crawl of your own.

Write a config¶

• Write a config for an HTML site: find the cards, follow the links, key the records.
• Crawl a JSON API: read paths instead of selectors, build item URLs, compute the next page.
• Authenticate requests: static headers, and a middleware that renews a token.
• Use a browser or your own transport: pages that need JavaScript, TLS fingerprints, proxies, and a fetcher of your own.

Run a crawl¶

• Pace requests and avoid blocks: concurrency, rate, retries, and the stop rules.
• Schedule and resume crawls: bounded runs, batches, exit codes, and CI.
• Recover from failures: failed URLs, blocked runs, coverage gaps, refused state.
• Revisit known records: fetch item pages again by age or on a change in the listing.

Use the data¶

• Query your data: current state, history, and listing sightings with DuckDB.