Concepts¶
A crawl walks the listing pages from the start URLs, reads every card it finds, and fetches each new record's item page. Revisits fetch known item pages again. Two stores keep the result: every card a listing page showed, and every distinct version of a record.
flowchart LR
roots["start_urls"] --> listing["Listing pages"]
listing -->|pagination| listing
listing -->|each card| observations[("observations/")]
listing -->|new item links| item["Item pages"]
revisit["Revisits: after_days or watch"] -->|known items| item
item -->|new or changed content| versions[("items/")]- How a crawl works: the queue, traversals, and what happens to a URL that fails.
- Observations and versions: the two kinds of row, their schema, and record keys.
- How a run ends: the stop and the outcome that every run reports.
- State and storage: what a storage path holds, and why records and state move together.
Extension points¶
Site behavior enters through three extension points, each for a different kind of problem:
| Extension point | Use it for | Where |
|---|---|---|
| Parser | Reading a value, building a URL | parsers = ["module"], as extractkit defines them |
| Middleware | Headers, recovery, changing items, ending a run | [[middleware]]; see the middleware reference |
| Fetcher | A transport that fetchkit does not offer | Your own fetcher |