Skip to content

Changelog

0.48.1 - 2026-11-14

Fixed

  • A batch showed each config's result only after the last config ended. run_many with more than one config now logs each result on databrew.progress when its config ends (#131).
  • run_many labeled the log records of a config given as a path with the file's stem, and the report with the config's name. Both carry the name now, and so do the records of run, which carried - (#135).

0.48.0 - 2026-11-13

A crawl now walks its listing pages as a traversal that resumes across runs, and every listing sighting is an observation. .state.db holds the queue, the traversals, and the failures.

Added

  • Traversals: the first is exhaustive, later ones are incremental, and an interrupted one resumes without repeating completed pages. A paused traversal keeps its start URLs, and a run with others fails before it changes listing progress. Failed listing requests permit a new traversal and remain visible as coverage gaps. Exhausted requests wait for reactivate or --rescan.
  • A listing page that no fetch has expanded, or whose last attempt failed or was cut off, follows its pagination in any traversal, so a page read after a failure, a reactivate, or an empty fetch reaches the pages behind it.
  • RunReport.traversal (paused or completed) and RunReport.coverage_gaps, in .last_run.json and the summary line, and Status.traversal with its lines in status and check.
  • databrew.Traversal, the type of Status.traversal.

Changed

  • Breaking: the top-level tables [listing] and [item] replace [extract.listing] and [extract.item], and a config with [extract] fails to load.
  • Breaking: [vars] and ${name} substitution are removed, and a config with [vars] fails to load. A child of extends sets its own values in place.
  • Breaking: --rescan and RunOptions.rescan replace --fresh, --full-crawl, fresh, and full_crawl. A rescan begins an exhaustive traversal and keeps item URLs, their failures, and the stores.
  • Breaking: .state.db is the record of the queue, the traversals, the failures, and each item's last fetch and validators. A missing one beside an earlier crawl, an unreadable one, or one of another schema fails the run. More stored versions rebuild only the item index; fewer require restoring records and state together. Stores written by 0.47 are incompatible; keep them with that version and use a new storage path for a new crawl. databrew no longer reads the contents of _failed_urls.json.
  • Breaking: an observation is one card of one response, keyed fetch_id:N, so each sighting is stored, on the same day and with unchanged values too.
  • Breaking: [policy].stop_on_no_new_items is removed, and a config that sets it fails to load. An incremental traversal stops a branch at an expanded page whose items are all known, on a site without item pages too.
  • Breaking: CrawlStats.record_drops is removed, CrawlStats.record_success takes no counts, and CrawlStats.record_listing_page requires start, which marks a start URL of the traversal.
  • A run fails for all-empty listing pages only when it fetched a start URL, so a resumed or --retry-failed run that reaches only an empty last page succeeds.
  • A failed listing page returns among listing pages in the order it was queued, ahead of revisits.
  • databrew init writes no key at its default value and leaves [fetch], [storage], and [policy] commented out.
  • An expanded listing page in an incremental traversal continues for a new item or a card that might be new: one without a key, or one discarded during extraction. Watched changes, revisits, and changed content do not extend it. Novelty uses the traversal-start baseline across roots and resumes; detail deduplication uses all stored items.
  • A listing page is charged a failed run in every run that it fails, before any success too, so only item pages are maybe_blocked. A run that fetches nothing while listing coverage gaps remain fails.
  • --retry-failed queues what a recovered page links to, and the revisits it finds, for the next run.

Fixed

  • --retry-failed skipped failed requests that a stop left queued or in flight.
  • items_dropped omitted cards discarded for missing required fields or item postprocess failures.
  • A listing page that failed after queueing item links could lose its pagination once those items were stored before it was fetched again.
  • A version count that differed from items/ lost the queue, the last unchanged check, and the validators.
  • read_reports returned urls_answered, listing_pages, and empty_listing_pages as zero.
  • Paced requests could start in a burst after synchronous work stalled the event loop.

0.47.0 - 2026-11-10

databrew now keeps a site's records current instead of snapshotting them once. Every change to a record is a version, a listing card is one observation a day, [revisit] fetches known records again, and each crawl operation has a command and a library call that do the same thing. 0.47 does not read a store that 0.46 wrote.

Added

  • [revisit]: after_days and watch queue known item pages after new work, with conditional requests on httpx and curl_cffi. watch compares a card with its values at the last fetch of its item page, and needs [extract.listing.items].key when [extract.item.items] has a key.
  • databrew parse and databrew.parse, which extract one page as a run does, through the middleware's pre_fetch, post_fetch, and post_extract, and write no crawl state.
  • databrew.check, status, reactivate, and read_reports, with the Plan and Status types.
  • -q on run, and a progress line per config per minute by default (#128).
  • [policy].max_requests_per_minute, an upper bound on the fetch rate.
  • databrew.StoreError for a store or _failed_urls.json that cannot be read and for a store that refuses [storage], with itemstore's message and its error as the cause. The CLI prints it in one line and exits 1.

Changed

  • Breaking: databrew requires extractkit 0.25.1, fetchkit 0.23.1, and itemstore 0.29. Each store holds one schema: key, item_key, url, fetched_at, and data as one JSON column.
  • Breaking: [extract.listing] and [extract.item] each hold one extractkit config and its type, and url_type says listing or item. They replace [extract].type, [extract.items], [extract.links], and items_from. [extract.item] needs [extract.item.items].
  • Breaking: an item-page item is a version keyed item_key:N, numbered per record and stored when its content differs from the latest version. A listing-page item is an observation keyed item_key:YYYY-MM-DD, and also a version on a site without [extract.item], which then needs [extract.listing.items].key.
  • Breaking: an item page holds one item and takes the key of the card that linked it, else its own key, else a digest of its URL. A page that yields several items fails.
  • Breaking: .state.db is a cache, rebuilt from the stores and _failed_urls.json when it is missing, is not a readable database, or has another schema, when that file differs from its last export, or when items/ holds another count of versions than it recorded. Failures live in its urls table.
  • A full crawl lives in the queue. It fetches each listing page once, past the caught-up and stop_on_no_new_items stops, goes on in the next run when one stops early, and goes on from a listing page that failed when a later run fetches it. A start URL new to the queue begins one, so a new store, a start URL added to a config, and the empty queue that --fresh or a rebuilt .state.db leaves are crawled in full. --full-crawl begins one for every start URL.
  • A start URL that a run stopped short of is served ahead of the ones it fetched, so a budget smaller than the list of start URLs reaches each of them over a few runs.
  • A reactivated URL is served behind fresh work, as every URL with a failure on record is.
  • The summary of degraded extraction is a warning, so the default output of run shows it.
  • Breaking: [policy] has 15 keys. max_requests names the budget everywhere, with -n/--max-requests and RunOptions.max_requests. max_retry_after caps every retry delay. A fixed table says which fetch failures retry, 408 among them, and adaptive_concurrency reads its own window.
  • Breaking: [storage] is path, format, compression, and part_size_mb. An existing store keeps the compression and the part size it recorded when the table names none.
  • Breaking: .last_run.json, RunReport, and the summary carry new_items, changed_items, observations, and total_items. A run is productive when it stored a new item or a change.
  • Breaking: check takes several configs and prints the plan that run --dry-run printed. It checks each config whatever the one before it did. status and check rebuild a missing or stale .state.db before they count.
  • Breaking: the default output of run leaves out per-page lines, which -v prints.
  • -v prints databrew's debug lines and leaves out those of the libraries under it.
  • Breaking: reactivate takes --url and --dry-run, and starts failed_runs over.
  • Breaking: --fresh clears the whole queue and the failures.
  • max_failed_runs counts failed runs in a row, since a success starts the count over.
  • Breaking: build_components no longer checks a supplied fetcher, which run checks before it loads the config.
  • A config that --deadline or a signal keeps from starting reports as failed and writes no .last_run.json, so its store's record still describes the last run that started.
  • The extra for the curl_cffi backend is named curl_cffi.

Removed

  • Breaking: [hooks], the --on-* flags, HookCallbacks, on_recover, replace_fetcher, and Stop.HOOK_ABORT. Middleware is the extension point for recovery.
  • Breaking: [observations], ObservationsMiddleware, .watch.db, revisions/, revisit_scheduled, and requeue_url.
  • Breaking: .failures.db and FailureStore.
  • Breaking: databrew.state.SchemaVersionError, since a .state.db of another schema version is rebuilt.
  • Breaking: run --dry-run, run --limit, RunOptions.limit, list --config-dir, and reactivate's --error, --status, --max-age, and --reset-failed-runs.
  • Breaking: the [policy] keys max_retry_delay, retryable_categories, min_requests_for_error_rate, stop_on_empty, stop_on_caught_up, caught_up_threshold, aimd_error_threshold, aimd_grow_step, and delay, and Stop.CAUGHT_UP.
  • Breaking: [stamp], [storage].state_db_name, the [extract].base_url alias, and [fetch].jitter.
  • Breaking: replace_extract_result, since ExtractResult takes dataclasses.replace.
  • Breaking: CrawlStats.summary, error_rate, urls_queued, urls_retried, last_success_at, and links_dropped.
  • Breaking: StateStore.add_pagination_url, add_item_url, url_completed_count, url_permanently_failed_count, read_items, and get_failed_urls, and the single-URL adds of UrlQueue, which only tests called.
  • Breaking: the StateStore methods that forwarded to its queue, such as mark_url_done and url_pending_count. StateStore.queue is the UrlQueue, whose count and restore take a status.
  • Breaking: the run_* hook methods of MiddlewareChain, which run(hook, ctx) replaces, and the full_crawl argument of Orchestrator, which serves one run.
  • The matching on itemstore's error messages (#99).

Fixed

  • An item page that answers 200 without an item stays out of the stop rules, as a 404 does (#130).
  • StateStore closes its connection when its constructor raises.
  • A command that finds the store held by a running crawl exits with a one-line error, and run leaves that store's .last_run.json to the crawl that holds it.
  • examples/configs/ethiopiapropertycentre.toml matches the redesigned site (#129).
  • run_many raises ValueError for parallel below 1, which hung, and a --deadline that passed while the configs loaded starts none of them.
  • A WebsiteConfig built in code stores under data/<name>, as one loaded from TOML does.
  • init with stdin at end of file reports the missing name, and the -c help names no fixed default.

0.46.0 - 2026-10-07

Breaking changes

Each entry ends with the edit a config or a program makes.

  • databrew requires fetchkit 0.21 and extractkit 0.22. The zendriver backend always navigates and rejects the mode and headers options. Its results carry the document's status, so a 403 or 429 block page fails the URL. Delete mode and headers from the [fetch] table of a zendriver config.
  • The PyDoll backend and the pydoll extra are gone. Set backend = "zendriver" and install the zendriver extra. Delete the PyDoll options wait_for_network_idle and network_idle_time, which zendriver rejects.
  • The [policy] option fetcher_close_timeout is gone, since zendriver's close finishes its cleanup even when cancelled. Delete the option.
  • load_config(path) builds no fetcher or extractor and raises ConfigError with a one-line message. databrew check and --dry-run build the components to check them. create_components, apply_cli_overrides, validate_fetcher, create_extractor, the validate_extractor and validate_fetcher arguments of load_config, and runner.run_crawl are gone. Replace a hand-built Orchestrator with databrew.run(config, options=RunOptions(...), middleware=[...], fetcher=...). Code that needs the parts calls build_components(config, fetcher) and open_state(config, fresh=, retry_failed=).
  • A run reports why it stopped as a Stop and how it went as an Outcome, both in a RunReport. Orchestrator.run() returns it, on_complete receives it, and CrawlPolicy.should_stop returns the Stop and its detail or None. CrawlResult, StopClassification, CrawlPolicy.classify_stop_reason, the stop-reason constants in databrew.core.policy, and write_last_run_record are gone. Read report.stop and report.outcome where code matched stopped_reason text, and let databrew.run write .last_run.json.
  • databrew.core.policy.CHALLENGE_CATEGORY is gone. Compare an error category with fetchkit.ErrorCategory.CHALLENGE, which equals "challenge".
  • Every run except --retry-failed adds the start URLs, also when the queue holds pending work, so a crawl that stopped on a throttle still fetches page 1 of each category the next day (#120). StateStore.url_resumable_count() and UrlQueue.resumable_count() are gone. Delete code that seeds start URLs itself, or call StateStore.seed_start_urls(urls, fresh=...).
  • The built-in LoggingMiddleware, HeaderMiddleware, UrlFilterMiddleware, RetryOnPatternMiddleware, ItemEnricherMiddleware, LinkFilterMiddleware, and CookieRefreshMiddleware, the MiddlewareChain.add method, the on_item, on_url_complete, and on_error arguments of Orchestrator, and the cookie-refresh example are gone. No config or known crawler used them. Copy a class you need into the project and list it in [[middleware]].
  • HookCallbacks, HookContext, run_hook, build_hook_callbacks, MiddlewareChain, UrlTask, CrawlComponents, Orchestrator, and StateStore leave databrew.__all__, which lists the supported API. Import them from their modules, or use databrew.run.
  • build_hook_callbacks takes (hooks, name, get_orchestrator, rebuild_fetcher), CrawlComponents holds fetcher, extractor, and middleware, and StorageConfig.itemstore_options() is gone. Code that wired these into an Orchestrator calls databrew.run.
  • Extraction reads the fetch result directly. Middleware that replaces ctx.response supplies an object with url and text, such as a fetchkit.FetchResult.
  • Orchestrator.pause(), Orchestrator.resume(), StateStore.url_failed_runs_sum(), and UrlQueue.failed_runs_sum() are gone. Delete the calls. Orchestrator.stop(reason) ends a run.
  • reset_failed_items of StateStore and UrlQueue is reset_failed_urls, which returns failed listing pages to the queue as well, and their reset_failed_for_retry_only is gone. Call reset_failed_urls.
  • The failures_db_name argument of StateStore is gone. The failure database is .failures.db, the name databrew reactivate reads. Drop the argument.
  • databrew status, check, reactivate, and run --dry-run exit with status 2 and one line on stderr when a config cannot load, where they exited with 1. When any of its configs cannot load, databrew run runs the others and exits with 2, whatever --fail-when says. A script that tells a bad config from a failed run checks for 2.
  • ${name} in a config string is a variable, and a name that [vars] does not set is a config error. Write a shell variable in a hook command as $NAME.
  • [storage].state_db_name and the observation store directories reject every name that databrew or itemstore uses at the storage root, which adds items, .writer.lock.db, and .last_run.json. Rename a state database or directory that uses one.
  • The module databrew.config.storage is gone, and ITEMS_DIR_NAME leaves databrew.config. Import StorageConfig from databrew.config and ITEMS_DIR_NAME from databrew.itemstore_adapter.
  • StateStore.stats() and UrlQueue.pagination_seen_count() are gone. Call the count methods of StateStore, such as url_pending_count() and item_count().
  • A run that empties its queue ends with the queue_empty stop, also when its last URLs reached max_consecutive_failures or its last fetch reached --limit, and its failed URLs stay in urls_failed. It used to end with consecutive_failures, which made a run without a new item failed, or with max_requests. A run with work left stops on those rules as before. A script that read either stop as work left reads the queue with databrew status.

Added

  • databrew.run(config, *, options, middleware, fetcher) runs one crawl as databrew run does and returns its RunReport. RunOptions holds what the databrew run flags express, and run_many(configs, *, options) runs several configs. The CLI is a thin caller of run_many, so a project that needs one middleware or fetcher keeps seeding, hooks, passes, the run record, and a clean stop at SIGINT or SIGTERM. run handles a signal on the main thread when the program has neither set a handler for it nor ignored it, and gives the signal its handler back when it ends. run raises ConfigError when the config cannot load. When the setup of a loaded config raises, it returns a failed report with the error stop.
  • databrew.hash_url(url) is the key of an item whose config sets no [extract.items].key.
  • [[middleware]] tables run middleware from the config file (#46). use names the class as "module:Class", from beside the config file when that module is there and from the installed packages otherwise, and the other keys go to its constructor. Loading checks the class and arguments. The config's directory is a package of its own: a module or package beside the config imports from it, two config directories can each hold a module of the same name, and a module imports its siblings relatively (from . import helpers). Entries run in file order after the observations middleware, a child's entries follow its parent's, and a middleware with close() or aclose() is closed when the run ends.
  • databrew run --parallel N runs up to N configs at once, and --deadline SECONDS stops every run that long after process start, as a signal does. The deadline is a positive finite number. --fail-when any|all picks the exit rule. Log lines name their config, the deadline's and a signal's warning included, and the run ends with one summary line per config. A config that cannot load, or whose setup raises, counts as a failed run, and the others run. run_many(configs, *, parallel, deadline, options) is the library form.
  • [vars] sets variables, and ${name} in any string of the config takes the value. A child's [vars] merges over its parent's through extends, so a shared parent plus a few lines per country or site replaces a file of repeated settings. A [vars] value is a string or a number, and a boolean is a config error. An unknown ${name} is a config error that names the key, and so is a ${name} inside a [vars] value. extends takes a literal path, since [vars] is known only once the files are merged.
  • [stamp] adds constant fields, such as country = "${cc}", to every stored item, observation, and revision. An extracted field of the same name wins. Without the table, stored payloads stay as they are.
  • databrew --version prints the installed version.
  • databrew report PATH... prints one line per store from its .last_run.json, and a totals line, in the form of the run summary. It writes nothing, reports a missing or unreadable record as failed, and reads records from databrew 0.45. --fail-when any|all picks the exit rule, and --stale-after N exits with 1 when any store has gone N runs without a new item, whatever --fail-when says.
  • Orchestrator.stop(reason) ends a run from outside the crawl loop.
  • MiddlewareContext.stop_reason lets middleware end the run with a reason, set on the context a hook returns, whether it is the one the hook received or a new one. The run takes no new URLs and ends once the batch in flight finishes.
  • .last_run.json, CrawlStats, and the run's log count the items and item links that extraction dropped, from extractkit's drop observations, as items_dropped and links_dropped. A drop changes neither the outcome nor degraded_responses.
  • StateStore.mark_url_maybe_blocked(url, error) gives a URL the maybe_blocked status, and StateStore.restore_maybe_blocked_urls() returns those URLs to pending. The first pass of the next run fetches them, a listing page after the other listing pages and an item URL after the other pending URLs. databrew status prints them as URLs maybe blocked.
  • Orchestrator.run(passes=...) runs the passes that --passes asks for. StateStore.mark_url_failed(count_run=False) records a failure without adding to failed_runs.

Fixed

  • A run without observations/.watch.db rebuilds the price-watch baseline from the stored observations before it starts. A runner that starts from a clean checkout records revisions instead of reading every listing as first seen.
  • A run writes a failed .last_run.json with the interrupted stop when it starts, and replaces it when it ends. A run killed by a CI timeout, and a config a batch never starts, leave a failed record instead of the previous run's. So does a loaded config whose setup raises, with the error stop.
  • A run whose listing pages all come back without an item or an item link is failed, and stopped_reason says so. A changed listing layout used to end as a successful run with zero items. A run whose listing pages link only to stored items stays a success.
  • A run in which every URL failed is failed and exits with status 1, also when it made fewer than min_requests_for_error_rate requests. A run that stored new items before a throttle stop is partial and exits with 0. So is a run that a deadline or a signal cut short after a URL succeeded.
  • An exception inside a run ends it with the error stop. .last_run.json and on_complete still record it. An exception while one URL's failure is recorded, such as database is locked, leaves that URL in_progress for the next run, and the other URLs of its batch finish. Every URL of a batch finishes before an exception leaves it, so no fetch runs on after the store closes.
  • The store closes before the run writes .last_run.json and runs on_complete. A store that cannot write its items at close, such as a read-only items/, ends the run with the error stop and keeps its stats, so the record, the summary line, and databrew report agree. It used to raise while the record said success.
  • A middleware hook that returns something other than a MiddlewareContext fails the URL with Middleware <Class>.<hook> returned <type>. It used to fail with an AttributeError that named no middleware.
  • A hook command that cannot be formatted, such as one with {}, fails the hook as a nonzero exit does, and the log names the command. It used to end the run with the error stop. A literal brace is written twice, as {{}}.
  • databrew status on a .state.db that is not a database, or that a newer databrew wrote, prints one line on stderr and exits with 1, as run refuses the store. It used to print a traceback.
  • A completed item URL that also appears as a start URL or pagination link stays completed. Requeueing pagination resets pagination rows only.
  • SIGINT and SIGTERM stop the run cleanly, also while it opens its state or closes its fetcher and store. Fetches in flight are cancelled and return to the queue, a running on_start or on_failure hook command is killed, .last_run.json records "Interrupted by SIGINT" or "Interrupted by SIGTERM", items are flushed, and the process prints no traceback. The exit status follows the run's outcome, so a signal after a URL succeeded exits with 0.
  • A blocked crawl records nothing against the URLs it could not fetch. A URL that fails for good before the run's first success is maybe blocked, uncharged, and the next run fetches it again. A retry pass and a --retry-failed run leave a maybe-blocked URL alone. A URL that fails that way within a --retry-failed run stays failed, uncharged, so the next --retry-failed tries it again. A 404, a 410, and a page that extraction cannot use are the site's answer about one URL, so any run charges them (#122).
  • A terminal 404 or 410 counts as a failed URL and stays out of max_consecutive_failures, max_error_rate, and the adaptive-concurrency window, since it says nothing about a block. A 404 counts toward max_failed_runs. HTTP defines a 410 as likely permanent, so a 410 shelves its URL in the run that sees it, whatever max_failed_runs is. A block of dead seed pages, such as a category the site removed, used to end every run before the pages behind it. CrawlStats.record_failure takes not_found, and CrawlStats.urls_not_found counts these failures.
  • A rate-limited response's Retry-After pauses every fetch until it expires.
  • A run retries the failed item URLs of earlier runs after its other pending URLs, and a listing page that failed or was maybe blocked after its other listing pages, the start URLs included. A group of item URLs that keeps failing used to be fetched first, and could end every run before it reached a listing page.
  • A listing page that fails returns to the queue on the next run, as a failed item URL does, and is shelved after max_failed_runs runs in a row that fail it. A plain run used to retry failed item URLs only, so a listing page that failed after the run's first success was not fetched again unless it was a start URL, and the pages behind it were never crawled. A start URL is fetched in every run whatever its state, since seeding returns it to the queue at the start URLs' priority with its failed_runs count started over.
  • databrew status and run --dry-run read .state.db without writing to the storage directory, so they leave _failed_urls.json untouched and status works while a crawl holds the writer lock. They list item files instead of the stored item count, which itemstore reports only to its writer.
  • Caught-up detection works with [extract.items].key in HTML configs. .state.db indexes the source URL of each keyed item, so a plain item link counts as scraped. JSON item links carry their key and skip the index. The index fills itself from stored items when it is empty, so stores written by earlier versions need no migration.
  • An item URL whose page yields no item fails with an extraction error that names the extractor's reason, such as a missing required field. A layout change shows up as failed URLs, and a later run fetches them again.
  • --passes auto ends when a retry pass recovers no URL, so a pagination URL that always fails cannot keep it looping (#123).
  • The passes of one invocation form one run. Hooks fire once, --limit counts requests across all passes, a URL's failed_runs grows by at most one, and a URL that fails in several passes is one failed URL. A short outage costs a URL one run of its max_failed_runs budget. max_consecutive_failures and max_error_rate count each retry pass from zero, and when one of them ends a retry pass the run keeps the stop of the pass before it.

Changed

  • Every store databrew opens, including the observation and revision stores, raises IncompatibleStoreError, a RuntimeError, when its part files predate itemstore 0.24. The message says how to migrate them.
  • databrew.run closes a fetcher passed to it, also when setup fails. An on_failure recovery keeps that fetcher instead of rebuilding one from [fetch].
  • --start-urls skips # comment lines, as start_urls = { file = ... } does, and a missing file or a line that is not an HTTP URL is a usage error, with or without --dry-run. So is --start-urls with --retry-failed, which seeds no start URL.
  • databrew requires pydantic 2.12, the first release with Python 3.14 wheels. HttpUrl is a class from 2.10 on, which RunOptions.extra_start_urls relies on.
  • .last_run.json gains stop and outcome. stopped_reason keeps its text.
  • .last_run.json gains consecutive_empty_runs: 0 when the run stored a new item, the previous record's value when the run fetched nothing, and otherwise one more. The run summary shows a streak above 0.
  • Pagination URLs added together are served in list order, also when they were queued before, so a reordered seed file takes effect (#124).

0.45.0 - 2026-07-28

Added

  • backend = "zendriver" selects fetchkit's second browser backend. It is registered in BROWSER_BACKENDS, so it gets the same max_tabs default from [policy].concurrency and the same page_load_timeout dry-run preview as pydoll. Install it with the new zendriver extra.

Changed

  • Upgraded Fetchkit to 0.18.0 for the Zendriver backend, lazy tab pooling, and cancellation-safe browser startup and shutdown.
  • CLI startup no longer pays for browser extras it does not use. Fetchkit 0.18.0 imports optional backends on first access, so import fetchkit drops from ~293 ms to ~57 ms with all three extras installed, and databrew --help from ~440 ms to ~190 ms.
  • Boundary tests now cover every name in BROWSER_BACKENDS rather than pydoll alone, and assert each one is a fetchkit backend that accepts the max_tabs and page_load_timeout options databrew supplies.

0.44.0 - 2026-07-28

Added

  • create_components(config, fetcher=...) accepts a caller-supplied fetcher and skips construction from [fetch], so a backend outside fetchkit's registry keeps databrew's resume, fresh, and retry-failed seeding without blanking fetch_config or reassigning CrawlComponents.fetcher (#118). The supplied object is checked for fetch and close before any state is opened, so a class or factory fails there rather than after the queue is seeded. A fetcher databrew builds is still closed if component creation fails; a supplied one is left open for the caller.
  • build_hook_callbacks(..., fetcher_factory=...) builds the replacement fetcher for on_recover. Without it, recovery rebuilds from [fetch], which swaps out a caller-supplied backend — or fails outright when fetchkit cannot build the configured name.

Changed

  • The browser tab-pool default (max_tabs from policy.concurrency) and the dry-run page-load timeout lookup now key on the BROWSER_BACKENDS set in databrew.config.fetch instead of the literal backend name pydoll, so another fetchkit browser backend needs one name added in one place rather than silently losing the default.

0.43.0 - 2026-07-26

Changed

  • Upgraded Fetchkit to 0.16.0 for browser lifecycle hardening and response-body access on failed HTTP responses.
  • Upgraded ItemStore to 0.26.0 for bounded catalog rebuilds and stricter enforcement of its frozen storage format.

0.42.0 - 2026-07-20

Added

  • databrew reactivate <config>: first-class force-retry for failed and permanently_failed URLs, with --url, --error, --status, --max-age, --reset-failed-runs, and --dry-run filters, plus a programmatic StateStore.reactivate(...) API (#76). Reactivation updates .state.db and .failures.db together and re-exports _failed_urls.json; the snapshot merge now orders records by newest event (last_failed_at vs. a new reactivated_at stamp), so a committed reactivation survives merging against a stale runner DB with a higher failed_runs count, while a failure recorded after the reactivation still wins back. Reactivated records reach the queue even when an item already exists in storage, and --dry-run previews against scratch copies so it writes no durable state.
  • Degraded-extraction visibility: CrawlStats gains degraded_responses and extraction_diagnostics, surfaced in summary(), .last_run.json (including the multi-pass aggregate), a completion summary line when nonzero, and per-diagnostic DEBUG context under -v (#93). Middleware can transform successful extraction results through the new replace_extract_result helper, which preserves diagnostics and any future ExtractResult fields (#94).

Changed

  • Failure-store timestamps are now recorded as aware UTC so cross-machine event ordering is immune to timezone skew; existing local-time stamps are converted (assuming the machine's own timezone) on the next open.
  • max_requests is documented as completed fetch attempts — retries and middleware-recovery attempts count toward the budget (#104).

Fixed

  • Failure-classified policy stops (consecutive failures, error rate) now end a multi-pass invocation immediately with a nonzero exit status instead of starting another pass; .last_run.json aggregates statistics across passes and records pass_count. A deliberate on_start abort remains a successful stop (#105).
  • Errors reclassified as challenges during general error recovery are redispatched through the challenge chain once.
  • Re-syncing failure state no longer re-stamps last_failed_at to the sync time; records keep the actual failure time, which --max-age filtering relies on.

0.41.0 - 2026-07-16

Fixed

  • Delayed retries now remain live work in the active crawl instead of making the queue appear empty until a later run. Middleware-cleared errors use the same bounded retry budget, and challenge recovery runs before general error recovery.
  • Incremental pagination now distinguishes item links that are already stored from links that merely already exist in the queue. Pending links from a resume or overlapping page no longer trigger a false caught_up result and truncate deeper pagination.
  • --passes auto compares the complete queue-state snapshot instead of a collision-prone scalar progress sum. Promoting one URL to permanently_failed can no longer hide progress on another failed URL and stop before its retry budget is exhausted.
  • Item URLs are marked durably as soon as they reach max_failed_runs, remain permanently failed if .state.db is lost, and are not resurrected by normal seeding. Failed pagination seeds are reactivated on later runs.
  • Terminal error-rate accounting excludes retryable attempts from both the numerator and denominator. Successful recovery starts a new policy epoch without erasing lifetime failure statistics.
  • Crawl setup and teardown now close partially initialized resources even when observation setup or fetcher shutdown fails. Fetcher shutdown is bounded by fetcher_close_timeout on every lifecycle path.
  • Malformed ports in extracted URLs no longer abort processing of the page's otherwise valid items and links.

Changed

  • Exceptions from data-path middleware hooks (pre_fetch, post_fetch, and post_extract) now fail the URL as non-retryable middleware errors instead of continuing with partially processed data. Recovery-hook exceptions are logged while the remaining recovery chain continues.
  • Configuration loading now rejects unknown top-level and component fields, invalid section shapes, extends cycles, unsafe state filenames, and colliding or escaping observation paths before crawl state is opened. Config-relative files continue to resolve from the user-facing config path, including when it is a symlink.
  • CLI limits must be positive integers, policy overrides are revalidated, and setup failures return a normal nonzero result. Shared config fragments whose names start with _ are omitted from databrew list.
  • Generated HTML and JSON starter configs are runnable list-page examples and safely encode TOML strings, including Unicode and control characters.
  • Supported component lines are capped at extractkit 0.21.x, fetchkit 0.13.x, and itemstore 0.24.x; Pydantic is capped below 3.0.

Migration

  • Existing configs with misspelled, unknown, or previously ignored fields now fail validation and must be corrected. Parser modules and hook commands are trusted executable input and must come from trusted sources.
  • Middleware that intentionally swallowed data-path exceptions must now return an explicit valid context or move recovery behavior into on_error or on_challenge.
  • The max_error_rate threshold now measures terminal URL outcomes only, so operators should review thresholds calibrated against retry attempts.
  • Shared example config base.toml is now _base.toml; update copied example configs that extend the old filename.

0.40.0 - 2026-07-16

Changed

  • Upgraded itemstore from 0.23.2 to 0.24.0 and replaced the removed StorageEngine/upstream StorageConfig integration with databrew-owned storage configuration plus the new ItemStore facade. URL hashing is now owned by databrew, preserving existing SHA-256 identities without importing an itemstore implementation helper.
  • Storage defaults now follow itemstore's periodic policy: flush at 500 pending operations or the target-size threshold. The catalog is fixed at .itemstore.db, item parts remain under items/, and backend is the storage backend TOML key.
  • Removed catalog layout and ephemeral-index settings. Configs using ephemeral_index_db, index_db_ephemeral, index_db_name, or items_dir_name now fail with a migration error instead of being ignored.

Migration

  • Itemstore 0.24 uses {op, key, recorded_at, data} operation envelopes and does not read 0.23 {v, key, stored_at, data} part files. Existing stores must be exported with itemstore 0.23 before switching dependencies, then reingested into a new 0.24 storage path. Starting a fresh path requires no migration; the old .index.db is no longer used.

0.36.1 - 2026-04-28

Changed

  • cli.py split across three sibling modules; closes #53. The argparse + dispatch surface stays in cli.py (905 → 171 LoC over the three slices); the command implementations move to dedicated modules where they can be tested and evolved independently.
  • databrew.templates now owns the databrew init TOML templates: .toml.tmpl package data loaded via importlib.resources and string.Template. New render_init_config(kind, *, name, url) exposes the renderer; kind is typed Literal["html", "json"]. Output is byte-identical to the prior version (#84).
  • databrew.runner owns crawl execution: setup_logging, _resolve_passes, _queue_counts, _run_single_crawl, _run_multi_pass, and run_crawl. Tests for the multi-pass loop and arg validation also moved (tests/test_cli.py → tests/test_runner.py) (#85).
  • databrew.display owns read-only CLI commands: show_status, check_config, list_configs, show_dry_run_info. Both status and dry-run now use the StateStore context-manager API (#86).
  • apply_cli_overrides lives in databrew.config (alongside load_config / create_store) and takes keyword-only limit / concurrency. The previous argparse.Namespace signature kept the CLI layer entangled with the config module; the new shape lets non-CLI callers (tests, library use) apply the same overrides without constructing a Namespace.

0.36.0 - 2026-04-27

Fixed

  • adaptive_concurrency was effectively a no-op under sustained 429 bursts. The AIMD signal counted only terminal failures, lagging the actual API health by the entire retry budget (~5-9 min per URL with max_retries=5 and Retry-After=60-120s), so AIMD never fired before max_consecutive_failures tripped and stopped the crawl. Outcome tracking is now split into two parallel windows: _outcome_window stays terminal-only and continues to drive error_rate / max_error_rate (transient bursts must not kill long-running crawls); new _aimd_outcome_window records terminal + retryable failures + successes and drives the new aimd_error_rate property the AIMD branch reads. Operators who calibrated aimd_error_threshold against the broken signal may see more aggressive halving and need to recalibrate (#74).

Added

  • Retry decisions now surface at INFO so operators can observe retry timing without dropping global level to DEBUG. The first retry per URL logs at INFO with an explicit attempt counter (Will retry <url> in <delay>s (retry N/max_retries)); subsequent retries stay at DEBUG to avoid flooding long backoff chains. Terminal failures emit a new WARNING line (Permanently failed: <url> - <message> (after N attempt[s])) so the should_retry → False transition is explicit (#75).

0.35.0 - 2026-04-27

Fixed

  • Server-instructed Retry-After delays were silently truncated by max_retry_delay (default 60s), guaranteeing the next retry would fire before the server's cooldown elapsed and waste a max_retries slot on a request the server already said would fail. The cap now splits into two fields with independent trust models: max_retry_delay (default 60s) caps locally-computed exponential backoff; new max_retry_after (default 600s) caps server-instructed Retry-After values. Configs that previously used max_retry_delay to also throttle Retry-After will see longer effective waits — set max_retry_after = max_retry_delay to restore the prior behavior (#73).

0.34.0 - 2026-04-27

Added

  • After every crawl, write a machine-readable <storage_path>/.last_run.json containing the run's stats record (name, urls_processed, urls_succeeded, urls_failed, items_extracted, error_rate, elapsed, urls_per_sec, stopped_reason). Replaces fragile parsing of the human-formatted Stats: {...} log line. Written atomically on both normal exits and on_start aborts; available to CLI, Python API, and CI consumers via jq .items_extracted data/<name>/.last_run.json (#45).
  • CrawlStats.as_record() returns a numeric-only snapshot suitable for JSON serialization. summary() continues to return human-formatted strings for log output.
  • Orchestrator(name=...) parameter labels the run in .last_run.json. Defaults to the storage directory's basename when not provided; the CLI now passes config.name.

0.33.0 - 2026-04-25

Removed

  • databrew.core.StrictModel is no longer re-exported from databrew.core. The class is owned by extractkit; importing it via databrew.core was a fragile public-API surface (an extractkit rename would silently break consumers). Import directly from extractkit instead: from extractkit import StrictModel (#58).

Changed

  • Split databrew/core/config.py into core/types.py (the UnitInterval and ItemsFrom primitives) and core/hooks.py (HooksConfig). Public re-exports from databrew.core are unchanged — HooksConfig, ItemsFrom, and UnitInterval still import the same way. Internal callers reaching into the old module path (from databrew.core.config import ...) must update to the new paths (#58).
  • load_config now routes through WebsiteConfig.model_validate instead of constructing the model from pre-cooked typed kwargs. Any field_validator or model_validator declared on WebsiteConfig (or a subclass) now fires reliably during loading, including mode='before' validators that need the full normalized payload. No observable change for the standard config path (#57).

0.32.0 - 2026-04-24

Fixed

  • RetryOnPatternMiddleware no longer accumulates per-URL retry counts without bound. State is stored in an LRU capped by the new max_tracked kwarg (default 10 000); evicted URLs start fresh on their next match. Previously the internal dict grew for every URL seen and persisted across Orchestrator.run() invocations — a URL that exhausted retries in one run was still at max_retries in the next one (#50).
  • RetryOnPatternMiddleware's "retries exhausted" branch now emits a non-retryable error so the orchestrator stops cycling the URL through policy.max_retries after the middleware has given up. Previously the exhausted error inherited retryable=True from FetchError.challenge(...), so the orchestrator could keep re-running the middleware, which kept firing the same exhausted branch.

Added

  • Middleware.reset() lifecycle hook (default no-op) and MiddlewareChain.run_reset(). The orchestrator calls run_reset() at the start of every run() so per-run state (retry counters, caches) can be cleared without recreating the chain. The base-class docstring now documents that middleware instances persist across runs.
  • New public databrew.CrawlError dataclass: canonical error shape carried on MiddlewareContext.error. Provides CrawlError.from_any to normalize a fetchkit.FetchError, an extractkit.CrawlError, or any exception-like object (#55).

Changed

  • Refactored Orchestrator._process_url (was 140 lines) by extracting _save_items, _collect_new_links, _build_pagination_flags, and _should_follow_pagination helpers plus a new PaginationFlags dataclass. The pagination-follow decision is now expressed as a documented priority cascade (retry_failed_only > caught_up > stop_on_no_new_items > stop_on_empty) instead of an anonymous if/elif chain. No behaviour change; covered by new unit tests in test_orchestrator_pagination.py (#54).
  • Breaking (middleware API): MiddlewareContext.error is now typed CrawlError | None (was object | None). Custom middleware that writes to ctx.error must set a CrawlError; previous code that stashed a raw FetchError should switch to CrawlError.from_any(err) (or construct a CrawlError directly). Built-in RetryOnPatternMiddleware now emits a CrawlError. The orchestrator no longer probes error objects with getattr: it normalizes fetch/extract errors at the boundary and passes a concrete CrawlError through _handle_failure (#55).

0.31.2 - 2026-04-24

Fixed

  • on_recover fetcher swap is no longer racy. The old path closed the fetcher before reassigning orchestrator.fetcher, so an in-flight _process_url holding the old reference could hit a closed object. New Orchestrator.replace_fetcher(new) installs the new fetcher before closing the old one via a shared _close_fetcher helper — in-flight tasks finish against a live fetcher and new fetches see the replacement immediately. hooks.py no longer reaches into the orchestrator's private module state; its components.fetcher assignment is guarded by finally so a cancel during the swap can't desync it from orch.fetcher.

Changed

  • Breaking (minor surface): CrawlPolicy.should_stop now requires error_rate: float. The total_failures argument and the None fallback that recomputed the rate internally are both gone — CrawlStats.error_rate is the single owner of the computation (windowed when error_rate_window is set, cumulative otherwise). Callers passing stats.error_rate (as the orchestrator already does) need no change; anyone calling should_stop directly must drop total_failures and pass a concrete error_rate.
  • _FETCHER_CLOSE_TIMEOUT (previously a private module constant in orchestrator) is now CrawlPolicy.fetcher_close_timeout (PositiveFloat, default 5.0 s). Users with slow-closing fetchers can tune it in TOML.

Added

  • Schema versioning for both state-layer SQLite DBs. StateStore and FailureStore now stamp PRAGMA user_version on a fresh DB and validate it on subsequent opens. A DB newer than the running code raises databrew.state.SchemaVersionError instead of silently corrupting state. Pre-versioning DBs (user_version = 0) upgrade in place on next open, so existing deployments require no manual step. STATE_SCHEMA_VERSION = 1 / FAILURE_SCHEMA_VERSION = 1.

0.31.1 - 2026-04-24

Fixed

  • StateStore.get_next_url is now atomic. The old SELECT followed by a separate UPDATE could let two writers sharing a .state.db claim the same URL; it now runs as a single UPDATE … WHERE url_hash = (SELECT … LIMIT 1) RETURNING …. The dequeue index is retuned to (status, priority DESC, created_at ASC, scheduled_at) to match the query's ORDER BY. Old idx_urls_status is dropped on first open.
  • reset_failed_items now clears scheduled_at = NULL, matching the behavior of reset_failed_for_retry_only. A shared _permanently_fail_exhausted_items helper de-duplicates the two reset paths so they can't drift again.

Internal

  • SQLite PRAGMA setup for StateStore and FailureStore now routes through a shared state/_connection.configure_connection helper. Both stores pick up PRAGMA busy_timeout=5000 so multi-process writers wait on the write lock instead of raising OperationalError. Drift-detection test asserts identical PRAGMA values across the two stores.

0.31.0 - 2026-04-24

Removed

  • Breaking: databrew migrate-url-keys subcommand and the databrew.migrations package. The one-shot rekey tooling introduced in v0.28 (and the older scripts/migrate_to_0_22_1.py) are gone — all known caller stores are already on canonical URL-hash keys. Users still on an un-migrated store should pin databrew<0.31 to run the migration, then upgrade.
  • docs/guide/migration.md removed from the docs nav alongside the subcommand.

0.30.1 - 2026-04-24

Fixed

  • normalize_url now produces the same hash for URLs that differ only in repeated-query-param order (?a=1&a=2 vs ?a=2&a=1) or in percent- encoding case (%2f vs %2F). Values within each query key are sorted, and %xx sequences in the path are canonicalized to uppercase per RFC 3986. Migration: existing URL-keyed item stores whose URLs contain lowercase %xx or repeated query params will see those rows rehash. Run databrew migrate-url-keys <config> --force to rekey under the new normalization — otherwise affected items miss their dedup lookup on the next crawl and produce one duplicate row per URL.
  • --limit 0 and --concurrency 0 are no longer silently ignored. The CLI override path used truthy checks; it now uses is not None, so an explicit zero is honored. The shared _apply_cli_overrides helper replaces the duplicated override block in _run_single_crawl and show_dry_run_info.
  • Crawl-failure log in _run_single_crawl uses logger.exception so the traceback is preserved instead of collapsing to a single line.

Internal

  • _already_migrated loop control clarified (break after sample counter bump; no behavior change).
  • Renamed error → error_info in Orchestrator._handle_failure to avoid the rebind-shadow read confusion.

0.30.0 - 2026-04-24

Changed

  • Behavior change: consecutive_failures counts only terminal failures. Retryable failures (e.g. 429, transient network errors) no longer advance consecutive_failures, so a concurrent burst at startup can't trip max_consecutive_failures before retries get a chance (#47). This also narrows the {failures} shell-hook template variable to terminal failures only; operators relying on the old number will see smaller values.

Fixed

  • --retry-failed runs now auto-restore deferred URLs back to the pending queue at the end of the run (including on KeyboardInterrupt or stop conditions), fixing a silent-park bug where active URLs would stay in deferred state after a retry pass (#47).

Added

  • Bump fetchkit>=0.10.0 and plumb the server-provided Retry-After value (now available via FetchError.retry_after) through the orchestrator's retry scheduling. CrawlPolicy.get_retry_delay accepts a retry_after argument that preempts exponential backoff when set (#47).
  • CrawlPolicy.warmup_seconds: linearly ramp effective concurrency from 1 to concurrency at startup. Logs a single INFO line each time the effective cap changes. Defaults to 0 (no ramp) for backwards compatibility (#47).
  • CrawlPolicy.error_rate_window: opt-in sliding-window error rate. When set, stats.error_rate and policy.should_stop evaluate the most-recent N terminal outcomes instead of lifetime cumulative. Retryable failures do not count toward the window — transient bursts heal out of the signal automatically (#47).
  • CrawlPolicy.adaptive_concurrency (AIMD): halve effective concurrency when the windowed error rate exceeds aimd_error_threshold; grow it by aimd_grow_step per batch when the rate is healthy AND the current cap is ≥ 90% saturated (utilization gate prevents grow-on-idle). Requires error_rate_window to be set (#47).
  • Middleware.on_challenge hook and MiddlewareChain.run_on_challenge: new lifecycle point for responding to CDN challenges (Cloudflare, bot checks) before the URL is marked failed. RetryOnPatternMiddleware now emits FetchError.challenge(...) so CAPTCHA-detection flows can opt in via the new hook. "challenge" is added to the default retryable_categories. _handle_failure is now async (#47).
  • Orchestrator.pause() / Orchestrator.resume(): freeze new task scheduling without aborting the run. In-flight tasks finish normally. Backed by an asyncio.Event gate at the top of the run loop (#47).
  • CookieRefreshMiddleware: built-in middleware that injects a Cookie: header from an in-memory dict and, on challenge, calls a user-supplied async callback to fetch fresh cookies. Concurrent challenges are serialized by an internal lock + cooldown window. Sample end-to-end Cloudflare-clearance flow in examples/middleware/refresh_cookies_middleware.py (#47).
  • databrew run --passes <N|auto>: drive the full primary → resume → retry-failed sequence from one CLI invocation. auto terminates when both pending and failed queues are empty, or when two consecutive passes make no forward progress. Incompatible with --retry-failed (the loop handles that per-pass) (#47).

0.29.0 - 2026-04-13

Breaking: upgrade to extractkit v0.12.0 with renamed config fields. This release separates item identity (dedup key) from URL template variable (#44).

Config field renames (no backward-compat aliases): - [extract.items].id → [extract.items].key - [extract.links].items_id → [extract.links].items_url_field - WebsiteConfig.id_field removed → use WebsiteConfig.key_field

New: natural-key mode. When [extract.items].key is set (e.g. key = "sku"), items are stored and deduped by that field's value instead of URL hash. This fixes the long-standing issue where has_item() dedup was a no-op in URL-hash-keyed stores. Sites that don't set key continue using URL-hash mode.

  • StateStore accepts key_field parameter; branches storage init and save_item() on natural-key vs URL-hash mode
  • migrate-url-keys skips stores that use natural keys (not applicable)
  • extractkit ItemLink.item_id → ItemLink.item_key reflected in orchestrator dedup loop
  • All example configs, CLI template, and docs updated

0.28.1 - 2026-04-12

  • Fix migrate-url-keys catch-22 on fresh checkout: migration no longer requires a pre-existing .index.db — it builds one from parquet files automatically (#42)
  • Fix false-positive "already migrated" detection: the check now validates actual key data instead of trusting store_config.key_field metadata
  • Add --force flag to migrate-url-keys to bypass the already-migrated check
  • Disable auto-compaction on the staging engine during migration to prevent ConnectionException on large stores (>70K rows)

0.28.0 - 2026-04-12

Breaking: id_field no longer affects storage keying. Items are now keyed by hash(normalize_url(source_url)) via a single path. The in-memory _url_to_key dict that silently broke incremental crawls after restart is gone (#41).

Existing stores created with id_field must be migrated before their first run under v0.28:

databrew migrate-url-keys path/to/site.toml

The migration is idempotent, keeps a .bak copy alongside the swapped store, and recovers the source URL from each row's _source_url field (or url fallback, stamped by extractkit). Configs that still declare [extract.items].id = "..." keep loading; the field is ignored with a deprecation warning pointing at databrew migrate-url-keys.

  • Fix incremental crawls walking the entire pagination archive after restart on sites with id_field set (#41). Affected HTML-scraped sites (realethio, ethiopiapropertycentre, ethiopianproperties) now stop at the first caught-up page as intended.
  • StateStore.save_item stamps _source_url on every row as a durable storage-layer invariant (separate from extractkit's url convenience). This decouples storage dedup from extractor conventions and makes future rekey/repair tools straightforward.
  • Orchestrator now reads the per-item source URL from item.get("url") before falling back to the task URL, so JSON items_url templates and HTML list configs with fields.url = ... selectors continue to get per-item dedup without relying on id_field.
  • New subcommand: databrew migrate-url-keys <config>..., with --dry-run, --drop-missing-url, and --no-keep-backup flags.

0.27.1 - 2026-04-08

  • Fix has_item_for_url() returning true for completed URLs with no stored item (#40)
  • Fix RetryOnPatternMiddleware silently accepting blocked pages after max retries (#40)
  • Fix databrew list silently hiding configs that fail to load (#40)
  • Bump itemstore to >=0.23.1 for read_items() buffered item fix (#39)

0.25.0 - 2026-03-30

Breaking: requires Python >=3.14 (was >=3.11).

  • Upgrade itemstore to v0.18.0 with envelope Parquet schema (#33, #35)
  • Fix broken import path for hash_key after itemstore refactor
  • Add StateStore.read_items() for reading back stored payloads
  • Remove 10 tests that duplicated itemstore's own coverage
  • Update docs for envelope schema, export_store(), compact_storage() params

Migration: existing flat Parquet stores need conversion. See the migration guide.

0.24.1 - 2026-03-23

  • Timeout fetcher.close() to prevent store flush hang (#32)
  • Upgrade itemstore to v0.9.0

0.24.0 - 2026-03-23

  • Batch URL seeding, relocate items_from, add stop_on_no_new_items (#31)

0.23.1 - 2026-03-22

  • Fix misleading log, store flush, and custom fetcher types (#27)

0.23.0 - 2026-03-14

  • Fix generated config templates: compression -> backend_options
  • Overhaul documentation for accuracy, tone, and real examples
  • Upgrade to extractkit 0.7.1

0.22.2 - 2026-03-14

  • Fall back to URL-based key when id_field doesn't resolve

0.22.1 - 2026-03-14

  • Fix caught-up detection and --fresh to preserve incremental crawl state

0.22.0 - 2026-03-13

  • Inject _scraped_at timestamp into every saved item