State and storage¶

Each config has one storage path, data/<name> unless [storage].path says otherwise. It holds the records and everything databrew needs to continue the crawl.

Path	Holds
`items/`, `itemstore.json`	The versions
`observations/`	The observations, as a store of their own
`.state.db`	The queue, the traversals, the failures, the last watched values, and each item's last fetch and HTTP validators
`.last_run.json`	The last run's report
`_failed_urls.json`	A readable export of the URLs with a failure on record, written when a run closes; absent when there are none
`.itemstore.db`	itemstore's catalog and journal

The record files follow itemstore's format. Databrew never reads _failed_urls.json back; it exists for people and diffs.

Records and state belong together¶

The records say what was stored, and .state.db says what is left to do. Neither can be rebuilt from the other, so back them up, restore them, and move them as one.

A run writes a version to the record store first and to .state.db second, and counts the versions it has stored in .state.db with each one. That order decides what happens if a process dies between the two writes:

The records hold more versions than .state.db counted. The next run or reactivate indexes the stored items again, then continues. This is also what happens after a checkout that adds record parts from another machine. Only the item index is rebuilt; the queue, the traversals, and the failures stay as they were.
The records hold fewer versions than .state.db counted. Records are missing, so databrew refuses to open the crawl. Restore the records and .state.db from the same snapshot.

Databrew never deletes, repairs, or migrates .state.db. A run, check, status, or reactivate refuses a storage path whose .state.db is missing beside earlier records, is not a readable database, or has a schema of another databrew version. The error says which.

Upgrading databrew

A release that changes the state schema cannot open storage paths written before it, and there is no migration; the changelog marks such releases as breaking. Keep an existing crawl on the version that wrote it, or start the new version at a new storage path.

One writer at a time¶

A storage path allows one writer. A second process that tries to run the same config fails that config at once, with a line naming the path, and leaves the running crawl alone. To crawl two sites at once, give each its own storage path; databrew run a.toml b.toml --parallel 2 runs them in one process.

Reading while a crawl runs¶

Readers can open the part files at any time, because parts are only ever added. The newest records may still sit in the journal in .itemstore.db until the run flushes them, at the latest when it closes, so a reader sees them after the run ends. A crawl never compacts its stores. If you compact them with itemstore's compact, do it while no reader and no crawl is using them, since compaction rewrites parts in place.

Changing a config¶

A config's records stay tied to its storage path, so some edits start over and some rewrite history:

Renaming the file changes the default name, and with it the default storage path, so the next run begins a new crawl. Set name, or [storage].path, to keep the old one.
Adding, removing, or reshaping a field changes the extracted content of every item, so each item's next fetch stores a new version, whether or not the site changed.
Changing the key splits each record's history: later observations and versions carry the new item key, and the earlier ones keep the old.
Changing the start URLs while a traversal is paused fails until it completes or a --rescan begins a new one, as traversals describes.