Skip to content

Crawl state

These calls read or change what a config's storage path holds, without crawling. State and storage describes the files.

databrew.status 

status(config: str | Path | WebsiteConfig) -> Status

Return what the storage path of config holds, as .state.db counts it.

Raises:

Type Description
ConfigError

The config cannot be loaded.
StoreError

.state.db is missing beside an earlier crawl, unreadable, or of another schema.
Source code in src/databrew/commands.py 
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
def status(config: str | Path | WebsiteConfig) -> Status:
    """
    Return what the storage path of ``config`` holds, as ``.state.db`` counts it.

    Raises:
        ConfigError: The config cannot be loaded.
        StoreError: ``.state.db`` is missing beside an earlier crawl, unreadable, or of another
            schema.
    """
    loaded = _loaded(config)
    storage_path = loaded.storage_path
    if not has_state(storage_path):
        return Status(loaded.name, storage_path, exists=False)
    counts, traversal = read_status(storage_path)
    statuses = ("pending", "completed", "failed", "maybe_blocked", "permanently_failed")
    return Status(
        loaded.name,
        storage_path,
        exists=True,
        part_files=count_parts(storage_path),
        total_items=counts["items"],
        **{status: counts.get(status, 0) for status in statuses},
        traversal=traversal,
    )

databrew.Status 

Status(
    name: str,
    storage_path: Path,
    exists: bool,
    part_files: int = 0,
    total_items: int = 0,
    pending: int = 0,
    completed: int = 0,
    failed: int = 0,
    maybe_blocked: int = 0,
    permanently_failed: int = 0,
    traversal: Traversal | None = None,
)

What a storage path holds. exists is False when no crawl has written to it.

name 

name: str

storage_path 

storage_path: Path

exists 

exists: bool

part_files 

part_files: int = 0

total_items 

total_items: int = 0

pending 

pending: int = 0

completed 

completed: int = 0

failed 

failed: int = 0

maybe_blocked 

maybe_blocked: int = 0

permanently_failed 

permanently_failed: int = 0

traversal 

traversal: Traversal | None = None

The latest traversal, or None before the first.

databrew.Traversal 

Traversal(
    number: int,
    exhaustive: bool,
    roots: tuple[str, ...],
    to_go: int,
    gaps: int,
)

A walk of the listing pages from a fixed set of start URLs, over as many runs as it takes.

An exhaustive traversal follows every pagination link of a nonempty page, and an incremental one stops a branch at an expanded page whose items are all known.

number 

number: int

exhaustive 

exhaustive: bool

roots 

roots: tuple[str, ...]

The start URLs, normalized and sorted.

to_go 

to_go: int

Listing pages that are queued or in flight with no failure on record, whichever traversal queued them.

gaps 

gaps: int

Listing pages of any traversal with a failure on record, retryable or exhausted, among them one that is queued for its retry or was cut off at it.

paused 

paused: bool

Whether listing work with no failure on record waits, so the next run resumes.

databrew.reactivate 

reactivate(
    config: str | Path | WebsiteConfig,
    *,
    url: str | None = None,
    dry_run: bool = False,
) -> list[str]

Queue the permanently failed URLs again, with failed_runs at 0, and return them.

url keeps URLs that hold the text, or match it entirely when it holds a glob character. With dry_run, nothing changes on disk, and a running crawl keeps its locks.

Raises:

Type Description
ConfigError

The config cannot be loaded.
FileNotFoundError

The storage path holds no crawl state.
StoreError

.state.db cannot be used.
BlockingIOError

Without dry_run, another process holds the store.
Source code in src/databrew/commands.py 
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
def reactivate(
    config: str | Path | WebsiteConfig, *, url: str | None = None, dry_run: bool = False
) -> list[str]:
    """
    Queue the permanently failed URLs again, with ``failed_runs`` at 0, and return them.

    ``url`` keeps URLs that hold the text, or match it entirely when it holds a glob character.
    With ``dry_run``, nothing changes on disk, and a running crawl keeps its locks.

    Raises:
        ConfigError: The config cannot be loaded.
        FileNotFoundError: The storage path holds no crawl state.
        StoreError: ``.state.db`` cannot be used.
        BlockingIOError: Without ``dry_run``, another process holds the store.
    """
    loaded = _loaded(config)
    if not has_state(loaded.storage_path):
        raise FileNotFoundError(f"No crawl state at {loaded.storage_path}; run the config first")
    if dry_run:
        return matching_urls(read_permanently_failed(loaded.storage_path), url)
    with create_store(loaded) as store:
        return store.reactivate(url)

databrew.StoreError

Bases: RuntimeError

A storage path holds state databrew cannot use, or a store that refuses [storage].

Unusable state is a .state.db that is missing beside an earlier crawl, unreadable, or of another schema, or a paused traversal that began from other start URLs. A store's refusal keeps itemstore's message, with itemstore's error as the cause.

databrew.hash_url 

hash_url(url: str) -> str

Normalize a URL and return its SHA-256 hex digest.

Source code in src/databrew/state/hashing.py 
57
58
59
def hash_url(url: str) -> str:
    """Normalize a URL and return its SHA-256 hex digest."""
    return hashlib.sha256(normalize_url(url).encode()).hexdigest()