Reports¶

A run returns a RunReport and writes it to .last_run.json in the storage path, where databrew report and read_reports read it back. The file's keys are those of RunReport.record, some of which differ from the attribute names: stopped_reason holds detail, pass_count holds passes, and consecutive_empty_runs holds empty_runs. How a run ends gives the meaning of each Stop and Outcome.

databrew.RunReport ¶

RunReport(
    name: str,
    stop: Stop,
    detail: str | None,
    outcome: Outcome,
    stats: CrawlStats,
    total_items: int,
    passes: int,
    empty_runs: int = 0,
    elapsed: float = 0.0,
    traversal: Literal["paused", "completed"] | None = None,
    coverage_gaps: int = 0,
)

The result of one run, which .last_run.json records under its own keys.

name ¶

name: str

The config's name.

stop ¶

stop: Stop

Why the run ended.

detail ¶

detail: str | None

The specifics of the stop, empty listing results, or exhausted listing retries, or None when the queue emptied without these notes.

outcome ¶

outcome: Outcome

How the run went.

stats ¶

stats: CrawlStats

The run's counts.

total_items ¶

total_items: int

The item keys in items/.

passes ¶

passes: int

The passes the run made.

empty_runs ¶

empty_runs: int = 0

The runs in a row that stored no new item and no change. A run that processed no URL keeps the previous count. A run killed before it ends leaves a record with the stop interrupted, the detail "Run did not finish", and the previous count plus 1.

elapsed ¶

elapsed: float = 0.0

Seconds the run took.

traversal ¶

traversal: Literal['paused', 'completed'] | None = None

The captured traversal state: paused when ready listing pages wait for the next run, completed otherwise, or None when setup failed or no traversal has begun.

coverage_gaps ¶

coverage_gaps: int = 0

Unresolved failed listing requests captured at crawl end, or zero when setup failed.

record ¶

record() -> dict[str, object]

The report as .last_run.json stores it.

Source code in src/databrew/report.py

def record(self) -> dict[str, object]:
    """The report as ``.last_run.json`` stores it."""
    return {
        "name": self.name,
        **self.stats.as_record(),
        "elapsed": round(self.elapsed, 1),
        "urls_per_sec": round(self.stats.urls_processed / self.elapsed, 1)
        if self.elapsed
        else 0.0,
        "total_items": self.total_items,
        "stopped_reason": self.detail,
        "pass_count": self.passes,
        "stop": self.stop,
        "outcome": self.outcome,
        "consecutive_empty_runs": self.empty_runs,
        "traversal": self.traversal,
        "coverage_gaps": self.coverage_gaps,
    }

from_record ¶

from_record(record: Mapping[str, Any]) -> RunReport

Rebuild a report from what record wrote; a key or value out of place raises.

Source code in src/databrew/report.py

@classmethod
def from_record(cls, record: Mapping[str, Any]) -> RunReport:
    """Rebuild a report from what ``record`` wrote; a key or value out of place raises."""
    stats = CrawlStats(**{name: record.get(name, 0) for name in RECORDED_COUNTS})
    return cls(
        name=record["name"],
        stop=Stop(record["stop"]),
        detail=record.get("stopped_reason"),
        outcome=Outcome(record["outcome"]),
        stats=stats,
        total_items=record.get("total_items", 0),
        passes=record.get("pass_count", 0),
        empty_runs=record.get("consecutive_empty_runs", 0),
        elapsed=record.get("elapsed", 0.0),
        traversal=record.get("traversal"),
        coverage_gaps=record.get("coverage_gaps", 0),
    )

databrew.Stop ¶

Bases: StrEnum

Why a run ended.

QUEUE_EMPTY ¶

QUEUE_EMPTY = 'queue_empty'

MAX_REQUESTS ¶

MAX_REQUESTS = 'max_requests'

CONSECUTIVE_FAILURES ¶

CONSECUTIVE_FAILURES = 'consecutive_failures'

ERROR_RATE ¶

ERROR_RATE = 'error_rate'

BLOCKED ¶

BLOCKED = 'blocked'

INTERRUPTED ¶

INTERRUPTED = 'interrupted'

ERROR ¶

ERROR = 'error'

databrew.Outcome ¶

Bases: StrEnum

How a run went.

SUCCESS ¶

SUCCESS = 'success'

PARTIAL ¶

PARTIAL = 'partial'

FAILED ¶

FAILED = 'failed'

databrew.CrawlStats ¶

CrawlStats(
    urls_processed: int = 0,
    urls_succeeded: int = 0,
    urls_failed: int = 0,
    urls_answered: int = 0,
    new_items: int = 0,
    changed_items: int = 0,
    observations: int = 0,
    degraded_responses: int = 0,
    extraction_diagnostics: int = 0,
    items_dropped: int = 0,
    listing_pages: int = 0,
    empty_listing_pages: int = 0,
    consecutive_failures: int = 0,
    started_at: datetime = now(),
)

The counts of one run, which the stop rules, adaptive concurrency, and the report read.

urls_processed ¶

urls_processed: int = 0

urls_succeeded ¶

urls_succeeded: int = 0

urls_failed ¶

urls_failed: int = 0

urls_answered ¶

urls_answered: int = 0

Terminal failures that are the site's answer about one URL: a 404, a 410, or a page extraction cannot use.

new_items ¶

new_items: int = 0

Items stored for the first time.

changed_items ¶

changed_items: int = 0

Later versions of items already stored.

observations ¶

observations: int = 0

Listing-page sightings stored, including unchanged values and repeated keys.

degraded_responses ¶

degraded_responses: int = 0

Successful responses with a parser_failed or required_missing diagnostic.

extraction_diagnostics ¶

extraction_diagnostics: int = 0

Those diagnostics across all successful responses.

items_dropped ¶

items_dropped: int = 0

Items that extraction dropped or that had no key, counted in every extracted response, retried and failed ones included.

listing_pages ¶

listing_pages: int = 0

Listing pages fetched successfully.

empty_listing_pages ¶

empty_listing_pages: int = 0

Listing pages that yielded no item and no item link.

productive ¶

productive: bool

Whether the run stored a new item or a change to one.

listings_empty ¶

listings_empty: bool

Whether a start URL was fetched and no listing page yielded an item or an item link.

Without a start URL, an empty page is the end of a branch that an earlier run or pass began, as when a run resumes a traversal or retries its last page.

databrew.read_reports ¶

read_reports(
    paths: Iterable[str | Path],
) -> list[RunReport]

Read every .last_run.json under paths without writing anything.

A path that holds no record, or a record that cannot be read, becomes a failed report named by its path.

Source code in src/databrew/report.py

def read_reports(paths: Iterable[str | Path]) -> list[RunReport]:
    """
    Read every ``.last_run.json`` under ``paths`` without writing anything.

    A path that holds no record, or a record that cannot be read, becomes a ``failed`` report
    named by its path.
    """
    reports = []
    for path in map(Path, paths):
        files = [path] if path.is_file() else sorted(path.rglob(RECORD_NAME))
        if not files:
            reports.append(failed_report(str(path), f"No {RECORD_NAME}"))
        for file in files:
            try:
                reports.append(RunReport.from_record(json.loads(file.read_text())))
            except (OSError, ValueError, KeyError, TypeError, AttributeError) as error:
                reports.append(failed_report(str(file), f"Unreadable: {error}"))
    return reports