Examples¶

These configs crawl real sites, and each shows a different shape of crawl. They live in the repository's examples/configs/, and the test suite builds each one with check. Run one from the repository's root:

databrew check examples/configs/realethio.toml
databrew run examples/configs/realethio.toml -n 25

The selectors follow the sites as they stood when the configs were written, and a redesign can leave one stale; databrew parse shows what a config reads from a page today.

An HTML bookstore¶

The tutorial's config: HTML cards, item pages, a next-page link, and a parser.

books.toml

name = "books"
start_urls = ["https://books.toscrape.com/catalogue/page-1.html"]

# Listing pages show 20 books, each in an article.product_pod card.
[listing.items]
selector = "article.product_pod"

[listing.items.fields]
url = { selector = "h3 a", attribute = "href" }
title = { selector = "h3 a", attribute = "title" }
price = { selector = ".price_color", parser = "parse_float" }

[listing.links]
items = "url"               # the card field that links to the book's page
pagination = ["li.next a"]  # the link to the next listing page

# Item pages show one book.
[item.items.fields]
title = ".product_main h1"
price = { selector = ".product_main .price_color", parser = "parse_float" }
availability = ".product_main .availability"

HTML listings with details¶

HTML listing and item pages, a pagination selector, required fields, key-value details, and lists.

realethio.toml

# Real Ethiopian Real Estate: HTML listing and item pages.
name = "realethio"
start_urls = ["https://realethio.com/search-results/?listing-view=list"]

# One card per property, each holding the link to its page.
[listing.items]
selector = ".listing-view h2.item-title"
fields.url = { selector = "a[href]", attribute = "href" }

[listing.links]
base_url = "https://realethio.com"
items = "url"
pagination = ["ul.pagination > li.page-item:nth-last-child(2) > a.page-link"]

# The whole page is one item.
[item.items]
key = "property_id"
fields.title = ".page-title-wrap h1"
fields.price = { selector = ".property-price-wrap .item-price", required = true }
fields.address = ".page-title-wrap address"
fields.description = "#property-description-wrap > div > div.block-content-wrap"
fields.listing_type = ".page-title-wrap .property-labels-wrap a"
fields.seller_name = "#sidebar .agent-name"
fields.seller_phone = { selector = "a[href*='tel:']", attribute = "href" }
fields.features = { selector = "#property-features-wrap ul > li > a", multiple = true }
fields.images = { selector = "[data-src]", attribute = "data-src", multiple = true }

[item.items.fields.details]
selector = ".detail-wrap li"
keys = "strong"
values = "span"

[item.items.fields.details.derive]
property_id = "Property ID"
bedrooms = "Bedrooms"
bathrooms = "Bathrooms"
property_size = "Property Size"

# A leading underscore leaves the JSON-LD itself out of the item.
[item.items.fields._ldjson]
selector = "script[type='application/ld+json']"
parser = "parse_ldjson"

[item.items.fields._ldjson.derive]
date_published = "datePublished"
date_modified = "dateModified"

[policy]
max_retries = 3
max_consecutive_failures = 10

[storage]
path = "examples/data/realethio"
format = "jsonl"

A JSON API with one endpoint per record¶

A parser that builds item URLs, [revisit] with watch, and extends, over the fragment _base.toml.

jiji.toml

# Jiji Ethiopia: a JSON API with one endpoint for listings and one for each advert.
extends = "_base.toml"

name = "jiji"
start_urls = [
    "https://jiji.com.et/api_web/v1/listing?slug=cars&page=1&sort=new",
    "https://jiji.com.et/api_web/v1/listing?slug=houses-apartments-for-rent&page=1&sort=new",
]
parsers = ["jiji_parsers"]

[fetch]
headers = { "Accept" = "application/json", "User-Agent" = "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36" }

# One card per advert, stored as one observation per response.
[listing]
type = "json"

[listing.items]
path = "adverts_list.adverts"
key = "guid"
fields.guid = "guid"
fields.price = "price_obj.value"
fields.url = { path = "guid", parser = "item_url" }

[listing.links]
pagination = ["next_url"]
items = "url"

# The whole response of the advert endpoint is one item.
[item]
type = "json"

[item.items]
key = "advert.guid"

# A price change on a card fetches the advert again.
[revisit]
after_days = 30
watch = ["price"]

[policy]
concurrency = 8
max_requests_per_minute = 120
jitter = 1.0
max_retries = 3
max_consecutive_failures = 10

[storage]
path = "examples/data/jiji"

_base.toml

[storage]
compression = "zstd"
part_size_mb = 65

jiji_parsers.py

"""Parsers of the jiji example."""


def item_url(guid: str) -> str:
    """Return the advert endpoint of a listing card."""
    return f"https://jiji.com.et/api_web/v1/item/{guid}"

A large JSON crawl¶

Start URLs from a file, request headers, and pacing for a large crawl.

engocha.toml

# Engocha Marketplace: a JSON API behind a list of category URLs.
name = "engocha"
start_urls = { file = "../engocha_categories.txt" }
parsers = ["engocha_parsers"]

[fetch]
headers = { "Content-Type" = "application/json", "User-Agent" = "Dart/2.19 (dart:io)" }

[listing]
type = "json"

[listing.items]
path = "listings.data"
key = "ListingID"
fields.ListingID = "ListingID"
fields.url = { path = "ListingID", parser = "item_url" }

[listing.links]
pagination = ["listings.next_page_url"]
items = "url"

# The whole response holds the listing, its category, and the parent category.
[item]
type = "json"

[item.items]
key = "listing.ListingID"

[policy]
max_retries = 5
max_consecutive_failures = 100
concurrency = 8
max_requests_per_minute = 240
jitter = 2.0

[storage]
path = "examples/data/engocha"
compression = "zstd"

engocha_parsers.py

"""Parsers of the engocha example."""


def item_url(listing_id: int) -> str:
    """Return the classified endpoint of a listing."""
    return f"https://engocha.com/api/v1/classifieds/{listing_id}"