Crawl a JSON API¶

Many sites load their listings from a JSON API, and crawling the API is faster and steadier than crawling the HTML: the data arrives structured, and an API's response format changes less often than page markup.

Find the API¶

Open the site with the browser's developer tools on the Network tab, filter by Fetch/XHR, and page through the listing. Find the request whose response holds the records, and note:

• the URL of the first page, which becomes a start URL;
• where the records sit in the response, such as data.items;
• where the next page comes from: a URL in the response, or a page number to compute;
• whether each record has its own endpoint, and how its URL is built.

Write the config¶

This config reads an API that pages through listings and has one endpoint per listing:

name = "market"
start_urls = ["https://api.example.com/v1/listings?page=1&sort=new"]
parsers = ["market_parsers"]

[listing]
type = "json"

[listing.items]
path = "data.items"  # where the records sit in the response
key = "id"           # the record's own ID

[listing.items.fields]
id = "id"
title = "title"
price = "price.amount"
url = { path = "id", parser = "item_url" }

[listing.links]
items = "url"
pagination = ["links.next"]  # the path to the next page's URL

[item]
type = "json"

[item.items]
key = "listing.id"

from typing import Any


def item_url(listing_id: int) -> str:
    """Return the endpoint of one listing."""
    return f"https://api.example.com/v1/listings/{listing_id}"


def next_page(data: dict[str, Any], current_url: str) -> str | None:
    """Return the URL of the page after this one, or None after the last page."""
    page = data["meta"]["page"]
    if page >= data["meta"]["pages"]:
        return None
    return current_url.replace(f"page={page}", f"page={page + 1}")

type = "json" makes a page type read JSON with dot paths instead of CSS selectors. The API returns an ID rather than the URL of the record's endpoint, so the item_url parser builds it, and [listing.links].items queues it. databrew init market --type json writes a starter with the [listing] tables; add [item] and items yourself. Stable keys and parsers work as in an HTML config.

Check what the config reads, quoting the URL because of its &:

databrew parse market.toml "https://api.example.com/v1/listings?page=1&sort=new"

Without items.fields, each record is stored as the API returns it, with a url added when it has none. That is a good default when you want everything: nothing is lost when the API adds a field, and you choose columns when you query.

Read the item endpoint¶

When the whole response is the record, [item] needs only type and [item.items].key, as in market.toml. An item page takes the key of the card that linked to it, so the card and the page describe one record. [item.items].key applies only when that card had no key; point it at the same ID. If the cards already hold every field you need, leave out [item] and [listing.links].items, and the cards are stored as versions.

Compute the next page¶

When the response has no next-page URL, a pagination builder computes it from the response and the current URL. next_page in market_parsers.py is one; name it in place of pagination:

[listing.links]
items = "url"
pagination_builder = "next_page"

The builder runs for every listing response and returns a URL, several URLs, or None when the listing has ended. See pagination builders for the rules.

Send the headers the API expects¶

Some APIs answer only requests that look like the site's own. Copy the headers that matter from the request in the developer tools:

[fetch]
headers = { "Accept" = "application/json", "User-Agent" = "Mozilla/5.0 ..." }

The HTTP backends send them with every request; the browser backend takes no headers. Headers that expire, such as a bearer token, belong in a middleware that refreshes them, as authenticate requests shows. When an API checks the TLS fingerprint of the client, the curl_cffi backend can impersonate a browser; see use a browser or your own transport.

Many listings at once¶

A site with many categories has many start URLs. Keep them in a file beside the config, one per line:

start_urls = { file = "categories.txt" }

All the start URLs belong to one traversal, and an item page is queued once however many categories list it.