Skip to content

Adding a New Site

Most sites need only a YAML profile. Custom code is required only for a bespoke parser, a non-trivial producer (bootstrap/pagination), or a typed site_config model.

Step-by-Step Guide

1. Create a Profile — profiles/<site>.yaml

This is the minimum requirement for any new site:

site: mysite
plugin: cloakbrowser
proxy: ${PROXY_URL}
headless: true

rate_limit:
  requests: 2
  window_seconds: 10.0

retry:
  max_retries: 3
  delay_seconds: 8.0
  backoff: fixed
  retry_on_status: [429, 503, 403]

stages:
  listing:
    plugin: cloakbrowser
    parser: generic
    container_selector: "article.item"
    selectors:
      title: ".title"
      price: ".price"
    follow_selectors: ["a.item-link"]
    follow_stage: detail
    next_page_selector: "li.next a"
    use_pool: true

  detail:
    plugin: cloakbrowser
    parser: generic
    selectors:
      title: "h1"
      price: ".price"
      location: ".addr"
    validation:
      required_fields: [title, price]
      strategy: drop

storage: [garage_s3, json]
storage_config:
  json:
    output_dir: ./output/mysite
    mode: jsonl

See Profiles reference for the full YAML schema.

2. Custom Producer — workers/producers/<site>.py (optional)

Only needed if URLs aren't a simple template — e.g.:

  • Dynamic pagination (idealista)
  • Browser bootstrap that captures tokens/session cookies (booking_com, facebook_marketplace)
  • API-based listing that requires specific request construction

Subclass Producer and publish payloads to url.queue. The system auto-detects custom producers — make scrape finds them automatically.

3. Custom Parser — plugins/parsers/<site>_*.py (optional)

Only if the generic CSS parser isn't enough:

  • Domain-specific field derivation
  • JSON extraction from embedded <script> state
  • Complex data transformation
from core.registry import register_parser
from plugins.base import BaseParserPlugin

@register_parser
class MySiteParser(BaseParserPlugin):
    plugin_id = "mysite_parser"

    def parse(self, raw: dict) -> list[dict]:
        # Extract structured data
        return [{"title": "...", "price": "..."}]

4. Typed site_config Model — core/profiles.py (optional)

If the profile has a custom site_config: block, add a Pydantic model and register it in SITE_CONFIG_MODELS so unknown keys are rejected loudly.

5. Capture a Structure Baseline

So the verify gate has something to compare against:

uv run python scripts/check_versions.py check mysite --via-workers --update

This creates fingerprint JSON files in tests/fixtures/mysite/.

Capture sample HTML/JSON responses for offline testing:

# Save actual responses as fixtures
cp response.html tests/fixtures/mysite/listing.html
cp detail_response.json tests/fixtures/mysite/detail.json

7. Add Prefect Deployment — prefect/register_flows.py (optional)

Add the site to SITE_SCHEDULES:

SITE_SCHEDULES = {
    "mysite": "0 5 * * *",       # daily 05:00
    # or None for manual-only
}

This creates a scheduled, gated scrape-mysite deployment on the next make prefect-up.

Checklist

  • [ ] profiles/<site>.yaml created
  • [ ] Profile loads: uv run python -c "from core.profiles import load_profile; p = load_profile('mysite'); print(p)"
  • [ ] Structure baseline captured: make version-check PROFILE=mysite UPDATE=1
  • [ ] Test run succeeds: make scrape JOB=test-mysite PROFILE=mysite FRESH=1
  • [ ] Custom parser (if needed) registered with @register_parser
  • [ ] Custom producer (if needed) auto-detected by make scrape
  • [ ] Test fixtures added to tests/fixtures/mysite/
  • [ ] Prefect deployment added to SITE_SCHEDULES
  • [ ] Makefile dashboard display name added (if desired)