Site Versioning (Structure Guard)¶

Anti-bot sites silently change their DOM / API shape. If a scrape runs against a changed structure, parsers store garbage and burn proxy budget doing it.

The structure guard fingerprints each site's structure and gates every scheduled scrape on a live check.

How It Works¶

graph TD
    A[Scheduled Scrape Starts] --> B{Fingerprint<br/>Current Structure}
    B -->|"Compare to Baseline"| C{Match?}
    C -->|"✅ ok"| D[Proceed with Scrape]
    C -->|"⚠️ changed"| E[Skip Scrape]
    C -->|"❓ unverified"| F{Gate Mode?}
    F -->|"fail-closed"| G[Skip - Alert]
    F -->|"fail-open"| D

    E --> H[Post Slack Alert]
    G --> H

Outcomes¶

Outcome	Exit Code	Gate Behaviour
`ok`	0	✅ Scrape proceeds
`changed`	1	⛔ Scrape skipped — fix selectors + re-baseline first
`unverified`	2	⛔ Skipped when `gate=fail-closed` (default); scrapes anyway on `fail-open`

Fingerprinting¶

Fingerprints are extracted from HTML or JSON responses and capture:

HTML Fingerprint (`extract_html_fingerprint`)¶

Script tag inventory — identifies data embedded in known script patterns:
__NEXT_DATA__ (Next.js apps)
application/ld+json (JSON-LD schema)
data-capla-store-data="apollo" (Apollo GraphQL state)
id="hermes-state" (Hermès-specific state)
JSON key paths — recursively flattened dot-paths of extracted JSON data
CSS selector inventory — class names and IDs from the profile's selectors are checked against the actual DOM; missing classes are flagged
DOM stats — element count, max depth, script/link/image counts
Hash — deterministic SHA-256 fingerprint (first 12 hex chars)

JSON/API Fingerprint (`extract_json_fingerprint`)¶

For API-based sites (booking_com, facebook_marketplace), the fingerprint is derived from JSON key paths only. GraphQL extensions envelopes are excluded because they contain session-scoped metadata that rotates between sessions.

Comparison¶

The compare_fingerprints function checks:

Change Type	Severity	Example
Script removed	Critical	`__NEXT_DATA__` missing — parser breaks
Script added	Info	New analytics script
JSON path removed	Critical	`data.product.price` missing
JSON path added	Info	New optional field
CSS selector missing	Warning	`.price-class` no longer in DOM
DOM stat change >30%	Warning	Elements: 500 → 1200

Running the Check¶

Live Check (via workers)¶

# What the Prefect gate uses — real fetch through the browser tier
make version-check PROFILE=idealista

# Accept current structure as new baseline (after intentional changes)
make version-check PROFILE=idealista UPDATE=1

Fixture Check (offline, CI-friendly)¶

# Check all fixture-backed profiles against saved HTML
uv run python scripts/check_versions.py check --fixture

# Refresh all fixture baselines after accepting parser changes
uv run python scripts/check_versions.py check --fixture --update

Scheduled via Prefect: check-site-versions runs at 02:30 daily.

Baselining¶

Before a site can be guarded, it needs a baseline fingerprint:

# Capture a new baseline (live)
make version-check PROFILE=mysite UPDATE=1

# Or offline from a fixture
uv run python scripts/check_versions.py check mysite --fixture --update

Baselines are stored as JSON in tests/fixtures/<profile>/<stage>_fingerprint.json:

{
    "profile": "idealista",
    "stage": "listing",
    "captured_at": "2024-01-15T10:00:00+00:00",
    "fingerprint": {
        "scripts": ["jsonld"],
        "json_paths": ["@type", "name", "description", "offers.price"],
        "css_selectors": [".item-info", ".price"],
        "dom_stats": {"elements": 850, "max_depth": 12, ...},
        "hash": "a1b2c3d4e5f6"
    }
}

[!IMPORTANT] Run --update only after you've confirmed the change is real and updated the parser — otherwise you're just freezing broken structure into the baseline.