Prefect Orchestration¶

In production, Prefect manages the scraping lifecycle — scheduling, gating, monitoring, and alerting.

Architecture¶

┌──────────────────────┐
│   Prefect Server     │ ← Web UI on :4200
│   (prefect-server)   │
└──────────┬───────────┘
           │
┌──────────▼───────────┐
│   Prefect Worker     │ ← Runs flows
│   (prefect-worker)   │
└──────────┬───────────┘
           │
┌──────────▼───────────┐
│   Pipeline Workers   │ ← Always-on (dedup/fetch/parse/store)
│   (docker-compose)   │
└──────────────────────┘

Prefect owns batch control — it triggers and monitors scrape runs. The pipeline workers (dedup, fetch, parse, store) are always-on services managed by Docker Compose.

Flows¶

`scrape-<site>` Flow¶

The main flow for each site. Registered as a deployment per site with its own cron schedule.

preflight → verify structure (GATE) → producer → wait for drain → llm-process → report

Step	Description
Preflight	Checks Redis connectivity and that all queues have consumers. Fails fast if workers are down.
Verify Structure	Runs a live structural fingerprint check. Gates the scrape — if the site changed, the scrape is skipped fail-closed.
Producer	Runs the producer to enqueue URLs. Waits for completion with a timeout (default 30 min, up to 2h for bootstrap-heavy sites).
Wait for Drain	Polls queue depths and job metrics until queues are empty and stats are stable past the store worker's flush window. Ensures the LLM step doesn't run on partial data.
LLM Process	Runs the chunk → embed → index pipeline.
Report	Logs final stats and posts a Slack summary (if configured).

`check-site-versions` Flow¶

Runs the fixture-based structural fingerprint check daily (02:30) before the earliest scrape. Catches structure drift proactively.

`daily-health-check` Flow¶

Checks site metrics and search API health daily at 09:00.

Deployments & Schedules¶

Registered in prefect/register_flows.py (single source of truth):

Deployment	Schedule	Description
`scrape-idealista`	Daily 03:00	Idealista property scrape
`scrape-g2`	Monday 04:00	G2 reviews
`scrape-hermes`	Tuesday 04:00	Hermès products
`scrape-trustpilot`	Wednesday 04:00	Trustpilot reviews
`scrape-booking_com`	Manual only	Booking.com hotels
`scrape-facebook_marketplace`	Manual only	Facebook Marketplace
`check-site-versions`	Daily 02:30	Structure fingerprint check
`daily-health-check`	Daily 09:00	Health check

Concurrency¶

The work pool has a concurrency limit of 1. Scheduled scrapes queue instead of fighting over the browser fleet / proxy budget.

Administration¶

# Start/stop the Prefect stack
make prefect-up          # Start server + worker + services
make prefect-down        # Stop everything

# Trigger a flow manually
make prefect-flow FLOW=scrape-site PROFILE=idealista

# Open the Prefect UI
make prefect-ui          # http://localhost:4200

# Rebuild the worker image (after code changes)
make prefect-build

# View logs
make prefect-logs        # Tail all Prefect logs

Alerting¶

If SLACK_WEBHOOK_URL is configured, the flow posts:

Job summary on completion (success/fail, items scraped, block rate)
Structure change alerts when a site's fingerprint has changed
Error notifications on pipeline failures