Scraping Framework¶
Modular, queue-based web scraping system for anti-bot-protected sites.
Producers enqueue URLs, a browser-based fetch tier bypasses WAF/anti-bot challenges, parsers extract structured data, and an optional LLM stage chunks + embeds results into a pgvector-backed search API. Orchestrated end-to-end with Prefect, and guarded by a live site-structure version check so a silently redesigned site never poisons your data.
Quick Overview¶
graph LR
Producer["Producer<br/>(enqueues URLs)"]
Dedup["Dedup Worker<br/>(Bloom filter)"]
Fetch["Fetch Worker<br/>(CloakBrowser / wreq)"]
Parse["Parse Worker<br/>(CSS / JSON / custom)"]
Store["Store Worker<br/>(S3 / Postgres / files)"]
LLM["LLM Pipeline<br/>(chunk → embed → index)"]
API["Search API<br/>(FastAPI + pgvector)"]
Producer -->|url.queue| Dedup
Dedup -->|fetch.queue| Fetch
Fetch -->|raw.queue| Parse
Parse -->|parsed.queue| Store
Store --> LLM
LLM --> APISupported Sites¶
| Site | Anti-bot Tech | Fetch Strategy | Success Rate |
|---|---|---|---|
| idealista | DataDome | CloakBrowser + ES proxy | 91.7% |
| g2 | Cloudflare + DataDome | CloakBrowser + IP rotation | 99.5% |
| hermes | DataDome | CloakBrowser + warmup | 84.1% |
| trustpilot | Cloudflare WAF | CloakBrowser + __NEXT_DATA__ |
98.4% |
| facebook_marketplace | GraphQL + login | Bootstrap → wreq GraphQL |
100% |
| booking_com | Akamai + GraphQL | Bootstrap → wreq GraphQL |
100% |
[!TIP] Blocked sessions are rotated automatically (fresh fingerprint/IP retries the URL), so block rate ≠ data loss.
Key Features¶
- Plugin-based architecture — fetchers, parsers, and storage backends are swappable
- Config-driven — most new sites need only a YAML profile; no code required
- Fail-closed structure guard — scrapes are automatically skipped if the target site's DOM/API shape has changed
- Resume capability — job cursors in Redis allow interrupted scrapes to continue
- LLM pipeline — semantic chunking, embedding (384-dim vectors), and reranked search via pgvector
- Prefect orchestration — cron-scheduled, gated, with Slack alerting
Tech Stack¶
| Category | Technology |
|---|---|
| Language | Python 3.12 |
| Fetch | CloakBrowser (Chromium), wreq (TLS-fingerprinted HTTP) |
| Queue | RabbitMQ |
| Cache / Dedup | Redis + RedisBloom |
| Storage | Garage (S3), PostgreSQL + pgvector, local files |
| ML/NLP | sentence-transformers (bge-small-en-v1.5) |
| Orchestration | Prefect |
| Search API | FastAPI |
| CI/CD | GitLab CI |
| Containers | Docker + Compose |
Next: Architecture — understand the pipeline in detail.