Fetcher Plugins¶

Fetchers are responsible for retrieving raw HTML or JSON from target URLs. The framework ships with two fetcher plugins.

CloakBrowser Fetcher¶

Plugin ID: cloakbrowser

A patched Chromium browser built for anti-bot evasion. Handles:

DataDome, Cloudflare WAF, Akamai challenges
Browser fingerprint rotation — each session gets a unique fingerprint
Residential proxy integration — routes traffic through rotating IPs
Session warmup — visits a homepage or seed URL before the target to establish cookies/identity
Block detection — heuristics to identify challenge pages and auto-rotate sessions

Configuration¶

stages:
  listing:
    plugin: cloakbrowser
    headless: true                 # Run without GUI
    use_pool: true                 # Share browser pool across requests
    warmup_url: "https://site.com/"  # Seed cookies before target
    concurrency: 3                 # Parallel browser sessions
    pool_size: 5                  # Max sessions in pool

Browser Pool¶

The BrowserPool class (core/browser_pool.py) manages CloakBrowser sessions:

graph TD
    BP[BrowserPool] -->|"acquire()"| S1[Session 1]
    BP -->|"acquire()"| S2[Session 2]
    BP -->|"acquire()"| SN[Session N]
    S1 -->|"release()"| BP
    S2 -->|"release()"| BP

    subgraph Reaper
        R[Background Reaper]
        R -->|"sweeps stale"| S1
    end

    subgraph Stats
        M[Metrics]
        M --> blocks[block count]
        M --> success[success count]
    end

Key features: - Sticky proxies — sessions are pinned to specific proxy IPs - Stale reaper — background task closes sessions that exceed age limits - Block detection — parses response for known challenge patterns - Health checks — periodic session health verification - Warmup — executes warmup URLs before first fetch per session

Wreq Fetcher¶

Plugin ID: wreq

A lightweight HTTP fetcher using TLS-fingerprinted requests. Used for:

JSON/GraphQL APIs — no browser overhead
Sites where a browser bootstrap already captured tokens
High-throughput API scraping

Configuration¶

stages:
  detail:
    plugin: wreq
    headers:
      User-Agent: "Mozilla/5.0 ..."
      Accept: "application/json"

Usage Pattern¶

For sites like facebook_marketplace and booking_com:

Bootstrap phase — CloakBrowser session logs in / navigates, captures cookies and API tokens
Scrape phase — wreq sends authenticated GraphQL POST requests using the captured session

Key Differences¶

Aspect	CloakBrowser	wreq
Engine	Patched Chromium	TLS-fingerprinted HTTP
Anti-bot	DataDome, Cloudflare, Akamai	None (relies on tokens)
Overhead	High (browser launch)	Low (HTTP request)
Sessions	Long-lived browser pool	Stateless
Block detection	Heuristic (HTML analysis)	Status code + response shape