Skip to content

Anti-Bot Strategies

Each site presents a unique anti-bot challenge. This document catalogs what protection each site uses and how the scraping system bypasses it.

pokemoncenter

Layer Technology How It Works
WAF / human behavior Imperva (Incapsula) JS challenge that evaluates mouse movement, scrolling, and timing patterns to distinguish humans from automation
TLS / fingerprint DataDome TLS handshake fingerprinting, browser canvas/WebGL/font fingerprinting, request header ordering checks

Challenge Flow

  1. Browser lands on https://www.pokemoncenter.com/
  2. Imperva fires a JS challenge (~5–10s) — evaluates human-likeness of the browser session
  3. If Imperva passes, DataDome issues a session cookie based on TLS fingerprint + browser profile
  4. Navigating to a search page (/search/pikachu) sets the proper referrer context
  5. Product data is fetched from the internal API (/tpci-ecommweb-api/search) which validates both the DataDome session cookie and request headers

Bypass Strategy

┌──────────────┐     ┌──────────────────┐     ┌──────────────────┐
│ 1. Homepage  │────▶│ 2. Search Page   │────▶│ 3. API fetch()   │
│  warmup      │     │  /search/pikachu │     │  via browser JS  │
│  (DataDome)  │     │  (referrer ctx)  │     │  (session cookie)│
└──────────────┘     └──────────────────┘     └──────────────────┘

Key tactics:

  1. Real browser, not Playwright Chrome. Uses CloakBrowser (patched Chromium) with a natural fingerprint — not Playwright's Chromium which DataDome flags as automation.

  2. Homepage warmup. The browser first navigates to the homepage and waits 15 seconds for the Imperva + DataDome challenges to resolve. Without this step, the API calls are blocked immediately.

  3. Search page context. page.goto() to /search/pikachu sets document.referrer and window.location.href — required by the API's ref_url and url parameters.

  4. In-browser API calls. The API is called via page.evaluate(fetch(...)) — the fetch runs inside the browser's JavaScript context, automatically carrying the DataDome session cookie and browser fingerprint. A direct HTTP request (even with the cookie) would fail because DataDome validates the full TLS fingerprint, not just the cookie.

  5. Dynamic pagination. Page 1's response includes numFound — the total number of products. The producer caps actual pages to min(profile_max_pages, ceil(numFound/rows)), never requesting a page beyond what exists.

  6. Lightweight by design. Only the producer uses a browser (for bootstrap + inline API calls). No fetch/parse workers needed — the JSON API response is parsed directly by the jsonpath parser and published to parsed.queue.

Producer Flow

workers/producers/pokemoncenter.py:
  1. launch cloakbrowser (headless)
  2. goto homepage → sleep 15s → check for block (geo.captcha-delivery.com)
  3. for each search term:
     a. goto /search/{term} → sleep 3s → check for block
     b. fetch API page 1 (start=0) via page.evaluate(fetch(...))
     c. extract numFound → cap total_pages
     d. fetch remaining pages
     e. publish items to parsed.queue
  4. close browser

Versioning Note

Pokemon Center's search page is a Next.js app. The HTML embeds a __NEXT_DATA__ script with Apollo cache IDs that contain dynamic hash segments (e.g. uf7da63854cad4d15b4ae6ebb25010c93) that change between deployments/sessions. The fingerprinting system filters out these dynamic segments automatically by skipping dict keys that are 32+ hex characters, so false-positive structure "changes" are avoided.

Metrics

Metric Value
Success rate 100.0%
Block rate 0.0%
Fetches 15
Items 480

idealista

Layer Technology
Geo-blocking ES-only access (blocked from non-Spanish IPs)
Anti-bot DataDome

Bypass: Spanish residential proxy (proxy_country: "es"), locationsSuggest warmup to seed DataDome cookies, same-session inline detail follows to avoid re-challenge.

g2

Layer Technology
WAF Cloudflare
Anti-bot DataDome (TLS fingerprint + browser fingerprint)

Bypass: Homepage warmup, random IP rotation per session, CloakBrowser with Windows fingerprint (--fingerprint-platform=windows), headed mode.

hermes

Layer Technology
Anti-bot DataDome (TLS fingerprint + browser fingerprint)

Bypass: Homepage warmup (warmup_url: "https://www.hermes.com/us/en/"), headless mode (DataDome bypass succeeds without proxy), fresh session per page (via pool_idle_timeout: 5).

trustpilot

Layer Technology
WAF Cloudflare

Bypass: CloakBrowser with homepage warmup, __NEXT_DATA__ extraction for data (no API calls needed), 2 concurrent sessions.

booking_com

Layer Technology
CDN / WAF Akamai
API GraphQL with signed persisted queries

Bypass: CloakBrowser bootstrap that performs a real destination search, captures session cookies + CSRF token + URL params (dest_id, dest_type, aid), then switches to wreq for lightweight GraphQL POST requests with the captured session.

facebook_marketplace

Layer Technology
Auth Login wall + session cookies
API GraphQL with doc_id + fb_dtsg token

Bypass: CloakBrowser bootstrap that navigates marketplace, selects location, intercepts GraphQL POST requests to capture fb_dtsg token + session cookies, then switches to wreq for lightweight API calls.