Profiles (YAML Configuration)

Each site is defined by a YAML profile in profiles/<site>.yaml. Most sites require no new Python code — just a YAML file.

Profile Structure

site: mysite                  # Site identifier
plugin: cloakbrowser          # cloakbrowser | wreq

proxy: ${PROXY_URL}           # Proxy URL (from env)
proxy_country: "es"           # Optional: injects _country-XX into proxy password
headless: true                # Run browser headless

rate_limit:                   # Per-stage rate limiting
  requests: 2
  window_seconds: 10.0

retry:                        # Retry policy
  max_retries: 3
  delay_seconds: 8.0
  backoff: fixed              # fixed | exponential
  retry_on_status: [429, 503, 403]

stages:                       # Pipeline stages (listing → detail, etc.)
  listing:
    plugin: cloakbrowser
    parser: generic           # Plugin ID for the parser
    container_selector: "article.item"   # CSS selector for item containers
    selectors:
      title: ".title"
      price: ".price"
    follow_selectors: ["a.item-link"]    # Links to follow → next stage
    follow_stage: detail
    next_page_selector: "li.next a"      # Pagination
    use_pool: true
    warmup_url: "https://mysite.com/"

  detail:
    plugin: cloakbrowser
    parser: generic
    selectors:
      title: "h1"
      price: ".price"
      location: ".addr"
    validation:
      required_fields: [title, price]
      strategy: drop           # drop | flag

storage: [garage_s3, json]     # Storage backends to use
storage_config:
  json:
    output_dir: ./output/mysite
    mode: jsonl                # jsonl | json

llm:                           # Optional LLM pipeline config
  enabled: true
  stage: detail
  fields:
    - preset: real_estate_listing_v1
    - name: custom_field
      type: string
      ...

Stage Configuration Reference

Field	Type	Description
plugin	cloakbrowser	wreq
parser	string	Parser plugin ID
container_selector	string	CSS selector for item containers (generic parser)
selectors	dict	Field name → CSS selector / JSON path map
follow_selectors	list	CSS selectors for links to follow
follow_stage	string	Target stage for followed links
next_page_selector	string	CSS selector for "next page" link
use_pool	bool	Use browser pool (shared sessions)
warmup_url	string	URL to visit before scraping
validation	object	Field validation settings

LLM Field Presets

Presets are defined in core/llm_presets.py and referenced by name in profiles:

Preset	Content Type	Fields
ecommerce_product_v1	Luxury fashion / retail	name, sku, price, description, attributes, care info
real_estate_listing_v1	Property listings	title, price, location, description, features
(enrich with all available presets)

Existing Profiles

Profile	Site	Plugin	Parser
idealista	idealista.com	CloakBrowser	generic + custom detail
g2	g2.com	CloakBrowser	custom listing
hermes	hermes.com	CloakBrowser	custom listing
trustpilot	trustpilot.com	CloakBrowser	nextdata parser
booking_com	booking.com	wreq (GraphQL)	(API response)
facebook_marketplace	facebook.com	wreq (GraphQL)	(API response)

Profile Resolution

The core/profiles.py module handles:

1. Environment variable interpolation — ${VAR} references in YAML are resolved from os.environ at load time
2. Proxy country injection — _country-XX suffix is appended to proxy passwords when proxy_country is set
3. Preset expansion — llm.fields references to presets are expanded into full field definitions
4. Site config models — custom site_config: blocks are validated against registered Pydantic models

---

[!TIP] See the Contributing guide for step-by-step instructions on adding a new site profile.