Parser Plugins¶

Parsers extract structured data from raw HTML or JSON responses. They are selected per-stage in the YAML profile.

Available Parsers¶

Plugin ID	File	Best For
`generic`	`generic_parser.py`	CSS-selector based extraction from HTML
`jsonld`	`jsonld_parser.py`	JSON-LD structured data in `<script>` tags
`jsonpath`	`jsonpath_parser.py`	JSONPath extraction from API responses
`apollo_json`	`apollo_json_parser.py`	Apollo GraphQL client-side state
`g2_listing`	`g2_listing_parser.py`	G2.com product/review pages
`hermes_listing`	`hermes_listing_parser.py`	Hermès product listings
`idealista_detail`	`idealista_detail.py`	Idealista property details
`trustpilot_nextdata`	`trustpilot_nextdata_parser.py`	Trustpilot `__NEXT_DATA__` extraction

Generic Parser¶

ID: generic

A CSS-selector-based parser that needs no custom code. Configured entirely in YAML:

stages:
  listing:
    parser: generic
    container_selector: "article.item"
    selectors:
      title: "h2.title"
      price: "span.price"
      url: "a[href] @href"      # Attribute extraction with @ prefix
    follow_selectors: ["a.detail-link"]

Selector Syntax¶

Text content: "h2.title" → extracts inner text
Attribute: "a[href] @href" → extracts the href attribute
HTML: "div.content @html" → extracts inner HTML
Nested: Sub-selectors within containers

JSON-LD Parser¶

ID: jsonld

Extracts structured data from <script type="application/ld+json"> tags commonly used for schema.org markup. Useful for products, reviews, and business data.

stages:
  detail:
    parser: jsonld
    selectors:
      name: "name"              # JSON path within the LD+JSON object
      description: "description"
      price: "offers.price"

JSONPath Parser¶

ID: jsonpath

Extracts data from JSON API responses using JSONPath expressions. Used with the wreq fetcher for API-based scraping.

stages:
  detail:
    plugin: wreq
    parser: jsonpath
    selectors:
      title: "$.data.product.title"
      price: "$.data.product.price.amount"

Apollo JSON Parser¶

ID: apollo_json

Extracts data from Apollo GraphQL client-side state embedded in <script> tags with data-capla-store-data="apollo".

Site-Specific Parsers¶

These parsers handle unique site structures that can't be expressed with generic selectors alone.

G2 Listing Parser¶

Extracts product information from G2.com category/review pages. Handles:

Product cards with ratings, reviews count, description
JSON-LD + microdata mixed content
Pagination state

Hermès Listing Parser¶

Extracts luxury product data from Hermès.com. Handles:

Product variant selection
Color/material options
Stock status

Idealista Detail Parser¶

Extracts property details from Idealista.com. Handles:

Property features and amenities
Location data
Price history
Agent information

Trustpilot NextData Parser¶

Extracts review data from Trustpilot's __NEXT_DATA__ embedded state. Handles:

Review text and ratings
Reviewer information
Pagination state within Next.js data

Custom Parser Quick Start¶

Create a new file in plugins/parsers/:

from core.registry import register_parser
from plugins.base import BaseParserPlugin

@register_parser
class MySiteParser(BaseParserPlugin):
    plugin_id = "mysite_parser"

    def parse(self, raw: dict) -> list[dict]:
        html = raw["html"]
        # Use selectolax, re, or json to extract data
        return [{"title": "Example", "price": "100"}]

Then reference it in the profile:

stages:
  listing:
    parser: mysite_parser