Storage Plugins¶

Storage plugins persist parsed items to various backends. Multiple storage backends can be active simultaneously.

Available Storage Plugins¶

Plugin ID	File	Format
`garage_s3`	`garage_s3_storage.py`	Parquet files in S3-compatible storage
`postgres`	`postgres_storage.py`	JSONB rows in PostgreSQL
`json`	`json_storage.py`	JSON / JSONL files on disk
`csv`	`csv_storage.py`	CSV files on disk
`markdown`	`markdown_storage.py`	Human-readable Markdown files

Configuration¶

Storage is configured in the YAML profile:

storage: [garage_s3, json]       # Enable S3 + JSON storage
storage_config:
  json:
    output_dir: ./output/mysite
    mode: jsonl                   # jsonl | json | pretty
  garage_s3:
    bucket: scraping-pipeline
    prefix: scrapes/
    site: mysite
    compression: zstd             # Parquet compression

Garage S3 Storage¶

ID: garage_s3

Stores items as Parquet files in Garage, an S3-compatible object store.

Columnar format — efficient for analytics and ML workloads
Zstd compression — configurable compression level
Partitioned layout — site=<name>/job_id=<id>/ prefix structure
Async I/O — uses aiobotocore for non-blocking S3 operations
Batch writes — items are accumulated and flushed as full row groups

Partition Structure¶

scrapes/
├── site=idealista/
│   ├── job_id=run_20240101/
│   │   ├── listing.parquet
│   │   └── detail.parquet
│   └── job_id=run_20240102/
│       ├── listing.parquet
│       └── detail.parquet
└── site=g2/
    └── ...

PostgreSQL Storage¶

ID: postgres

Stores items as JSONB rows in the scraped_items table.

CREATE TABLE scraped_items (
    source_url  TEXT PRIMARY KEY,
    job_id      TEXT NOT NULL DEFAULT '',
    data        JSONB NOT NULL,
    scraped_at  TIMESTAMPTZ NOT NULL DEFAULT now()
);

Upsert semantics — source_url is the primary key, so re-scraping updates existing rows
JSONB flexibility — schemaless; any parsed fields are stored
Async driver — uses asyncpg for non-blocking access

JSON Storage¶

ID: json

Writes items to local JSON or JSONL files.

output/
└── mysite/
    ├── listing_20240101.jsonl
    └── detail_20240101.jsonl

Modes:

jsonl — one JSON object per line (good for streaming/bulk import)
json — single JSON array
pretty — formatted JSON (human-readable)

CSV Storage¶

ID: csv

Writes items to CSV files. Flattens nested fields into dot-notation columns.

output/
└── mysite/
    ├── listing_20240101.csv
    └── detail_20240101.csv

Markdown Storage¶

ID: markdown

Creates human-friendly Markdown files, one per item.

output/
└── mysite/
    └── detail/
        ├── 20240101_item_001.md
        ├── 20240101_item_002.md
        └── ...

Useful for: - Manual review of scraped content - Quick debugging without loading structured data tools - Documentation / reporting