Setup & Configuration¶

Requirements¶

Docker + Docker Compose (for infrastructure and containerized workers)
Python 3.12 and uv (for host-side runs)
A residential/rotating proxy (for anti-bot sites)

Quick Start¶

1. Environment Variables¶

Copy .env and fill in the required keys:

cp .env.example .env

Required variables:

PROXY_URL=...                          # residential proxy (required for anti-bot sites)
REDIS_URL=redis://localhost:6379/0
RABBITMQ_DSN=amqp://guest:guest@localhost:5672/
GARAGE_S3_ENDPOINT=http://localhost:3900
GARAGE_S3_ACCESS_KEY=...
GARAGE_S3_SECRET_KEY=...
GARAGE_S3_DEFAULT_BUCKET=scraping-pipeline
POSTGRES_DSN=postgresql+asyncpg://user:pass@localhost/scraper
SLACK_WEBHOOK_URL=...                  # optional: job summaries + structure-change alerts

2. Local Development¶

Host workers with hot-reloadable code, infra in Docker:

make up                                   # infra: redis, rabbitmq, postgres, garage
make workers                              # dedup + fetch + parse + store (background)

make scrape JOB=my-run PROFILE=idealista FRESH=1
make stats  JOB=my-run                    # pipeline progress + per-pool stats

make llm-process JOB=my-run PROFILE=idealista   # chunk + embed + index (optional)
make api                                  # search API on :8000 (foreground)

make kill-workers                         # stop host workers
make down                                 # stop infra

FRESH=1 clears the job cursor, dedup filter, and prior parquet output for a clean run.

3. Production (Containerized)¶

make up-workers        # infra + fetch/dedup/parse/store worker containers
make up-api            # search API container on :8000
make prefect-build     # build the Prefect worker image
make prefect-up        # Prefect server (:4200) + services + worker

Makefile Reference¶

Infrastructure¶

Command	Description
`make up`	Start infra services (Docker)
`make up-workers`	Start containerized pipeline workers
`make up-api`	Start search API container
`make down`	Stop all services
`make down-v`	Stop + nuke all volumes

Workers¶

Command	Description
`make workers`	Start all workers in background (host)
`make kill-workers`	Stop all background workers
`make restart-workers`	Restart all background workers

Scraping¶

Command	Description
`make scrape JOB=<id> PROFILE=<name> [FRESH=1]`	Run a scrape
`make stats JOB=<id>`	Pipeline progress
`make dashboard`	Cumulative per-site metrics
`make version-check PROFILE=<name> [UPDATE=1]`	Live structure check

LLM¶

Command	Description
`make llm-process JOB=<id> PROFILE=<name>`	Process into LLM-ready chunks
`make llm-status JOB=<id>`	LLM pipeline status

Search API¶

Command	Description
`make api`	Start API on port 8000
`make api PORT=9000`	Start API on custom port

Docker Services¶

The docker-compose.yml defines the full stack:

Service	Image	Purpose
`rabbitmq`	`rabbitmq:3.13-management`	Message broker
`redis`	`redis/redis-stack:latest`	Cache + dedup + state
`postgres`	`pgvector/pgvector:pg16`	Primary DB + vectors
`garage`	`dxflrs/garage:v2.3.0`	S3-compatible object store
`garage-webui`	`khairul169/garage-webui`	Garage admin UI
`fetch-worker`	custom (×2 replicas)	Browser-based fetching
`dedup-worker`	custom	URL deduplication
`parse-worker`	custom	HTML/JSON parsing
`store-worker`	custom	Data persistence
`api`	custom	FastAPI search server

Logs¶

Local dev — rotating JSON files under logs/:

tail -f logs/fetch_worker.log
tail -f logs/parse_worker.log logs/store_worker.log logs/dedup_worker.log
tail -f logs/producer_idealista.log
tail -f logs/site_versioning.log

Production — stdout via docker compose logs:

docker compose logs fetch-worker -f
docker compose logs -f parse-worker store-worker dedup-worker
docker compose logs api -f

[!TIP] This file is a starting point. Enrich with example .env files, troubleshooting steps, and platform-specific notes using Claude.