Search API

A FastAPI server providing semantic search over scraped and LLM-processed data.

make api                          # Start on port 8000
make api PORT=9000                # Custom port

Endpoints

POST /search

Semantic search across processed chunks.

Request:

{
    "query": "new development flat with pool in Madrid",
    "job_id": "my-run",
    "site": "idealista",
    "stage": "detail",
    "chunk_types": ["description"],
    "top_k": 5,
    "min_similarity": 0.0,
    "include_metadata": true,
    "rerank": true,
    "deduplicate": true
}

Response:

{
    "results": [
        {
            "id": "uuid-...",
            "content": "Bright 3-bedroom apartment with communal pool...",
            "chunk_type": "description",
            "source_url": "https://idealista.com/...",
            "site": "idealista",
            "stage": "detail",
            "score": 0.87,
            "metadata": {
                "price": 350000,
                "location": "Madrid",
                "size": 120
            }
        }
    ],
    "total": 1
}

GET /health

Health check endpoint.

{
    "status": "healthy",
    "vector_count": 15420,
    "models_loaded": true
}

GET /job/{job_id}/status

Check if a job's data has been fully processed.

{
    "job_id": "my-run",
    "status": "completed",
    "total_chunks": 523,
    "sites": ["idealista"],
    "completed_at": "2024-01-15T10:30:00Z"
}

Query Parameters

Parameter	Type	Default	Description
query	string	—	Natural language query (required)
job_id	string	—	Filter to specific scrape job
site	string	null	Filter by site
stage	string	null	Filter by pipeline stage
chunk_types	array	null	Filter by chunk type
top_k	int	5	Max results (1–100)
min_similarity	float	0.0	Minimum cosine similarity
rerank	bool	true	Apply cross-encoder reranking
deduplicate	bool	true	One result per source URL

Search Flow

graph LR
    Q[Query] --> EM[Embedder]
    EM -->|"vector"| VS[Vector Store]
    VS -->|"top 50"| RR[Reranker<br/>(cross-encoder)]
    RR -->|"top 5"| RES[Response]

1. Query is embedded using the same bge-small-en-v1.5 model
2. Approximate nearest-neighbor search via pgvector HNSW index returns top 50
3. Optional cross-encoder reranker re-scores candidates for precision
4. Optional dedup: only highest-scoring chunk per source URL
5. Metadata fields (non-chunked) are included for context