{"id":"hybrid-search-retrieval","slug":"hybrid-search-retrieval","title":"Hybrid Search Retrieval","description":"Keyword search finds documents containing the exact terms in a query but misses conceptually related content expressed in different words. Dense vector search captures semantic similarity but can underweight precise term","category":"modules","tags":["modules","ai"],"lastModified":"2026-04-14","source_ref":"content/modules/hybrid-search-retrieval.md","url":"/developers/hybrid-search-retrieval","htmlPath":"/developers/hybrid-search-retrieval","jsonPath":"/api/docs/modules/hybrid-search-retrieval","markdownPath":"/api/docs/modules/hybrid-search-retrieval?format=markdown","checksum":"467803e56ad3a2e9d42dfcff3d6ab06ff449c7b7e2171d3636fa6c70bfec69be","headings":[{"id":"overview","text":"Overview","level":2},{"id":"key-features","text":"Key Features","level":2},{"id":"use-cases","text":"Use Cases","level":2},{"id":"integration","text":"Integration","level":2},{"id":"open-standards","text":"Open Standards","level":2}],"markdown":"# Hybrid Search Retrieval\n\n## Overview\n\nKeyword search finds documents containing the exact terms in a query but misses conceptually related content expressed in different words. Dense vector search captures semantic similarity but can underweight precise term matches that matter in intelligence and legal work. Running both in parallel and fusing the ranked results combines the precision of lexical search with the recall of semantic search, producing a ranked list that neither method achieves alone.\n\nThe Hybrid Search Retrieval module combines PostgreSQL full-text search (BM25-style ts_rank scoring) with Cloudflare Vectorize dense semantic search, then merges the two ranked lists using Reciprocal Rank Fusion (Cormack et al. 2009, SIGIR, DOI: 10.1145/1571941.1572114). Both retrieval paths execute in parallel, and the fused ranking is returned in a single response. The alpha parameter controls the relative weight of dense versus BM25 results, defaulting to 0.5 for equal contribution. All queries are scoped to organization_id, satisfying EDF/PESCO data sovereignty requirements.\n\n```mermaid\ngraph LR\n    A[Query] --> B[BM25 PostgreSQL ts_rank]\n    A --> C[Dense Vectorize Embedding]\n    B --> D[RRF Fusion]\n    C --> D\n    D --> E[Ranked Results]\n```\n\n**Last Reviewed:** 2026-04-14\n**Last Updated:** 2026-04-14\n\n## Key Features\n\n- **Parallel Retrieval**: BM25 full-text search and dense vector search execute concurrently. If either path fails, the system degrades gracefully to the results of the surviving path rather than returning an error.\n\n- **Reciprocal Rank Fusion**: The two ranked lists are combined using the RRF algorithm (Cormack, Clarke, and Buettcher, SIGIR 2009). Each document receives a score of the form 1 / (k + rank), where k=60 is the standard smoothing constant. Documents appearing in both lists accumulate contributions from each, which promotes results with broad cross-system relevance.\n\n- **Configurable Alpha Weighting**: The alpha parameter sets the relative contribution of dense search versus BM25. alpha=0.5 gives equal weight. alpha=1.0 is dense-only. alpha=0.0 is BM25-only. Operators can tune this per deployment profile without rewriting queries.\n\n- **Organization Scoping**: Every PostgreSQL query includes organization_id in the WHERE clause, and every Vectorize query includes an org_id metadata filter. Cross-tenant result leakage is structurally prevented at the query level.\n\n- **Graceful Degradation**: If Vectorize is not configured or returns an error, the module falls back to BM25 results only. If PostgreSQL full-text search fails, dense results are used. Both failures are logged at WARNING level with the org_id context.\n\n- **GraphQL Query Surface**: The hybridSearch query accepts query, topK, and alpha arguments and returns a HybridSearchResultPayload containing a list of results with id, score, documentType, and title fields.\n\n## Use Cases\n\n- **Evidence Retrieval**: Find evidence items matching an investigator's query using both exact terminology and semantic similarity, improving recall when evidence descriptions use varied language.\n- **Intelligence Document Search**: Locate intelligence reports relevant to a named entity or concept regardless of whether the exact name appears in the document text.\n- **Cross-Language Concepts**: Dense search bridges gaps where translated or paraphrased content does not share vocabulary with the query.\n- **Incident Response**: Quickly surface related alerts, reports, and evidence across a case using a natural-language query when urgency does not permit iterative keyword refinement.\n\n## Integration\n\n- **Cross-Encoder Reranking**: The fused results can be passed to the Cross-Encoder Reranking module for a second-pass precision improvement before returning the final top-k to the caller.\n- **Search Index Management**: Indexes managed through the Search Index domain are the document source for BM25 search.\n- **Cloudflare Vectorize**: Dense search uses the VectorizeService client to query per-organisation vector indexes.\n- **AI Embedding Generation**: Query embeddings are generated via the EmbeddingService before being submitted to Vectorize.\n\n## Open Standards\n\n- Cormack, G.V., Clarke, C.L.A., and Buettcher, S. (2009). Reciprocal Rank Fusion Outperforms Condorcet and Individual Rank Learning Methods. SIGIR 2009, pp. 758-759. DOI: 10.1145/1571941.1572114\n- PostgreSQL full-text search using to_tsvector / plainto_tsquery / ts_rank (PostgreSQL 15+)\n- Cloudflare Vectorize (cosine similarity, L2, dot product)\n"}