Tech AI News - 테카이

Hybrid search: why a vector DB alone isn't enough

· 5 min read

This post was translated from the Korean original by AI.한국어 원문 읽기 →

A common story: someone puts embeddings into a vector DB and wires up search, then types in an error code or a product model number and gets back completely irrelevant results. Semantic search is strong at finding things that are "similar in meaning," but weak at queries that need an exact string match. Hybrid search is the idea of combining semantic search with keyword search to cover that weakness.


Background — why this term now

Embedding-based vector search converts sentences or words into numeric coordinates to find things that are "similar in meaning." But this conversion creates a problem: because it compresses (pools) the whole set of tokens into one fixed-size vector, the identity of strings that only matter when they match exactly — error codes like ERR_SSL_VERSION_OR_CIPHER_MISMATCH, or product model names like RTX-4090 — gets blurred. tianpan.co cites these kinds of identifiers, SKUs, function names, and rare proper nouns as classic cases vector search often misses.

Keyword search based on an inverted index (BM25), on the other hand, has the simple structure of finding a match only if the exact token is present — so searching for "vacation" won't surface "PTO." Hybrid search runs both searches in parallel to fill in each other's gaps, and has become an established way to raise retrieval quality in RAG (retrieval-augmented generation) pipelines.

Key data & current state

How hybrid search is implemented varies a bit from platform to platform.

PlatformFusion methodNotes
LangChain EnsembleRetrieverCombines BM25Retriever and a vector retriever with weights (e.g. 0.5:0.5), then re-ranks with RRF (Reciprocal Rank Fusion)Recommended to tune weights to your data
WeaviateNative BM25 + vector search, with alpha (0–1) controlling the mixSince v1.24, relativeScoreFusion, which normalizes and combines scores, is the default; previously rankedFusion, which uses only rank, was the default (Weaviate official docs)
PineconeTreats sparse and dense as one vector, scaling directly at query time by multiplying the dense value by alpha and the sparse value by (1-alpha)For natural-language queries, alpha 0.75 (favoring the vector side) is the default recommendation (Pinecone official docs). The community has also reported that relevance drops when alpha is below 0.5 (Pinecone community), so it needs to be measured directly
ElasticsearchRRF as default fusion + per-field weight tuningIn a 2025 case study on the Wands furniture dataset, default RRF raised NDCG about 1.3% over BM25 alone, and staged tuning — weighting exact matches 100x and partial matches 10x — raised it to 7.5% (tianpan.co)

Some reports say adding reranking makes the improvement even larger. According to an Analytics Vidhya piece citing an Anthropic benchmark, combining contextualized embeddings with contextualized BM25 and then running a reranker reduced the retrieval failure rate — measured on the top 20 chunks — from 5.7% to 1.9%, about a 67% drop. However, this figure was measured by Anthropic on its own data, and it's hard to say it applies as-is across every domain.

Deep dive

Hybrid search feeds the same query into two retrievers at once. One is dense search, which finds documents "similar in meaning" using cosine similarity between embedding vectors; the other is BM25 (sparse) search, which finds documents "containing the exact words," correcting for document length and word frequency in an inverted index. After getting result lists from both, you typically combine them one of two ways to solve the problem that their score scales differ. RRF discards the original scores and uses only rank, rescoring with 1/(k + rank) (commonly k=60); the relativeScoreFusion family normalizes both scores to a 0–1 range and then takes a weighted sum.

2) A common misunderstanding — why it reads this way

A common misunderstanding is that "vector search is smarter, so it's always superior to keyword search." This comes from the impression that embedding models understand sentence meaning, but in reality the very process of compressing meaning is the weakness. A single rare identifier in a query gets its signal weakened, buried among other tokens during the averaging (pooling) process. This problem stands out more in domains with tightly controlled vocabulary, like legal, medical, or technical documents. Another misunderstanding is thinking "just bolt on hybrid and you're done" — default fusion parameters like RRF or alpha don't fit every dataset.

3) What actually matters when judging it

Deciding whether to adopt hybrid search, and how to weight it, starts with looking at actual query logs. The first criterion is what share of queries contain identifiers that need exact matches — error codes, product numbers, function names, proper nouns. If that share is low and most queries are natural language, dense search alone may be enough. Second is how controlled the domain vocabulary is. Third is whether to use default weights (alpha 0.75, RRF k=60, etc.) as-is or tune them directly with labeled query-answer pairs. According to Bruch et al. (2022), cited by tianpan.co, tuning the weighted-sum ratio with about 40 labeled pairs consistently outperformed RRF. Fourth is whether there's room to add a reranker, which can push accuracy further at the cost of added latency and compute.

4) When it helps and when it doesn't matter

In areas like log search, code search, or product catalogs, where identifiers, numbers, and proper nouns frequently show up in queries, hybrid search is close to essential. Conversely, when queries are entirely natural language with almost no identifiers — a typical Q&A chatbot, say — it makes more sense to first measure dense-only performance. Analysis: if the dataset is small enough that a full comparison finishes quickly, the complexity of running two retrievers and tuning fusion parameters may outweigh the benefit.

In practice — 5 principles for reading hybrid search correctly

  1. Before adopting hybrid search, analyze actual query logs to check the share of queries containing identifiers like error codes, product names, or function names. The higher that share, the more pronounced the benefit of going hybrid.
  2. To solve the problem that the two searches' score scales differ, RRF is a safe default. If a platform supports score normalization methods like relativeScoreFusion, it's worth testing both approaches.
  3. Don't just keep the default alpha or weights (0.5–0.75) — tuning directly with a few dozen labeled query-answer pairs consistently gives better results.
  4. Don't stop at hybrid fusion alone; adding a cross-encoder reranker on top can widen the improvement, though it comes with added latency and cost.
  5. If queries are mostly natural language with few identifiers, or the dataset is small, measuring dense-only performance first and filling in only where it falls short keeps complexity down.

References

  • #ai
  • #hybrid search
  • #bm25
  • #vector search
  • #rag
  • #dense retrieval
  • #search engineering