RAG: Hybrid Search Based on Two Indexes

The proposition I will be talking about in this article is something I already have implemented and I am currently testing in a personal project.

What is hybrid search?

Hybrid search in the context of RAG and vector database means searching chunks of documents that can help answer a question using both a semantic search based on embeddings and a full text search on the content of those chunks.

The limitations

While hybrid search should give better results than a pure semantic approach as it should gives more relevant chunks of text by highlighting those that contains some keywords present in the research, there is still room for improvement.

As keywords are search in the same text that was used to calculate the embedding, what happen when a chunk A of a document contains the keywords while another chunk B of the same document is semantically close to the query and so should help to answer it?

We would like to have chunk B to be part of the documents returned by the retriever, but with a standard hybrid search that might not be the case.

Some existing solutions

Self-querying retriever

This kind of retriever is based on metadata filtering. Key information that might help to filter the vector database content are added to metadata. Using a LLM call with both the query and the description of the content of the metadata values, instructions are determined to add filters to the vector database query.

I do believe self-querying is a good idea, as long as you don’t have too much metadata value to filter on.

Using multiple vector stores and a routing solution

In this approach, multiple vector databases are build, each one responsible with a knowledge domain. A LLM call with both the query and the description of the content of each vector database is perform to determine the vector store the most likely to contain text helping to answer the query.

This solution can also help getting better results by reducing the size of the knowledge base used to find text chunks.

The proposition

My solution is a mixture of several techniques and is quite easy to implement.

The main components to build the solution

I build my knowledge base using two different indexes:

• The first one is text based only and contains either full documents or a summary of a document and an ID to identify this document. (When I work with arXiv results I store article abstracts and use the article URL as a unique ID for the document)
• The second one is embedding based and contains chunks of original documents. (When I work with arXiv results I store chunks taken from full content of article PDF files)

Querying the knowledge base

Illustration by the author

• Using an LLM call, we begin by determining keywords that should appear in documents that help answer the query.
• Those keywords are used to perform a search on the text based index, it gives us the ID of documents containing those keywords
• We combine the ID list find in previous step with the original query to construct a vector query with metadata filter
• The query is performed on the vector database and we retrieve chunks of documents that are most likely to answer the query.

What is expected from this proposition

I expect this technique to help in cases a question is asked about a specific element but only chunks about other element than then one asked are retrieved because the content were most semantically similar to the query.

We can also increase the score of most recent documents in the text index to focus on most recent n documents containing the expected keywords and so help to get up to date data by then searching k chunks only on those n documents.

This should be really helpful while working with large knowledge bases.

Update: the second part!

Thats all, feel free to clap, comment, or reach me!