Retrieval-Augmented Generation: Why naive RAG is not enough (and some ways to improve it)
Naive RAG is a term in vogue used to define basic RAG systems. This article will help you understand their limitations and how to fix them.
At ingestion time, one technique that can be used if, for a given chunk, ask an LLM to determine what questions this chunk is answering to. Then the embeddings will be computed according to those questions instead of using the text of the chunk.
Illustration by the author. Generating questions for the chunks and use them to compute embeddings.
At query time, the HyDE (Hypothetical Document Embedding) technique is another solution. Instead of using directly the user query for the semantic search, a first LLM call is made to try to imagine a chunk that would answer the question, even if the LLM hallucinate, the generated document should be semantically close to a real one. Then it is this hypothetical document that is embedded and used to search for similar chunks.
Illustration by the author. HyDE technique in action.
Issues linked to chunking documents
Even now where LLM have large context windows, we need to split an original document into smaller pieces often called chunks. The RAG technique won’t use only one of those chunks to build an answer but retrieve the top-k results.
Now what happen when the answer of your query is on two distinct chunks of the same document? Let’s say one that contains the name of the A380 plane and one that say it’s cruise altitude?
That’s the issue, the semantic search might not fetch those two chunks because other ones, close but not answering the query, might looks more similar to what is looked for.
Illustration by the author. While the green chunks are what might help to answer the question about the cruise altitude of the A380 plane, the blue chunk is more likely to be the top-most chunk retrieved.
One solution would be to add a context to chunks…
… to remember elements about the document. ie: for a knowledge base about technical data for planes, you would add the name of the plane into each chunk. This is already done by some frameworks like LlamaIndex that include metadata into both the text used for embedding and the one used for generation.
Issues linked to having too much documents
The larger your knowledge base, the greater the risk of having chunks that are semantically close to what’s being searched for but don’t answer the question!
The solution for that would be to pre-filter your semantic search…
… based on full document content. (ie: only includes chunks inside documents talking about the A380)
(You can see links to two articles I wrote about it)
RAG: hybrid search based on two indexes
RAG: hybrid search based on two indexes — part 2
On a personal note, I recently participated to a webinar about how to use LLM to construct SQL queries (text-to-sql). An interesting point was the use of an hybrid search for keywords that allowed to create filters to reduce the knowledge domain.
ie: if the query is about what is eaten in Iceland, the hybrid search would retrieve a document describing that “Iceland” is a keyword that can be present in the “country” metadata field. It helps filtering the knowledge base and then decrease the risks of having only non-relevant chunks retrieved.
To conclude…
Semantic based RAG is far from being perfect but there exists solutions to improve it!
That’s all folks!
Feel free to contact me if you are curious or my employer (French company) if you had like to work together.