Why Retrieval-Augmented Generation Is Still Relevant in the Era of Long-Context Language Models
In this article we will explore why 128K tokens and more models can't fully replace using RAG.
Some researchers even found (July 2024) out that it might give better results than purely relying on RAG.
Retrieval Augmented Generation or Long-Context LLMs? A Comprehensive Study and Hybrid Approach
While the article was focused on “Self-ROUTE” a proposition to determine wether to use a Long-Context prompt or RAG to answer a query, they did start by determining than in most cases the Long-Context choice might provide better results than relying only on RAG. The final goal of their experimentation is both to increase the quality of the answers, and to reduce costs
A most recent article (September 2024) suggests that RAG is still relevant and the limitations encountered in the previous article were mostly about the order in which found relevant text were added to the prompt. (They recommend to keep those chunks in the same order they were in the original document).
In Defense of RAG in the Era of Long-Context Language Models
This second article has a different conclusion than the other and say that stuffing too much information in the context results in degrading the quality of the answer. It also provides food for thought on the importance of the order of elements in building context in the prompt.
To continue on this subject, I recommend reading a third article, although it is a little older (July 2023). As nowadays most LLMs remain transformer based, this article is still relevant to help us understand some limitations of using long long-context prompts.
Lost in the Middle: How Language Models Use Long Contexts
To summarize, they used a context where only one of the document was relevant, and they looked for the quality of the answer according to the position of this relevant document. They then repeated the experiment with an increasing number of chunks in the context. Changing the LLM had no impact on the overall shape of the curves.
Illustration by the author, original graphs are in the article.
The LLM is most likely to be able to use an information if it is at the start of the prompt than if it is in the middle. It improves in the end. Also increasing the number of “non-relevant” documents reduces the capacity of the model to retrieve an information.
To conclude, my thoughts about it
In my humble opinion Retrieval-Augmented Generation will still stay relevant for a very very long time, the main reason being… money.
The longer the prompt is the more computation time is needed to process the context. Consequently, using RAG to limit the prompt to what is needed reduces the cost compared to feeding the LLM with all your company knowledge.
In the future, when using LLM with long context, what we might expect from RAG might be not to find out relevant parts of documents to answer a query, but to filter-out irrelevant parts in order to reduce the costs and increase the quality.
But I do think that the future of RAG will lie in the use of smaller, more specialized models rather than models designed for general use.
That’s all folks!
Feel free to clap, follow me or give your opinion!