Retrieval-Augmented Generation (RAG), or the art of feeding the prompt
To increase the capabilities of chatbot based on LLM, like ChatGPT, you can use a technique called RAG to provide a context to answer from.
Picture by Hert Niks on Unsplash
When OpenAI released ChatGPT on November 30, 2022, the public discovered what Large Language Models are, or to be more precise, they learn about one of the possible uses of LLMs. But there are some misconceptions remaining about what a LLM can and cannot do.
The first misconception is to assume a LLM can directly access to Internet or a kind of database to perform searches.
While the LLM can be trained using data from internet and databases, there are no direct access to anything like that when actually running the model.
It is basically a closed box with a text input and a text output. This box is meant to imagine a text to complete the input, without real garantee this output text will be factually right.
It is called hallucinations when the LLM gives answers that are erroneous.
The second misconception is to assume LLM really understands what it is reading/writing.
LLM are basically meant to provide text, not to give facts, that is why they can tell things like 1+1=3 and actually ‘believe’ it.
How to get relevant answers?
That is the question. (the answer isn’t 42)
On a 2020 paper Meta proposed a solution. This solution is called Retrieval-Augmented Generation, or RAG in sort.
Instead of simply asking a question about a subject to a LLM and hoping the answer will be correct, we give a context to the prompt and ask to answer using this prompt.
“Using the following context … answer this question …”
By asking to use the context, we constraint the response and both reduce hallucination risks and help to get up-to-date answers.
I have many times read misconceptions about what is RAG and how to make it in place. The main misconception is that RAG is about using embeddings, which is far from the truth. RAG is about providing a context, embeddings are only a tool among many to get this context.
What is the provided context?
The provided context is always a text, as LLM can only work with them. If you want to use data coming from a database, you need to provide a textual representation of this data.
What to provide?
That is the difficulty of the task, you must provide a context with elements allowing to answer the question. That is to say you must first retrieve relevant text pieces (often called chunks).
There are several ways to find those pieces like using:
- a semantic index,
- a knowledge graphs index,
- a keyword index,
- an hybrid search,
- an api call (ie: searching on google, wikipedia, calling a weather api)
- “classic” database search,
- …
In semantic search, all contents are associated with a vector representation, this “embedding” is meant to capture the semantic meaning of the text. The question is also converted to an embedding, and we then look for the contents those embeddings are the closest to the question one. This is usually performed in a vector database.
This method is quite good for general question answering related to a knowledge base but not so good when you need to narrow the answer to specific content like a product.
With knowledge graphs, we capture the relationship between concepts, allowing many kind of queries like :
- what do we know about the concept x?
- what are the relationship between x and y?
- whose concept has the following relation to y?
We can add that knowledge graphs make it easier to make inferences, that is to say to combine information to determine new information.
The main issues are :
- the complexity to build the knowledge graph
- the complexity to build a query from the prompt
Fortunately, there are solutions to make these parts easier with frameworks like langchain or llamaindex.
With keyword index, we extract keywords both from the contents and the question to find contents that might correspond to the question. The main issue is that this does not guarantee the found content really help to answer the question. (A content about Paris city current demography won’t help solve a question about Paris history)
Hybrid search is a generic concept of combining multiple search methods, like semantic and keyword search to try to get the benefits of all of them without their weak points.
As for API calls and classic database search, the main pro is not having to prepare content before hand, the main weak point remains it is difficult to be sure the content we will get and put in the context part of the prompt will actually help to answer the question.
What to choose in the end?
Unfortunately there is no absolute answer to this question.
What you will need to use depends on what kind of questions/queries you are expected to answer and of the kind of content that are at your disposition to help provide a relevant context.
You might need to use multiple solutions at the same time and select for each question/query what might be the most adapted solution to use.