(LangChain) Conversational Retrieval Chain, how does it work?

Picture by Laura Ockel on Unsplash

I am someone very curious. I like understanding how things are made. So I dove into the LangChain source code to understand how this feature, the conversational retrieval chain, works. I will now share with you what I find out.

What is the ConversationalRetrievalChain?

Well, it is a kind of chain used to be provided with a query and to answer it using documents retrieved from the query. It is one of the many possibilities to perform Retrieval-Augmented Generation.

But it won’t only answer your last query, it will also use the chat history to improve the quality of the RAG by taking into account past queries and answers when :

• retrieving documents,
• feeding the LLM with those documents and asking it to answer a question.

Sequence Diagram

First step: query rephrasing

What you need to understand that the query that will be used is not always the one that you gave the chain, but one that will be constructing using your query and the conversation history, which allow the chain to “remember” what you are asking about.

Let’s give an example, we will ask two questions :

Who was the first American President?

“The first American President was George Washington”

When did he became president?

“Georges Washington became the first President of the United States on April 30, 1789”

Under the hood the conversational retrieval chain will for each question (except for the first) rephrase the query to take into account the chat history using the following prompt:

Given the following conversation and a follow up question, rephrase the follow up question to be a standalone 
question.

Chat History:
{chat_history}
Follow Up Input: {question}
Standalone question:

So when we asked:

“When did he became president?”

… it has probably being rephrased like…

“When did Georges Washington became president?”

Second step: relevant document retrieval

In the second step, the chain will use the provided ‘retriever’ to find document relevant to the question.

A retriever is basically an object with a function taking a query and returning a list of documents.

Most RAG tutorials use a retriever constructed from a vectorstore, that is to say a vector database.

In those cases, the retrieval use a similarity search based on embeddings (vector representations of text) to find documents that seems to be the closest to talking about the same subject than the question.

Third (and last) step: the generation

Context + Question = Answer

In this last step, we will basically ask the LLM to answer the rephrased question using the text from the found relevant documents.

The chat model is given the following system prompt, where context is the text from the relevant documents.

Use the following pieces of context to answer the users question.
If you don't know the answer, just say that you don't know, don't try to make up an answer.
----------------
{context}

The chat model is also provided with the rephrase question.

At the end of the generation, the question and the answer are added to the chat history to be used in the next question.

Thanks for reading, and do not hesitate to leave a comment or ask if you have any question!