RAG: hybrid search based on two indexes — part 2
Previously, we discussed implementing a strategy involving a first retrieval system, which employed a keyword-based approach to limit the…
Previously, we discussed implementing a strategy involving a first retrieval system, which employed a keyword-based approach to limit the number of documents used in the subsequent, vector-based retrieval system.
During that period, the code employed for testing this method consisted of hard-coded components and was designed as an individual task within my DAG-based platform. Consequently, it lacked the flexibility for customization.
With the enhancement of the platform’s configuration capabilities for tasks, and following the publication of an article detailing how chat memory functioned in LLM-based chat systems, it was appropriate to experiment with a more adaptable methodology utilizing multiple straightforward tasks rather than a solitary intricate one. In both cases LlamaIndex was used.
The tasks involved
DAG content
There are nine tasks involved in total, which can be divided into three main parts.
- The first part pertains to query rewriting. This process allows for the generation of follow-up questions using the last user’s query and the chat history.
- The second part is responsible for the retrieval mechanism. It accepts a follow-up question (text) as input and returns a list of potential nodes.
- Lastly, the third and final part involves the generation phase. During this stage, a prompt is constructed based on the final question and an LLM (Large Language Model) is asked to generate an answer.
Illustration by the author.
There are two separated tasks, Select LLM and Log condensed.
Select LLM is designed to easily allow users to specify which LLM to use (LLM configurations are archived in a database). In our implementation and during the test, we utilize a single LLM — a Mistral-7B instruct model that runs locally with LlamaCPP — for both query rewriting and final generation.
As for Log condensed, it is an helper task to provide a log as a system message in the chat interface.
The task parameters
The Condense task was provided during the tests with the following prompt. That is a small modification of the prompt you can find on LlamaIndex source-code.
Given a conversation (between Human and Assistant) and a follow up message from Human,
rewrite the message to be a standalone question that captures all relevant context
from the conversation. Do not add things that are not part of the conversation. Do not expand acronym values.
As for the Answer generation task this was the used prompt.
The following is a friendly conversation between a user and an AI assistant.
The assistant is talkative and provides lots of specific details from its context.
If the assistant does not know the answer to a question, it truthfully says it does not know.
Here are the relevant documents for the context:
{context_str}
Instruction: Based on the above documents, provide a detailed answer for the user question below.
Answer “don’t know” if not present in the document
Question: {query}
The Summary retriever was configured to use the arxiv-articles-2-ft index definition, which act as a factory to provide either:
- a LlamaIndex VectorStoreIndex using the ElasticsearchStore,
- or a BaseRetriever configured for text search.
Screenshot by the author.
class LiElasticFullTextIndex(LiIndex):
# ...
def as_index(self, context: "Context") -> "VectorStoreIndex":
vector_store = ElasticsearchStore(
index_name=self.index_name,
retrieval_strategy=AsyncBM25Strategy(),
es_url=Config().get(self.elastic_url_config),
)
# a text splitter that... does not split text
text_splitter = PassthroughTextSplitter()
storage_context = StorageContext.from_defaults(vector_store=vector_store)
return VectorStoreIndex.from_vector_store(
vector_store,
llm=None,
embed_model=MockEmbedding(embed_dim=1),
storage_context=storage_context,
text_splitter=text_splitter,
node_parser=None,
)
def as_retriever(self, context: "Context", **kwargs) -> "BaseRetriever":
kwargs.setdefault("vector_store_query_mode", VectorStoreQueryMode.TEXT_SEARCH)
return self.as_index(context).as_retriever(**kwargs)As for the Content Retriever, it is given the arxiv-articles-2-vector index definition, which act as a factory to provide either:
- a LlamaIndex VectorStoreIndex using the ElasticsearchStore,
- or a BaseRetriever configured for text search.
Screenshot by the author.
class LiElasticVectorIndex(LiIndex):
# ...
def as_index(self, context: "Context") -> "VectorStoreIndex":
vector_store = ElasticsearchStore(
index_name=self.index_name,
es_url=Config().get(self.elastic_url_config),
)
# the node parser is loaded from its reference
node_parser = (
None
if not self.node_parser
else self.load_reference(context, "node_parser", LiNodeParser)
)
storage_context = StorageContext.from_defaults(vector_store=vector_store)
return VectorStoreIndex.from_vector_store(
vector_store,
llm=None,
embed_model=self.embed_model_str,
storage_context=storage_context,
node_parser=node_parser,
)
def as_retriever(self, context: "Context", **kwargs) -> "BaseRetriever":
filter_value = kwargs.pop("filter", None)
vector_store_kwargs = None
if filter_value: # if a filter object was provided, we prepare the query to take it into account
def custom_query(es_query, query):
es_query["knn"]["filter"] = {"terms": filter_value}
return es_query
vector_store_kwargs = {"custom_query": custom_query}
return self.as_index(context).as_retriever(
**kwargs, vector_store_kwargs=vector_store_kwargs
)Through configuring indices in a database, we are able to utilize the same index for various distinct processes (such as ingestion and querying) with ease.
It’s time to play
Screenshot by the author.
After each user query, there appears a “system” message indicating which question was actually addressed in the response.
Addendum
Changes from the original article
The initial article proposed employing a third LLM call to derive keywords from the query and subsequently deliver them exclusively for full-text search in the summary database. However, this approach was eliminated due to its insignificant enhancement compared to directly incorporating the condensed question into the search.
Illustration by the author, the original proposition add another LLM call to extract keywords.
Moreover, during the preliminary assessments for the initial article, OpenAI APIs were employed rather than a locally installed model for the Language Model (LLM) component.
Tweaking LlamaIndex’s LLM usage
When employing LlamaIndex’s LLM API directly, most parameters are specified upon instantiating the LLM object rather than during generation. This posed a challenge for us since we intended to utilize the same LLM instance repeatedly within the same task graph, with varying configurations, primarily concerning the stop list which is derived from the prompt. We created an abstraction layer that allows us to establish these parameters during the inference process to resolve this issue.
Reducing dependency on LlamaIndex
Through the implementation of separate tasks rather than utilizing a chat engine directly from LlamaIndex, we gain the flexibility to substitute components of the process with alternatives not provided by LlamaIndex.
Does not skip the first query rewriting
The tested Directed Acyclic Graph (DAG) always executes the query rewriting part, even for the initial user query. In the upcoming version, we will incorporate a check to determine whether or not to bypass this process. The platform already allows tasks to be routed to alternative ones based on specific conditions.
That’s all Folks!
Feel free to comment or contact me if you had like a demonstration.