AI powered assistant: going further than RAG chatbot

Knowledge is information that has been processed by a human mind to help him do tasks or take decisions.

Data vs information vs knowledge

An assistant is a person or a tool that provides help by performing tasks or helping performing them. They are here to help make the work faster and/or easier.

The state of the art

Currently the state of the art consist of using a user query to look for document, then provide those documents and the query to ChatGPT to formulate an answer. This method is called RAG for Retrieval-Augmented Generation.

A Basic RAG process

Used alone, ChatGPT and LLM tools have knowledge limited to what was used to train them, which leads to not up-to-date information and risk of hallucinations as those tools are made to generate text, not to tell facts.

Their reasoning capacities are limited and only based on text, there is no understanding of the physical world or of mathematics.

With the RAG, we provide the LLM a context and ask it to answer a question using this context. This is a solution to:

• increase the amount of available information for the LLM, providing with up-to-date documents,
• reduce risk of hallucination.

The challenges facing RAG

They are mainly located at the document source level:

• be able to retrieve relevant information from a written request in natural language,
• search methods generally require prior indexing of information and/or analysis/understanding of the request to retrieve the parameters to be passed to a search API.

---

Basic RAG systems rely on vector databases to find text based on its meaning, encoded as a vector. The request is also converted into a vector, and the database will retrieve the texts whose associated vector is closest to that of the request.

Information retrieval based on vector databases

This technique works very well in the case of a question-answer format, where the text needed to answer the question is already present in the database. But for context-sensitive queries, such as finding the nearest restaurant, these systems are unsuitable. The other drawback is that the content needs to be ingested before being used.

---

More advanced systems are based on the provision of several search systems, with automatic selection of the most suitable data source(s) for the application.

---

As for vector-based data sources, the text used to calculate the embedding is not necessarily the original text or the one passed to the LLM:

• during indexing/ingestion, we can create for a given document questions that are answered by the document and use those questions to create the embedding,
• or during query, we can ask the LLM to generate an hypothetical document that answer the question, and use this document to search real documents that are the closest to this “false” one (the HyDE technique)

Using an LLM at ingestion time to derive questions from document

Using an LLM at query time to derive an hypothetical document and use it for similarity search (HyDE)

Both those techniques propose to query the vector store with a content of the same nature than the one that was used to get document embeddings:

• having questions in the first case,
• having documents in the second one.

Specifications for our assistant

Define tasks

I believe that a good assistant should above all be able to perform tasks that meet the user’s needs.

These tasks should go beyond simply answering questions with the RAG. They should be able to be triggered both manually,

• via a button, or a natural language request

or automatically:

• scheduled or triggered by an event internal or external to the system.

---

Examples of tasks that could be performed:

• generate a notification every morning with the day’s appointments and reminders,
• automatically update the content of data sources used by the RAG

Add more context to chat based tasks like RAG

In RAG we provide the prompt (what is asked ChatGPT) with information to be used to generate an answer.

What about also providing information about the user itself in order to have a more personalized answer? Such as providing instructions on how to address the user.

This extended context could also be used to:

• restrict the knowledge/data access to only what is relevant to the user,
• convert units and values into the measurement system used by the user.

I believe this extended context should be considered both at the conversation level and at the user one.

RAG using a user associated context

Do not use RAG as default to handle input and also process answer

I believe that before using RAG we should first ensure that que input given by the user should really be handled as a RAG query, that is to say:

• we should ensure it does not ask to launch a task (like ingesting new knowledges),
• we should ensure it does not give instruction to update the user context or the conversation context.

Basic configuration with a first processing to determine if user or rag agent should be used.

I also think that the answer given by the RAG should also be analyzed to propose actions based on the LLM output.

For example, if the RAG task answer “I don’t know”, our assistant should answer that he doesn’t know for the moment, but also propose solutions to help it answer the query, like performing a search on wikipedia or arxiv to increase it’s knowledge base.

Our assistant should also be able to generate diagrams using the last or previous answer (because images are often easier to understand than text)

Do not work with only one agent in chat mode

That is the continuation of the previous point. I think we could prepare multiple agents, defined by:

• the tools they can access,
• restrictions on their knowledge base access,
• the persona given for text generation.

We could them choose to have a “conversation” with only one of those agents or multiple-one at the same time. In the latter case for the system to know which agent should handle an input, we could use an “at” annotation “@agent” as an indication. We could also ask an agent to give the result of a task to another agent and so implement roleplay based tasks like it is described in the CAMEL paper.

Example of using agent notation to give instructions to an agent to increase the knowledge domain of the RAG.

In the above screenshot, by starting the first message with the system notation, we ensure it is treated by the system agent which will understand he need to get articles from arxiv API.

The connections between the assistant and the user

The assistant should be available from anywhere

I think we all agree that a good assistant should be available around the clock and from anywhere, which means it should be available both from a computer or from a smartphone/tablet.

The assistant should have access to other elements

As I think the assistant should be allowed to perform tasks on a scheduled basis or following an event, I also think those tasks could include performing actions on other systems, like executing a script on a server or on the user’s computer.

For security reasons I think those actions should first be reviewed by the user before being executed. After a learning phase, we could then consider letting some tasks/scripts run without external supervision.

To conclude

I will say that those new tools can give a big boost in productivity to those who will know both what they can and can’t do.

Do not hesitate to leave a comment or reach me if you disagree on some point or would like to add your ideas/opinions.

Thanks for reading!