RAG, Knowledge Bases: transposing programming principles

You might want to take a look at this previous article if you were interested in this section.

https://medium.com/@jerome.o.diaz/rag-prepare-your-knowledge-domain-86291b36a04f

Interpreted programming languages equivalent in the context of Knowledge Base

We could put here all cases of retrieval mechanisms that does not need to have specific indexes build to search for data to answer a question.

“Classic” Database query based retrievers/agents

Those mechanisms include constructing SQL queries from a user’s question in order to find rows in an existing database and then use those results as the context to answer the question in a RAG system.

Both LangChain and LLamaIndex have tools for that (https://python.langchain.com/v0.1/docs/use_cases/sql/agents/ and https://docs.llamaindex.ai/en/stable/examples/index_structs/struct_indices/SQLIndexDemo/ )

External API Based mechanisms

In those cases you prepare parameters from the question and then call an external API. Even if the tool you can use might under the hood indexes, those are not build specifically for your usage and you can freely use the provided data without an ingestion from your part.

Just-in-time (JIT) ingestion

In the case of compilation, JIT is combining both interpretation and compilation. The source code is first interpreted but then all or part of it can be converted in direct machine language at run-time.

In the case of ingestion it would be:

1. using the user’s query to build parameters for an API call that would return large documents,
2. ingest those documents for semantic and/or full text search,
3. answer the user question after having selected relevant part from those documents.

Data persistency

Lazy loading in case of JIT ingestion

Lazy loading in programmation is a technique that will fetch a data the first time it will be accessed. It is often used to help the software to be in a usable state sooner, even if it will be slower until all lazy variables have finished being called.

In the case of JIT ingestion it would be persisting the ingested documents in the 2. step instead of having them in memory only during the question answering. Next time the same raw documents would be returned in the 1. step, we could directly use them in the 3. step (using a filter ensuring to only use those documents).

Result caching in case of ingestion

While lazy loading in case of ingestion is about the documents being used to answer a question, result caching is about the answer being produced.

After having answered a question, both the question and the answer are stored in a knowledge base. If later a similar enough is asked, the past answer will be returned or help build a new answer.

---

To be effective, an answer result caching must first ensure that the original question was indeed answered.

The non-absolute comparison

This comparison will only concern the retrieval part of a RAG and does not include the cost of using an API based LLM or the need for high-performance machine to run a local LLM.

Made by the author

The pros for AOT ingestion solutions

As ingestion is a quite demanding task, AOT ingestion allows to use a high-performance machine only during ingestion one, and a cheaper one later for answering. Also answers might be given faster than in the case of JIT ingestion.

The cons for AOT ingestion solutions

For me the bigger issue is that you do not have an immediate access to new content and must update your knowledge base quite often to keep it the most possible up-to-date. Also having two much data in your knowledge base can reduce the quality of your answers if you do not use mechanisms to filter and refine retrieved chunks.

The pros for pure non-ingestion based solutions

Those solutions does not require any ingestion from your part and does not require high-performance machines for the retrieval part of your RAG system. They give your users access to new content immediately.

The cons for non-ingestion based solutions

The main issue I am concerned with for those is the dependency on things you might not have control on. The second one is it can be quite complexe and time expensive (even if it is performed by an external LLM in most cases) to build the most adapted parameters from the user’s question.

The case of JIT ingestions solutions

JIT ingestion helps you to build a knowledge base you have control on, but does not reduce the complexity and time of building parameters for querying the API/Database. It allows to have a real-time access to new content, but it requires to keep a high-performance machine to be able to ingest new knowledge at a reasonable enough speed for the user expecting an answer to its question.

---

In conclusion there is no perfect solution and you need to choose what is best adapted to your needs. You might also want to consider using multiple solutions at the same time to mix multiple knowledge base.