(RAG) Using a website to build your knowledge base: what you need to know

Photo by Ilya Pavlov on Unsplash

This article is not a tutorial and will not provide you with source code, but it will give you tips, and maybe ideas, to help you conceive your own scrapping mechanism, adapted to the site you want to ingest in your knowledge base.

Websites can be often updated

Do not assume scrapping is a one time operation!

Your scrapping solution must be ready to periodically ingest new or updated content, but also to remove those who disappeared from the website.

All pages on your website are not good to be ingested

Some pages are very poor in term of knowledges, like “listing” pages.

Let’s take a concrete example, you are scrapping a blog. While individual article pages are often great to be ingested, pages listing those articles aren’t. Those pages only provide you with the title and a short text about the article, nothing usable if their contents were proposed by a retriever in the context of a RAG tool.

If you use something like a RecursiveUrlLoader to scrap the website, you should both :

• scrap those listing pages to extract the links they contains,
• ignore the content of those pages and not use them to provide “Documents” to your knowledge base.

---

If you use something like a SitemapLoader you only have to ignore the content of those pages.

---

But in both cases, the issue is about how to ignore those listing pages! Well, it’s up to you. Maybe you will use a black list of identified listing pages, maybe you will parse the page content to detect you shouldn’t ingest it.

Sometimes the content isn’t in the HTML source code!

When the webpage use AJAX queries to load the content, a solution based on a simple HTTP GET request won’t be able to load the displayed content.

For those cases you need to use a real browser to get the HTML DOM after the page has been rendered. Like the SeleniumURLLoader from langchain.

Not all page content is suitable for ingestion

Let’s take our reasoning a step further. If all pages are not good to be ingested, is all elements of a single page really relevant?

The answer here is also NO.

All elements related to navigation or common across pages should be ignored, ie: headers, navigation bars, footers… Not forgetting blocks used to list content, like related articles, related products, …

Although they provide links to other content, they do not provide information in themselves.

So like the previous part, you might parse those elements, but you shouldn’t include their content to the Document you will build for this page.

Do not assume using a simple vector database will cover all your use-cases

Vector database, embeddings, many tutorials are all about that.

Using embeddings is like making a very blurred version of a photo and comparing it with other blurred photos to find the most similar.

It works quite well with fairly generic questions like asking what kind of product can help you with a task.

The problem arises when the questions concern specific elements such as asking the size of a particular product. As the question is “blurred”, the details disappear, and the semantic search alone cannot filter out the product mentioned.

The larger the knowledge base, the greater the problem, your retriever might give documents non relevant to answer the asked question.

---

What to do then? Well, it’s once again up to you, but first you should identify the intended used case of your RAG system.

You might want to build multiple “smaller” databases and use a routing solution to use the most relevant one for a given query.

Otherwise you might want to build a single “bigger” database and use a routing solution or a prior query analysis to determine metadata values to filter on.

---

I hope you found this article interested, feel free to comment if you want to add/correct things or contact me if you need help.