Keyword based similar content with vector databases

UPDATED on September 14, 2024

Concept of mixing keyword extraction with embeddings for fast similarity search.

Picture by Bruno Wolff on Unsplash

It is an idea of using a similar approach to sentence embeddings to determine if two documents (webpages, PDFs, …) have a similar content or not.

Original concept

First step: determine the keywords to use

You might already have keywords associated with your documents and use them. If your keywords are associated with a taxonomie, you might consider the parents term of your keywords in your keyword list, just use a ponderation to reduce their impact in the search results.

You might have to use an agent like ChatGPT or another LLM to extract keywords from the content for you. In this second case it would be relevant to count the number of occurrences of each keyword, in a given document, and across all documents.

You might want there to reduce the number of keywords. A keyword appearing a few times and only in a few documents might be ignored. Convert any plural keywords to singular and remove duplicates.

When you have your final list of keywords, the fun can start.

Second step: build the vector space

We will consider a vector space, each dimension is normalized between 0 and 1 (one being a perfect match) and there are as much dimensions as keyword found in the previous step.

Each document we want to index will be split in small pieces.

For each piece and each keyword, we will use an embeddings model to determine the proximity.

This will give use a vector. We can repeat this for each pieces of the document and get a mean vector for the document.

This vector will then be stored in a vector database with associated metadata (like an unique ID for the source document)

Third step: query the vector space

At that step you want to know which documents are the nearests from a given one?

Just start getting from your vector database the vector associated to your first document.

Then just do a similarity search using this vector to get the nearest documents (and of course ignore your given document from the list).

I think using DOT product distance should perform great at that task.

Updated proposition / other idea

(Part added on September 14, 2024)

The idea

This new proposition is an idea I got after testing fine-tuning models for sentence-classification.

The new idea is about assigning (by hand) a main category to each one of your content. Then you will train a text-classification model on the summary of those contents. When your model is ready you can use it for two tasks:

• automatically classify a new content,
• build a vector using the score for each categories for a given summary.

Those models never give a 100% score for a label and 0% for all others, so using the full output will contains information about the most probable main category as well as secondary categories. IE: an article about machine learning will most probably have a big score for “machine learning” and smaller, but non near zero, scores for “mathematic”, “statistics” and so on.

It is this vector that you can index in order to find similar contents.

Benefits of this approach

• Speed: While the training of the text-classification model can take some time, including the time to prepare the training dataset, this method is way faster on the long run. (Only one inference per documents instead of many).
• Double usage: the trained model can be used not only to help find similar contents, but also to auto-label new content with a main category but also sub-categories (categories those scores is above a threshold).

Please fill free to give me your honest opinion about it!