LangChain/RAG semantic splitting explained!

single_sentences_list = re.split(self.sentence_split_regex, text)

distances, sentences = self._calculate_sentence_distances(single_sentences_list)

if self.number_of_chunks is not None:
    breakpoint_distance_threshold = self._threshold_from_clusters(distances)
    breakpoint_array = distances
else:
    (
        breakpoint_distance_threshold,
        breakpoint_array,
    ) = self._calculate_breakpoint_threshold(distances)

indices_above_thresh = [
    i
    for i, x in enumerate(breakpoint_array)
    if x > breakpoint_distance_threshold
]

chunks = []
start_index = 0

# Iterate through the breakpoints to slice the sentences
for index in indices_above_thresh:
    # The end index is the current breakpoint
    end_index = index

    # Slice the sentence_dicts from the current start index to the end index
    group = sentences[start_index : end_index + 1]
    combined_text = " ".join([d["sentence"] for d in group])
    # If specified, merge together small chunks.
    if (
        self.min_chunk_size is not None
        and len(combined_text) < self.min_chunk_size
    ):
        continue
    chunks.append(combined_text)

    # Update the start index for the next group
    start_index = index + 1

# The last group, if any sentences remain
if start_index < len(sentences):
    combined_text = " ".join([d["sentence"] for d in sentences[start_index:]])
    chunks.append(combined_text)
return chunks

You may have heard of semantic splitting, but do you know how it works?

This article is based on LangChain implementation, but LlamaIndex one is based on the same principles.

Why to use text splitting in the first place?

When you ingest new documents into your knowledge base (often a simple vector database), you usually don’t add the document as a whole. What you will want to add is smaller parts of the documents, often called chunks. It’s those chunks that are retrieved and used to create the context to answer a query in the context of Vector Based RAG.

The size DOES matter!

The length of those chunks is something very important!

If they are two smalls, you will lose information that would have been useful for the LLM to perform its intended task.

If they are too big, fetching the chunks based on semantic similarity will be less accurate.

Illustration by the author, the relevant parts from a document to answer a query straddle two chunks.

Some approaches to text splitting

Basic splitters will usually only try to build chunks with a size as near as possible to a given token limit while keeping sentences together.

More advanced splitting might use a recursive approach and start by splitting by paragraphs/sections and then perform additional splitting until reaching an acceptable length.

And there is also the semantic splitting that is not based on an expected token length, but on the similarity between consecutive sentences. The idea here is to keep together sentences that have the same semantic meaning.

Let’s focus on semantic splitting!

You can have access to the current LangChain implementation at the following page: https://python.langchain.com/api_reference/_modules/langchain_experimental/text_splitter.html#SemanticChunker

let’s see the split_text method

The first thing that is done is to split the text into sentences.

after some securities to ensure the code won’t break if there is only one or two sentences, the similarity is processed

let’s dive in the _calculate_sentence_distances method

The _calculate_sentence_distances method does the following:

• first it combines the sentences by groups of self.buffer_size (parameter given when creating a new SemanticChunker and whose default value is 1,
• then it will use the mandatory embeddings model given as first parameter to get an embedding representation of each sentence group,
• finally it will use the calculate_cosine_distances function to get the distance between a sentence group and the next one using the cosine distance. (a bigger value means a greatest semantic difference)

back to the split_text method

The method will then use a strategy to determine the distance above which to consider too sentence groups are too distinct to be put in the same chunk. (See code below)

Finally the method create the chunks by combining successive sentences similar enough.

When to use semantic splitting?

Semantic splitting is interesting for long unstructured text, that is to say text with no section/paragraph that can be used as boundaries for the chunking strategy.

Well, when the text in a section/paragraph is long you can then split it further using semantic chunking.

That’s all folks,

Feel free to comment or show your appreciation!