LLMs can't remember conversations, so how can ChatGPT work?

What lies behind ChatGPT is primarily an LLM, a language model that has been trained for a single task: from a text, predict the next word (in fact, what is predicted is a token, a small part of the text). It does not remember previous interactions.

Photo by David Clode on Unsplash

LLMs are just about performing text completion

One thing you might not realize when using ChatGPT is that LLMs have no memory. Every time you use an LLM, it will only complete the text (prompt) you’ve just given it, nothing more, nothing less.

Preparing a basic chat interface

We will configure on the P6 KMS/platform a chat room with a basic prompt.

Screenshot from the P6 KMS (in development but already operational). Illustration by the author.

In the screenshot, you can see that the LLM used will be a Mistral 7B model running on my computer. The prompt concerns being a python expert.

You can see that a stop word “user:” has been added, to prevent the model from anticipating the next question. (Yes this is a real issue)

There are only two tasks, the one that prepares the LLM instance, and the one that performs the prediction/text-completion.

Using the chat room

We will first ask to create a ‘car’ class, then ask for it to inherit from another class.

With a basic LLM usage, we don’t have real conversations, only question-and-answer pairs. (Illustration by the author).

As you can see from the screenshot, the LLM constructed the second response without taking into account the first interaction.

Inserting history to the prompt

The trick to having a conversation is to allow the LLM to access previous messages by adding them to the prompt.

This time, we’ll start by looking at the conversation, then explain what’s changed.

Screenshot of the chat room

As you can see the queries are the same but…

LLM call result with chat history added to the prompt. (Illustration by the author).

… this time, the answer to the second question follows on from the first!

The configuration that was used

Screenshot from the P6 KMS (in development but already operational). Illustration by the author.

You can see that there are three additional tasks in this case.

• ChatListTask, which retrieves messages passed to the database,
• ChatBufferListTask, will create a buffer from the messages retrieved in the previous step, including up to 2,500 tokens (to count the tokens, we need access to the tokenizer of the LLM instance that will be used),
• ChatHistoryStringTask, will convert the contents of the buffer into a string and make it accessible using the “history_str” variable.

As there is a limit to the text size that can be stored in the LLM context window, we might need to remove older content, this is what chat memory buffer are used for.

For the prompt, we’ve simply added the {history_str} placeholder.

For the curious, the current ChatBufferListTask source code (the _process method signature may change in the near future).

class ChatBufferListTask(Task["ChatBufferListTask.InputModel"]):
    
    class Parameters(BaseModel):  # To declare task parameters for both the task and the UI
        token_limit: Optional[int] = Field(ge=0)

    class InputModel(BaseModel):  # The easy (static) way to declare data needed for this task
        class Config:
            arbitrary_types_allowed = True

        chat_history: List[ChatMessage]
        llm_model: LiLlm

    class OutputModel(BaseModel):  # The easy (static) way to declare data provided by this task
        chat_history: List[ChatMessage]

    async def _process(
        self, context: "Context", input_data: Mapping[str, Any]
    ) -> Mapping[str, Any]:

        input_model = self.input_object(input_data)
        parameters = cast(ChatBufferListTask.Parameters, self.merge_params(input_data))
        raw_token_limit = parameters.token_limit

        from llama_index.core.memory.chat_memory_buffer import ChatMemoryBuffer

        token_limit = raw_token_limit if raw_token_limit else None

        memory_buffer = ChatMemoryBuffer.from_defaults(
            chat_history=input_model.chat_history,
            llm=input_model.llm_model._llm,
            token_limit=token_limit,
        )

        return {"chat_history": memory_buffer.get()}

Regarding the declaration of the complete task graph:

with TaskDAG(
    id="llm_test_predict_history", label="Prediction", required_worker_tag="MM1"
):
    llm = LLMPrepareTask(id="get_llm", is_passthrough=True)
    predict = LLMPredictTask(id="llm_predict_task")
    llm >> predict

    chat_list = ChatListTask(id="chat_list")
    chat_buffer = ChatBufferListTask(id="chat_buffer")
    chat_history_str = ChatHistoryStringTask(id="history_str")

    chat_list >> chat_buffer >> chat_history_str >> predict
    llm >> chat_buffer

That’s all folks!

Feel free to comment or contact me if you’d like to know more about how LLM can be used and why task orchestration is crucial.

Addendum: when playing with LLM on the platform what is used is mostly based on LlamaIndex for the content of basic tasks, but for the task orchestration it is the platform itself that is used as it allows to change parameters without having to edit the source code.

P6 stands for Pinceau6, I conceived the first Pinceau systems on 2010 during my internship at the INSERM (French institute for health and medical research). It is a platform that enable to easily access database content, handling inheritance, and perform tasks related.