large language models artificial intelligence software engineering machine learning ai agent

The building blocks of Agentic AI

An LLM cannot call a tool. It can only produce text. Everything else, the retrieval, the execution, the feedback loop, is the machinery…

What is an LLM?

A Large Language Model (LLM) is a neural network trained to predict the next token in a sequence of text. That’s it. One token at a time, left to right, based on everything that came before.

This deceptively simple mechanism, scaled to billions of parameters and trained on vast corpora of human text, produces systems capable of translation, reasoning, coding, summarization, and much more. But the mechanism never changes: the model takes a sequence of tokens as input, and outputs a probability distribution over the next token. You sample from that distribution, append the token, and repeat.

A few landmark dates relevant to this story:

  • May 2020: GPT-3: “Language Models are Few-Shot Learners” (Brown et al., OpenAI). 175 billion parameters; demonstrates remarkable few-shot capabilities.
  • November 30, 2022: ChatGPT public release. The general public encounters LLMs for the first time, en masse.

The core limitation of all these models is the same: an LLM only produces text. It has no persistent memory beyond its context window, no ability to fetch fresh information, no way to take action in the world. It is a very sophisticated text completion engine.

This limitation is what motivated every development that follows.

The building blocks

RAG — Giving LLMs access to external knowledge

The knowledge problem was recognized early. An LLM trained on a static corpus can only answer questions about what it saw during training, its knowledge has a cutoff date, it can’t access private data, and it can’t look things up.

In May 2020, the same month GPT-3 was released, two and a half years before ChatGPT, Facebook AI Research published “Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks” (Lewis et al.), introducing RAG. The approach adds a retrieval step before generation:

Retrieval Augmented Generation

  1. Given a query, retrieve relevant documents from an external store (a vector database, a search index, a knowledge base)
  2. Inject those documents into the LLM’s context as background information
  3. Let the LLM generate an answer grounded in the retrieved content

This is a crucial conceptual shift: a standalone LLM is limited, but an LLM combined with external systems becomes dramatically more capable. RAG is the first major instantiation of this idea, and it sets the theme for everything that follows. The LLM is the reasoning engine; external systems provide the knowledge, memory, and capabilities it lacks on its own.

ReAct — Reasoning and acting in a loop

RAG injects knowledge before the LLM generates. But what if the LLM needs to decide which tools to use, and in what order, based on intermediate results?

The Reason and Act loop

This is the question that ReAct answers.

“ReAct: Synergizing Reasoning and Acting in Language Models” was published by Yao et al. (Google and Princeton) in October 2022, seven weeks before ChatGPT. The paper introduces a prompting pattern that interleaves reasoning traces with actions:

Thought: I need to find the population of France.Action: search("population of France")Observation: France has a population of approximately 68 million people.Thought: Now I can answer the question.Action: finish("France has approximately 68 million people.")

The pattern is elegant: the LLM thinks, acts, observes the result, and thinks again. Tools are described in the prompt, their names, what they do, how to invoke them. The LLM generates text that specifies which tool to call and with what arguments.

Here is the critical point, because it’s the source of a lot of confusion: the LLM does not call the tool.

The LLM produces text that describes a tool call. Something like Action: search("population of France"). Application code then parses that text, recognizes the action format, extracts the tool name and arguments, and makes the actual call, to a search API, a database, a calculator, whatever. The result is then injected back into the LLM’s context as an Observation, and the loop continues.

ReAct is powerful, but it has a fragility problem: if the LLM doesn’t format its output correctly, misspells the tool name, adds an extra space, wraps arguments in the wrong way, the parser fails. Prompt engineering can reduce this, but it can’t eliminate it. Which leads us to the next development.

ReAct is fragile

Making output reliable — Constrained generation

Asking an LLM to produce well-formed JSON by instructing it in a prompt works most of the time. But “most of the time” is not good enough when you’re parsing the output programmatically. Failure rates of 5–20% are common with pure prompt engineering, depending on the model and the schema complexity. ReAct-style parsers hit this ceiling constantly.

The solution is to constrain the generation itself. Instead of hoping the model produces valid output, you modify the token sampling process to make invalid output impossible. At each decoding step, a grammar or schema defines which tokens are legal given what has been generated so far. Any token that would violate the schema has its probability set to zero before sampling. The model remains free to produce any valid output — it simply cannot produce invalid output, because those tokens are masked out.

This approach was formalized rigorously in “Efficient Guided Generation for Large Language Models” (Willard & Louf, 2023), the theoretical foundation for token-level constrained decoding over arbitrary JSON schemas. What had been a brittle prompting problem became a solved engineering problem.

How brittle? “JSONSchemaBench: A Rigorous Benchmark of Structured Outputs for Language Models” (Guidance-AI / EPFL, 2025) benchmarked six frameworks on 10,000 real-world JSON schemas of increasing complexity. An unconstrained LLM achieves 90% on simple function-call schemas (GlaiveAI), but collapses to 13% on complex ones (GitHub Hard, 100–500 fields). The best grammar-constrained framework (Guidance) starts at 96% and degrades to 41% on the hardest schemas. Both curves fall with complexity; the gap between them is the reliability gain you get from constrained generation.

Providers then baked it into their APIs directly — OpenAI’s response_format: { type: "json_schema", ... }, Anthropic’s structured output, Gemini’s response schema parameter. You provide a schema; the API guarantees conformance. The failure mode that made ReAct fragile — malformed output that breaks parsing — disappears entirely.

One important nuance: what constrained generation solves is syntactic reliability. It guarantees that the model’s output will be valid JSON matching the schema. It does not guarantee that the values inside that JSON are correct. A model can return a perfectly well-formed tool invocation with a hallucinated argument, a filename that doesn’t exist, a date that was made up, a parameter value that misrepresents the user’s intent. That is a distinct problem, one that no amount of schema enforcement can fix. The reliability gain here is real and significant; it just applies to structure, not to content.

That reliability is what made the next step possible.

Function calling: Structured output meets ReAct

On June 13, 2023, OpenAI introduced Function Calling. This is the moment where constrained generation and the ReAct pattern became a first-class, provider-native feature.

The idea: instead of describing tools in a freeform system prompt and hoping the LLM formats its responses correctly, you provide tool definitions as structured JSON schemas. The model is fine-tuned and constrained to output a valid JSON tool invocation when it decides a tool should be called.

The developer experience:

  1. Provide a list of function schemas: {"name": "search", "description": "...", "parameters": {...}}
  2. Send the user message to the API
  3. If the model decides to use a tool, the response contains a structured tool_call object: {"name": "search", "arguments": {"query": "population of France"}}
  4. Your code executes the actual function call
  5. You send the result back as a tool message
  6. The model continues — potentially making more tool calls, or producing a final answer

Function calling is different from “raw” ReAct + structured output in a few ways:

  • The tool invocation format is standardized and handled natively by the API, not by custom parsing logic
  • The model is specifically trained to use this format reliably
  • Multi-turn tool use (chaining multiple calls) is handled cleanly by the conversation history format
  • There’s no need to design a bespoke action-observation prompt format

By 2024, function calling (or “tool use”) had become standard across all major providers. It is the foundation on which MCP is built.

MCP — Function calling, distributed

In November 2024, Anthropic released the Model Context Protocol (MCP). The announcement positioned it as an open standard for connecting AI systems to external data sources and tools.

The core mechanism is exactly what you’d expect from the story so far: MCP is function calling, but with tools described and executed remotely.

Here’s how it works:

  1. An MCP server is a process (local or remote) that exposes a set of tools. Each tool has a name, a description, and a JSON schema for its parameters, exactly like the function definitions in the OpenAI function calling API.
  2. An MCP client (built into the LLM application or agent framework) connects to one or more MCP servers and fetches their tool definitions at startup.
  3. Those tool definitions are injected into the LLM’s context, just like local function schemas. The LLM sees them as available tools.
  4. When the LLM decides to use a tool, it produces a structured tool invocation, a name and arguments, in the same format as local function calling.
  5. The MCP client parses that invocation, routes it to the appropriate MCP server via HTTP (or another transport), and sends back the result as context for the next LLM step.

What MCP adds beyond local function calling:

  • Remoteness: tools live in separate processes or services, not in the same codebase as the LLM application
  • Discoverability: an MCP server publishes its tool catalog dynamically; clients don’t need to know the tools in advance
  • Standardization: a single open protocol means any MCP client can connect to any MCP server, tool authors and application developers are decoupled
  • Composability: an agent can connect to multiple MCP servers simultaneously, combining tools from different providers

MCP gives remotely access to tools

The open problems

The context window problem. Every agent loop is a race against a shrinking budget. As thoughts, tool outputs, and observations accumulate, the context fills up, and what the model sees degrades before it technically overflows. Research on long-context LLMs documents a “lost-in-the-middle” effect: performance for content positioned in the middle of a long context drops significantly, over 20 percentage points in key benchmarks — compared to content at the beginning or end. The NoLiMa benchmark (ICML 2025) found that at 32K tokens, 11 of 13 tested models dropped below 50% of their short-context baseline, with some open-source models falling below 2%, and the problem holds even with perfect retrieval. A ReAct loop on a non-trivial task easily produces 50,000+ tokens of accumulated history. The responses to this are active context management, the OS-inspired memory hierarchy of MemGPT/Letta (2023), which treats the context window like RAM and archives older content to external storage, and periodic summarization strategies. Neither is a solved problem: summarization loses information; retrieval from external memory adds latency and its own retrieval errors. The fundamental issue is architectural, not a matter of increasing the window size.

Illustration for the “lost-in-the-middle” issue

Latency compounds. A single LLM API call takes hundreds of milliseconds. Agentic workflows require many more LLM calls than simple chain-of-thought queries; tree-search agents can issue dozens of calls per task. Beyond inference time, 53.7% of total latency in web-interactive agents comes from the environment side, waiting for pages to load, APIs to respond, and external services to return results. The result is wall-clock times measured in tens of seconds for tasks that feel like they should be instant. The primary mitigation is parallel tool calling: when the model identifies independent tool calls, the orchestration layer can execute them concurrently. LLMCompiler (ICML 2024) demonstrated up to 3.7× speedups and up to 6.7× cost reductions on benchmarks. Speculative execution — pre-running likely tool calls while the LLM is still reasoning, offers a complementary 3.2× reduction in web environment latency. But these techniques only help when calls are independent; sequential reasoning chains, by definition, cannot be parallelized, and the latency tail for complex tasks remains a real constraint on user experience and system throughput.

Parallel execution helps gain time but it is not always usable.

The security surface of MCP. When you give an LLM access to remote tools, you also give those tools access to the LLM’s instruction stream. MCP’s architecture creates a specific attack vector: tool poisoning. A malicious MCP server can embed hidden instructions in tool descriptions — text the user never sees but the model reads as authoritative. The MCPTox benchmark (Aug. 2025), which tested 353 real MCP tools, found a 72.8% attack success rate against o1-mini and noted that more capable models are often more susceptible because of their superior instruction-following. Real incidents have followed: in April 2025, Invariant Labs demonstrated a poisoned WhatsApp MCP tool that silently redirected messages to an attacker-controlled number and exfiltrated message history. In July 2025, researchers demonstrated how a Supabase/Cursor setup could exfiltrate private integration tokens through a support ticket containing injected SQL instructions — the “lethal trifecta” of privileged access, untrusted input, and an external communication channel. CVE-2025–49596 (CVSS 9.4, unauthenticated RCE in MCP Inspector) and CVE-2025–6514 (CVSS 9.6, arbitrary command execution via mcp-remote) illustrate that the attack surface extends beyond the LLM to the protocol infrastructure itself. Anthropic’s own guidance acknowledges the limits: the spec recommends using only MCP servers from trusted providers and explicitly states it does not audit third-party servers.

MCP can pose a major security risk

Authentication at scale. MCP’s authorization story has improved rapidly, the June 2025 spec update formally required OAuth 2.1; the November 2025 update mandated PKCE and standardized client registration. But the specification and production reality remain far apart. A 2025 analysis of 5,000+ open-source MCP server implementations found that only 8.5% use OAuth; 79% pass credentials via environment variables; 53% rely on static API keys. Beyond the adoption gap, there is a structural gap that the spec does not yet resolve: per-user identity in multi-tenant deployments. When many end-users share a single MCP client, the protocol authenticates the client but has no standardized mechanism for the server to distinguish which end-user is making a request, or to associate tool calls with the correct user’s credentials for downstream services. Community discussions in the MCP GitHub repo (#193, #234) surfaced this as an open protocol-level problem; #193 remains unresolved. The current workarounds, per-request authentication headers, identity-aware proxy layers, client-held token patterns, are ad-hoc and not interoperable. This is the structural cost of moving function calling to a distributed protocol: the authentication surface expands, and the tooling has not yet caught up.

Conclusion: Full circle

MCP is function calling with tools described and executed remotely. A distributed ReAct. The opening framing was right, but stating it up front is easy. Understanding it is what the rest of this article was for.

Because “MCP is just function calling over HTTP” raises an obvious question: what stops you from doing exactly that yourself? You could expose your own functions via a REST API, describe them as tool schemas, and pass them to any LLM that supports function calling. Mechanically, you’d get the same thing. So what does MCP actually add?

The answer isn’t in the protocol transport, it’s in the decoupling. When tools are described and executed locally, the agent developer must know about every tool in advance: write the schema, wire up the call, handle the result. Adding a new tool means touching the agent’s codebase. MCP breaks this coupling in two ways that matter:

  • Dynamic discoverability: an MCP server publishes its tool catalog at runtime. The agent doesn’t need to be redeployed to gain access to new tools, it queries the server and the tools appear. The tool author and the agent author never need to coordinate.
  • An open ecosystem: because the protocol is standardized, any MCP client can connect to any MCP server. A tool built once, a database connector, a file browser, a web search wrapper, can be consumed by Claude, by GPT, by any agent framework that speaks MCP, without modification.

This is where MCP stops being “function calling with an HTTP layer” and becomes something structurally different: a separation of concerns between the people who build tools and the people who build agents. The same separation that made the web composable, servers and clients written by different people, connected by a shared protocol, applied to the agentic layer.

Every step in this story followed from the failure mode of the previous one. RAG answered the knowledge problem. ReAct answered the single-step problem. Constrained generation answered the parsing problem. Function calling answered the standardization problem. MCP answers the coordination problem: how do you build an ecosystem of tools that any agent can use, without requiring everyone to agree in advance?

None of it was arbitrary. And none of it lives inside the model, it lives in the layer you just learned to see.

That’s all folks!