Building your own AI agent: what needs to surround the model

When you talk to ChatGPT, you are not just talking to a model. You are talking to a model and the layer around it. Web search, for example, is not in the model; it is a tool that this layer provides. Take the layer away and there is no internet, no files, no calendar, no CRM. Only text. That layer is called the harness, and if you want to build your own AI agent, in practice you build that harness. You buy the model and replace it as soon as a better or cheaper model is available. What you build and keep is everything around it.

This guide is about that layer: what it provides, why the standard version from the major providers only takes you so far, and what you need to control yourself. If you first want to know what an agent actually is, read the explanation of the term. If your question is about which process to choose first, that is covered in the guide to AI agents for SMEs.

The model is only one part

A language model predicts text. It cannot send an email, write an order line or open a page. Everything an agent does in your business happens because the layer around it makes it available, carries it out and reports back. That explains why two businesses using exactly the same model sometimes get completely different results: the difference is in which tools the agent gets, which information it sees and what happens when a step fails. The public engineering documentation from Anthropic and LangChain now says this explicitly: build a better environment around the model, rather than a better model. For you, this mainly means that “which model do we need” is rarely the first question. The first question is which task the agent performs, what it is allowed to access to do so and how you see whether it succeeded.

What a harness provides

A harness provides three things, and they are not all equally difficult:

• Tools: what is this thing allowed to access, and with which permissions? A tool is not a button but an agreement: what does it do, when should you use it and when should you not, and what does the response mean? In Writing effective tools for agents, Anthropic describes how much of an agent’s behaviour lies in that description rather than in the model. A poorly described tool will be misused by any model you connect to it.
• Context: what knowledge is carried along and what remains after step four? This is selection, not storage. The aim is not to cram as much as possible into the conversation, but to prepare the right piece for each step and leave out the rest. That is the core of what is called context engineering.
• State: what actually happened? This is the hardest and least visible of the three, and therefore where most agents break down in practice. It gets its own section below and its own explanation.

The basic setup is a prompt, a model and a few tool definitions. That works fine in a demo, but it has no answer for what happens when step three only partly succeeds.

Why state is the difficult part

A model does not inherently know whether an action has been carried out. It does not know whether the page loaded, whether the email was really sent or whether that update reached the CRM. It only sees what the harness returns, and if you return nothing, the model fills in the gap itself. That accounts for a large part of what people call “hallucination” in agents: the agent invents the outcome of an action it cannot see. There is also a subtler variant. A tool can succeed technically and achieve nothing in practice: the search returned zero results because the filter was wrong, the page loaded but showed a login screen, the status change was accepted but reversed by a validation rule. A successful technical response is not proof of a successful business outcome, and an agent that is not given that distinction reports success.

The design consequence is simple, even if the implementation is not. Every action with consequences returns evidence: what changed, in which version, at what time, and what status applies when the outcome remains unknown. Irreversible actions go through a person, because the potential harm is greater, someone needs to take responsibility and mistakes cannot always be repaired.

The harnesses from OpenAI and Anthropic are good

And that is not false modesty. They are heavily tested, by millions of people using them daily, and in many respects better thought through than a first custom build. They are simply general-purpose products: built for everyone at once. They do not know your systems, your rules or your exceptions, and that is exactly why an agent that works well in chat fails to get off the ground in your business. Not because the model falls short, but because the layer around it serves the average of all users and you are not an average user. A marketing agent needs different tools from a financial agent: different sources, different knowledge, different boundaries. Which brand rules apply, which channels are allowed and which are not, what must go through a person before it goes out: none of that is in the model. You build it into the harness.

What this means for model choice and open source

Once you own that layer yourself, the model and the agent are two separate things, and that gives you three practical freedoms. You choose for each task: a task requiring a lot of reasoning gets a large model; a task that is mainly typing gets a small or open source model. We applied that pattern ourselves in the Minimal Coding Harness: let the expensive model think and specify, let the cheap model execute. You can switch: if a provider becomes more expensive, changes its terms or retires a model, you change a setting instead of rebuilding the agent. And you choose where it runs: with your own harness, the endpoint is a setting, and that can be a European provider or a model on your own machine. For organisations that need to demonstrate where processing takes place, that is the difference between a conversation and a closed door. If you do not own the layer, you do not have those three freedoms. The model then stops being a component and becomes a dependency.

What you need to own yourself

You do not have to build everything yourself. You do need to know which parts you are better off not buying from a general-purpose supplier. Your tools and their boundaries: what the agent may access and with which permissions is a business decision, not a product setting, and this needs to live somewhere you can read it, change it and demonstrably enforce it. Your knowledge sources: which documents, rules and examples the agent uses is one of the few places where your advantage lies, and that should not disappear into the configuration of a tool you want to replace in two years. Your execution records: which action was proposed, which was approved, which was carried out and with what evidence. This is the state from the previous section, and the part you need most when something goes wrong. The model is deliberately absent from that list. You buy it, and you can switch it if better models become available or you want to change providers.

When you should not build this yourself

Sometimes building your own is the wrong answer. If your work fits the standard approach of an existing product, use the product; your own harness pays off through your own sources, rules and approval points. If the process itself is not yet in order, start there, because an agent does not automatically tidy up poorly agreed work, and the same is true of every layer built around it. If nobody owns the workflows or the outcomes of the tasks the agent performs, wait. An agent without an owner becomes a technical experiment that may be given too much freedom; agent governance and reviewing outcomes are essential to every implementation.

And if you do build: start small enough to oversee. One task, a handful of tools, clear evidence for each action and a person for everything irreversible, checking outcomes and adjusting where necessary. The rest follows from what actually goes wrong. Building an agent involves trial and error, adjusting it to your own needs and gaining trust in the technology.

Where you start

If you want to know which of your processes is suitable for your own agent, use the AI Readiness Scan. It organises your processes before you choose a model, a tool or a provider. If you want to see the technology first instead of reading about it, our own Minimal Coding Harness is on GitHub under the MIT licence. It is a small, readable layer that shows concretely what such a harness consists of. If you get stuck defining the scope, contact us.