What is human-in-the-loop in AI agents?

An AI agent preparing a response to a customer enquiry searches, summarises and drafts an answer. At that point it stops, and an employee reads the proposal and clicks send. That point is called human-in-the-loop. It is a step in the process that the agent cannot pass without a person’s decision.

Why that point needs to be there

An agent can do a lot, but it cannot be responsible. If a customer receives an incorrect quote, a payment goes to the wrong account or an employee receives a message that should never have been sent, the agent is not the one who explains it. Someone in your organisation is. Human-in-the-loop makes sure that person has actually seen it before it happens. It is the point where a person checks whether the proposal is correct, decides whether it may proceed and thereby takes ownership of the outcome.

On top of that, an agent is poor at spotting its own mistakes. It works with what it is given, and if something is missing or wrong, it often does not notice. Human review is not an automatic solution, however. People can approve too quickly or assume the proposal must be correct. The review only works if the employee has enough context and time, is authorised to decide and can actually change or reject the proposal.

The EU AI Act requires appropriate human oversight measures for high-risk AI systems under Article 14. Whether a system falls into that category depends on its intended use and its effect on people. An internal system for employee selection, for example, may be high risk. Aside from that legal classification, human intervention is also a broader design choice: when an agent acts on behalf of your organisation, someone must be able to intervene before a risky decision or action takes effect.

Where that point belongs

Look beyond reversibility to risk, impact and uncertainty as well. Human approval is appropriate, for example, for:

• external communication on behalf of your organisation;
• payments and deletions;
• decisions about employees or customers;
• exceptions and situations where the agent has little certainty;
• actions that automatically start other processes.

An agent can usually do preparatory work such as searching, reading, summarising and drafting independently. Use as few approval points as the risk responsibly allows. If every step needs approval, the person ends up doing the work themselves with an extra screen in between.

A good approval step shows the proposed action, the source data used, the expected consequences and any uncertainty. The employee has enough time and can approve, change or reject the proposal.

Where that point is enforced

This is the point that most often goes wrong. A rule in the instruction to the model, such as “always ask for confirmation before sending an email”, is a request rather than a guarantee. The model can apply the instruction incorrectly or wrongly assume that permission has already been given. Approval therefore needs to be technically enforced in the harness, the software around the model: the tool that sends only works with an approved command, or the action stays in a queue until someone releases it. Then it does not matter what the model thinks was agreed. How that choice relates to the rest of the build is covered in the guide to building your own AI agent. The other agreements you record about permissions, logging and ownership are covered in the guide to AI agents for SMEs.