AI Agents: What They Are and Where They Break

The word "agent" is used loosely enough to be nearly meaningless in marketing material. The distinction that actually matters is simple: a system that produces text for a person to use, versus a system that takes actions in the world.

Crossing that line changes the risk profile entirely, and it deserves a deliberate decision rather than a feature upgrade.

The difference

An assistant drafts an email. You read it, adjust it, send it. If the draft was wrong, you noticed.

An agent is given a goal — "resolve this support request" — and works towards it in steps: reading the request, looking up the customer, checking an order, deciding on a response, and sending it. It decides its own sequence, and it acts.

The essential shift is that errors are no longer caught by a person before they have effect. They have effect first.

Where agents genuinely help

Multi-step tasks where the steps vary by case are a poor fit for conventional automation, which needs the path specified in advance. Where the path depends on what is found along the way, an agent can be a good answer.

Investigating and gathering — pulling together everything relevant to a question from several systems — is another sound use, particularly because the output is still presented to a person.

Where they break

Errors compound. This is the fundamental problem. If each step is right ninety-five percent of the time, a ten-step sequence is right about sixty percent of the time. The arithmetic is unforgiving, and it explains why demos of three-step agents look excellent while real ten-step processes disappoint.

Recovery is hard. When a step fails, the agent may not recognise it and will proceed on a false premise — sometimes producing an elaborate, confident sequence of actions built on an early mistake.

Actions are not always reversible. A sent email cannot be unsent. A refund issued must be recovered. Before granting the ability to act, ask what the worst available action is and whether it can be undone.

Debugging is genuinely difficult. When a deterministic process fails, you read the code. When an agent fails, you read a trace of decisions it made for reasons that are not fully inspectable, and which may differ on the next run.

Instructions can arrive in the data. If an agent reads external content — an email, a web page, a document — that content may contain text that reads as an instruction. An agent with real permissions and untrusted input is a security question, not merely a reliability one.

Building them responsibly

Keep sequences short; every additional step multiplies the failure rate. Grant the narrowest permissions that allow the task, and prefer reversible actions. Require confirmation before anything with external effect — sending, paying, deleting. Log every step so failures can be reconstructed. And define what the agent must never attempt without a human, explicitly.

A reasonable position

For most companies, the right first deployment is an agent that gathers, analyses and proposes — but where a person performs the final action. You capture most of the value while errors remain recoverable.

Grant autonomy later, narrowly, for specific actions you have watched behave correctly over time. That is a defensible progression. Starting with full autonomy because a demo was impressive is not.

All Articles
Let’s Talk

about the process
AI should run.