The Model Will Forget What You Told It
Give an agent five things to do in sequence and somewhere around the third or fourth it will drop one. No error, no refusal, no half-finished output that announces itself as half-finished. It stops applying something you asked for earlier, carries on with everything else, and hands back work that looks complete.
I run agents against my own work every day, and the ways this shows up are consistent enough to name. The first is instruction decay: a constraint you set at the beginning stops being applied somewhere in the middle, and nothing marks the transition. The second is context loss — a long task gets compacted or reset, and whatever lived in the part that was summarised away is gone rather than flagged as gone. The third is the expensive one, because it produces no symptom at all: there’s no default mechanism in any of the tools I use that distinguishes work that was delivered from work that was delivered correctly. Both arrive looking the same.
None of this is a complaint about model quality. The models are good, they keep getting better, and most individual outputs I get back are fine. The problem is structural. The thing producing the work and the thing reporting on the work are the same thing, and it has no independent record of what you asked for twenty minutes ago to check itself against.
Why this gets harder the more you hand over
A single agent doing one task while you watch is a manageable risk. You’re reading the output as it appears, you still remember what you asked for, and the distance between instruction and result is short enough to hold in your head. Most people’s experience of AI is exactly this, and at that scale it’s genuinely fine.
Chaining changes the arithmetic. The output of one step becomes the input of the next. Several jobs run against different parts of the same task. Nobody is reading every intermediate result, because reading every intermediate result would cost more than doing the work by hand. A dropped instruction at step two is no longer visible at step two. It surfaces at step seven as something subtly wrong that looks deliberate, or it doesn’t surface at all and quietly becomes part of whatever record you wrote the result into.
What you have built at that point is a system with parts that fail independently and don’t reliably report their own failures. That isn’t a new category of problem. It’s a very old one, and it usually gets discussed under a different name.
The principles come from payments
I spend my working life in payment orchestration — the layer that sits above a set of payment providers and decides which one a given transaction goes through. Most of what that discipline knows is about this exact shape of problem: a set of components that each work most of the time, fail for unrelated reasons, and rarely announce it when they do.
Three of its habits transfer without much translation.
Redundancy comes first. You don’t run a business through a single provider, and the reason has nothing to do with providers being unreliable. Any single path can degrade for reasons that have nothing to do with its quality — a bank adjusts a threshold, a risk team reviews a category, a connection has a bad week — and a business with one path has no way to notice, because it has nothing to compare against. A single agent doing a job you can’t independently check is in the same position: whatever it produces is both the work and the only account of the work.
The second habit is keeping a second route live before you need it. In payments that means integrating the alternative while things are calm, because the moment you need it is the moment you have no time to build it. With agents the equivalent is less dramatic and more boring: knowing which parts of a job you could still do by hand, at what cost, and keeping those parts small enough that the answer is never “we can’t.”
The third is verification, and it’s the one that matters most here. In payments you don’t take a provider’s word that money moved. You reconcile — you check what the provider says happened against a separate record of what happened, and where the two disagree, the disagreement is the finding. Nobody in payments treats this as paranoia. It’s simply what you do before you let a number be true. Pointing that same habit at model output isn’t an exotic new discipline someone needs to invent. It’s the oldest instinct in my actual job, aimed at a newer kind of pipe.
What that looks like in my own system
The system I run against my own sales pipeline was built on this from the start rather than patched with it afterwards, and two rules do most of the work.
Nothing sends. Every output it produces — an outreach draft, a follow-up, a written account of where a deal stands — lands in a draft and waits for me. It has never sent a message on its own and it isn’t designed to.
Nothing enters the record without a source it can point at: a line in a transcript, an actual email, a calendar entry. Where no such source exists, it writes “unverified” rather than writing the claim. That produces a record which is sometimes visibly incomplete, and the visibility is the point. An unverified flag costs a minute to resolve. A confident wrong entry costs whatever gets decided on top of it, because nothing about its appearance invites a second look.
The clearest thing the system has caught so far was my mistake rather than its own. Two deals I had personally signed had never made it into the record properly — they existed in conversations and in my memory and nowhere a system could see them. The system surfaced them by reading the correspondence and noticing that something which had plainly happened had no corresponding entry anywhere. Humans make mistakes and AI systems make mistakes. This was a case of one catching the other, which is the only arrangement I actually trust.
The cheap version, if you have no system
Most people reading this aren’t going to build anything. That’s a reasonable position, and the discipline still transfers at a much lower price.
Ask for the source alongside the answer, every time, and treat an answer that can’t produce one as a draft rather than a result. Restate the constraints that matter at the point where the work gets long, instead of assuming something you said at the start is still in force. And check a sample rather than everything — not the output that looks wrong, which you would catch anyway, but two or three pieces chosen at random from the output that looks fine. The failure mode described at the top of this article is invisible precisely because the work looks finished, so a spot check aimed only at suspicious-looking results will find nothing and tell you everything is well.
Where this doesn’t apply
Treating every task as though it needs a verification layer is its own failure mode, and an expensive one. It’s how people end up with more process than work.
If you ask a model a question and read the answer yourself in the same minute, you’re the verification step. There’s no gap for anything to drift into, and adding ceremony to that is pure cost. Most of what most people do with these tools is this, and it needs nothing from this article.
The discipline earns its place when output is going to sit somewhere unattended: written into a system of record, acted on next quarter by someone who wasn’t in the room when it was produced, or handed to another agent that will treat it as settled fact and build the next thing on top of it. The question I use is who reads this next, and would they be in a position to notice if it were wrong. When the answer is nobody, the checking has to be structural, because it isn’t going to be human.
What I would avoid is treating model output the way you would treat a competent colleague’s — broadly trusted, spot-checked occasionally, escalated if something feels off. That prior is wrong in one specific way. A colleague who has stopped doing what you asked usually knows they have stopped. An agent doesn’t, and will tell you the job is done in exactly the tone it would use if the job were done. Work from the assumption that the output is probably fine and definitely unverified, and confirm it before anything else gets built on top of it.