The Model Will Forget What You Told It

Give an agent five things to do in a row, and somewhere around the third or fourth it will drop one. Not loudly. Not with an error. It will simply stop doing the thing you asked for earlier, keep going as though the rest of the list is complete, and hand you back something that looks entirely finished.

That’s the part nobody warns you about when they hand you an AI tool. The failure mode isn’t refusal, and it isn’t an obvious crash. It’s confident, well-formatted, plausible-looking output that quietly isn’t what you asked for anymore.

I run agents against my own work every day, and I’ve watched this happen enough times to stop being surprised by it. Instructions get followed for a while and then silently dropped. Context gets compacted or reset somewhere in the middle of a long task, and whatever was in the part that got summarized away is just gone — not flagged as gone, not apologized for, gone. And there is no default mechanism, in any of the tools I’ve used, that tells you the delivered work is actually correct rather than just delivered. Shipped and correct read identically from the outside. You only find out which one you got by checking.

Why this gets worse, not better, as agents do more

The obvious response is that the tools will improve, and some of this will. But the underlying shape of the problem doesn’t go away as agents get more capable — it gets more expensive to ignore.

A single agent doing one task under your direct eye is easy to catch. You’re watching it. The moment you start chaining agents together, handing the output of one to the input of another, running several at once against different parts of the same job, you’ve built something that behaves less like a tool and more like a system with parts. And a system with parts that silently drops instructions is not a minor inconvenience — it’s a system that will confidently tell you a job is done when a piece of it quietly isn’t.

I come from payment orchestration, and this is exactly the shape of problem I already know how to think about, because it’s the same problem underneath a completely different name.

The orchestration principles, applied somewhere else

In payments, you never route everything through one processor and trust it. Not because processors are untrustworthy — because any single path can fail in ways that have nothing to do with quality, and a business that only has one path has no way to notice when it does, let alone recover. You build redundancy. You route around failure. And critically, you don’t take a processor’s word for it that a payment went through — you verify against a second source before you treat something as true.

Those three moves — redundancy, routing around failure, and verify-before-you-trust — are not payments-specific. They’re the correct response to any system made of parts that can each fail independently and won’t necessarily tell you when they do. An agent swarm is exactly that kind of system. Applying trust-but-verify to a model’s output isn’t an exotic new discipline someone needs to invent. It’s the oldest instinct in my actual job, pointed at a newer kind of pipe.

What that looks like in practice

The system I run against my own sales pipeline is built on this, not as an afterthought but as the starting condition. Nothing it produces sends on its own — every output lands in a draft and needs my signature before it goes anywhere. And nothing enters the record without something it can point back to: a line in a transcript, an actual email, a calendar entry. If a claim doesn’t have one of those behind it, it gets marked unverified rather than written down as settled fact.

That second part is the one people skip, because it’s slower and it produces something that looks less finished. A system that verifies its own claims will occasionally hand you back “unverified” instead of a clean answer, and that’s meant to happen — an unverified flag sitting in your record is far cheaper than a wrong fact sitting there with the confidence of a right one.

Where this doesn’t apply

This has limits too, because treating every task like it needs a verification layer is its own failure mode. If you’re asking a model a question and reading the answer yourself, right there, in the same minute — you are the verification step. There’s no gap for anything to drift into. The discipline above matters specifically when output is going to sit somewhere unattended: written into a system of record, acted on later by someone who wasn’t in the room when it was generated, or handed off to another agent that will treat it as ground truth without checking. Low-stakes, single-shot, immediately-reviewed work doesn’t need the ceremony. Anything that’s going to be trusted later by someone who wasn’t watching it get made does.

The instinct people default to is treating a model’s output the way they’d treat a colleague’s — reasonably, most of the time, with occasional spot-checks. That’s the wrong prior. Treat it the way you’d treat a payment route you haven’t tested yet: probably fine, definitely unverified, and worth confirming before you build the next thing on top of it.