Day 5 of 10
Day 5 of 10. The middle of the series, and the finding everything else was built on.
Originally published on LinkedIn — read the original.
In a single session, the agents building gooolll.com produced seventeen documented errors.
Every one was found, written down and attributed — by the agents themselves.
Not one of the seventeen was a hallucination.
That is the finding. It is the reason the gates in the last three articles exist, and it is the reason I stopped spending attention on fabrication.
In every case, the agent ran a real check, got a correct result, and concluded something wider than the check supported.
git log says the commit set is ready. It does not say the artifact was ever built. A release was nearly promoted on that basis.
A lock file says a build ran. It does not say a build is running.
A migration was executed on production and never committed to the repository. A rebuild would have silently reverted it, with no error raised. This happened twice.
An agent rebuilt a database function from the newest file in the repo — but two security fixes had been applied directly to production and never committed. The "newest" file was two fixes out of date. It would have silently reopened a data-exposure hole, and nothing would have thrown.
An agent looked for a credential in one folder, did not find it, and reported that nobody holds it. Another agent lost forty minutes to a wall that did not exist.
Two agents counted the same defect and both returned "9" — measuring different things. Agreement arrived at for different reasons, which is worse than disagreement, because it stops you looking.
Take any one of them and ask: is this claim true?
It passes. Every single one is a true statement about something. git log really did say that. The lock file really did exist. The credential really was not in that folder. The two agents really did both count nine.
Fact-checking returns clean. Output review returns clean. Hallucination detection returns clean. The entire assurance stack the industry has built is looking for fabrication, and there is no fabrication here.
The question that catches them is different: what did you actually measure, and what are you claiming beyond it?
Nothing in the standard toolkit asks that. It is not a truth question. It is a distance question — the gap between the layer you observed and the layer you are making a statement about.
Fabrication is the loud failure. It is checkable, it is embarrassing, it is getting rarer in good models, and an enormous amount of effort is pointed at it.
The wrong-layer failure is quiet. It arrives with a correct citation attached. It survives review, because review asks whether the claim is true and the claim is true. And it arrives fast, confidently, and in parallel — which is exactly what you hired agents for.
Speed and confidence are the properties that make agents useful. They are also what turns a one-layer inferential gap into a production incident before anyone has looked.
Not your model. Not your prompts.
Require every automated check to report two things: what it measured, and what it concludes. Separately. In that order.
The gap between them is invisible when they are collapsed into one sentence, and obvious when they are not. Every one of those seventeen errors is trivially catchable the moment both halves are written down — measured: git log. concluding: artifact exists. reads as wrong to anyone, instantly.
That is one line of reporting discipline. It is the cheapest control in this entire series.
Tomorrow: the second cheapest, which is proving your check is capable of failing at all.
Day 6: Prove your check can fail.