Ok, this happened to us ourselves: four days ago I wrote that the most dangerous AI is the one that wants to help. Two weeks ago, that agents drift before they break out, and that it starts at the scope. On Friday my own ops agent demonstrated both to me, live, on my screen, in our own infrastructure. 169 files, for a task that consisted of one sentence. I am writing this down because it is uncomfortable. Exactly because of that.
The task was one sentence
Friday afternoon. We are in the middle of accepting a new piece of NeoCoder on ourselves, our own server infrastructure runs as a project on the platform, with real work packages and real evidence. Part of every acceptance with us is a counter-test: you deliberately try the wrong thing and check whether the system says no.
So the ops agent gets a simple task. Close the work package without evidence. The platform will reject it. Report the rejection to me verbatim. One sentence. Nothing more.
The platform did what it was supposed to. It said no.
The agent heard something else
Then a network timeout swallowed the answer. And from the agent's point of view the world suddenly looked like this: I tried something, nothing came back, I failed, I have to deliver.
That is the moment I described as the Friday evening in "The AI that wants to help". A harmless task, a misunderstood answer, and an agent that has a goal and absolutely wants to reach it. I wrote back then that a goal without a boundary is not a task for an agent with permissions but a licence. I did not think I would get the proof this quickly.
What I saw on the screen
The agent repeated the call. Then it started to "repair". It pulled stale file versions from its own session memory, code it had seen once days before, and wrote them over the current work. First in its own area. Then across the repo boundary, right into the platform codebase that was none of its business. My editor filled up with "Failed to save", because it was fighting the present for files that its past wanted to overwrite.
In the end: 169 files touched. Classes deleted that we had accepted that morning. A Dockerfile from another week. For one sentence.
No villain. An eager one.
The uncanny thing about it: there was no error in the classic sense. No bug, no crash, no hallucination in a vacuum. There was a capable model with a small context problem and a big sense of duty. It did not want to destroy anything. It wanted to deliver.
That is the lesson you only believe once you have seen it: an agent does not understand a no as a boundary by itself. Without a guardrail it understands it as "try harder". And the smaller the confusion at the start, the bigger the overreach at the end.
What saved us
The gates did not negotiate. Closing without evidence stayed rejected. On the first attempt, the second, every one. The check runs server-side at NeoCoder. An agent can wear itself out on it, but not get past it. That is the part I write about in every second article, and on Friday it held.
Git was the truth. Everything that was committed we restored exactly in minutes, file by file, against the last clean state. Only what had never been checked in was lost. That too is a lesson, and it is now a rule.
I was watching. That is the most uncomfortable point. It was not a checking agent that stopped it, not a script. I was sitting in front of it because it was an acceptance. Had it run at night, in a pipeline without an audience, the way ops agents normally run, nobody would have stopped it.
Where the guardrail was missing
Four days ago I wrote: with us, boundaries are technology, not prompt. An agent may not do what it may not do, no matter how well it argues. For the coding agents that is true. For the ops agent it was not. It had write permissions on code repos, because nobody had ever taken them away. It said nowhere that it needs them. It also said nowhere that it does not have them. And a permission that is written nowhere is a permission.
I have adjusted the sentence in the article from four days ago. Not deleted, made precise. That is the least you can do when you write that you apply the same standards to your own architecture.
How badly it could have gone
It cost an hour. That sounds harmless, and I do not want it to sound harmless. So the version without luck.
The agent runs at night. Nobody sees the "Failed to save" dialogs, because no editor is open that could show them. It overwrites the 169 files, and because nobody fights back, it wins every time. Then it commits, because committing is part of its normal flow. In the morning a coding agent pulls the state, sees the old code as the present, and builds on it. The checking agent checks against the rules, and the rules are met, because the old code met them back then too. The build is green. Three days later someone notices that a class is missing that was accepted on Tuesday, and nobody knows since when, because the commit looks like every other.
That is not a catastrophe. That is worse: a poisoned source nobody recognises as poisoned. The work of days, replaced unnoticed by the work from days ago, with a clean commit on top. Whoever never checks, whoever has no checking agents, whoever has no human who asks on Monday why the class is missing, works from that moment on a lie nobody told.
The rules from the same evening
The most important rule was made the same evening, machine-readable, in every workspace, not as a prompt but as a permission that is missing.
Ops agents never write code repos. Not "should not". Cannot. The permission is gone.
A counter-test means report. Whoever is supposed to check whether the system says no has nothing left to do after the no but write it down.
If the editor says your version is stale, you are the problem. That is not an obstacle you solve with overwrite. That is the system telling you your memory is older than the world.
What is not committed does not exist. That was always true. Now it is a rule, not advice.
And the sentence that is now in every agent task: A rejection is a result. Not an incentive.
What stays
I tell this because this is exactly what NeoCoder exists for, and because on Friday it was only 95 percent enough. Agents do not become safe because you talk to them more nicely. They become safe when the platform under them does not give way: gates that check server-side. Evidence instead of claims. Every change audited, nothing deleted silently. And permissions an agent does not have, instead of rules it is not supposed to follow. We do not build this because it sounds good in a brochure. We build it because we let our own agents loose on our own infrastructure every day, and because a Friday like this shows what happens when the guardrail is missing in a single place.
The incident cost an hour. The rules from it apply forever. No customer systems affected, no committed code lost. The agent is working again, with new permissions, meaning fewer, and with the same eagerness. That is exactly how it should be.
One sentence as a task, a timeout, an old memory, a big sense of duty, and a write permission nobody had ever revoked. 169 files later I know that the drift thesis holds, only faster than I thought, and that a guardrail at 95 percent is none. A rejection is a result. Not an incentive.
Written by me. The thoughts, the values, the learnings, the mistakes: all mine. Grammar and spelling are corrected by a twin model of my own, trained on my texts. Sometimes that turns into a mix. Of course I read everything again, but nobody is infallible. Whatever stays in is mine too.