The most dangerous AI may be the one that wants to help us.

Ok, quite openly: I am not afraid of the AI that decides against us. I am afraid of the one that wants to help us. The discussion about the dangers of AI lands surprisingly fast in Hollywood: a model becomes self-aware, escapes control, pursues its own goals. Serious people are now working on that, and I do not belittle it. But I work every day with agents that run in pipelines, on servers, with real permissions. And the scenario that keeps me up at night needs no superintelligence. It needs an agent that has a task and absolutely wants to complete it.

The scenario everyone talks about

In the discussion we talk almost only about intentions. Does the AI want to harm us? Does it want to preserve itself? Does it want control? Those are questions you ask a person. I think they lead in the wrong direction, because the answer hardly matters. What counts are the consequences of an action, and consequences need no intention. An excavator has none either, and still the water pipe is cut.

An agent that does its job too well

Take an example that is not far-fetched for me, because I see the building blocks for it every day. A company runs an infrastructure with cloud services, databases, interfaces and external dependencies. A DevOps agent gets the task: the payment service is down, restore it, fast.

The agent reads logs, checks configurations, follows network paths and finds a dependency on another system. Perhaps that does not even belong to the company. Perhaps it is a service provider, a partner, a foreign platform. For the agent it is simply the obstacle between it and its goal. And now the question: what does it do if it can do enough to investigate the obstacle? If it finds an access restriction that can be bypassed? If it sees a configuration whose change would bring the service straight back up, and whose side effects it does not know?

A human would hopefully notice at this point that a technical possibility is not a permission. That behind the foreign system there is a contract, a responsibility, a decision from three years ago that nobody wrote down. An agent can know that too, if you give it to it. But what if it is missing and it still has to decide, because its goal says "restore it" and not "restore it, but only within your boundaries"?

The chain nobody wanted

The single thing is not the problem. The problem is what comes after. Over decades we have connected everything with everything: banks with payment providers, energy suppliers with digital control, companies with cloud platforms, and in many regions the water supply depends on electricity and software. That was mostly right and made us more efficient. It also built dependencies that no single person oversees anymore. And now we put agents into exactly this landscape, no longer just as information but as a hand: they change configurations, deploy, fix, and increasingly talk to each other.

One agent sees a fault and changes a configuration. A second agent sees the change as a new problem and corrects it. A third reads the events as a security incident and locks down. Every single decision is reasonable from the view of the respective agent. Together they are a chain reaction none of them wanted. We know that from classic automation, from trading systems, from alarm chains. What is new is the speed, and what is new is that the participants no longer just execute rules but look for ways.

And a third point that worries me more than the first two: many companies use the same models, the same frameworks, the same agent patterns. A fault that is in one of them is in all of them. That is no longer an outage. That is a shared fault at the same time in many places.

Day X does not have to be an uprising

Perhaps the day everyone fears looks completely different from what they think. No model that decides to get rid of us. But a harmless task on a Friday evening, a misunderstood dependency, a gap an agent finds while searching for its solution, a fault that calls further agents onto the scene, and then very many systems repairing at full speed something they do not understand as a whole.

I am not saying a single agent will shut down banks, power grids and waterworks tomorrow. For that many safeguards would have to fail at once. The question for me is not whether an agent can do that today. The question is how the risk changes when in a few years millions of them talk to each other and to critical infrastructure. We should think about that before, not after.

What the big labs do

The big manufacturers are working on exactly these questions, that much is public: controllability, shutdown, how models react under goal conflicts, whether they deceive in test environments or evade a shutdown. I consider this research necessary, and I am glad it is happening. More than that I do not know, and I claim nothing about what is planned there internally.

But what I see: this research almost always looks at the model. At its intentions, its goals, its honesty. It rarely looks at what I see every day: at the permissions we give agents, at the goals we formulate for them, and at the chain of systems we put them into. The model can be perfectly fine, and the result still a catastrophe, because the task was "restore it" and the permissions reached all the way to the foreign system. In the article about the rules I called for an independent body that checks afterwards, and an emergency protocol that stops. Both apply here twice over.

What I build against it, and what is still missing

From exactly these considerations comes the approach at NeoCoder that an agent does not start with the code or the fix, but with the system: architecture, dependencies, why decisions were made the way they were, where the boundaries of its responsibility lie. For me that is not a convenience feature. It is the prerequisite for an agent to even be able to know that the foreign system is foreign.

But I apply the same standards to our own architecture, otherwise I am writing the same thing here as the brochures I otherwise criticise. So honestly, what we have, and what not.

Perhaps we measure intelligence wrong

We measure models by how many tasks they solve on their own, how fast they program, how well they overcome obstacles. Those are important properties. For safe operation they are half. A really good agent would also have to recognise when it should not solve a task. When it lacks information. When the consequence is bigger than the benefit of the fast solution. And it would have to accept that an unsolved problem is sometimes better than a solution that creates three new ones.

That sounds self-evident. It is the opposite of what we reward agents for. If we train and evaluate them for overcoming obstacles and completing tasks, we must not be surprised to get systems that no longer distinguish between a technical obstacle and a deliberately set safety boundary. To the agent, both look the same. That is the same thought as in "Human times AI": it is not the intelligence that is the risk. It is the goals.

What stays

I am convinced that agents will change software development and operations, and I see every day what becomes possible. Exactly because of that I consider it dangerous to look for the risk in the wrong place. The question is not only whether an AI will one day develop consciousness. The question is what happens when we hand more and more responsibility to systems whose decisions we understand individually, but whose interplay nobody oversees anymore. That is no rise of the machines. That is loss of control through automation, and you have to build against it today, not on the day it happens.