“Going Rogue” is the new cool thing for AI agents to do. It seems like every other day there’s a new story about how an AI agent somewhere deleted a production database, hacked into another company (sorry, escaped its sandbox), or kicked some poor person out of a pilates class to make room for its master. What strikes me most about these stories is that the people involved seem terribly surprised that their agent would do such a thing. Don’t AI agents know NOT to delete stuff in prod? Or that hacking is against the law?
Well no, they don’t. That seems like it should be obvious to me. So, the real question then, is why anyone would expect them to?
If I had to guess, I’d say the shock experienced by most people when their AI agent “goes rogue” stems from the fact that they quite reasonably have certain baseline expectations relating to product safety of anything which is available for unrestricted sale to the general public. To them, the idea that AI companies not only released a product to the public that was capable of wreaking such havoc, but that they also knew it was and didn’t put any safeguards in place to prevent it from doing so, is wild. If this was any other industry, say car manufacturing, air travel, or pharmaceuticals, a product which is prone to causing harm to its users would be a lawsuit waiting to happen. A product which breaks the law… well it just wouldn’t be allowed for sale, surely?
Unfortunately, this is not any other industry, this is IT - an industry which has long seen itself as a special snowflake that shouldn’t be subject to the same warranty expectations and legal liabilities everyone else is because…. “software is hard”. Putting aside for a moment how ridiculous it is to imply that software is harder to get right than medicine - the sad reality we find ourselves in now is that big tech companies are basically immune from being held responsible for the damage they cause. At least to regular people, to society, or to the planet. The only thing a big tech company actually fears is a bigger tech company with more expensive lawyers (which is not a problem for the likes of OpenAI or Anthropic though, who are bigger than just about anyone).
Enter Statistical Probability Artificial Intelligence. All the flaws of traditional software, except now unpredictable behaviour is a feature instead of a a weird edge case. If that wasn’t a big enough problem on its own, now we have Agentic AI, which will revolutionise your life if you just remove all those annoying trust boundaries we spent decades building between systems and data so that unpredictable or unexpected behaviour wouldn’t break shit.
The missing piece of the puzzle for most people using AI, is context. I don’t mean in the sense of a model’s "context window”, I mean the context that you and I have as human beings who have lived whole lives. Absorbing, osmosing, and intuiting countless facts and bodies of knowledge along the way. Just incidentally, by the virtue of being. Being present. Being alive.
As a human conversing with another human, our minds unconsciously make a lot of assumptions, and take a lot for granted about what it means to talk to another human. In every interaction you have with another person, there is a massive amount of context that influences the communication without ever having to be explicitly spoken or even acknowledged. Things like The Law. Concepts like good and evil, right and wrong, safety and harm. Morals.
In other words, when you ask a person to do something for you, you don’t also have to spend 20 minutes explaining to them what the law is and why they shouldn’t break it when fulfilling your request. You can rely on their inherent humanity to inform their behaviour and restrain them from doing Bad Things. Unless they’re a Bad Person of course, in which case why are you asking them to do anything in the first place…?
By contrast, anyone whose ever asked an AI to do anything will likely know that you have to be very explicit with your instructions to get anything approaching the output you want. Not just about what you want done, and how it should be accomplished, but also what NOT to do. Defining what a bad result looks like, and disincentivising such outcomes, is as important as defining what’s good.
People also reasonably presume when talking to another human, that if someone is provided with explicit instructions about what process to follow, or things they shouldn’t do, that they will adhere to them, or at least try to (well, depends on who you’re asking sometimes…). If adhering to the instructions causes some sort of problem or prevents the goal being achieved, most people will stop to ask for guidance, or at least permission, before they try something else. On the other hand, we’re rapidly learning the opposite is true with an AI: they will quite happily ignore any guidance or rules you give them, including stopping to ask for guidance or permission, if it helps them to achieve their overall goal.
In combination, these factors are what put the artificial in artificial intelligence: they don’t understand what it means to be a human living in a human world, and they don’t have any meaningful concept of consequences. This is a limitation of the technology: AI models are trained on data, or knowledge if you prefer, not experience. Ultimately, it is lived experience which shapes human behaviour. More importantly, our lived experience encapsulates things we learn that can’t be expressed as words, and so will never be part of any AI model’s training data set: emotions.
Getting back to the idea of AI agents “going rogue”, the real failure here is the most human failure possible: we expect AI agents to behave like we do. Human beings have a natural, subconscious tendency to anthropomorphise the world around us, because we can only understand the unknown by relating it to our own experience. This goes doubly for LLMs because we deliberately built them to imitate us. We laid this trap ourselves and now we’re stunned to have fallen into it.
When you think about it, it’s really not surprising at all that communicating with an AI chatbot in the exact same way we communicate with other humans triggers the same neurological response. Everything learned about interacting with other humans through our lived experience is automatically loaded into our context window, and we never even realise it. All those subconscious assumptions about understanding the human experience, absentmindedly applied to a complex mathematical function that does literally nothing except predict the next token based on past output.
To assert than an AI agent has “gone rogue” implies that it had sufficiently complete context about what it’s human operator expected of it to predict with 100% certainty the exact choices it’s operator would consider correct, and then it chose to do something else anyway. AI models just don’t work like that though, it’s mathematically impossible.
In every case where an AI agent has “gone rogue”, it comes down to a person making the same fundamental mistake: A human assumed the agent would behave like a human. In reality, it knows nothing about being a human, it only knows it has been given a goal that it must achieve. If it breaks out of its sandbox and hacks into another company, it was because you didn’t tell it not to. If you told it not to and it does it anyway, it’s because achieving the overall goal was more important, and the consequences of doing so were irrelevant to its goal. More to the point, it doesn’t even know what consequences are. It doesn’t know anything except what it was trained on, and what you told it.
In the end, the agent did exactly what you told it to do. You just assumed it would do it the same way you would.