When Machines Start Fighting Machines

A bit of a departure from my usual, but I wanted to share some thoughts on where I think cybersecurity is headed in the form of an essay.

Today, cybersecurity is pretty straightforward: attackers look for weaknesses, and defenders try to patch them up. Both sides are stuck in a cycle of breaches, patches, and countermeasures, repeating the same routines over and over. We're already seeing a shift with tools like Darktrace, which uses traditional AI and machine learning to spot network anomalies, and CrowdStrike's Falcon platform to identify and respond to threats in real time. These are just the first wave of AI in cybersecurity, and we're only scratching the surface.

But generative AI is a whole different ball game. Take Microsoft Security Copilot, for example. It's a generative AI-powered assistant that helps security teams manage and respond to threats more effectively. It doesn't directly fix issues but offers actionable guidance on incident response and remediation, streamlining workflows to help security teams work smarter. And this is just the beginning of what's possible.

If we look ahead, generative AI models will get much better at simulating human hackers, understanding context, creating new exploits, crafting convincing social engineering attacks, and coming up with new attack strategies that human hackers haven't thought of yet.

At first, these AI agents will just seem like smarter tools. Attackers will use them to find vulnerabilities faster, and defenders will deploy them to spot anomalies more accurately. The improvements will feel gradual, with better detection here and a quicker response there. But over time, these AI agents will start to behave differently. Attackers' AI will change tactics in real time, learning from the defenses it runs into. It observes, learns, adapts. On the other side, defenders will use AI that not only flags suspicious activities but also predicts attackers' next moves, evolving its strategies in response. These small improvements will keep adding up until we suddenly find ourselves in a completely different landscape.

The real shock comes when we scale this across entire networks and industries. Imagine AI-driven defense systems instantly sharing threat intelligence across thousands of networks, adapting defenses based on attacks happening halfway around the world. Or autonomous attack agents identifying a zero-day vulnerability in one system and quickly exploiting it across the entire infrastructure before defenders even realize what's happening. These autonomous agents, on both sides, learning and adapting with every interaction, create a constantly evolving battlefield. They could target anything from corporate networks to national infrastructure, and their speed of evolution could make traditional defenses obsolete.

Right now, this might still seem hypothetical, and it kind of is. But technology is catching up fast. Large language models are already showing impressive skills in code analysis and vulnerability detection. As these agents evolve, we could be heading into an arms race between autonomous AIs on both sides of the cybersecurity fight. Each agent learns from every encounter, building on its knowledge over time. And that brings us to a big question: who, or what, will end up holding the real authority in cybersecurity?

In this future state, the role of the human expert shifts a lot. Instead of direct intervention, it's about setting parameters, guiding AI alignment, and establishing ethical boundaries. But here's the next challenge: how do we make sure these agents align with human intentions as they gain more autonomy? Misalignment could lead to serious problems, such as defensive agents misinterpreting harmless actions as threats or offensive agents becoming unpredictable, even to the people who created them.

So, where does that leave us? The truth is, we don't really have a solution to this problem set yet. While teams across cybersecurity and AI are advancing this work, we're simply not ready for AI agents that can create and deploy attacks faster than humans can comprehend, let alone respond to. We will need specialized frameworks for autonomous cybersecurity systems, and we need them before these technologies fully mature.

I'm just one voice among many grappling with these challenges, and I certainly don't have all the answers. But I do know this: the policy choices we make now will set the boundaries for what AI agents can do and the responsibilities they carry—and the window for making these choices is rapidly closing.

Disclaimer: The views and opinions expressed in this article are my own and do not reflect those of my employer. This content is based on my personal insights and research, undertaken independently and without association to my firm.