Generative AI: The New Spy vs Spy

Our real world is sort of becoming a metaverse in the inverse. Recently an AI generated photograph won the Sony World Photographer Awards. The photographer rejected the award, saying he was only trying to make a statement and spark a debate. A lot of content and interactions in here, out there, over and under, is going to be created through generative AI. Only we just won’t be able to tell the difference.

Anything that you do not physically see and touch can potentially be an AI at work. Any content on the web, people online, photos, videos, emails, phone calls and teleconferences, the lot. There seem to be hardly any exceptions. Your voice and photographs can be used to impersonate you - to the point that you need to establish “safe words” with near and dear for safety. Add to this scenario social engineering techniques that use malicious AI models that are also experts in psychology – they will know exactly what to say in a situation.

What are some possible strategies to combat AI-driven misinformation, scams, attacks, and malware?

For one, there will be regulatory and ethics frameworks. These might include the use of legal disclaimers that say you are interacting with an AI. Like you might hear from call centers: “This conversation is being recorded for quality purposes”. The European Union has embarked on a strong legislative framework for AI, which hopefully will set a good example.

There also needs to be technology-driven solutions. Such as wider use of digital signatures and certificates. Cameras need to digitally sign images and videos they generate. Media creation software could also add their own information and digital signatures, creating chains. In ancient times people had seals to attest to content authenticity, like those Sumerian or Akkadian cylinder seals. Likewise people might also want to use multiple personal digital certificates to login, sign content and do transactions. The idea of zero trust, popular mainly in the enterprise, will need to extend to everyday mundane transactions happening in social media and other platforms. The notion of a “verified user” will take on some urgency. Web sites will need to prioritize and later demand digital content from identifiable sources. Popular media and file formats, and some protocols will need to be extended to support digital identification. PKI (public key infrastructure) will need to extend and scale further, to become truly public. And there is of course the need to switch to quantum-proof algorithms.

Today AI can write good code. On the flip side, AI will have the ability to scan open source code (among others) for vulnerabilities and generate exploits automatically. This vastly expands the attack surface and can turn out to be a serious national security issue. Blocking zero day attacks coming out of these will be hard. An attacker can select any arbitrary target, but defenders need to protect anything and everything that could be attacked. This makes cyber defense many orders of magnitude harder. Effective DevSecOps with software bill-of-materials (BoM) analysis will need to become universal in all software development.

In addition, significant user training and advocacy will be needed to handle social engineering and attacks like phishing. Standards such as ISO 27001 and SOC 2 will need revisions. Software supply chain security frameworks like those from Mitre will need updates. Training courses and certification exams will all need to change considerably.

AI detection is another avenue. Indeed, one should ideally use AI to combat AI – a kind of next generation AI warfare. This can turn out to be an arms race, including between state actors. Actions or output from proprietary and custom trained AI models will be harder to detect. Those offering popular models could try and provide detectors for the same.

Established and popular AI models themselves need to be subject of research for us to better understand how they work. Today one can identify well over 130 emergent capabilities that these very large models seem to exhibit. No one truly understands how these come about. It appears that if we make artificial neural networks large enough (beyond some threshold number of parameters), and also highly scalable allowing massive training, these emergent behaviors will show themselves, regardless of specific architectures such as the Transformer models used today. But how or why would that be? Nor can we really tell when these large models are hallucinating. The ability to explain why a model behaved in a specific way is also hard. These unknowns make them easier prey, and hallucination attacks will be new forms of threat vectors.

For instance, you should be able to hypnotize an AI neural network. The term ‘hypnotize’ of course will have a different connotation here, but it will essentially be a kind of prompt engineering that jail breaks protections and gets the model to “believe” and behave according to the will of the prompter. Indeed, I tried an exercise with ChatGPT which seemed successful enough, but the avenue itself was later blocked, presumably by human instruction tuning. But one can imagine AI conversing with AI, trying to get over each other’s defenses. ?

Then there is AI-driven misinformation per se. Combating misinformation is much harder than fending off attacks and malware. These cut to the core of what people believe in and converse about, and there are fundamental freedoms at stake that should not be infringed. AI simply needs to trigger and catalyze misinformation in the right places. People then carry and spread these around all by themselves, because we are also (partly) self-perpetuating ideas and thought structures. This kind of AI-human thought transference is not easy to fight. And that makes me wonder. Perhaps a better way to combat misinformation and conspiracy theories is not to suppress them, but to introduce even more! Craft them pseudo-scientifically using AI in increasing numbers, saying even more preposterous things. The ensuing information driven bewilderment could hopefully result in a healthy skepticism, rather than cynicism!

Academic curricula need to include healthy information consumption and critical thinking methods a required series of courses from an early age. Today these are difficult to do in the US, mainly because of political polarization.

The areas of concern listed above are obviously far from exhaustive. Embodied AI including military capability, big brother surveillance, and nation spy craft (among others) will require whole essays in themselves.

But one thing is clear. We are in for a ride.