Countering The Rise of AI Criminals

By Amit Yossi Siva Levi , Principal Researcher at Forter (previously CTO and co-founder of bot detection company Immue)

Have you experimented with generative AI tools like ChatGPT yet? If not, you should. From beating writer’s block to composing ad copy, creating travel itineraries, and kickstarting code snippets, there’s something for everyone. Unfortunately, “everyone” includes criminals.?

Generative AI & Cybercrime

Cybercriminals are early adopters. If there’s a shiny new technology to try, you can bet that crooks will explore how to use it to commit crimes. The earlier, the better because that means they can make the most of the time before defenses are put in place to block their nefarious activities. If tech helps boost the scale or sophistication of criminal attacks, it’s extra attractive. So cybercriminals have been loving tools like ChatGPT.

There are many examples of problematic uses of generative AI already, from malicious code to phishing email composition to CAPTCHA solving. Yes, CAPTCHAs — the checks that are supposed to confirm that you’re human and not a bot. Ironic, isn’t it??

What was so interesting about the CAPTCHA story was that it involved ChatGPT using social engineering to trick a human into solving its challenge. In this case, the AI pretended to have a visual impairment that made it difficult to see the images, supposedly legitimating its need for assistance.?

That means we’re not just up against AI; we’re up against the fact that AI can use humans to overcome barriers against it.?

Dealing With the Challenge of the Present

The combination of generative AI and social engineering is a worrying one. Fraudsters, who are already used to incorporating social engineering into their schemes, may pick up on this possibility and attempt to use generative AI to scale up their attacks. The non-human weakness of the AI could be balanced out by tricking humans into covering for them where necessary.

Before you get excited about using AI to beat AI (on the principle of setting a thief to catch a thief), don’t get too excited; look at another example of a CAPTCHA failure to see what can go wrong.?

AI doesn’t yet know more than humans. This challenge is significant because it’s intelligence rather than the ability to process and use vast amounts of needed information. Solutions need to be creative, not formulaic. New, not amalgamations of old ideas.

? Related: Let's Talk About Generative AI & Fraud

Right now, my perspective, based on working with and against bots for many years now, is that short-term solutions need to be based in innovative technological angles, finding specific weaknesses in the specific generative AI services that currently exist, and making sure that those are exploited and the AIs blocked from being misused for fraud.?

I’m working on a solution of this type right now, which I hope to be able to share more about shortly. Initial results are looking great, so keep your fingers crossed.?

Prepare Now For What’s Ahead

I think the most effective solutions will be relatively narrow in the short term. That’s the fastest and most effective way to deal with the threat before it becomes a severe problem.?

We will need to be far more strategic in the long term. Generative AI will change fraudsters’ options and daily work, just like in other industries. And just like with other industries, that time of future change is not far away.?

Now is the time to dive deep into thinking about where things might go and start exploring the tools and changes in processes and attitudes we might need to protect against new attack types and methods.?

It’s already time to look beyond the attacks we’re used to. Deepfakes using the voices of CEOs on phone calls and WhatsApp calls have been seen in the wild. Deepfake images and videos have entered the mainstream. With the wealth of data on the internet at its disposal, generative AI has all the material it could need to help fraudsters scale attacks using these sorts of tricks before most people are even aware that that’s a risk.?

Since fraudsters have been branching out into more areas of the business than checkout, targeting post-checkout flows such as returns fraud and pre-checkout fields such as ad tech, it seems probable that generative AI will also play a role there.?

When it comes to crime, the possibilities for generative AI tools seem to be limitless. Fortunately, so is the human capacity for ingenuity in preventing crime. It’s time to get our thinking caps on.