ChatGPT's Likely Impacts on Malware and Fraud

I am seeing a lot of headlines around ChatGPT lately; some seem accurate, and some seem like click-bait and fear mongering. I've been working with Artificial Intelligence (AI) for a long time and writing this to cut through the noise, sharing my opinion on what ChatGPT likely will and won't do.

Executive Summary

• OpenAI, ChatGPT's creator, is trying to do the right things and limit ChatGPT being used for "bad" things, but people are finding ways around those limitations
• ChatGPT makes it easier for people to create malicious content, such as Malware and Fraud (Phishing, BEC, et cetera). I expect the volumes of those attacks to increase, as tools like ChatGPT reduces the barrier to entry for use
• That said, ChatGPT uses a Tranformer based on Neural Networks, meaning the content ChatGPT generates is “known;" it's all contained in its training set. It won't invent radically new malware or Fraud. As an industry, we should, over time, be able to detect it and defenses should learn to capture it
• While the threat from ChatGPT can be ring-fenced, other forms of AI may create unknown/new malware code and Fraud over time; so we need to stay vigilant

High level, how ChatGPT works

ChatGPT is based on a "transformer" called GPT-3. Keeping this very high level - The easiest use case to explain transformers is language translation. You can give the transformer plenty of examples of sentences in one language to another. The transformer learns concepts such as which words connect together, the order words go in various languages, gender and pluralizing adjustments, and other language concepts.

ChatGPT was built using this transformer, but instead of using it for just language translation, it was fed 40TB of data to learn, to make connections and inferences on!

Again, without going into details, some key things to know about transformers:

• Transformers use Neural Networks at its core. Neural Networks use (Self-)Supervised Learning: When you train a Neural Network, you give it all the cases its learns on, its "training set." If something isn't in the training set, the Neural Network won't "invent" it. Neural Networks only rely on their training. (If asked to do something outside of its training set, a Neural Network will still give output - but the further you deviate from what it knows, the lower accuracy the output.)
• Transformers build sentences by using probabilities - what's the likely next word given the previous words in the output. Transformers are not perfectly deterministic, a fancy word that means a specific input gives the same output every time. (Think about math: 1+1 always = 2. Math is deterministic.) The underlying parts of transformers, Neural Networks, are deterministic though, so there really is a good amount of consistency between identical queries to transformers

Ok, I kinda get how they work, what does this mean for using ChatGPT?

This means that no matter how someone queries ChatGPT to generate an answer, ultimately everything ChatGPT outputs is found somewhere in its training set. Ask it to output code, or a legal document, or just look up some facts... you might vary the question a bit here and there to get different answers, but ultimately those answers come from that "training set." Now to be fair, ChatGPT was trained on a mind-blowing amount of data, but still, it's a set.

There are plenty of AI algorithms that can "invent" or explore new directions, such as Genetic Algorithms and DeepMind/AlphaZero. But ChatGPT isn't either of those, it's a Neural Network based transformer. It won't "invent" beyond its training set; it will try to predict answers but fail. If you want to add data to the training set, it takes time to adjust the Neural Networks to consider the incremental changes to the training set - it's not instantaneous.

Does ChatGPT replace a web search tool?

In some regards, it might replace some aspects of using web search tools. Go back to the point of a web search tool - it's to help you find websites, news, other content, based on your query. Back when Altavista (ok, I'm old), Yahoo!, Google, et cetera, all started, there weren't great ways to search the web. They opened the web, they made it more usable for the masses. But their point wasn't to get you an answer per se, rather help you find websites that would have the answer.

In some sense, ChatGPT is the next wave of accessibility and consumption of the internet. Rather than having to navigate to different websites and interpret or find the info you want, ChatGPT can cut through that noise and "just answer the question." But there's a problem... remember, ChatGPT is only as good as its training. It doesn't know current events, or events that happened after its training date. And if you want to learn more about a topic, or don't know what you should be asking, well it can't help you there either. So, while really really good, there are big limitations that keep it from replacing web searches outright.

Ok... but the headline is about security... what's the bad news here?

The bad news - ChatGPT will lower the barriers to entry for a bunch of "bad" concepts, such as Malware and Fraud. (An aside, OpenAI, ChatGPT's creator, is trying to do the right things and limit ChatGPT being used for "bad" things, but people are finding ways around those limitations.)

• Malware: Rather than someone needing to read big programming manuals to write code, ChatGPT will be able to interpret those manuals, use cases, etc., and write code for the user. It reduces the barriers for entry. This means that the volume of malware attacks may rise - more people will have access to malicious code.
• Fraud: ChatGPT can help (especially non-native speakers to a language) build e-mails, texts, ... that look more realistic than what people create today. There will be less typos, better grammar, etc. It will be easier to fool the victims.

Is it all doom and gloom? There an upside?

It's not all doom and gloom. Remember, all of the output from ChatGPT is in its (vast) training set. We can/should over time be able to detect Fraud emanating from ChatGPT. There's only so many ways ChatGPT can generate code and that too will be detected. We're already seeing examples of just this at work.

And there are so many positive uses of ChatGPT! Good programmers can find answers to their questions easier than searching copious Q&A boards. Security researchers can easily research terms, technical concepts, etc. Overall, ChatGPT could reduce time finding answers, and allow us to be more efficient in solving the problems!

So, that's it? Nothing to see here?

Well... not exactly. There definitely could be smaller uses of GPT-3/Transformers for more nefarious uses outside of ChatGPT, so we need to stay current on that. And while Neural Network based transformers really don't go outside the scope of their training set, other forms of AI might be able to do that. It's not hard to envision a DeepMind/AlphaZero system learning to invent new malware code.

Overall, ChatGPT is just another step in the game of cat and mouse between attackers and defenders. It's a fascinating new step, and while volumes will increase, hopefully it won't take long for our defenses to catch up. And nothing here precludes the most important step: doing the basic steps well to defend, educate, protect, etc. Sure, ChatGPT is fancy, but nothing beats the basics.