The emerging threat of ChatGPT and AI

I've been talking for a while now about the emerging threats coming from the use of ChatGPT, AI & and attacks like Deepfakes. Yesterday I finally got a chance to play around with some of this technology directly.

What is the threat from AI, machine learning and technologies like ChatGPT?

• More sophisticated attacks using AI, such as reconnaissance and password attacks (Port scans, vulnerability analysis and basic attacks like password sprays) against infrastructure and cloud services.
• Evolution of malware using AI - providing the ability for malware to leverage AI to adapt and evolve to prevent detection and minimise organisational containment. (I should note at the same time, Endpoint Protection is keeping up using AI in its detection technologies for a lot of vendors now)
• Faster weaponisation of exploits and attacker tools. Here is an example of openAI being used to convert Javascript code to Python Code which I generated via the openai playground:

• Attacks mimicking people like deepfakes - for example your accounts payable person gets a teams call from the 'CEO' which looks and sounds pretty much identical to the real person asking for a payment to be made.
• Attacks utilising AI to affect company or individual reputation / image - for example, this post showing how midjourney and AI created a fake photo of Boris Johnson being arrested. A very believable image and a technique that can be used easily in disinformation campaigns in military and commercial uses and easily adapted for cyber attacks.

(P.S if you havent seen Midjourney and ai, it is amazing! check this post Midjourney V5 Playground: Create Photorealistic Images of Famous Celebrities with AI | by Michael King | Mar, 2023 | Medium)

Show me it in action

Here is a basic example of the technology being used in generating a phishing email. For this example I've used the openai (openai.com) platform. I've asked the platform to write me a very basic phishing email. AI took about 1 second to create me this basic email.

Ok lets get a bit fancier, Now using ChatGPT for some recon, I've asked it to get me the contact details for Nexon (so I can send through a phishing email). It pulled the wrong nexon, :) but still powerful:

I tried to get it to pull me employee names and emails but it has safeguards in place. But it does provide some useful information.

Same deal for phishing emails, but that doesnt mean that safeguards cant be removed or changed but on the current chat.openai platform it's blocked. If using a different AI platform it most definitely would be allowed.

It's also easy to create images, for example using the images API (API Reference - OpenAI API & Image generation - OpenAI API) you can send a a post request like such

curl https://api.openai.com/v1/images/generations \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $OPENAI_API_KEY" \
  -d '{
    "prompt": "A baby sea otter",
    "n": 2,
    "size": "1024x1024"
  }'        

and AI will generate an image on the fly.

It's important to remember that although AI has risks, AI can also have many benefits as well. For example, AI and machine learning is being used in Security Operation Centres (SOCs) for faster and more accurate threat hunting and to speed up detection and response to cyber-attacks. It's also being used with Endpoint Protection products as mentioned previously. AI is now also being used to allow faster and more efficient vulnerability identification and analysis.

In medical uses, AI is now being used to more accurately identify skin cancers such as melanoma and AI is now being used in conjunction with robots in surgery trials across the globe.

So what do these technologies mean to my business?

Firstly it means that you and your staff need to be aware of the uses and risks associated with AI, machine learning and technologies like ChatGPT against the organisation. This includes:

• Adding these technologies as a recognised organisational risk when it comes to cyber, and ensuring that safeguards are put in place to address these technologies and future advancements.
• Educating your staff on the risks from these technologies, for example educating employees to not believe everything they see, that images can be quickly doctored and used in disinformation campaigns.
• Ensuring standard security safeguards, staff awareness training and phishing, pentests, mfa and other technical controls etc.
• Putting in place processes to verify people and requests are real. Calling people before making payments, giving employees a secret code to provide to IT in support requests, these sorts of extra verification mechanisms.
• You should be fighting fire with fire, adopting technologies into your security roadmap that leverage AI and machine learning, such as Endpoint Protection Technologies using AI, Nextgen Firewalls, and ensuring your SOC provider is using these technologies in its detection and response. For example, microsoft have adopted AI as part of the Copilot security technology to assist defenders (Microsoft launches Security Copilot in private preview (cnbc.com))

AI, machine learning and ChatGPT really is a fascinating area and you can spend hours getting lost in it and playing around with the technologies, so much fun to be had! It's highly worth looking into and getting your head around it, as it is most definitely the future.

We have also started introducing these technologies into our pentest tooling to provide our clients an engagement far beyond that of traditional toolsets.

#ai #cybersecurity #cyberattacks #chatgpt #openai #deepfakes #danweis #hackproofyourself