Unmasking the Dark Art: How Hackers Exploit GenAI and Deepfake to Steal Money and Commit Fraud

In a recent article published by Datanews/Knack and the South China Morning Post, a chilling case of phishing was described, highlighting the alarming potential of GenAI and deepfake technology in the hands of hackers. An internationally operating company has fallen victim to a staggering €23.6 million fraud. The incident unfolded when an employee at the company's Hong Kong branch was deceived by a deepfake video, wherein an individual posing as the company's CFO instructed the employee to transfer funds.

In mid-January, an employee of the multinational corporation received an email from the CFO, urging them to carry out a covert transaction. Initially skeptical, the employee's doubts were dispelled when they were invited to a video conference. During the meeting, the employee encountered what appeared to be the CFO and other colleagues, both in appearance and voice, leaving the employee convinced of their authenticity.

This incident serves as a stark reminder of the growing threat posed by the combination of generative artificial intelligence (GenAI) and deepfake techniques. In this article, we delve into real-world examples of how hackers have exploited these technologies to steal money and commit fraud, while also providing recommendations to safeguard against such threats.

Sumsub has released its third annual Identity Fraud Report, providing a comprehensive analysis of identity fraud based on millions of verification checks and over 2 million fraud cases between 2022 and 2023. Key findings include a significant 10x increase in the number of deepfakes detected globally. Spain was the most targeted country for deepfakes, while ID cards were the most frequently exploited for identity fraud. The online media industry experienced the highest increase in identity fraud.

AI-driven fraud remains a prominent challenge, with crypto being the main target sector, followed by fintech. The report highlights the alarming tenfold increase in AI-generated deepfakes, paving the way for identity theft, scams, and misinformation campaigns. AI safety is set to become an integral part of companies' activities, as regulations focusing on AI are expected in 2024.

Complex fraud schemes, such as money muling and forced verification, have become more common and sophisticated. Account takeover incidents increased dramatically in 2023.

The predicted continued proliferation of account takeover and money muling schemes, emphasizing the need for robust countermeasures and regulatory responses. Non-document verification and alternative methods for identity validation are expected to gain importance.

Unleashing the Power of GenAI and Deepfake

GenAI, the rapidly advancing field of artificial intelligence, enables the creation of highly realistic and convincing content, including images, videos, and audio. When combined with deepfake techniques, which involve manipulating or synthesizing media to create deceptive content, hackers gain a powerful toolset to deceive and defraud unsuspecting victims.

1. Impersonation Attacks:

Next to the example cited above, we can see that the deepfake-technique is already around for some time, and now even getting more powerful. Even already in 2019, a CEO of a UK-based energy firm was targeted by hackers who used deepfake technology to mimic his voice. The hackers called the company's financial department, authorizing a fraudulent transfer of $243,000 to their own account. The convincing impersonation left the employees unaware of the scam until it was too late.

2. Phishing and Social Engineering:

In another case in 2020, a deepfake video of a bank representative was circulated, urging customers to update their account information urgently. Unsuspecting victims fell prey to this scam, providing their personal details, including login credentials, which were then used to access their accounts and carry out fraudulent transactions. Also fake ID cards are popping up everywhere.

3. Manipulating Financial Transactions:

In a recent incident, hackers used GenAI to create counterfeit invoices that appeared identical to those from a reputable supplier. The manipulated invoices led to payments being made to fraudulent accounts, resulting in substantial financial losses for the targeted organization.

Safeguarding Against GenAI and Deepfake Threats

As the threat landscape evolves, it is crucial to adopt proactive measures to mitigate the risks associated with GenAI and deepfake technologies. Here are some recommendations to safeguard against potential fraud:

1. Awareness and Education:

Stay informed about the latest developments in GenAI and deepfake technologies. Educate yourself and your organization about the risks and warning signs associated with these techniques to recognize potential threats.

2. Multi-Factor Authentication:

Implement robust multi-factor authentication mechanisms to add an extra layer of security. By requiring additional verification steps, such as biometrics or one-time passwords, the risk of unauthorized access or fraudulent transactions can be significantly reduced.

3. Vigilance in Communication:

Exercise caution when receiving requests for sensitive information or financial transactions, especially if they seem urgent or unusual. Verify the authenticity of the request through alternative means, such as contacting the individual or organization directly using verified contact information.

Real-world examples of fraud through GenAI and deepfake highlight the urgent need for heightened security measures. By understanding the potential threats and adopting proactive security measures, individuals and organizations can protect themselves against the growing menace of GenAI and deepfake-enabled fraud. Remember, staying informed and vigilant is the key to safeguarding your digital assets in this ever-evolving landscape.