The Real Story Behind AI Security Incidents

Headlines scream about the latest "AI threat." But our analysis of 243 documented AI security incidents/issues between 2015 and 2024 reveals a surprising truth: most of these aren’t AI-specific attacks at all. They’re conventional security failures that just happen to affect companies and software working with AI.

The Numbers Tell a Different Story

Let’s cut through the hype with some hard numbers from our research:

• Nearly 89% of incidents are researcher demonstrations or ethical hacking exercises, not actual attacks by malicious actors.
• Only 17.7% are genuine AI-specific attacks.
• A substantial 82.3% of incidents are traditional vulnerabilities found in AI software that is incorrectly labeled as “AI vulnerabilities.”

These numbers show a clear pattern of traditional security issues being sensationalized as “AI security threats.”

The Reality Gap: Three Key Insights

White Hat is defined as researcher published vs Black Hat actual exploitation against real companies.

1. Traditional Vulnerabilities Dominate

The most common vulnerabilities in AI systems are traditional security issues:

• File System Access: Multiple cases of path traversal in AI platforms like MLflow, Anything-LLM, and ZenML
• Authentication Issues: Numerous cases of privilege escalation and authentication bypass
• API Vulnerabilities: Common IDOR and SSRF issues across multiple platforms

Example: The Anyscale Ray incident (March 2024) resulted in nearly $1 billion in computational resources being exposed - not through an AI-specific attack, but through a basic unauthenticated API endpoint.

2. AI-Specific Attacks Are Less Common But Growing

Although AI-specific attacks by the numbers are small, they are growing and as more AI applications are built this will surge:

• Prompt Injection: Cases like the Chevrolet chatbot ($76,000 car for $1) and Air Canada refund manipulation
• Model Theft: Incidents like the May 2024 attack affecting multiple cloud LLM providers
• Training Data Extraction: Google researchers demonstrating extraction of ChatGPT training data (note: GP2 and there was an older bug)

3. Infrastructure Vulnerabilities Are the Biggest Risk

The most damaging attacks target infrastructure rather than AI models:

• Resource Hijacking: Multiple cases of crypto mining using stolen compute resources
• Data Exposure: Breaches exposing customer data and model training information
• Cloud Misconfigurations: Leading to unauthorized access to AI training infrastructure

Emerging Trend in 2024

Proliferation of Development Framework Vulnerabilities

Everyone is targeting the data pipeline software. In any fast growing market these tools are coming out quick and loose with very little rigor around security. The majority of AI vulnerabilities are all standard security issues that exist in this software.

Not only are vulnerabilities found in popular ML frameworks, adversaries are creating malicious ML models and hoping others will download. See "model insights": https://protectai.com/insights

• Numerous vulnerabilities discovered in popular ML frameworks
• Common patterns of insecure file handling and authentication issues
• Trend toward targeting development and deployment tools

A Tale of Two Vulnerabilities

Traditional Security Issues (82.3% of Real Incidents)

These are standard cybersecurity problems that could affect any company:

Data Breaches

• Example: Clearview AI's 2020 breach exposed 3 billion images through basic security failures.
• Example: The 2023 Removal.ai incident exposed 14 million user records due to standard database vulnerabilities.

Resource Hijacking

• Example: Anyscale Ray's 2024 incident, where attackers exploited a conventional vulnerability (CVE-2023-48022) to hijack computing resources for crypto mining. Estimated damage: Nearly $1 billion in computational resources at risk.

Cloud Misconfigurations

• Example: Replicate AI container registry exposed allowing malicious container uploads (May 2024)
• Example: Multiple 2024 incidents where AI companies’ data was exposed through improperly secured S3 buckets.

API Security Issues

• Example: Vanna SQL database API vulnerability enabled arbitrary code execution (July 2024)
• Example: Google Cloud's Vertex AI allowed data exfiltration through malformed API requests (October 2024)

True AI-Specific Attacks (17.7% of Real Incidents)

Prompt Injection: Welcome to this era’s SQL Injection. Prompt Injection reigns as king and will continue to do so for a long time as currently there is no true solution for this issue.

• Example: The 2024 Chevrolet chatbot manipulation ($76,000 car offered for $1 - August 2024).
• Example: ChatGPT's 2024 persistent spyware vulnerability allowing continuous data exfiltration. (September 2024)

Model Manipulation

• Example: Researchers' extraction of training data from ChatGPT (several megabytes for about $200 - November 2023).

• Example: The 2024 Hugging Face incident where malicious models were uploaded containing backdoors. (March 2024)

Training Data Poisoning

• Example: ByteDance AI intern maliciously interfered with model training tasks (October 2024)
• Example: Protiviti client experienced attacker attempting to manipulate AI system input data (June 2024)
• Example: Nightshade tool allowing artists to poison training data for generative AI models (November 2023)

Adversarial

• Example: Tesla autopilot tricked into accelerating 50mph through modified speed limit sign (February 2020)
• Example: Google's image recognition system fooled to classify turtle as rifle through pixel modifications (November 2017)
• Example: Google Cloud Vision API tricked into misclassifying images through subtle corruptions (2019)

Why This Matters

1. Misallocation of Security Resources: Companies focus on exotic AI threats while leaving basic security holes unplugged.
2. False Sense of Novelty: Standard security failures get rebranded as novel AI threats.
3. Distracted Defense: Security teams chase theoretical AI vulnerabilities instead of addressing fundamental security practices.

Looking Ahead

The AI security landscape is evolving, but not quite in the way headlines suggest. While AI-specific attacks are a real and growing concern, the data shows that basic security failures remain the primary threat to AI companies and systems.

Key Takeaways:

1. Most "AI attacks" are conventional security failures affecting AI companies.
2. True AI-specific attacks exist but are less common than reported.
3. Basic security measures prevent most current real-world incidents.

As we continue to monitor this space, it's crucial to maintain this perspective: while AI brings new security challenges, old-school security problems haven’t gone away. In fact, they’re still causing most of the damage.

Methodology Note

This analysis is based on 243 documented security incidents involving AI companies or systems between 2015 and 2024. Incidents were classified based on the actual attack vector used, not the media representation or target company’s industry.

What’s your experience with AI security? Have you noticed this gap between headlines and reality? Share your thoughts in the comments below.