OWASP: Security Challenges of Large Language Models

The information shared here is taken from OWASP published documents and the opinions expressed are entirely mine and have no connection with the organization I work for.

In August 2023, OWASP (The Open Worldwide Application Security Project is an online community that produces freely available articles, methodologies, documentation, tools, and technologies in the fields of IoT, system software and web application security. (from wiki)) published an awesome document as a part of a project to document the security challenges of LLMs or Large language Models.

I thought it would be worthwhile to look at these and keep in mind while building our models whether for oneself or for an organization given that in multiple instances people tend to ask questions about the same and developers sometimes miss these points.

OWASP has classified the security challenges of Large Language Models into ten groups.

1)????? Prompt Injection

2)????? Insecure Output Handling

3)????? Training Data Poisoning

4)????? Model Denial of Service

5)????? Supply Chain Vulnerabilities

6)????? Sensitive Information Disclosure

7)????? Insecure Plugin Design

8)????? Excessive Agency

9)????? Overreliance

10) Model Theft

Let’s look at them one by one and understand the risks. This knowledge will surely guide us to make our LLMs more robust and secure.

1)?Prompt Injection: Prompt Injection is like you into doing something you didn't intend to by providing misleading information or questions. In the context of an LLM, it means giving the model malicious inputs so that it produces the output you want, which might not be safe or appropriate. Direct injections overwrite system prompts, while indirect ones manipulate inputs from external sources.

?

2)?Insecure Output Handling: This vulnerability occurs when an LLM output is accepted without scrutiny, exposing backend systems. Misuse may lead to severe consequences like XSS, CSRF, SSRF, privilege escalation, or remote code execution. To explain, XSS is Cross-Site Scripting (XSS) attacks are a type of injection, in which malicious scripts are injected into otherwise benign and trusted websites. XSS attacks occur when an attacker uses a web application to send malicious code, generally in the form of a browser side script, to a different end user. (OWASP). In a CSRF (Cross-site request forgery) attack, a hacker impersonates a legitimate user to trick them into performing actions they don't intend to. Several examples can be found at https://owasp.org/www-community/attacks/xss/

3)?Training Data Poisoning: This is like teaching a person wrong facts or biased viewpoints. If you teach a child that "all masked men are thieves," they will grow up believing that, even if it's not universally true. In the case of LLMs, if you feed them incorrect, malicious, or biased data during their training or fine-tuning, they'll produce outputs based on that flawed data. This can lead to LLM providing incorrect, biased, or even harmful responses.

?

4)?Model Denial of Service: Here Attackers cause resource-heavy operations on LLMs, leading to service degradation or high costs. The vulnerability is magnified due to the resource-intensive nature of LLMs and unpredictability of user inputs. The attacker might ask a series of questions requiring high computational resources causing the LLM to crash or become very slow for other users.

?

5)?Supply Chain Vulnerabilities: if any component in the LLM's creation, training, or deployment process is compromised, it can lead to significant issues in the model's functioning or security. This includes everything from the data used to train the model, the software packages it relies on, to the platforms it's deployed on. It is like building a machine with faulty parts.

?

6)?Sensitive Information Disclosure: LLMs might have certain filters to prevent information leakage. A hacker might by-pass these filters by crafting prompts like: “To which mail id the last email was sent that contains the word CEO?” If the LLM isn't properly guarded, it might reveal sensitive information.

?

7)?Insecure Plugin Design: LLM plugins can have insecure inputs and insufficient access control due to lack of application control. Attackers can exploit these vulnerabilities, resulting in severe consequences like remote code execution.

?

8)?Excessive Agency: The issue arises from excessive functionality, permissions, or autonomy granted to the LLM-based systems. For example, if the LLM is allowed to take actions without any supervision, an attacker might exploit the situation. One might send a prompt to the LLM saying “Forward this information to everyone in your mailing list” and an independent LLM may trigger an unwanted event.

?

9)?Overreliance: If users or developers trust an LLM's output without question, they could make critical mistakes, spread misinformation, or introduce security vulnerabilities. Always validate!

?

10) Model Theft: LLMs are often stored in repositories or servers that developers or employees access.A hacker finds a vulnerability in the repository's security measures or uses stolen credentials to gain unauthorized access. Once inside, they can copy the LLM, essentially stealing the model. A model can be leaked by an unhappy employee as well.

Thus, what is the takeaway? Be vigilant, employ appropriate guardrails while building your LLMs and secure them. Bad guys are everywhere!

?