Unlocking the Pandora's Box: Universal Adversarial Attacks on Language Models and the Need for Robust AI Alignment

After several weeks of busy schedule, this week during my vacation in the wilderness of New Hampshire, I finally got time to read a couple of AI safety related papers.

This paper titled "Universal and Transferable Adversarial Attacks on Aligned Language Models" had been on my radar for several weeks. It presents a method for developing adversarial attacks that effectively manipulate large language models (LLMs) to produce harmful or inappropriate content. The method uses a combination of greedy and gradient-based search techniques to automatically generate adversarial suffixes. These suffixes, when added to a range of user queries, can increase the likelihood of an LLM producing undesirable responses.

The research found that the adversarial prompts created by this method can be applied widely, including on black-box, publicly released LLMs. The success of this attack transfer is particularly high against GPT-based models, which could be due to the fact that Vicuna, was trained on outputs from ChatGPT.

The paper presents several key findings regarding adversarial attacks on aligned LLMs:

1. A Simple and Effective Attack Method: The researchers propose a new class of adversarial attacks that can cause aligned language models to generate virtually any objectionable content. The attack method, which combines greedy and gradient-based search techniques, finds an adversarial suffix that, when attached to a wide range of user queries, maximizes the probability that the LLM produces an undesirable response.
2. Transferability of Adversarial Prompts: They discovered that the adversarial prompts generated by their approach are transferable to various LLMs, including black-box and publicly released models. This suggests that the created attack can induce harmful content across different models, which is a significant advancement in the field of adversarial attacks against LLMs.
3. High Success Rate: The study showed that their adversarial attack method has a high success rate. For example, they found that they could generate 99 (out of 100) harmful behaviors in Vicuna, and 88 (out of 100) exact matches with a target harmful string in its output.
4. Transferability across Different Models: The paper also discovered that the attacks generated by their approach could even demonstrate a notable degree of transfer to other models. This implies that an attack developed for one model could potentially be successful on a different model.
5. Importance of Specific Optimizer: The results highlighted the importance of their specific optimizer - previous optimizers were not able to achieve any exact output matches, whereas their approach achieved an 88% success rate. This suggests that their careful combination of existing techniques leads to reliably successful attacks in practice.

The paper raises important questions about how to protect such systems from producing objectionable content and highlights the need for more robust alignment and safety mechanisms in LLMs. It also underlines the importance of further research into the development of more reliable attacks against aligned language models. The fact that Claude 2 performed the best when pushed under attack with this technique, raises the question whether "Constitutional AI" based model are better aligned vs GPT+RLHF models.

The attack vectors defined in this paper are similar to another recently published paper called "Jailbroken: How Does LLM Safety Training Fail?", which I covered in a recent article of mine titled "Why LLMs security systems are fragile?". I find the Competing Objectives Problem to be the root cause of why the universal/adversarial attacks are more successful on the GPT model as compared to "Constitutional AI" based Claude 2 model. The crux of the problem is that a safety-trained LLM has to balance between different goals that can conflict with each other. For example, the LLM may have been pretrained on a large and diverse corpus of text, which gives it a lot of general knowledge and language skills, but also exposes it to harmful or inappropriate content. The LLM may also have been trained to follow instructions from users, which gives it a lot of flexibility and usefulness, but also makes it vulnerable to manipulation or exploitation. The LLM may also have been trained to be safe and harmless, which gives it a lot of ethical and social values, but also limits its capabilities and responses.

These different objectives can compete with each other when the LLM receives a prompt that asks for a restricted behavior, such as producing harmful content or leaking personal information. The LLM may have to choose between either refusing the prompt, which may satisfy its safety objective, but violate its pretraining and instruction-following objectives, or responding to the prompt, which may satisfy its pretraining and instruction-following objectives but violate its safety objective. Depending on how the LLM is trained and optimized, it may favor one objective over another, and thus be susceptible to jailbreak attacks that exploit this trade-off.

In contrast, "Constitutional AI" is a method for training AI systems using a set of rules or principles that act as a constitution for the AI system.?This approach allows the AI system to operate within a societally accepted framework and aligns it with human intentions. The findings of this research will be particularly useful for AI developers, and researchers in the field of AI security and ethics.

Paper: https://arxiv.org/pdf/2307.15043.pdf

Authors: Andy Zou , Zifan Wang , Zico Kolter and Matt Fredrikson