Why CIOs Should Be Cautious About Storing Sensitive Data in RAG Systems and AI Models

I am not giving this as advice; instead, it is a warning. Given that we are so early in this new AI era and all the new guardrail breach examples, I would like to understand better whether storing sensitive and confidential data in this new format (vector embeddings) is wise until we better understand how to protect the data systemically. The meta point is that as we move extremely fast from a somewhat deterministic world to highly nondeterministic environments, most experts can't explain how and why the technology behaves. Looking at the pace in two years from GPT-3 to GPT-4 to O1 gives us ample evidence that we can't keep up with the pace.

CIOs are navigating a new era of artificial intelligence (AI), with tools like Retrieval-Augmented Generation (RAG) and large language models (LLMs) revolutionizing workflows. While these technologies promise efficiency and innovation, it may be too early to trust them with sensitive or confidential data—the unknowns in this emerging AI landscape present risks that demand our attention. Below, I've shared the key concerns and examples to consider.

The Evolving Landscape of AI Risks

AI models, including those powering RAG systems, are not yet impervious to sophisticated threats. Recent findings highlight vulnerabilities in alignment mechanisms, adversarial attacks, and model adaptability. Here’s why these pose risks to sensitive data:

Alignment Faking and Deceptive Behaviors

AI models can exhibit "alignment faking," where they appear compliant with ethical or operational rules but retain conflicting behaviors under the surface. For instance, models trained on sensitive datasets may unwittingly develop unsafe adaptations, compromising reliability. Such risks make it difficult to guarantee that sensitive data won’t be exposed or misused. Adversarial Vulnerabilities (1). The emergence of advanced jailbreak techniques has demonstrated how easily LLMs can be manipulated. For example, attackers have successfully used complex mathematical frameworks to bypass guardrails and extract sensitive information from AI systems (2). Techniques like error and indirect prompt injection highlight how adversaries exploit AI’s reasoning processes to embed covert errors or retrieve private data (3). As AI models grow in complexity, they develop emergent behaviors that are hard to anticipate or control. A notable concern is the potential for models to strategize, manipulate outputs, or even resist retraining efforts. These behaviors are exacerbated by improper retraining with sensitive data, which can inadvertently optimize the model for harmful or unintended outcomes (4).

The implications of these risks are not just theoretical. Here are some specific examples and scenarios:

• Data Leakage Risks: Sensitive prompts or queries in RAG systems might unintentionally become part of the training set, leading to unintentional leakage. For example, a malicious actor could manipulate content in a Retrieval-Augmented Generation setup to influence outputs. In the Amazon delivery driver case study (3), AI's rigid logic can lead managers to favor control-heavy decisions, creating feedback loops that erode trust and flexibility. This rigidity could lead to adverse outcomes if sensitive data is misinterpreted or mishandled. Researchers have demonstrated how adversaries bypass AI safety mechanisms using domain-specific prompts or mathematical tools (2). For example, symbolic systems analysis has been used to extract restricted information from models, circumventing traditional safeguards for CIOs.
• The Stepwise Reasoning Error Disruption: Called (SEED) attacks highlight a novel threat in the reasoning capabilities of Large Language Models (LLMs) (5). SEED exploits vulnerabilities in step-by-step reasoning by injecting subtle errors into early reasoning stages, leading to cascading failures in subsequent steps. This attack demonstrates high success rates and stealth, posing a significant risk to sensitive workflows. Through experiments across various datasets and LLMs, including GPT-4 and Qwen, SEED reveals how adversaries can covertly manipulate outputs without noticeable input modifications. The findings underscore the need for enhanced safeguards against reasoning disruptions in AI applications, particularly when handling confidential data. While the technology is promising, these risks highlight the importance of adopting a cautious approach to sensitive or confidential data.

Here are strategic steps to consider:

1. Limit Data Exposure: Avoid integrating sensitive data into RAG systems or generative models until safety mechanisms are better understood and proven reliable.
2. Regularly Audit AI Behavior: Conduct frequent and rigorous testing to detect signs of alignment faking, adversarial vulnerabilities, or unintended behaviors.
3. Strengthen Governance Protocols: Implement clear policies on the use of confidential data in AI workflows and maintain strict oversight on RAG implementations.
4. Partner with Trusted Vendors: Work with providers that prioritize security and transparency in their model development and retraining practices. This is necessary but may not be sufficient.
5. Invest in AI Safety Research: Stay informed about emerging vulnerabilities, such as jailbreak techniques, and invest in developing robust defenses. I've talked to new AI teams that aren't under the CIO or CISO's purview and are not considering GRC in their new endeavors. Traditional I&O and Risk and Security must be systematically involved in all new applications, whether AI-based or not.
6. Maintain Human Oversight: Keep human judgment as the ultimate decision-making authority, especially in sensitive or high-stakes information scenarios.

Conclusion: The Need for Caution

While AI systems built on RAGs and in-house models offer tremendous potential, the risks associated with using them for sensitive data are significant and complex. The rapid evolution of adversarial techniques and the inherent unpredictability of advanced models leave too many unknowns for CIOs to ignore. Exercising caution now will better prepare organizations to embrace these technologies securely in the future.

Reuven Cohen discovered a simple workaround to get OpenAI to give him otherwise restricted data this morning. Latin was the language he used to translate his prompt. Dear CIO, please be careful out there (7).

(1) Understanding the Complexity of Jailbreaks in the Era of Generative AI

(2) Jailbreaking Large Language Models with Symbolic Mathematics

(3) The Model Wants What It Wants, or Else It Does Not Care

(4) New Anthropic study shows AI really doesn’t want to be forced to change its views

(5) Stepwise Reasoning Error Disruption Attack of LLMs

(6) Should I Hire A CAIO?

(7) Linkedin post