Secure corporate LLMs using only 3 patterns

Three design patterns are all it takes to secure a Corporate LLM in 2024. CISOs will perceive this as a welcome relief.

Corporate LLM IT risks are much simpler than they look

There is just one IT risk which is actually novel, and that is prompt injection. All the rest is run-of-the-mill and should be readily handled by your existing application, IAM and infrastructure security controls.

• Protect integrity of the model, its versioning and its lifecycle (build + run)? Run of the mill;
• Protect the training environment and process? Run of the mill;
• Detect and prevent vulnerabilities in the model? Run of the mill;
• Prevent denial of service or overspending through throttling and monitoring? Run of the mill;
• Ensure tenant isolation, privacy, geolocation, resilience, perimeter security? Run of the mill;
• Govern risks, educate management? Run of the mill;
• Prevent bias or hallucinations? Data quality job, not security!
• Compliance? Ethics? Regulation? Take inputs from legal, don't wear their shoes!

Staying focused

Cybersecurity analyst who need to ingest the flurry of risks & recommendations dealing with AI risks can easily get lost and find it hard to quantify residual risks with accuracy. Even NIST itself published its own taxonomy of AI related risks in January |6].

For corporate LLMs, my one recommendation is simple:

DON'T focus on the many scary scenarios. Focus on prompt injection!

However, the bad news is that we can't deal with natural language malevolence upfront and efficiently. Don't play casino: the current so-called mitigations are just too early and too experimental to be trusted. Human language is way more subtle and complex than computer protocols used in machine-to-machine communication. Putative "LLM message parsers" have no way to filter natural language messages efficiently. We cannot just settle for poor filtering.

Always keep in mind: a single natural language instruction can be all it takes to cripple your organization.

We need checkpoints to "prevent always", not just guardrails that "detect sometimes".

So how to prevent prompt injection?

Reason like a security architect

Corporate LLMs feed on corporate data. There are basically two non-exclusive ways: fine-tuning and making calls (APIs / events / queries, including RAG).

For fine-tuning, corporate data have been pre-injected into LLM "memory".

When making calls, corporate data are read, modified or acted upon with the agency of a plug-in.

The crucial point to acknowledge is that

Fine-tuning and remote calls (RAG,...) are completely independent. They deserve completely independent mitigations.

Keeping this in mind, the security architect in our heads is going to introduce one pattern tailored for fine-tuning, and two patterns tailored for remote calls.

Pattern #1: In fine-tuned LLMs, prevent prompt injection with Knowledge Reflection

By default, fine-tuned LLMs gather too much corporate information and are too talkative.

They must be broken down (like in anti-trust laws) following a very special kind of least privilege pattern, especially coined for AI:

Fine-tuned LLMs must only contain the corporate data that their consumers are allowed to know.

So, if several consuming populations have different rights, LLM instances must be isolated accordingly. LLMs will be bounded to provide a reflection of their consumer's own knowledge.

With Knowledge Reflection, LLMs act as (very) sophisticated mirrors. No more jailbreaking!

In a nutshell: horizontal trust boundaries must be applied to fine-tuned LLMs, and Knowledge Reflection serves as a guidance for designing these boundaries.

When RAG or other remote calls are required, prevent prompt-injection with Conversational Integrity (pattern #2) and Linguistic Isolation (pattern #3)

In their inevitable interaction with their environment, LLMs are not autonomous, they need plug-ins (API gateways with workflows). Plug-ins, by definition, have the ability to do gating, hence to prevent.

(Image credits: Ashish Bhatia , click on image to get to his post)

For gating, plug-ins have a mighty weapon at their disposal, it's statefulness and we're gonna need it to prevent prompt injection, because, in the end of the day, the stateful nature of plug-ins is what fosters Conversational Integrity.

But what is Conversational Integrity? It's 3 things into one!

1. de-confusion: in any conversation, plug-ins make and continuously maintain their own, independent authoritative opinion on the identity of the consumer (human, app), the producer (LLM instance) and the unique session ID of that conversation.
2. lockdown: in any conversation, plug-ins must adjust their own entitlements to those of the consumer. Thus, even if the plug-in has relatively wide ranging entitlements, the authorization tokens used to push/pull corporate data must be lowered to match the consumer's. Producers are locked down from corporate data, they have no rights at all. This concept is similar to what Oliver Cronk calls "Walling the Bot" [7]
3. funneling: in conversation, the orchestration and execution of potential remote calls is forced through a non-bypasseable chokepoint: the conversational graph. The conversational graph is a state machine whose transitions are fired by the plug-in. The most robust and resilient graphs are cyclic graphs. They are used by state-of-the art risks prevention systems like Azure Policy [9]. Very recently, langChain updated its state machine to support cycles in its langGraph library [8]

We have a necessary condition, but it is not sufficient. We need one extra mitigation, tailored-made for LLMs: linguistic isolation.

Linguistic isolation

One can see an LLM plug-in as the modern version of Janus:

• one face, the front end, is highly fluent at human conversations.
• the other face, the back end, only speaks machine language.

Information passing to and fro the Janus brain of the plug-in must stick to 2 simple principles:

1. Natural language is only allowed by the plug-in if the consumer is human (observe that knowledge reflection is a pre-requisite here) ;
2. All operations (read/write/act) [3] must be re-formulated in computer language (REST verb/nouns, JSON, serialized C structures, all strongly typed, excluding strings, varchar, binary, opaque types...)

Put together, these 5 rules (the 3 ones from Conversational Integrity, and the remaining 2 from Linguistic Isolation) enforce an efficient design which prevents semantic confusion (leading to data corruption and arbitrary decision making) and noxious payload reentrance (leading to ??self poisoning?? and ??cross LLM contamination??) in the case of RAG and remote calls.

In a nutshell: vertical trust boundaries must be applied to remote calling LLMs. Plug-ins serve as custodians of vertical isolation, provided they enforce Linguistic Isolation and Conversational Integrity.

Conclusion & takeaways

An LLM subjected to Knowledge Reflection, Conversational Integrity and Linguistic Isolation prevent is a tutored AI.

A tutored AI prevents prompt injection, this is a highly desirable feature in a Corporate environment [5].

As of early 2024, AI tutoring is ONE possible state of the art reference architecture for securing enterprise LLMs enriched with corporate data with fine-tuning, RAG and APIs integration.

Recall that:

• data protection of fine-tuned instances can be handled by Knowledge Reflection, which drives horizontal trust boundaries
• data protection of RAG and remote calls is driven by plug-ins enforcing conversational integrity and linguistic isolation as part of their gating nature. This drives vertical trust boundaries to prevent propagation of semantic confusion/harm throughout the information system.

These boundaries & mitigations are VERY different from the usual IT security WAFs and filters.

The rest is not disruptive: risks must be governed, assessed, mitigated, refused or accepted. Strings must be sanitized. Models must be versioned. Logs must be monitored. API calls must be authenticated, scoped and "least-privileged". AI actions to back-end systems [4] must have human-in-the-loop (for now), etc [2].

A final recommendation: don't treat Corporate LLMs like LLMs for individuals, or LLMs on-device. The focus of today's article is businesses exclusively. If you are seeking guidance for securing digital assistants, which is the main anticipated need for individuals, I warmly recommend you read the thorough blog [1] that Daniel Miessler recently published in Unsupervised Learning, his newsletter.

State-of-the art, the references

[1] Daniel Miessler, https://danielmiessler.com/p/ai-predictable-path-7-components-2024

[2] Rob van der Veer et alter, OWASP AI Exchange, https://owasp.org/www-project-ai-security-and-privacy-guide/owaspaiexchange.html

[3] Christophe Parisel, https://www.dhirubhai.net/pulse/ai-security-2024-christophe-parisel-km2ie/

[4] Q&A with AI Safety Alliance Chair Caleb Sima , https://pureai.com/Articles/2023/12/27/Caleb-Sima-QA.aspx

[5] Peter Gostev , Prompt injection and LLM Security a Deep Dive

[6] NIST Publication, Adversarial Machine Learning

[7] Scott Logic , How we de-risked a genAI chat bot

[8] LangChain AI, the langGraph library

[9] Microsoft Learn, Understand Azure Policy Effects