LLMs and Sensitive Data

My colleague Victoria Gamerman, PhD recently shared an article from Tamer Chowdhury about architecture for using sensitive, access-controlled data in the context of ChatGPT and other Large Language Models (LLM).? This is vitally important area of development for these models, since many use cases in healthcare, business, and government are going to require a more nuanced approach than “everyone sees everything.”? The particular model that was being used couples an embedding database with the LLM.? Rather than cost-prohibitive retraining of the model, potentially relevant information is retrieved from the database and passed to the model as context at the time of query. This provides a point of access control, since you can simply not retrieve information to pass to the model that the user should not have access to.

This is a great solution for ensuring sensitive information doesn’t get out there. However, because of the way that an LLM deals with data, it is useful to think about degrees of knowledge when handling the problem. Take, for example, an employee who is looking for sensitive data from a study that they shouldn’t have access to.? If thoughtful engineering isn't put into managing how context is provided, it is possible that not only would the employee not get the sensitive information (good!), but they would not be aware of the existence of the data. Worst of all, they might get a totally incorrect answer, based on the best available data that the LLM is provided, without any awareness that this could be improved!

As a more concrete illustration, say a company had sensitive data on patients. I’m going to exaggerate and put in theoretical, entirely descriptive data. To a person with full access, the passage might be:

John Smith participated in the Megacompany trial for Awesometumumab, and received a single injection of 500 mg. His initial body weight was 300 kg at week 1, and was 280 kg at week 12.? Mild nausea was reported as an adverse event.

Someone with individual patient-level access might instead get the following context - this is the redaction of patient names model that was proposed in the Tamer Chowdhury article, and should be actually fairly straightforward to automate:

Patient 1001 participated in the Megacompany trial for Awesometumumab, and received a single injection of 500 mg. His initial body weight was 300 kg at week 1, and was 280 kg at week 12.? Mild nausea was reported as an adverse event. Data has been anonymized.

Other interesting cases come up where it would be appropriate to share single data points, devoid of context, for assembling aggregate statistics

A patient in the one trial had a body weight of 300 kg.? A separate body weight reading was 280 kg. For details about specific trials, contact Jane Coordinator.

Or, alternately:

Mild nausea was a side effect in a patient.? For details about specific trials, contact Jane Coordinator.

Providing these different contexts to ChatGPT and then asking to whom the antibody was administered and in what dose parses things with the appropriate level of information to the recipient.? And also inserts the relevant contact person, when the data is possibly available, but hidden based on access rights.

For this reason, I believe that with LLMs, it is more important than ever not to simply remove data that shouldn’t be available, but, in the ideal case, to redact or aggregate the data to the appropriate level of security, and provide that to the model.? Also, providing context for how to actual access the data can reduce some of these silos. Since LLMs are more conversational than traditional databases, it is important to manage the risk of people assuming they are getting accurate and complete information, without some of these guardrails in place.