Navigating the Irreversibility of Sensitive Data in LLMs: A Particular Challenge in the Legal Domain

In the legal domain it is a prerequisite for lawyers to make sure any sensitive data they have received and are entrusted with is kept confidential. In the last edition of the Legal Informatics Newsletter we looked into technical ways to manage such data safely. In this edition the risks related to sensitive data being introduced to a LLM application are the focus.

Introduction

Large Language Models (LLMs) have become very trendy tools across various sectors, including the legal domain, due to their ability to process and generate human-like text. However, a critical issue arises: once LLMs learn sensitive data, it becomes practically impossible to delete or unlearn this information.

In the legal field, the sensitivity of data is paramount. Legal documents often contain privileged communications, client information, and confidential details or secrets. If such data is inadvertently included in LLM training sets, it can lead to severe consequences, such as data breaches and unintended disclosures. The inability to easily unlearn or delete this sensitive data from LLMs exacerbates these risks, creating significant challenges for legal compliance and data protection.

It needs to be noted that also anonymization alone might not suffice, as legal documents often contain other sensitive information, such as context-specific details or metadata, that cannot be easily masked. Additionally, sophisticated re-identification techniques can sometimes de-anonymize data, further complicating the protection of sensitive information.

So there is a need for awareness about the risks related to making sensitive data available to LLMs. In this article we want to explore this issue in more detail.

The Challenge of Deleting or Unlearning Data in LLMs

Deleting or unlearning data in LLMs presents a formidable challenge due to the inherent nature of these models. LLMs, such as GPT-4, are trained on vast datasets, absorbing patterns, structures, and information to generate human-like text. Once trained, the model integrates this data deeply into its neural network, making it virtually impossible to isolate and remove specific pieces of information without retraining the entire model.

Another layer of complexity arises from the fact that retraining an LLM to exclude specific data is resource-intensive and impractical for most organizations. The training process of advanced LLMs involves substantial computational power, time, and financial resources. In the context of public operators like OpenAI there is also no obvious legal leverage a user would have to force a retraining of the model.

The crux of the problem lies in how LLMs learn. These models do not store information in distinct, easily separable units. Instead, they encode data across numerous parameters and layers in a highly interconnected manner. This distributed representation means that the knowledge is not localized but spread throughout the network of nodes and edges containing relevant pieces of information. Attempting to remove specific information without affecting the overall model's functionality and coherence is not a possibility.

Moreover, the sophisticated algorithms that power LLMs are designed to generalize from the training data they receive. This means that sensitive information might not only be stored directly but also inferred indirectly through patterns and associations. For instance, even if explicit names or details are not retrievable, the model might generate outputs that reflect the underlying sensitive information it was exposed to during training.

Implications

It is obvious in the context of data from the legal domain, that the inability to remove data from a LLM poses a significant risk. When such data is inadvertently included in the training set of an LLM, it becomes part of the model's foundational knowledge. Unlike traditional databases where specific records can be deleted, LLMs do not have discrete, removable units of data.

Therefore, the inability to delete or unlearn data from LLMs underscores the critical need for stringent data handling practices before the training phase. A proactive approach is essential to mitigate the risks and ensure compliance with data protection regulations in the legal sector.

In case sensitive data has been exposed to a LLM there are some mitigation strategies that may help to a certain extent.

Mitigation Strategies

When retraining an LLM is not a viable option due to resource constraints or other limitations, alternative strategies can mitigate the risks associated with sensitive data. These methods focus on controlling the model's outputs and attempting to obscure or minimize the influence of sensitive information. Effective mitigation involves a combination of techniques that address the problem from multiple angles.

Output filtering is a crucial strategy that involves implementing automated filters to review and sanitize the model's outputs before they are delivered to end users. These filters can detect and remove sensitive information or flag potentially problematic content. Additionally, incorporating human review for high-risk outputs can ensure that subtle or context-specific sensitive information is manually assessed and redacted. While effective, this approach can be time-consuming and costly, and it may not scale well for large volumes of output.

Negative prompting is another useful technique that involves designing prompts to instruct the model to avoid generating sensitive information. By using negative constraints, such as "Do not mention client names" or "Avoid discussing confidential case details," the model can to some extent be guided to avoid certain topics or phrases.

Reinforcement Learning with Human Feedback can further enhance this approach by training the model to prioritize safe outputs through rewards and penalties. However, this method requires significant initial setup and ongoing management.

Data deletion techniques, although challenging, can also be employed to mitigate the impact of sensitive information. Selective deletion attempts involve manually or programmatically identifying and removing sensitive data entries from the model’s training dataset. Embedding space manipulation might be another method that alters the model’s parameters to down-weight the influence of sensitive data. Both techniques can reduce the prominence of sensitive data in the model’s outputs without full retraining but fall short of completely erasing the information.

The effectiveness of these non-retraining strategies varies. Output filtering is effective as a post-hoc measure but depends on the thoroughness of the filters and review processes. Negative prompting helps steer the model away from sensitive topics but requires constant refinement and vigilance. Data deletion techniques can reduce the impact of sensitive data but often cannot guarantee complete removal, posing ongoing risks.

Each mitigation strategy has its risks and limitations. Incomplete mitigation remains a significant challenge, as none of these methods ensure the complete removal of sensitive data. They also require substantial resources and ongoing management, hence are costly.

Overall, mitigating the effects of sensitive data in LLMs requires a comprehensive and multi-faceted approach. While retraining is the most effective method, alternative strategies such as output filtering, negative prompting, and data deletion techniques can provide significant protection. Legal professionals should prioritize proactive data management and continuously evaluate the effectiveness of their mitigation strategies to ensure compliance and protect sensitive information.

Even despite robust mitigation strategies, the problem of retrieving sensitive information from LLMs through re-engineering or aggressive prompting remains a significant concern. Attackers can use sophisticated techniques to manipulate the model into revealing confidential data. Re-engineering involves analysing the model’s outputs to infer underlying data patterns, while aggressive prompting uses carefully crafted queries to elicit specific information. These methods can bypass standard mitigation measures like output filtering and negative prompting and potentially lead to the leakage of sensitive information.

Moreover, these techniques exploit the very nature of LLMs, which are designed to generate human-like text based on patterns learned from vast datasets. Even with measures like differential privacy and data masking, traces of sensitive information can persist in the model’s responses. The interconnected and distributed nature of neural networks means that once sensitive data is embedded within the model, it can be extremely difficult to fully eliminate its influence. As a result, the threat of data leakage through re-engineering and aggressive prompting underscores the need for continuous advancements in mitigation strategies and the importance of minimizing the exposure of sensitive data from the outset.

Obviously significant risks remain in case sensitive data have become available to LLMs. So, the best mitigation strategy is to prepare data before making it available for training.

Preparing Data with Sensitive Content Before Using LLMs

Implementing robust data preparation strategies can significantly mitigate the risk of exposing sensitive information when working with LLMs. This involves techniques like data segregation, classification, anonymization, masking, minimization, and encryption.

First, data segregation and classification are essential steps in protecting sensitive information. By separating sensitive data from non-sensitive data and implementing a robust data classification system, organizations can create clear boundaries and ensure that only appropriate data is used for training. Automated tools and manual reviews help accurately label data according to its sensitivity, facilitating targeted data management practices and reducing the risk of cross-contamination.

Data anonymization and masking are additional techniques that help protect sensitive information. Anonymization involves removing or altering personally identifiable information to prevent re-identification, while data masking replaces sensitive data elements with non-sensitive placeholders. Both techniques are effective in obscuring sensitive information and preventing its exposure. However, they require continuous evaluation and updates to address advanced re-identification techniques and ensure comprehensive protection.

Data minimization and encryption further enhance data security. Data minimization involves collecting and using only the minimum amount of data necessary for training LLMs, reducing the volume of sensitive data at risk. Encryption provides a robust layer of security by protecting sensitive data both in transit and at rest using strong encryption standards and key management practices. These strategies help ensure that sensitive information remains secure, even if accessed without authorization.

Implementing these data preparation strategies requires a multi-faceted approach that combines multiple techniques to achieve the highest level of data protection. Legal professionals should establish clear data handling policies, conduct regular training for staff, and utilize advanced tools for data management. Continuous monitoring and auditing of data practices are essential to identify and address potential vulnerabilities promptly. By meticulously preparing data with sensitive content, organizations can significantly reduce the risks associated with using LLMs and ensure compliance with legal and regulatory requirements.

Using hybrid clouds, as discussed in a previous edition of the Legal Informatics Newsletter, can enhance data security by allowing organizations to keep sensitive information in a private cloud while leveraging public LLMs for processing less sensitive data. This approach ensures that sensitive data remains protected within the private cloud environment, reducing the risk of exposure while benefiting from the computational power of public LLMs.

Conclusion and Outlook

Mitigating the risks associated with sensitive data in LLMs is a complex and ongoing challenge. While current strategies such as output filtering, negative prompting, and data segregation offer some protection, they are far from adequate in the legal domain. The persistent threat of data leakage through sophisticated techniques like re-engineering and aggressive prompting highlights the limitations of these measures.

Legal professionals must therefore adopt a proactive approach, protecting sensitive data by robust data preparation with vigilant monitoring of model outputs and continuous improvement of mitigation strategies. The alternative would not to use LLMs at all. Sensitive data in public model is a no-go for any serious legal professional.

Looking to the future, advancements in AI and machine learning hold promise for more effective mitigation techniques. Intense research is ongoing into developing models with built-in privacy safeguards and improved methods for data anonymization and encryption. Innovations such as federated learning, where models are trained across multiple decentralized devices or servers without sharing raw data, may also enhance data security. Until these advancements become mainstream, it is crucial for organizations to remain cautious and prioritize the meticulous handling and preparation of sensitive data. By staying informed and proactive, legal professionals can better protect sensitive information and ensure compliance with evolving data protection standards.