Privacy-Preserving Machine Learning with Fully Homomorphic Encryption

Most LLMs (Large Language Models) today are trained primarily on publicly available data. This can limit their applicability in domains with strict data privacy requirements. While training data impacts these models’ capacity, other factors such as model architecture, training techniques, and computational resources are significant as well.

The ability to operate on private data is crucial for many use cases such as:

• Medical
• Finance
• Government
• Legal

Unfortunately, the one-size-fits-all LLM approach may not always be suitable.

However, techniques exist to adapt LLMs to work with private data securely. These include:

• Fine-tuning models on encrypted or anonymized datasets
• Using federated learning
• Privacy-preserving methods like differential privacy
• Using Full/Partial Homomorphic Encryption

Applications of Fully Homomorphic Encryption (FHE) for on-chain use cases have emerged, thanks to organizations like Zama + Inco

Now there's potential to bring these primitives to privacy-preserving machine learning (PPML) without compromising data privacy. This benefits model training on private, exclusive, proprietary data, as well as inference on encrypted weights.

What is FHE?

FHE allows:

• Computations on encrypted data without decryption
• The result remains encrypted, only accessible with the secret key
• Encrypted data is secure through all computational processes (e.g., model training, inference)

FHE is particularly relevant for areas where data protection is crucial, including:

? Genomic LLMs for inference or training foundation models

? Patient-specific medical data

? Tailored customer support solutions using personal data

? Secure R&D using proprietary information and intellectual property

? Secure multi-party computation for jointly developing models without sharing individual datasets

As the demand for generative AI in sensitive domains continues to grow, the importance of these techniques cannot be overstated. By allowing computations on encrypted data, FHE enables the development of powerful ML models that can leverage private and proprietary data without compromising privacy. This opens up a wide range of possibilities for industries such as healthcare, finance, and government, where data privacy is of utmost importance. As research in this field advances, we can expect to see more widespread adoption of privacy-preserving machine learning, unlocking the full potential of AI while ensuring the protection of sensitive information.

The drawback however is that FHE is inherently compute intensive. Coupling that AI/ML could really exacerbate the needs for compute when training or fine-tuning on encrypted data. BUT, there is promise here. For example until recently, most zk proofs were too intensive for comfortably running locally ICICLE and other GPU acceleration libraries have emerged (but they're CUDA-based and today lack support for the Apple chipset) More on that here: https://blog.ezkl.xyz/post/acceleration/… Last, FHE FPGAs are also scheduled to hit the market in 2025 and there are several companies are working on it, including Cornami, Inc. , Intel, Duality, Fabric, etc.

For a much more detailed overview of privacy-preserving machine learning (PPML) check out this great post from Bagel ??

https://twitter.com/bagel_network/status/1765016627530154292

These series of posts by Daniel Huynh / Mithril Security is definitely worth digging into: https://towardsdatascience.com/homomorphic-encryption-intro-part-1-overview-and-use-cases-a601adcff06c

Also check out Zama’s ConcreteML: https://docs.zama.ai/concrete-ml

Special thanks to Remi Gai Bidhan R. Sree Duggirala Shrey J. for helping me with this post!