Run Large Language Models (LLMs) locally on consumer-grade hardware.

"Can I run a large language model (LLM) locally on consumer-grade hardware?"

When a customer asked our Data Science team this question just several months ago, we answered with a resounding "Not a chance!". However, every single week in AI brings progress that makes us revisit what we know and question our assumptions about the future.??

Our Senior Data Scientist Mattia Sanna is leading our research into the innovations and collaborations that are democratising cutting-edge technology at lightening pace. The recent development of smaller open-source LLM models are a perfect example of this innovation. These models are giving everyone the opportunity to build on-premise local solutions and flipped our response to a resounding “Yes you can!”.??

Naturally, being able to meaningfully develop LLMs with consumer-grade hardware will lead customers to ask, "Can I fine-tune an LLM locally on my data, on a specific task, on consumer-grade hardware?"?

Thankfully, there’s even more good news to report. Recent research implemented in the matrix decomposition algorithm called Low-Rank Adaptation - LoRa is making this fine-tuning possible. This talk organised by DeepLearning.AI and presented by FourthBrain expands on this new and exciting development.?

Cristian Genes is our in-house expert in matrix low-rank decomposition, and he is really excited by the incredible efficiency this research has unlocked. One of the most valuable parts of fine-tuning an LLM with LoRa for a specific task is that you train just a small subset of the neural network's weights (usually less than 1%!), before injecting these new specialised weights into your original model, to perform the downstream task. This enables 2 key benefits:?

• The specialised weights require way less memory (sometimes in order of MB instead of GB).?
• You can pick and swap specialised weights to accomplish a task, instead of changing or loading different models.?

This is fantastic, and it opens new horizons for the commercial use of LLMs on sensitive data, using more cost-effective hardware. We believe this is another crucial step in bringing fine-tuned Large Language Models closer to reality for our customers in Defence and Public Sector.??

If you’d like to learn more or discuss how your organisation can begin your journey towards an AI-assisted future, please get in touch: [email protected]