Building a Private AI Assistant with Local LLMs — A Practical Guide

In the past few years, the development of private AI assistants has been accelerating, especially as companies seek to implement solutions that maintain data privacy and allow for custom integrations. But running a local LLM (Large Language Model) isn’t just about downloading the latest model and hitting run — it requires careful consideration of hardware, architecture, and prompt engineering. Today, I’ll take you through our journey of setting up a private AI assistant using local models like Gemma from Google or Llama2 from Facebook through the Ollama server, detailing the challenges and solutions that made it work.

Why Local Models?

The allure of running AI models locally is all about control — control over the data, the configuration, and the deployment environment. Whether you’re concerned about data security or just want to avoid the high costs of querying cloud-based models, local models provide a way to build AI systems that operate entirely within your infrastructure.

But the path isn’t straightforward. Running LLMs requires balancing hardware constraints, managing embeddings, and designing architectures that can serve real-time queries. Here’s how we tackled these challenges.

Running LLMs Locally — Understanding Hardware Trade-offs

To start, one of the biggest factors when setting up a local LLM is understanding your hardware. Models like Gemma can technically run on any machine, but their performance is highly influenced by whether you have a GPU or not.

When we first ran Gemma on a Mac M1 with GPU (Metal API), we hit limitations when we tested the biggest Models like Llama2:70b . While you can run these models on a CPU-only setup, the speed for creating embeddings and generating responses will likely become a bottleneck. In contrast, using a G4DNX family on AWS (which comes equipped with a GPU) dramatically improved the processing time.

On the other hand, since this type of EC2 family type is expensive is worth having a mechanism enabled to turn off those instances when they are not in working hours, to reduce costs. Here a resource scheduler with a System Manager in AWS will help you.

Key takeaway: If your machine doesn’t have a GPU, don’t waste your time, you will need it very soon, and in case you are limited in GPU you can use a smaller version of a model like Llama2:7b instead of the 70b. Investing in GPU-powered instances or using smaller, optimized models is a must for production-ready applications.

Building a Private Knowledge Base with Local Models

A private AI assistant is only as good as its knowledge base. We tackled this by setting up a Q&A system with a structured private knowledge base using a vector database. Here’s a brief rundown of the process:

1. Data Ingestion: We took large documents and split them into manageable chunks, translating these into embeddings that the LLM can work with. This required optimizing the document size and running multiple iterations to ensure that the chunking preserved the context.
2. Vector Storage: Once transformed into embeddings, these chunks were stored in a vector database, making it easier for the model to perform similarity searches when a query comes in.
3. Query Processing: When a user query is made, it’s matched against the most relevant chunks in the vector database. The LLM then uses this context to generate a more informed and accurate response.

Pro tip: If you’re building a similar system, invest time in fine-tuning the chunking and embedding strategies. The quality of your embeddings will directly impact the assistant’s ability to answer questions accurately.

Architectural Blueprint — Building with AWS Copilot

Next, we designed an architecture using AWS Copilot, making it easier to deploy and scale the services. The architecture was divided into two main flows:

• Ingest private knowledge information.
• Request for a task or question.

Our main architecture components are:

1. Load Balancer Service: Responsible for handling incoming requests and routing them to the appropriate backend service.
2. Backend and Worker Services: Here, the backend service processes embeddings, while the worker service handles the heavy lifting of creating embeddings from the documents.
3. Storage and Vector Database: We used a vector database to store the embeddings, making it easy to retrieve relevant information based on user queries.
4. Ec2 with a G4dn family type enabled to run the models. We can run our public LLM through the Ollama server in our private network.

Pro tip: We recommend implementing a web socket or subscription to handle the LLM stream response to drastically improve the general performance and usability of your assistant. In this way you can stream out each token at the time is generated by the LLM, generating a live response behavior and avoid to wait the LLM to complete generating the response.

Prompt Engineering — The Secret Sauce

One of the key elements we focused on was prompt engineering. This isn’t just about asking the right questions — it’s about fine-tuning the responses by adjusting variables like temperature and system prompts. In our setup:

• LLM: Regarding the type of tasks you want to accomplish you will need to choose the more relevant LLM available since they have different training and expertise. Notice that the LLM will have an impact on the size of your embeddings and you need to match your query embedding with your knowledge base embeddings.
• System Prompts: These were used to define the assistant’s role and behavior. For example, setting the assistant as an “experienced software architect” ensured that the answers were relevant and professional.
• Temperature Settings: We adjusted these based on the type of query. Lower temperatures were used for factual queries, while higher settings allowed for more creative problem-solving scenarios.
• Increase the amount of relevant documents in case of big context domain questions. The default value used to be 4 documents, we had good results until 24 documents, when we started having hallucinations.
• ReAct agent with custom tools tailored for your domain with an iterative behavior to request all the information needed within your scope. You can integrate with other internal APIs or give the agent no built-in functionalities.

If you’re building your assistant, spend time experimenting with different prompts and configurations. It can drastically change how the model interprets and responds to queries.

Pro tip: A monitor layer will help you to understand how your assistant works, what are the retrieved documents, which model is in use, associated costs, response times, time to generate the first token, and recursive behavior in the case of a ReAct Agent. In this context, LangSmith from Langchain could be a game changer with ReAct agents.

Closing Thoughts

Building a private AI assistant using local LLMs is a rewarding yet complex endeavor. From managing hardware constraints to designing robust architectures and fine-tuning prompts, the process requires both technical acumen and a deep understanding of how these models operate.

But the benefits are clear: complete control over your data, flexibility in deployment, and the ability to build custom solutions that cater to specific business needs. As new models like Llama, Mistral, and Gemma emerge, it’s exciting to see how these capabilities will evolve and offer even more opportunities for innovation.

For those looking to dive into this space, my advice is simple: start small, iterate quickly, and don’t be afraid to experiment. The potential of local AI assistants is immense, and we’re just scratching the surface of what’s possible.

Happy building!