A Sunday Evening Activity with my drink: Bringing LLM to My Personal Data

Last weekend, I tried to to run a Large Language Model (LLM) on my Apple MacBook with an M1 chip and 8GB RAM. Given my machine's limitations, I knew running large, sophisticated models wasn't feasible. While I could use Google Colab, however, I decided to run on my personal computer. Here's how I made it work, inspired by PrivateGPT:

Leveraging LLMs with Personal Data

LLMs provide a powerful natural language interface between humans and data. While they come pre-trained on extensive public datasets, they aren't tuned to your specific data, which might reside behind APIs, in SQL databases, or trapped in PDFs, DOCs, Excel sheets, and slide decks.

My Approach

I utilised LlamaIndex to ingest, parse, index, and process my private documents stored locally. LlamaIndex supports HuggingFace embedding models to generate embeddings (vectors) before storing them in a database.

To ensure my generative model could answer questions using my internal data, I used the Retrieval Augmented Generation (RAG) approach. This way I could use my private data as context, which is then stored locally in a vector database. I chose Qdrant (though alternatives like Milvus, ChromaDB, MongoDB are also few options).

For building the app, I used the FastAPI framework and wrapped everything in Gradio UI (I also experimented with Streamlit UI but did not proceed further).

The Workflow

So here is the final flow, ask a question based on your private documents, it will create embeddings for your question using SentenceTransformers.

Then it will use LangChain to search the vectorstore for the most similar embeddings from your documents.

It will return a number of chunks (sources) that match your question best.

Then it will use the LLM model to generate an answer for your question based on these chunks. It will feed the chunks and the question into the LLM model one by one (or in batches), and get an output from the model.

The output will be a natural language answer that tries to satisfy your question.

The Experiment

I kept unstructured data (doc, ppt, pdf) in a local folder and ingested all using the python script. I additionally trained the already pre-trained model on my personal data, I chose my CV, academic documents from high school to my Master's, and other private documents. Then, I had a fun session asking the model quirky questions about myself, enjoying a relaxed Sunday evening with a nice drink and my customised AI model, running small on my limited resource laptop. Next is when I will try on Colab.

Takeaway

With the right tools and inspiration from projects like PrivateGPT, even basic hardware can be leveraged to create powerful, personalised AI experiences. If you're curious about LLMs and have some private data to experiment with, give it a try! There are many GitHub projects and you can leverage one of those, if you need more compute for free (TNVIDIA Tesla T4 GPU with 16GB of VRAM) use colab. That's next for me.