Run AI Chatbot Locally with Open-Source Tools ??

Concerned about sharing personal information with online AI chatbots? I was too! In this article, we'll explore how to run AI chatbot locally on your machine (it's easier than you think ??) using open-source tools, ensuring your data remains private and secure.

Few Advantages:

• Easy setup: No coding skills required! Install and start running your local AI chatbot with ease ??.
• Data privacy: Run locally, and your data never leaves your PC (even works without wifi).
• Upload documents: Chat with your documents too!
• Example use case: Improve your resume without sharing data with third-party services.

Few Disadvantages:

• Speed: Depends on your hardware. The better the GPU, the faster the response time.
• Limited capabilities: Minified version of LLMs (for faster response), but still effective. (using LLama3 in this guide with 8B parameters by Meta)

Tools Breakdown (in simple way possible)

• Ollama: Runs LLMs on your PC (no GUI, but has terminal access to chat)
• Open-WebUI: Provides a ChatGPT-like UI for Ollama. (alternat UIs: AnythingLLM, LLaMA UI, LLM Studio, etc..)
• Llama3: A Large language model by Meta. (you can download different models e.g mistral, gemma2 etc... as well from the OLLama site)

Prerequisites

• WSL (Windows Subsystem for Linux): Install using PowerShell command

wsl --install        

• Docker: Download and install Docker Desktop to run Open-WebUI docker container.

Step-by-Step Guide

Ollama Installation

• Download and install OLLama from their site.
• Run Ollama using the .exe file or PowerShell.
• Verify successful installation by visiting localhost:11434 in your browser. It should say "ollama is running". Ollama runs on port 11434 by default.

Download and Run Llama3 Model

• Download and run llama3 LLM using PowerShell command:

ollama pull llama3
ollama run llama3        

Installing UI for AI Chatbot - Open WebUI

• Run the following command inside PowerShell to install Open-WebUI.

docker run -d -p 3000:8080 --add-host=host.docker.internal:host-gateway -v open-webui:/app/backend/data --name open-webui --restart always ghcr.io/open-webui/open-webui:main        

This will run the web-based chatbot UI on port 3000 of your local machine, which is mapped to port 8080 of the Docker container, where Open-WebUI is running. For more info visit Open-WebUI github page.

Now, open a browser and navigate to localhost:3000 to create a local admin account and start chatting with your AI chatbot. You can also upload files in chat, securely and privately, as they will remain on your local machine! Note: If you have downloaded multiple LLMs, you can select your preferred model from the dropdown menu in the chat interface.

Enjoy & share it with your friends ??.