Run your local AI ChatBot with Podman Desktop

I wrote this article first on my newly created personal technical blog where I hope to share more insights, demos and information like this.

Do you want to get your hands on GenAI and experiment with LLM without the need for huge computing resources?

In this guide, I’ll walk you through setting up a local AI environment using Podman Desktop and its AI Lab extension. By the end, you’ll have a functional chatbot application connected to an AI model, ready to respond to your queries. This tutorial is designed for beginners, requiring no prior experience with Podman or container technologies.

Introduction to Podman Desktop and AI Lab Extension

Podman Desktop is a free and open-source graphical interface that simplifies container management across Windows, macOS, and Linux platforms. It provides an intuitive way to handle containers, making it accessible for both technical and non-technical users.

The AI Lab extension enhances Podman Desktop by offering a curated set of resources tailored for common AI tasks, such as chatbot development and text summarization.

Key Features:

• User-Friendly Interface: Manage containers effortlessly across different operating systems.
• Curated Recipes Catalog: Access open-source AI recipes for various applications.
• Model Management Tools: Utilize features for model tuning and performance monitoring.

Step 1: Installing Podman Desktop

Minimum System Requirements:

• Memory: 4 GB of available memory
• CPU: 4 vCPU cores
• Operating Systems:
• Windows
• macOS
• Linux distributions with a desktop environment

Installation Process:

For Windows:

1. Download the Windows Installer: Visit the official Podman Desktop website and download the installer for Windows.
2. Run the Installer: Open the downloaded file to start the installation process.
3. Follow On-Screen Instructions: Proceed through the installation wizard to complete the setup.

For macOS:

1. Download the macOS Installer: Access the Podman Desktop website and download the installer for macOS.
2. Install the Application: Open the downloaded .dmg file and drag the Podman Desktop app to your Applications folder.

For Linux:

1. Install via Flatpak: flatpak install flathub io.podman_desktop.PodmanDesktop
2. Launch Podman Desktop: After installation, start the application from your applications menu.

For detailed installation instructions, refer to the official documentation

Step 2: Installing the AI Lab Extension

Open Podman Desktop: Launch the application on your system, if it’s your first time launching Podman Desktop you’ll find a big banner on top recommending installing “Podman AI Lab”.

Otherwise, you can always install it from the extensions as follows:

• Access Extensions: Click on the Extensions icon in the sidebar (represented by a puzzle piece).
• Search for AI Lab: In the search bar, type “Podman AI Lab” and press Enter.
• Install the Extension: Click the download button (downward-pointing arrow) next to the AI Lab extension.
• Verify Installation: Ensure successful installation by checking for a green indicator and the new AI Lab icon in the sidebar.

Step 3: Setting Up a Chatbot Application

Accessing the Recipes Catalog:

The Podman AI Lab extension significantly enhances the capabilities of Podman by offering a curated Recipes Catalog. This catalog provides users with access to a variety of open-source recipes designed to address common AI use cases, such as chatbot development, code generation, and text summarization. Each recipe includes detailed explanations and sample applications that can be executed using various large language models (LLMs).

In the Recipe Catalog, you can find several pre-defined AI use case, you can view more details on each use case.

To create your first AI application in this case a ChatBot:

1. Open AI Lab: Click on the AI Lab icon in the sidebar.
2. Navigate to Recipes: Go to the Recipes Catalog section.
3. Select ChatBot Recipe: Choose “Clone Recipe” on ChatBot recipe from the list.

After the use case is cloned, click the “More Details” link and you’ll be taken to all the information related to the AI use case including how you can download a model, build a model service, build the application, deploy the application and interact with it, if you decide to do everything by yourself using command line.

But you also have the option to run the recipe directly from the extension by clicking the “Start” button on the top right corner of the page.

This will take you to the Model selection where normally a Granite 8b LLM is used but you have the option to select other LLMs from the dropdown list and then “Start ChatBot Recipe”

This process will pull the ChatBot recipe and download the LLM which may take some minutes to finish.

Hint: You can later see all the available LLMs on your local machine from the AI Lab menu > Model > Catalog > Downloaded

After downloading the LLM, the recipe will start the inference server and load the needed configuration then build and deploy the ChatBot application.

Step 4: Running and Interacting with the Chatbot

After the application is ready, you’ll see a button “Open Details” where you’ll see the information of the running ChatBot use case and which port the ChatBot is listening to that you can use to access the ChatBot from any browser by going to https://localhost:<port>/ , you can also directly open the app from this view from the Actions on the right-hand side.

Start Interaction: Begin conversing with the chatbot by typing your questions and receiving responses.

Bonus Tip: You can see the running ChatBot POD from the Pods view from the left side menu, from there you can access the POD summary and logs.

Step 5: Tuning Model Performance

Podman AI Lab gives you the possibility to tune the performance of the locally used LLMs, this is done through the Playground feature which helps in exploring the capabilities and accuracy of various models and aids in finding the best model for the use case at hand.

You can optimize the chatbot’s responses, by adjusting several parameters like:

• Temperature: Controls the randomness of the model’s responses. A lower temperature results in more deterministic outputs, while a higher temperature increases variability.
• Max Tokens: Sets the maximum length of the generated response.
• Top-p (Nucleus Sampling): Determines the cumulative probability for token selection, balancing between diversity and relevance.

How to Change Parameters:

1. Use the Playground feature in AI Lab to experiment with these parameters.
2. Adjust the sliders or input fields corresponding to each parameter to see how changes affect the model’s responses.
3. Test different configurations to find the optimal settings for your specific use case.

Conclusion

By following this guide, you have successfully set up a local AI environment using Podman Desktop and the AI Lab extension. You now have a functional chatbot application connected to an AI model, ready to handle queries. This setup not only simplifies AI application deployment but also provides a robust framework for further exploration and development.

Next Steps:

• Explore other recipes in the AI Lab.
• Experiment with different models and configurations.
• Join the Podman community for support and collaboration.

Stay tuned for more tutorials on leveraging Podman Desktop for advanced AI tasks!