Using AutoGen and LM- Studio with Any Open-Source LLM: A Tutorial

What is Agent in context of AI?

In the context of artificial intelligence (AI), an agent is an entity that can perceive its environment through input and act upon that environment through actuators to achieve specific goals(execute the workflows).

What is Microsoft AutoGen?

MS AutoGen (https://autogen-studio.com/autogen-studio-ui) is an open-source framework designed to simplify the development of applications using large language models (LLMs). The framework emphasizes an autonomous agent-based architecture, facilitating the automation of tasks and the coordination of multiple language models. Key features and components of AutoGen include:

1. Framework for Autonomous Agents: It allows developers to create, customize, and manage agents that can independently perform specific tasks.
2. Simplified LLM Application Development: AutoGen reduces the complexity involved in integrating LLMs into applications, providing tools and abstractions to streamline the process.
3. Collaboration among LLMs: The framework supports the coordination of multiple LLMs, enabling them to work together to achieve complex goals.
4. Open-Source: AutoGen is available as open-source software, encouraging community contributions and collaboration.

What is lm-studio?

LM Studio (https://lmstudio.ai/) is a desktop application that simplifies the process of running and interacting with large language models (LLMs) on personal computers. It provides a user-friendly interface for deploying and utilizing LLMs, particularly those optimized to run on local hardware without needing extensive cloud resources. Key features of LM Studio include:

1. Local Deployment: Allows users to run large language models on their local machines, making it accessible without the need for powerful cloud servers.
2. User-Friendly Interface: Provides an intuitive and easy-to-use interface for interacting with and managing language models.
3. Support for Various LLMs: Compatible with different open-source language models, offering flexibility in terms of the models users can work with.
4. Resource Management: Optimizes the use of local computational resources to ensure efficient operation of the language models.

What is Ollma?

OLLMA, short for Open LLM App, is an open-source project focused on creating applications that leverage large language models (LLMs). It aims to provide tools and resources for developers to build, deploy, and manage applications powered by these models (locally). Key features of OLLMA include:

1. Open-Source Framework: OLLMA offers a comprehensive open-source framework that facilitates the development and deployment of LLM-powered applications.
2. Customization and Flexibility: Developers can customize and adapt the framework to suit their specific needs, ensuring flexibility in application development.
3. Integration with LLMs: The framework supports integration with various open-source LLMs, allowing for a wide range of use cases.
4. Community-Driven: Being open-source, OLLMA encourages contributions from the community, fostering innovation and collaboration.

Hope the above small descriptions will help to understand the basics concept, lets get started.

1. Step-by-Step Guide to Setting Up OLLMA, Autogen Studio, and LM-Studio on Windows

Download and Install OLLMA for Windows:

• Get the installer from this link.
• Note: While this guide uses Windows, OLLMA is also available for Linux and macOS.
• After installation, you'll see a small OLLMA icon in the system tray at the bottom right of your screen. Alternatively, you can check if OLLMA is running by entering https://localhost:11434/ in your web browser.

Download and Install Autogen:

• After downloading, install Autogen.
• Run the Command Prompt as an administrator.
• Start MS Autogen by entering autogenstudio ui --port 8081.
• To access Autogen Studio locally, open your preferred web browser and go to https://127.0.0.1:8081.

Download and Install LM-Studio:

• Install LM-Studio on your system.
• Open LM-Studio and search for the model you wish to download.
• For this learning, the "Phi-3-mini-4k-instruct-q4.gguf" model was downloaded for experimentation and learning.

By following these steps, you'll be able to set up and run OLLMA, Autogen Studio, and LM-Studio on your Windows machine, allowing you to explore and experiment with various models.

To run the model locally, follow these steps:

• Open LM-Studio.
• Navigate to the Local Server section (?, located as the second last item on the left side).
• Select the desired model from the top.
• Click on "Start Server."

This will start the local server on https://localhost:1234/v1 with the default API key set to "lm-studio."

Now we need to register the model using the base URL and API key from the previous step:

1. Open AutoGen Studio.
2. In the left menu, click on "Model."
3. Click on "New Model."
4. For Model Type, select "OpenAI" for this example.
5. Enter the base URL and API key obtained in the previous step.

Give the model a name, enter the API key, and input the base URL. Providing a description is optional but recommended to avoid confusion. Finally, click on "Test Model" to ensure everything is set up correctly.

Great! Now that our model is running on the local server, we need to create an agent to communicate with it. Follow these steps:

1. In AutoGen Studio, select "Agent" from the left menu.
2. Click on "New Agent."
3. Select "User Proxy" as the agent type, as this will allow the model to take input from a user. You can also explore the multi-agent group chat option if needed.

Okay, now that we have downloaded the model and created the user proxy agents, the final step is to create a workflow to execute the agent. Here's how:

1. In AutoGen Studio, select "Workflow" from the left menu.
2. Click on "New Workflow."
3. Give the workflow a name.
4. Configure the workflow by setting the "Initiator" as the User Proxy agent and the "Receiver" as the agent you created in step 8.

Great! We are done. You can now test your workflow by clicking on "Test Workflow" and giving a prompt to the agent to see it in action. Alternatively, you can:

1. Go to "Playground."
2. Create a new session.
3. Select the agent.
4. Give the prompt.

Enjoy experimenting with your model!

Note: The response time will depend on the machine you are using to run the model. Machines with higher configurations will generally respond faster.

?

##### Begin of Discussion #####

# from skills import Discussion # Import the function from skills.py

from typing import List

import uuid

import requests # to perform HTTP requests

from pathlib import Path

from openai import OpenAI

def generate_and_save_images(query: str, image_size: str = "1024x1024") -> List[str]:

"""

Function to paint, draw or illustrate images based on the users query or request. Generates images from a given query using OpenAI's DALL-E model and saves them to disk. Use the code below anytime there is a request to create an image.

:param query: A natural language description of the image to be generated.

:param image_size: The size of the image to be generated. (default is "1024x1024")

:return: A list of filenames for the saved images.

"""

client = OpenAI() # Initialize the OpenAI client

response = client.images.generate(model="dall-e-3", prompt=query, n=1, size=image_size) # Generate images

# List to store the file names of saved images

saved_files = []

# Check if the response is successful

if response.data:

for image_data in response.data:

# Generate a random UUID as the file name

file_name = str(uuid.uuid4()) + ".png" # Assuming the image is a PNG

file_path = Path(file_name)

img_url = image_data.url

img_response = requests.get(img_url)

if img_response.status_code == 200:

# Write the binary content to a file

with open(file_path, "wb") as img_file:

img_file.write(img_response.content)

print(f"Image saved to {file_path}")

saved_files.append(str(file_path))

else:

print(f"Failed to download the image from {img_url}")

else:

print("No image data found in the response!")

# Return the list of saved files

return saved_files

# Example usage of the function:

# generate_and_save_images("A cute baby sea otter")

#### End of Discussion ####