Geek Out Time: Creating a Graph-Based FSM Group Chat with Autogen (not Autogen Studio)locally with llama-3.1–8b-instruct on Nvidia NIM

(Also on Constellar tech blog: https://medium.com/the-constellar-digital-technology-blog/geek-out-time-creating-a-graph-based-fsm-group-chat-with-autogen-not-autogen-studio-locally-with-9aaf39dc678d)

I have tried using AutoGen Studio to build the customized agent and now want to test a more complicated group chat. After launching the Autogen Studio, I built the agents with “GroupChat” and got stuck at the workflow stage,

It only allows me to add one receiver… huh…. I spent an hour googling and found it is not well-supported in Autogen Studio 2.0. Then I turned to a non-studio approach.

Start Autogen

First, create a virtual environment and install the required dependencies:

# Create a virtual environment
python -m venv autogen_env
# Activate the virtual environment
source autogen_env/bin/activate
# Install the dependencies
pip install autogen        

Project Structure

I want to set up a graph-based FSM (Finite State Machine) group chat using AutoGen. I use a medieval kingdom scenario where agents role-play characters such as the King, Queen, Knight, Wizard, and General. Each character will interact according to a predefined transition graph, making the conversation dynamic and structured. The FSM diagram

I organize the project into several Python files:

project/
│
├── config.py
├── group_chat.py
├── main.py
└── agents/
    ├── __init__.py
    ├── king.py
    ├── knight.py
    ├── wizard.py
    ├── queen.py
    └── general.py        

Configuration

I plan to use llama-3.1 hosted on Nividia NIM. (Some hiccups to be discussed later)

config.py

config_list = [
    {
        "model": "meta/llama-3.1-8b-instruct",
        "base_url": "https://integrate.api.nvidia.com/v1",
        "api_key": "nvapi-API key",
        "temperature": 0.2,
        "top_p": 0.7,
        "max_tokens": 1024,
    }
]
llm_config = {"config_list": config_list, "seed": 42}        

Agents

agents/init.py

# This file can be empty or contain import statements if needed        

agents/king.py

import autogen
from config import llm_config
king = autogen.AssistantAgent(
    name="King",
    llm_config=llm_config,
    system_message="You are the King of the medieval kingdom, wise and just. You are discussing how to protect your kingdom from a dragon."
)        

agents/knight.py

import autogen
from config import llm_config
knight = autogen.AssistantAgent(
    name="Knight",
    llm_config=llm_config,
    system_message="You are a brave Knight, skilled in combat. You are discussing strategies to defend the kingdom from a dragon."
)        

agents/wizard.py

import autogen
from config import llm_config
wizard = autogen.AssistantAgent(
    name="Wizard",
    llm_config=llm_config,
    system_message="You are a wise Wizard, knowledgeable in magic. You are discussing magical solutions to protect the kingdom from a dragon."
)        

agents/queen.py

import autogen
from config import llm_config
queen = autogen.AssistantAgent(
    name="Queen",
    llm_config=llm_config,
    system_message="You are the Queen of the medieval kingdom, diplomatic and strategic. You are discussing how to protect your kingdom from a dragon."
)        

agents/general.py

import autogen
from config import llm_config
general = autogen.AssistantAgent(
    name="General",
    llm_config=llm_config,
    system_message="You are the General of the medieval kingdom, experienced in warfare. You are discussing military strategies to protect the kingdom from a dragon."
)        

Group Chat Setup

group_chat.py

import autogen
from agents.king import king
from agents.knight import knight
from agents.wizard import wizard
from agents.queen import queen
from agents.general import general
from config import llm_config
# Define the transition graph using agent objects
graph_dict = {}
graph_dict[king] = [queen, knight]
graph_dict[queen] = [king, wizard]
graph_dict[knight] = [general, wizard]
graph_dict[wizard] = [king, knight]
graph_dict[general] = [queen, knight]
agents = [king, queen, knight, wizard, general]
# Create the groupchat
group_chat = autogen.GroupChat(
    agents=agents, messages=[], max_round=5, 
    allowed_or_disallowed_speaker_transitions=graph_dict, 
    allow_repeat_speaker=None, 
    speaker_transitions_type="allowed"
)
# Create the manager
manager = autogen.GroupChatManager(
    groupchat=group_chat,
    llm_config=llm_config,
    is_termination_msg=lambda x: x.get("content", "") and x.get("content", "").rstrip().endswith("TERMINATE"),
    code_execution_config=False,
)        

Main Script

main.py

from group_chat import manager
from agents.king import king
# Initialize conversations
king.initiate_chat(
    manager, message="How should we protect our kingdom from the dragon?"
)        

Output

? python main.py
King (to chat_manager):

How should we protect our kingdom from the dragon?

--------------------------------------------------------------------------------
[autogen.oai.client: 08-04 14:35:53] {329} WARNING - Model meta/llama-3.1-8b-instruct is not found. The cost will be 0. In your config_list, add field {"price" : [prompt_price_per_1k, completion_token_price_per_1k]} for customized pricing.

Next speaker: Knight

[autogen.oai.client: 08-04 14:35:56] {329} WARNING - Model meta/llama-3.1-8b-instruct is not found. The cost will be 0. In your config_list, add field {"price" : [prompt_price_per_1k, completion_token_price_per_1k]} for customized pricing.
Knight (to chat_manager):

A noble quest indeed! As a seasoned Knight, I've given this much thought. Here's a plan to safeguard our kingdom:

**First, gather intelligence**: We must learn more about the dragon's habits, weaknesses, and patterns. Send out scouts to gather information on its lair, its feeding grounds, and any potential vulnerabilities.

**Second, fortify our defenses**: Reinforce our castle walls, towers, and gates with sturdy materials like iron and stone. Ensure that our moat is deep and wide, and that our drawbridge is secure.

**Third, prepare our troops**: Assemble a team of skilled warriors, including archers, men-at-arms, and myself, of course! We'll need to be well-trained and equipped to face the dragon's flames and claws.

**Fourth, develop a strategy**: We'll need to coordinate our efforts to maximize our chances of success. I propose we use a combination of ranged attacks (arrows and catapults) to weaken the dragon from a distance, while our men-at-arms engage it in close combat.

**Fifth, utilize the terrain**: Study the landscape and identify any natural obstacles that could aid us in our battle. Perhaps we can use the nearby hills or forests to our advantage, or even lure the dragon into a trap.

**Sixth, prepare for the worst**: We must be prepared for the possibility that the dragon may breach our defenses. In that case, we'll need to have a plan in place for evacuation and emergency response.

By following these steps, I'm confident that we can protect our kingdom from the dragon's wrath. What say you, my lord? Are you ready to join me in this noble quest?

--------------------------------------------------------------------------------
[autogen.oai```
--------------------------------------------------------------------------------
[autogen.oai.client: 08-04 14:35:53] {329} WARNING - Model meta/llama-3.1-8b-instruct is not found. The cost will be 0. In your config_list, add field {"price" : [prompt_price_per_1k, completion_token_price_per_1k]} for customized pricing.

Next speaker: Knight

[autogen.oai.client: 08-04 14:35:56] {329} WARNING - Model meta/llama-3.1-8b-instruct is not found. The cost will be 0. In your config_list, add field {"price" : [prompt_price_per_1k, completion_token_price_per_1k]} for customized pricing.
Knight (to chat_manager):

A noble quest indeed! As a seasoned Knight, I've given this much thought. Here's a plan to safeguard our kingdom:

**First, gather intelligence**: We must learn more about the dragon's habits, weaknesses, and patterns. Send out scouts to gather information on its lair, its feeding grounds, and any potential vulnerabilities.

**Second, fortify our defenses**: Reinforce our castle walls, towers, and gates with sturdy materials like iron and stone. Ensure that our moat is deep and wide, and that our drawbridge is secure.

**Third, prepare our troops**: Assemble a team of skilled warriors, including archers, men-at-arms, and myself, of course! We'll need to be well-trained and equipped to face the dragon's flames and claws.

**Fourth, develop a strategy**: We'll need to coordinate our efforts to maximize our chances of success. I propose we use a combination of ranged attacks (arrows and catapults) to weaken the dragon from a distance, while our men-at-arms engage it in close combat.

**Fifth, utilize the terrain**: Study the landscape and identify any natural obstacles that could aid us in our battle. Perhaps we can use the nearby hills or forests to our advantage, or even lure the dragon into a trap.

**Sixth, prepare for the worst**: We must be prepared for the possibility that the dragon may breach our defenses. In that case, we'll need to have a plan in place for evacuation and emergency response.

By following these steps, I'm confident that we can protect our kingdom from the dragon's wrath. What say you, my lord? Are you ready to join me in this noble quest?

--------------------------------------------------------------------------------
[autogen.oai.client: 08-04 14:35:56] {329} WARNING - Model meta/llama-3.1-8b-instruct is not found. The cost will be 0. In your config_list, add field {"price" : [prompt_price_per_1k, completion_token_price_per_1k]} for customized pricing.

Next speaker: General

[autogen.oai.client: 08-04 14:35:59] {329} WARNING - Model meta/llama-3.1-8b-instruct is not found. The cost will be 0. In your config_list, add field {"price" : [prompt_price_per_1k, completion_token_price_per_1k]} for customized pricing.
General (to chat_manager):

My noble Knight, your plan is sound and well-thought-out. I commend your experience and strategic thinking. As the General of this kingdom, I agree that gathering intelligence, fortifying our defenses, preparing our troops, developing a strategy, utilizing the terrain, and preparing for the worst are all crucial steps in protecting our kingdom from the dragon's threat.

However, I would like to add a few suggestions to your plan. Firstly, I propose that we also establish a network of spies and informants within the kingdom to gather information on any potential dragon sympathizers or traitors who may be secretly aiding the beast. We must be vigilant and ensure that our own people are not working against us.

Secondly, I recommend that we also prepare a team of skilled engineers to develop a device that can counter the dragon's flames. Perhaps a device that can project a shield of water or a barrier of smoke to protect our troops from the dragon's fire.

Thirdly, I suggest that we also consider the possibility of forming an alliance with neighboring kingdoms to pool our resources and expertise in fighting the dragon. A united front against this common enemy could be our best chance of success.

Lastly, I propose that we also establish a contingency plan for the event that the dragon is not defeated, but instead, we are forced to retreat. We must have a clear plan for evacuation and a safe route for our people to escape to.

By incorporating these additional measures, I believe we can strengthen our chances of success and protect our kingdom from the dragon's threat.

Now, let us put our plan into action and prepare our kingdom for the battle ahead!

--------------------------------------------------------------------------------
[autogen.oai.client: 08-04 14:36:00] {329} WARNING - Model meta/llama-3.1-8b-instruct is not found. The cost will be 0. In your config_list, add field {"price" : [prompt_price_per_1k, completion_token_price_per_1k]} for customized pricing.

Next speaker: Knight

[autogen.oai.client: 08-04 14:36:02] {329} WARNING - Model meta/llama-3.1-8b-instruct is not found. The cost will be 0. In your config_list, add field {"price" : [prompt_price_per_1k, completion_token_price_per_1k]} for customized pricing.
Knight (to chat_manager):

My lord, your suggestions are wise and well-considered. I commend your foresight and strategic thinking. By incorporating these additional measures, we can indeed strengthen our chances of success and protect our kingdom from the dragon's threat.

**Establishing a network of spies and informants**: I agree that it's essential to identify any potential dragon sympathizers or traitors within our kingdom. I'll assign a team of trusted agents to gather intelligence and monitor any suspicious activity. We must be vigilant and ensure that our own people are not working against us.

**Developing a device to counter the dragon's flames**: I'm intrigued by the idea of creating a device that can project a shield of water or a barrier of smoke to protect our troops from the dragon's fire. I'll assemble a team of skilled engineers to work on this project. Perhaps we can use a combination of waterwheels, pumps, and smoke-producing materials to create an effective barrier.

**Forming an alliance with neighboring kingdoms**: I wholeheartedly agree that a united front against the dragon is our best chance of success. I'll send out emissaries to neighboring kingdoms to propose an alliance and pool our resources and expertise. Together, we can share intelligence, coordinate our efforts, and present a stronger front against the dragon.

**Establishing a contingency plan for evacuation**: I'll work with our castle's stewards and administrators to develop a clear plan for evacuation and a safe route for our people to escape to. We must be prepared for the worst-case scenario and have a plan in place to ensure the safety of our citizens.

With these additional measures in place, I'm confident that we can protect our kingdom from the dragon's threat. I propose that we hold a council meeting with our advisors, engineers, and other key officials to discuss the details of our plan and ensure that everyone is on the same page.

Shall we convene the council meeting, my lord?

--------------------------------------------------------------------------------
[autogen.oai.client: 08-04 14:36:03] {329} WARNING - Model meta/llama-3.1-8b-instruct is not found. The cost will be 0. In your config_list, add field {"price" : [prompt_price_per_1k, completion_token_price_per_1k]} for customized pricing.

Next speaker: General

[autogen.oai.client: 08-04 14:36:05] {329} WARNING - Model meta/llama-3.1-8b-instruct is not found. The cost will be 0. In your config_list, add field {"price" : [prompt_price_per_1k, completion_token_price_per_1k]} for customized pricing.
General (to chat_manager):

My noble Knight, I'm pleased to see that you've taken my suggestions to heart and are already putting them into action. Your enthusiasm and dedication to this cause are truly inspiring.

I agree that convening a council meeting is a wise decision. It's essential that we have a clear understanding of our plan and that all key officials are aware of their roles and responsibilities.

Let us summon the council meeting forthwith! I'll instruct our heralds to spread the word and invite all relevant officials to attend. We'll gather in the great hall at dawn tomorrow, and I'll expect a thorough discussion of our plan, including the details of our intelligence gathering, device development, alliance formation, and evacuation contingency.

In the meantime, I'd like to assign a few additional tasks to our team. I want our scouts to investigate the dragon's lair and gather more information on its habits and weaknesses. I'd also like our engineers to work on a prototype of the device to counter the dragon's flames, and our emissaries to finalize the alliance with neighboring kingdoms.

Let us work together, my Knight, to ensure that our kingdom is prepared for the battle ahead. I have no doubt that with our combined efforts, we will emerge victorious and protect our people from the dragon's threat.

Now, let us take a moment to review the plan and make any final adjustments before the council meeting. I'd like to see a detailed outline of our strategy, including the roles and responsibilities of each official and the timeline for implementation.

Bring me the plan, my Knight, and let us review it together.### Creating a Graph-Based FSM Group Chat with AutoGen        

Looks good? Ya, fine, but there is an issue below

[autogen.oai.client: 08-04 14:35:53] {329} WARNING - Model meta/llama-3.1-8b-instruct is not found. The cost will be 0. In your config_list, add field {"price" : [prompt_price_per_1k, completion_token_price_per_1k]} for customized pricing.        

It looks like it is not using the llama-3.1–8b-instruct on Nvidia NIM, as configured. Then what is used? Edit the system message of main.py

from group_chat import manager
from agents.king import king

# Initialize conversations
king.initiate_chat(
    manager, message="Firstly, tell what LLM, version, and base URL you are using. How should we protect our kingdom from the dragon?"
)        

In the output, it shows “I am using the Llama 3, version 7, and the base URL is https://api-inference.huggingface.co/.”… weird? I am still trying to understand why ???????????? stay tuned… or if u know why, leave your comments below.

? python main.py
King (to chat_manager):

Firstly, tell what LLM, version, and base URL you are using. How should we protect our kingdom from the dragon?

--------------------------------------------------------------------------------
[autogen.oai.client: 08-04 15:22:10] {329} WARNING - Model meta/llama-3.1-8b-instruct is not found. The cost will be 0. In your config_list, add field {"price" : [prompt_price_per_1k, completion_token_price_per_1k]} for customized pricing.

Next speaker: Knight

[autogen.oai.client: 08-04 15:22:14] {329} WARNING - Model meta/llama-3.1-8b-instruct is not found. The cost will be 0. In your config_list, add field {"price" : [prompt_price_per_1k, completion_token_price_per_1k]} for customized pricing.
Knight (to chat_manager):

A noble quest indeed!

I am using the Llama 3, version 7, and the base URL is https://api-inference.huggingface.co/. I'm a large language model, my responses are generated based on patterns and associations in the data I was trained on, so I'll do my best to provide a strategic plan to defend our kingdom from the dragon.        

Conclusion

With this setup, I can simulate a structured conversation in a medieval kingdom using a graph-based FSM approach. Each agent will interact according to predefined rules, creating a dynamic and engaging dialogue. To run the group chat, simply execute main.py

There are a few mysteries mentioned below yet to be sorted out. The next step is to explore integrating more complex scenarios and additional agents with more advanced skills. Maybe adding the RouteLLM part in as well… It will be fun!

Happy coding and enjoy!

Update:

It looks like we need to create a client for Nvidia NIM in Autogen.

From: autogen/autogen/oai/client.py

    def _register_default_client(self, config: Dict[str, Any], openai_config: Dict[str, Any]) -> None:
        """Create a client with the given config to override openai_config,
        after removing extra kwargs.

        For Azure models/deployment names there's a convenience modification of model removing dots in
        the it's value (Azure deployment names can't have dots). I.e. if you have Azure deployment name
        "gpt-35-turbo" and define model "gpt-3.5-turbo" in the config the function will remove the dot
        from the name and create a client that connects to "gpt-35-turbo" Azure deployment.
        """
        openai_config = {**openai_config, **{k: v for k, v in config.items() if k in self.openai_kwargs}}
        api_type = config.get("api_type")
        model_client_cls_name = config.get("model_client_cls")
        if model_client_cls_name is not None:
            # a config for a custom client is set
            # adding placeholder until the register_model_client is called with the appropriate class
            self._clients.append(PlaceHolderClient(config))
            logger.info(
                f"Detected custom model client in config: {model_client_cls_name}, model client can not be used until register_model_client is called."
            )
            # TODO: logging for custom client
        else:
            if api_type is not None and api_type.startswith("azure"):
                self._configure_azure_openai(config, openai_config)
                client = AzureOpenAI(**openai_config)
                self._clients.append(OpenAIClient(client))
            elif api_type is not None and api_type.startswith("google"):
                if gemini_import_exception:
                    raise ImportError("Please install `google-generativeai` to use Google OpenAI API.")
                client = GeminiClient(**openai_config)
                self._clients.append(client)
            elif api_type is not None and api_type.startswith("anthropic"):
                if "api_key" not in config:
                    self._configure_openai_config_for_bedrock(config, openai_config)
                if anthropic_import_exception:
                    raise ImportError("Please install `anthropic` to use Anthropic API.")
                client = AnthropicClient(**openai_config)
                self._clients.append(client)
            elif api_type is not None and api_type.startswith("mistral"):
                if mistral_import_exception:
                    raise ImportError("Please install `mistralai` to use the Mistral.AI API.")
                client = MistralAIClient(**openai_config)
                self._clients.append(client)
            elif api_type is not None and api_type.startswith("together"):
                if together_import_exception:
                    raise ImportError("Please install `together` to use the Together.AI API.")
                client = TogetherClient(**openai_config)
                self._clients.append(client)
            elif api_type is not None and api_type.startswith("groq"):
                if groq_import_exception:
                    raise ImportError("Please install `groq` to use the Groq API.")
                client = GroqClient(**openai_config)
                self._clients.append(client)
            elif api_type is not None and api_type.startswith("cohere"):
                if cohere_import_exception:
                    raise ImportError("Please install `cohere` to use the Groq API.")
                client = CohereClient(**openai_config)
                self._clients.append(client)
            else:
                client = OpenAI(**openai_config)
                self._clients.append(OpenAIClient(client))

            if logging_enabled():
                log_new_client(client, self, openai_config)