Supercharging Semantic Kernel with AutoGen: Integrating the "best of both worlds" for Advanced AI Workflows

In the rapidly evolving landscape of artificial intelligence, combining powerful tools can unlock unprecedented capabilities. Today, we're exploring how integrating Semantic Kernel with AutoGen through a plugin can revolutionize the way we handle complex AI workflows. This integration not only streamlines processes but also amplifies the potential of both frameworks when used in tandem.

Introduction

Semantic Kernel is a versatile AI orchestration SDK that simplifies the development of complex Generative AI applications. AutoGen, on the other hand, is a framework designed for building agent workflows, allowing for more dynamic and flexible AI solutions. By integrating these two, we can create a system where multiple AI agents collaborate efficiently, enhancing both performance and scalability.

Imagine a scenario where you need to generate an article, have it reviewed by multiple experts simultaneously, and then receive a consolidated feedback summary. Traditionally, orchestrating such a workflow would be cumbersome. However, with Semantic Kernel and AutoGen working together, this process becomes streamlined and highly efficient.

The Technique Simplified

At the core of this integration is a Semantic Kernel plugin that wraps an AutoGen critic agent. This setup abstracts a complex workflow where:

1. An Article Writer Agent creates content (Semantic Kernel).
2. The content is passed through a Plugin to a Critic Wrapper Agent (AutoGen).
3. The Critic Agent delegates the article to four specialized reviewer agents: Ethics Reviewer, Legal Reviewer, SEO Reviewer, Style Checker (AutoGen)
4. Each reviewer provides feedback in parallel (fan-out).
5. A Meta Reviewer Agent aggregates the feedback (fan-in).
6. The consolidated feedback is returned to the Article Writer Agent for refinement.

This approach leverages the strengths of both Semantic Kernel and AutoGen. Semantic Kernel handles the orchestration and plugin management, while AutoGen manages the agent workflows and inter-agent communications.

Diving into the Code

Let's break down the key components of the code to understand how this integration works.

Setting Up the Semantic Kernel with AutoGen Plugin

We start by creating a static class SemanticKernelWithAutoGenPluginChatFactory that initializes the Semantic Kernel and imports necessary plugins.

public static class SemanticKernelWithAutoGenPluginChatFactory
{
    // Lazy Kernel initialization
    private static Kernel? _kernel;
    public static Kernel Kernel => _kernel ??= CreateKernel();

    // Create the Kernel lazily using the environment variables
    private static Kernel CreateKernel()
    {
        var builder = Kernel.CreateBuilder();
        builder.Services.AddSingleton<IFunctionInvocationFilter, SearchFunctionFilter>();

        Kernel kernel = builder.AddAzureOpenAIChatCompletion(
                            deploymentName: EnvironmentWellKnown.DeploymentName,
                            endpoint: EnvironmentWellKnown.Endpoint,
                            apiKey: EnvironmentWellKnown.ApiKey)
                        .Build();

        // Importing Bing search plugin and custom plugins
        BingConnector bing = new BingConnector(EnvironmentWellKnown.BingApiKey);
        kernel.ImportPluginFromObject(new WebSearchEnginePlugin(bing), "bing");
        KernelPlugin updateArticlePlugin = KernelPluginFactory.CreateFromType<UpdateArticle>();
        kernel.Plugins.Add(updateArticlePlugin);

        // Add the AutoGen plugin 
        KernelPlugin askForFeedbackAutoGen = KernelPluginFactory.CreateFromType<askForFeedbackAutoGenPlugin>();
        kernel.Plugins.Add(askForFeedbackAutoGen);

        return kernel;
    }
}        

Here, we're setting up the kernel with necessary plugins, including our custom askForFeedbackAutoGenPlugin which bridges Semantic Kernel with AutoGen.

Creating the Chat Agents

We define an ArticleWriterAgent that follows a structured process for writing and refining articles.

public static IAgentGroupChat CreateChat(int characterLimit = 2000, int maxIterations = 1)
{
    string projectDetails = Context.Facts;

    ChatCompletionAgent ArticleWriterAgent = new()
    {
        Instructions = $"""
            You are a writer. You write engaging and concise articles (with title) on given topics.
            You must polish your writing based on the feedback you receive and provide a refined version.
            Only return your final work without additional comments.
            Also you will always follow the same process when writing articles:
            1. Research using the bing plugin search engine on the topic (or topics) of the article.
            2. Write the article based on the research and the input.
            3. Ask for feedback on the article by using the AskForFeedback function in the askForFeedbackAutoGenPlugin plugin, providing the article.
            4. Update the article based on the feedback.
            5. Update the article by using UpdateArticle plugin and ask the user for feedback.
            6. Update the article based on the user's feedback.
            7. If the user is satisfied, you are done. If not, go back to step 3 unless the user asks for more research - then go back to step 1.
            """,

        Name = "CodeCrafterAgent",
        Kernel = Kernel,
        Arguments = new KernelArguments(
            new OpenAIPromptExecutionSettings()
            {
                ToolCallBehavior = ToolCallBehavior.AutoInvokeKernelFunctions
            }),
    };

    IAgentGroupChat chat = new AgentGroupChatExt(ArticleWriterAgent)
    {
        ExecutionSettings = new()
        {
            TerminationStrategy = new ApprovalTerminationStrategy()
            {
                Agents = [ArticleWriterAgent],
                MaximumIterations = maxIterations,
            }
        }
    };

    return chat;
}        

This agent uses the askForFeedbackAutoGenPlugin to request feedback, which is where AutoGen comes into play.

We create the rest of the agents like this, but no need to paste here all the code right? GitHub link is at the end ;)

The AutoGen Plugin

The askForFeedbackAutoGenPlugin is a Semantic Kernel plugin that invokes an AutoGen workflow.

public class askForFeedbackAutoGenPlugin
{
    [KernelFunction, Description("Performs a code review through different experts")]
    public async Task<string> AskForFeedback(string article)
    {
        string articleReview = await AutoGenChatWorkflow_AskForFeedback.Execute(article);

        return articleReview;
    }
}        

This plugin calls the AutoGenChatWorkflow_AskForFeedback.Execute(article) method, which orchestrates the review process using AutoGen.

Orchestrating the AutoGen Workflow

The AutoGenChatWorkflow_AskForFeedback class handles the execution of the review workflow.

public static class AutoGenChatWorkflow_AskForFeedback
{
    public static async Task<string> Execute(string article)
    {
        // Agent creation
        var CriticWrapperAgent = AutoGen_CriticWrapperAgent.CreateAgent(article);
        var EthicsReviewerAgent = AutoGen_EthicsReviewerAgent.CreateAgent();
        var LegalReviewerAgent = AutoGen_LegalReviewerAgent.CreateAgent();
        var SEOReviewerAgent = AutoGen_SEOReviewerAgent.CreateAgent();
        var StyleCheckerAgent = AutoGen_StyleCheckerAgent.CreateAgent();
        var MetaReviewerAgent = AutoGen_MetaReviewerAgent.CreateAgent();

        // Middleware creation
        var middleware = new NestedChatReviewerMiddleware(
            CriticWrapperAgent,
            EthicsReviewerAgent,
            LegalReviewerAgent,
            SEOReviewerAgent,
            StyleCheckerAgent,
            MetaReviewerAgent);

        // Register the middleware and setup message printing
        var middlewareAgent = CriticWrapperAgent
            .RegisterMiddleware(middleware)
            .RegisterPrintMessage();

        var message = new TextMessage(Role.User, $"The code to review is: {article}");

        IMessage reply = await middlewareAgent.GenerateReplyAsync([message]);

        return reply.GetContent();
    }
}        

This method sets up the agents and middleware, then initiates the review process.

The Middleware Handling Parallel Reviews

The NestedChatReviewerMiddleware is where the fan-out and fan-in logic resides.

public class NestedChatReviewerMiddleware : IMiddleware
{
    // Agents declarations...

    public NestedChatReviewerMiddleware(
        IAgent criticWrapperAgent,
        IAgent ethicsReviewerAgent,
        IAgent legalReviewerAgent,
        IAgent seoReviewerAgent,
        IAgent styleCheckerAgent,
        IAgent metaReviewerAgent)
    {
        // Assign agents...
    }

    public async Task<IMessage> InvokeAsync(
        MiddlewareContext context,
        IAgent critic,
        CancellationToken cancellationToken = default)
    {
        var messageToReview = context.Messages.Last();
        var reviewPrompt = $"""
            Review the following article:
            {messageToReview.GetContent()}
            """;

        // Initiate parallel review tasks
        var reviewTasks = new[]
        {
            critic.SendAsync(receiver: EthicsReviewerAgent, message: reviewPrompt, maxRound: 1).ToListAsync().AsTask(),
            critic.SendAsync(receiver: LegalReviewerAgent, message: reviewPrompt, maxRound: 1).ToListAsync().AsTask(),
            critic.SendAsync(receiver: SEOReviewerAgent, message: reviewPrompt, maxRound: 1).ToListAsync().AsTask(),
            critic.SendAsync(receiver: StyleCheckerAgent, message: reviewPrompt, maxRound: 1).ToListAsync().AsTask(),
        };

        // Await all review tasks in parallel
        await Task.WhenAll(reviewTasks);

        // Collect all reviews
        var allReviews = reviewTasks.SelectMany(task => task.Result);

        // Meta reviewer aggregates feedback
        var metaReview = await critic.SendAsync(
            receiver: MetaReviewerAgent,
            message: "Aggregate feedback from all reviewers and give final suggestions on the article.",
            chatHistory: allReviews,
            maxRound: 1)
            .ToListAsync();

        var lastReview = metaReview.Last();
        lastReview.From = critic.Name;

        // Return the summarized reviews
        return lastReview;
    }
}        

In this middleware, the reviews from different agents are processed in parallel (fan-out), then aggregated by the Meta Reviewer Agent (fan-in).

The Power of Combining Semantic Kernel and AutoGen

By integrating Semantic Kernel with AutoGen, we're able to create a sophisticated system where:

• Scalability: Multiple agents can process tasks simultaneously, improving efficiency.
• Modularity: Agents and plugins can be added or modified without disrupting the entire system.
• Abstraction: Complex workflows are encapsulated within plugins, simplifying the overall architecture.
• Flexibility: The system can adapt to different tasks by changing the agents or workflows involved.

This combination leverages the orchestration capabilities of Semantic Kernel with the dynamic agent workflows of AutoGen, resulting in a powerful and flexible AI solution.

The Code

The code can be found here: https://github.com/joslat/SemanticKernelAgenticChatWorkflowsExtension

If you like what you see please give it a star and a like to the article, as well as feel invited to reshare it in your network.

And if you read it until here - thank you!!

Conclusion

Integrating Semantic Kernel with AutoGen opens up new possibilities for handling complex AI tasks efficiently. The ability to delegate work to multiple specialized agents and then aggregate their feedback allows for more thorough and nuanced processing of information.

This integration not only streamlines workflows but also enhances the potential of AI applications by harnessing the strengths of both Semantic Kernel and AutoGen. Whether you're developing AI-driven content generation, complex data analysis, or any task requiring advanced orchestration, this approach can significantly boost your application's capabilities.

If you like what you read, I invite you to follow me on LinkedIn and Twitter, also have a YouTube channel but haven't had time to put too much on it, but happy to be motivated.

I'd love to have your feedback, suggestions and would definitely love having some motivation to put more content on my YouTube Channel :)