Agentic AI - What it is and how we can create some!

This article is based on my video https://youtu.be/UYJ539hgDS0. It was created based on the video transcript, edited with GPT-4o hosted in Azure AI Foundry and then given some human love after!

In this article I want to dive into agentic AI—what it is—and then look at some examples of how we create agentic AI using both low-code and pro-code solutions.

AI Agents

AI agents are everywhere today, and we often hear the term "agentic," but it’s not always clear what it actually means.

When I talk about AI agents, we’re dealing with various levels of functionality. At the core of all these agents is a generative model. Often, we think of a large language model, but it could also be a smaller language model. Today, there are some fascinating advancements in distillation, where a training set from a larger model is used to help a smaller model emulate much of the intelligence and behavior of the larger one. Regardless of size, the model is always a critical part of the solution.

Remember, the model is trained on a large corpus of data, giving it existing knowledge that it can use during interactions. The simplest level of interaction involves using the model as it is, without additional knowledge or external tasks. In this case, the model generates responses based solely on its pre-trained knowledge and the prompts it receives. However, this knowledge is limited. It only knows about the data it was trained on up to a specific cutoff date. It doesn’t know anything beyond that point, nor does it have access to private data, such as your emails or organizational information. Its knowledge is essentially a vast body of public information.

Because of these limitations, we often want the model to incorporate some element of retrieval and our next level of AI agent. This is where retrieval-augmented generation (RAG) comes into play. RAG allows the model to access additional information from an external body of knowledge and integrate it into its responses. For instance, this knowledge could reside in a SharePoint, a Dataverse, or a file you’ve provided.

Since we’re using natural language for interactions, we typically layer a semantic index over this data. This involves using an embedding model to create a vector—a mathematical representation of the semantic meaning of the data. These vectors have thousands of dimensions and allow the system to match meanings rather than exact words, which is crucial for natural language interactions. When you ask the model a question, it generates a vector for your query and compares it to the stored vectors, finding the closest matches in meaning. This ensures the most pertinent data is retrieved.

Beyond retrieval, there are scenarios where we want the AI to perform actions. These actions could take many forms, such as calling an API or interfacing with serverless capabilities. This functionality allows the AI to execute tasks, retrieve additional data, and incorporate it into its responses. At this stage, the AI becomes a task agent, capable of more than just generating responses or retrieving information.

Whether the AI is generating responses, retrieving knowledge, or performing tasks, there’s always a human in the loop.

Agentic AI

This brings us to the concept of agentic AI. Agentic AI can act autonomously and handle long-running, complex tasks. Unlike traditional workflows, which follow predefined steps, agentic AI dynamically plans how to achieve its goals. It uses reasoning to determine the optimal path to success based on the instructions it receives.

A key feature of agentic AI is its ability to create complex plans to achieve its objectives. It may call actions, retrieve additional knowledge, or even collaborate with other agents. Often, agents are designed to specialize in specific tasks, and one agent may rely on another to complete a larger goal.

While agentic AI can operate autonomously, it doesn’t necessarily exclude human interaction. For instance, in certain scenarios—such as those involving risk or significant resource expenditure—the AI might pause and seek human approval before proceeding. This variability in human involvement is an important consideration.

Another hallmark of agentic AI is its autonomy. It is typically triggered by an event, which could take many forms. For example, the AI might monitor your email for specific keywords, such as "YouTube," and then check if you have a relevant video, analyze its transcript, and provide a helpful response. Alternatively, it might detect a new record in a spreadsheet, such as a new hire, and automatically provision accounts or order equipment. The trigger could even be speech—perhaps it hears you mention travel plans and starts researching options—or a recurring schedule.

One attribute often associated with agentic AI is its ability to improve over time. However, this is not yet common. Current models are stateless and have no memory. Any history or additional knowledge they appear to have is actually provided to them as part of the prompt. While it’s possible to store state externally—for instance, in a Cosmos DB database—this is separate from the model itself. The AI might query this external store to recall previous interactions or preferences, such as your seating preference on flights or your dislike of salad when booking restaurants. This gives the illusion of improvement over time, but the model itself remains unchanged.

In the longer term, we might see advancements like reinforcement learning, where feedback on good or bad outcomes is used to fine-tune future models. However, this is not a common practice today.

To summarize, agentic AI is a subset of AI agents. All agentic agents are AI agents, but not all AI agents are agentic. Think of it like this: all elephants are gray, but not all gray things are elephants.

Many Expert Agents

When designing agentic AI, it’s common to create many specialized or "expert" agents, each focused on a specific task. This approach reduces the likelihood of hallucination, which can occur when an agent’s scope is too broad. Instead of one massive, general-purpose agent, we build multiple expert agents, each excelling in its designated area. This modular approach is why you often hear people talk about having hundreds of agents. While it might seem excessive, this specialization ensures accuracy and effectiveness in handling diverse activities. For very complex tasks an expert agent may call one or more other agents, it has that ability to decide that is way to solve the task. There could be a whole set of different interactions among them. Often, we’ll see the idea of many agents working together. For example, in a multi-agent scenario, one of them might be very creative, while another might act as a critic, checking the work. So, I have an author-critic, a maker-checker dynamic.

The more autonomous the system is—the more I’m taking the human out of the interaction—the more confident I want to be in the plans it’s making and the output it’s performing. To ensure this, we can have a different agent validate what another agent is doing. It’s going to be very common to have multi-agent systems working together in a more complex environment. I may have an orchestrator agent that has the primary mission, the prime knowledge, and the actions. This orchestrator can then call other agents and trigger them based on what is needed.

Back to Basics

Now, even though this is a generative AI, all your normal considerations still apply. You must ensure these agents are healthy and behave as you want. Every single component would still undergo full testing. You’d conduct unit testing to ensure the right parameters are in place—that’s more of a pass/fail type of testing. You’d also test the agents with different types of inputs and conduct evaluations, often using a large language model as the judge. I discuss this in my Azure AI Foundry video, where I explain how to determine whether the output aligns with expectations.

We can still have full logging. I can use tracing from Azure AI Foundry by simply adding the tracer decorator to almost anything. This allows for detailed tracing, which is all stored in Log Analytics via Application Insights. This way, I can get a complete view and interaction with all these wonderful things we’re seeing in action.

Creating an Agentic AI

What I wanted to do here is actually demonstrate what these agentic, autonomous systems with complex planning capabilities look like. I thought it would be fun to compare a low-code solution with a pro-code solution. For the low-code solution, I’ll use Copilot Studio and for pro-code Semantic Kernel. To get the most of this you really should watch the video where I walk through it but I'll cover the key aspects here.

For Copilot Studio there are two critical elements we will use in our new agent.

• Under orchestration: I’ve enabled the use of generative AI to determine how best to respond to users and events. One absolute requirement for mapping to this idea of a complex plan is enabling generative orchestration. This allows the agent to decide how to handle certain tasks, such as where to place certain values or how to use specific capabilities.
• Now, the autonomous part: We add triggers. For an agent to be autonomous, it must have events to trigger its actions. For example, an event might occur, and the agent will work out what it should do to solve the problem on its own. There’s a wide variety of triggers available. If we look at the triggers, there are featured ones like recurrence, a new response submitted on a form, an item created on SharePoint, a file modified, an RSS feed update, a new channel message added, an email arriving in Gmail, or a data-driven alert triggered. There are many options.

That’s our low-code way of creating an agentic AI. What about the pro-code approach? There are many ways to do this. You could write everything yourself, but there are also solutions like Semantic Kernel. Semantic Kernel is Microsoft’s enterprise-ready orchestration engine, which I often leverage for AI endeavors.

Semantic Kernel provides connectors. These connectors handle various aspects, such as AI providers that interact with different models and services, memory for accessing vector stores, and functions that can be called. Semantic Kernel abstracts the specific details, making it easier to work with different components of an AI solution. It’s an open-source solution from Microsoft, available for .NET, Java, and Python. It’s designed for enterprise developers and is production-ready. One of its standout features is its specific capabilities for agents.

At a basic level, you might just have a model tied to an AI provider. If all you want to do is work with a model, Semantic Kernel offers a chat completion-based agent. However, if you need additional state storage, other types of storage, or tooling, you would need to plumb in these elements. To integrate all those components natively, you would need another type of agent. This agent includes both the model and built-in state, storage and tooling, and it is called the assistant agent. Depending on the type of agent interaction you require, you can choose between the chat completion agent for basic interactions or the assistant agent for more complex tasks. If you are working on something beyond simple chat interactions—such as interpreting code or completing a specific workflow—the assistant agent is designed to handle those tasks for you.

Another concept to consider is the multi-agent solution. In the semantic kernel, this involves creating individual expert agents (Agent A, Agent B, Agent C, etc.) and then combining them in what is referred to as a group chat. This group chat allows the agents to collaborate and converge on a solution to a given problem. While the terminology might seem unusual, the idea is straightforward: the agents work together to achieve a shared goal.

When implementing a multi-agent solution, several factors must be considered. For instance, how do the agents know when they have completed their task? Large language models are excellent at communicating with one another, but they need a termination strategy to determine when their goal has been met. Without such a strategy, the agents might continue interacting indefinitely. To prevent this, you can set a maximum number of iterations—for example, ten attempts—after which the process is deemed unsuccessful. Additionally, you might use a coordinator agent to outline the workflow and call the specialized agents as needed.

The multi-agent capabilities of the semantic kernel are particularly powerful because they allow for autonomous operation. Once triggered—whether by a human, another agent, or a scheduled event—the agents can execute complex plans without constant human intervention. However, it is crucial to establish proper guardrails. These include defining the tasks, specifying the knowledge and actions the agents can access, and ensuring appropriate permissions and protections are in place.

A great example of a multi-agent for semantic kernel can be seen at https://github.com/microsoft/semantic-kernel/blob/main/python/samples/getting_started_with_agents/chat_completion/step5_chat_completion_agent_group_chat.py (which is the example I walk through in the video).

Closing Thoughts

Data security is another critical consideration. Many organizations struggle with discovering, classifying, and protecting their data. While this is not an AI-specific issue, it becomes more pressing when using AI systems. Proper data governance ensures that only authorized individuals can access sensitive information. Additionally, any actions performed by the agents will still adhere to existing data loss prevention policies and other security measures.

I hope this overview was helpful. Until next time, take care! ??