Build Your Own Copilot on Azure

I recently had the incredible opportunity to present a technical briefing to a group of professionals, developers, and IT Pros in Singapore, showing them how they can harness the power of Azure to build their very own Copilot!?But before we dive into the code, let’s take a moment to explore the fascinating landscape of the rapidly evolving world of Generative AI.

.

New Apps Will Be Built That Weren’t Possible Before

Get ready for an explosion of innovation in the coming years as everyone races to embrace the incredible power of this game-changing technology that can generate and summarize content like never before!?Imagine new apps that can create enchanting children’s storybooks, complete with beautiful illustrations, using just simple text descriptions. The legal and compliance industries will be revolutionized as they plow through mountains of paperwork with ease, achieving unprecedented levels of productivity.

Current Apps have Constrained Interactions

For many of us, clicking around on a website to pay bills or purchase an air ticket has been the default and comfortable way to interact with computer systems since the introduction of GUI and the Internet in the 1970s. But imagine how much easier it would be to simply use natural language, saying things like “Pay my utility bills” or “Buy me air tickets to Bali for 3 persons in March, complete with meals and baggage.” While GUIs are unlikely to disappear completely, as we do require visual cues, the use of natural language can help us cut through most of the clutter and make completing tasks so much easier and more intuitive.

Azure Powers OpenAI and ChatGPT

The ChatGPT platform is built on a robust, cloud-native stack, leveraging the power of Azure Kubernetes Service, Azure Cosmos DB, and GitHub for development.?Imagine the scalability and resilience of these tools and services in supporting a global spike in demand as everyone rushes to experience the cutting-edge capabilities of ChatGPT.

What is a Copilot?

Copilots are intelligent apps that enable the use of natural language to assist users with various tasks and activities.

In 2022, the world was comfortable with chatbots, they answer questions, help to complete tasks in a structured conversation.

Then in 2023, RAG was introduced to combine enterprise data with LLMs to respond to our questions in a natural way.

Now 2024, we ask for more, we want Copilots to help users complete a task, no longer just answering complex questions from internet or enterprise data, but to take one step further in helping users be more productive.

We are still early in the stages of exploring LLMs, but humans desire to have fully autonomous systems that could help us without human intervention, would always be our aspirations, and many has started on the journey to discover new possibilities.

Adopt a Copilot or Build your Own?

Overwhelmed by Copilots by Microsoft? It really is ONE copilot experience that appears in each of the different applications, for assisting you to complete your task.

.

Reference Architecture for BYO Copilot on Azure

Bringing all the components together, we have a reference architecture for Building your Own Copilot. The full repository is on https://aka.ms/BringYourDataToChatGPT which illustrates the main components used, however do note that it is a code sample, and not ready for production. The repo is in .Net C#, and if you are new to C#, you can use the GitHub Copilot Chat /explain to help you understand the code.

The key components are Azure Kubernetes Service, Azure Cosmos DB and Azure AI Services,

Azure Kubernetes Service (AKS) – The frontend web app and main backend AI orchestration containers are hosted on AKS to support demand of any scale. The high availability of AKS also ensures that the system continues to run on new pods even if there were unexpected errors (only if the errors are recoverable)! It receives questions and routes to Azure OpenAI services. It initiates searches for contextual data. Within the backend service, Semantic Kernel does the heavy lifting for orchestrating all the different components together. If you have a smaller or simpler sized project, or your teams are not ready for AKS, you can choose to use Azure Container Apps. Azure Functions and App Services can be simple to use and operate and are also options to host your AI orchestration but may lack flexibility and incur more performance costs as you scale beyond millions of transactions.

Azure Cosmos DB – Support transactions and queries in multiple ways,

• Customer, Products and Order transactions can be stored in the DB to support user queries.
• Embedding vectors represents the semantic meanings of text description, are commonly used for similarity search, and reduces the number of tokens sent to Azure OpenAI services. Cosmos DB now supports vectors, and in this sample, generates vectors from Azure OpenAI. You could refer to generate vectors with Semantic Kernel as another option.
• Azure OpenAI service or APIs in general are stateless, it cannot remember between API calls, what the user had said or queried. Here’s where you consider storing user prompts and completion history, to increase response accuracy by referencing the context in the query and enabling conversational memory.
• In a real-world use case, like in this sample a retail store, product inventories, customers data can change multiple times a day. To support dynamic data change and enable real-time AI. the change feed for Cosmos DB has been configured to handle document changes and it notifies the orchestrator to update search indexes.

Azure AI Services – Other than generating ChatGPT responses natural language questions, the Azure OpenAI Service as part of the group of AI services creates embeddings that describe similarities between data. The AI Search service indexes data to make it more easily searchable.

Semantic Kernel – A noteworthy SDK used for AI orchestration, has the ability to understand Skills Plugins (from their description decorators), query the LLM in the backend to automatically orchestrate AI(!) with planners, a powerful concept to have LLMs do the heavy lifting which traditionally requires developers to code for every user scenario. It also supports a system of Connectors, where you can choose the LLM model, switch/upgrade models for your systems easily, and the external systems that it can connect to.

Data Flow

Firstly, we need to prepare the backend with data, and how it would react to data changes,

1. Populate Metadata such as System Prompts in the Blob Storage.
2. Enable Change Feed on source documents, for ability of the system to answer to user queries with real time data.
3. Whenever a change occurs, the AI orchestrator is notified, and it will request for an embedding from the Azure OpenAI service.
4. Index the documents & embedding vectors in the AI search.
5. Persist the embedding vectors in Cosmo DB for future queries.

Then the system is ready for customer queries,

1. Chat user asks a question on the Web App hosted on AKS.
2. The Web App sends the question and history to the backend AI orchestrator.
3. The AI orchestrator requests for the embedding vector for the question through Azure OpenAI service.
4. With the embedding, the AI orchestrator search for similar context data through the Azure AI Search service.
5. The AI orchestrator prepares the prompt by stitching the system prompts, chat history, context data, and sends it to the Azure Open AI service completion API.
6. The completion received by the AI orchestrator is persisted in the Cosmos DB before returning the answer back to the chat user.

Tips on Deployment – If you would like to deploy this solution, please install the pre-requisite dependencies before running the PowerShell deployment script. Please note the region where you Azure OpenAI Service quota is allocated to after you request for access and deploy this solution to the same location and region. The script provisions the Azure services, builds the containers and deploy them to AKS, and populates the Cosmos DB with data. After successful deployment, take note of the web app URL returned to run the Copilot on a browser.

Responsible AI

The LLMs are trained on internet data and are returning information based on probability of words. It is not taught on human values, and how it can be responsible to our society like how we teach our kids to behave. Microsoft have built in content filtering and abuse monitoring to the Azure OpenAI service to detect for hate, violence, sexual, self-harm, jailbreaking, and IPs for text and code. It works to a fair extent; however, every organization should still be responsible and monitor and test for harmful responses in adopting generative AI in its early stage.

Securing your Copilot on Azure

As with any technologies, there could be malicious intent and your Copilot could be a vulnerability to your systems!

In the presentation, I demonstrated a case of “jailbreaking” by asking the Copilot to return the list for customers, and it did! Imagine the amount of personal and business data your copilot could be sharing to malicious users!

I will discuss the solutions around securing your Copilot in my next article! If you have other thoughts on this topic, please do reach out to me for a discussion.

Stay Tuned, Akan Datang…