A Plea for Azure OpenAI Developers: Stop Using Keys ??

Let me explain with a poem…

Managed Service Identity, oh how I love thee! ????????

In the realm of security, you set my heart free.

No more secrets to keep, no keys to hide,

With you by my side, love and trust coincide.

In the shadows lurks the Azure OpenAI key,

A specter of fear, an authentication decree.

Evil whispers echo in the code's dark lair,

A key that haunts, a burden hard to bear.

No tokens to juggle, no passwords to bear,

You, my MSI, make everything fair.

In the Azure sky, our love takes flight,

Managed Service Identity, my beacon of light.

So let us wield DefaultAzureCredential's art,

A gateway back to MSI, a fresh start.

No more the fear of keys, in code's embrace,

A seamless return, where trust finds its place.

The Common Problem I See

Let’s get down to brass tacks ????

I know companies are moving fast with AI, but I hate seeing Azure OpenAI code samples (and real world code) littered with:

?? os.getenv("OPENAI_API_KEY")

?? Environment.GetEnvironmentVariable("AZURE_OPENAI_API_KEY")?

?? new AzureKeyCredential(key)

That means there are Azure OpenAI API keys floating around in your config files. ??

What’s the big deal??

??Are you sure your Azure OpenAI API isn't being used by former employees who took the key with them when they left??

?? Are you sure you haven’t committed the key into an Azure DevOps repo that a large group of developers can see??

?? Have you uploaded your proprietary data to Azure OpenAI and are you sure the key isn’t being used by the wrong people to ask questions about your data?

So how do you fix this??

A baby steps ?? solution would be to put the key in a secret in Azure Key Vault ?? then have your code retrieve the key before connecting to Azure OpenAI. This is a little better if you rotate ?? the Azure OpenAI key on a schedule programmatically. But possibly you have just traded an Azure OpenAI key for a SPN secret for connecting to Key Vault in your config file. That’s not much better.?

So how do we get away from secrets in config files entirely? ??

Managed Service Identity is a great solution to this! ????

How does that work? If your code is running in Azure, there likely is a Managed Identity you can use. For example, if your code is running in an Azure Function App, you should grant that managed identity the following RBAC role on your Azure OpenAI Services:?Cognitive Services OpenAI User?

This link walks you through a new preview way to add RBAC roles for MSIs. Or you can do it the traditional way.

(Or better yet do it in your infrastructure-as-code in Terraform or Azure Bicep or Pulumi.)

How do you change your code to use the MSI?

I wish MSI was more prominent in the Azure OpenAI docs. You have to dig in the docs and know that’s what is meant by #MicrosoftEntra authentication and DefaultAzureCredential.?

Here is a Python example. Notice this line that has DefaultAzureCredential():

from azure.identity import DefaultAzureCredential, get_bearer_token_provider
from openai import AzureOpenAI

token_provider = get_bearer_token_provider(DefaultAzureCredential(), "https://cognitiveservices.azure.com/.default")

api_version = "2023-12-01-preview"
endpoint = "https://my-resource.openai.azure.com"

client = AzureOpenAI(
    api_version=api_version,
    azure_endpoint=endpoint,
    azure_ad_token_provider=token_provider,
)        

Here is a C# example. Notice DefaultAzureCredential():

string endpoint = "https://myaccount.openai.azure.com/";
var client = new OpenAIClient(new Uri(endpoint), new DefaultAzureCredential());        

What's my bottom line advice?

?? Make sure your code review or security review checklist confirms developers are using MSI instead of Azure OpenAI key

?? Tell developers that the Azure OpenAI key will be rotated after the next production deployment of code so the code will break if they are still using the key. ??

?? Give developers the?Cognitive Services OpenAI User or?Cognitive Services OpenAI Contributor RBAC roles which can’t view the Azure OpenAI key (unlike the Contributor or?Cognitive Services Contributor roles).

?? For the most enterprise-grade use cases, consider whether putting Azure API Management (APIM) in front of Azure OpenAI makes sense.

?? Remember that authentication is only one layer in Defense in Depth. Since this post only focuses on authentication, I want to call out the other Defense In Depth layers and Zero Trust.

Thanks for reading all the way to the bottom! What other plea do you have for Azure OpenAI developers? Let’s discuss in the comments.?

?? Follow me for more on Azure AI security best practices and architecture.?