Access Azure Key Vault secrets from MS Fabric Notebooks

Below are steps on how to fetch and use secrets from Azure Key Vault in a Microsoft Fabric environment. We will try to understand:

1. Why use Key Vault in Fabric.
2. Method 1: Using a Service Principal (client secret).
3. Method 2: Using User Credentials (interactive).
4. Current Workspace Identity functionality and limitations.
5. Final Thoughts and trade-offs.

Why Use Azure Key Vault in Microsoft Fabric?

Azure Key Vault is a secure store for secrets, keys, and certificates. In a Fabric:

• You often need to authenticate to external services (databases, storages, APIs) without embedding credentials in code.
• Storing secrets in Key Vault helps maintain centralized and secure control of sensitive data.

Method 1: Using a Service Principal (Client Secret)

Create an Azure Key Vault

Below is an Azure CLI example for creating a Key Vault. Adjust --name, --resource-group, and --location to your needs:

az keyvault create --name <unique-keyvault-name> \
  --resource-group <resource-group-name> \
  --location <azure-region>        

Store a Secret in Key Vault

Use the az keyvault secret set command to add a secret. For example:

az keyvault secret set --vault-name <unique-keyvault-name> \
  --name AZURE-AI-SERVICES-KEY \
  --value "XXXXXXXXXXXXXXXXXXXXXXXXXXXX"        

Create a Service Principal (SP) and Client Secret

Create the Service Principal

az ad sp create-for-rbac --name "<YourSPName>"        

This command will output several important values:

• appId (the client_id)
• tenant (the tenant_id)
• password (the client_secret)

Grant Key Vault Access to the SP In the Azure Portal (or via CLI), give the SP appropriate roles in Key Vault. Common roles:

• Key Vault Administrator (for managing secrets/keys)
• Key Vault Secrets User (for reading secrets)

Command approach below( or you might set an Access Control directly in the Key Vault settings if you’re using the older model)

az role assignment create \
  --role "Key Vault Secrets User" \
  --assignee <appId-of-SP> \
  --scope /subscriptions/<subId>/resourceGroups/<rgName>/providers/Microsoft.KeyVault/vaults/<unique-keyvault-name>        

Use the Service Principal in Fabric

In your Fabric notebook, install the required Python packages (if needed):

%pip install azure-identity azure-keyvault-secrets        

Then authenticate using the ClientSecretCredential and retrieve your secret:

from azure.identity import ClientSecretCredential
from azure.keyvault.secrets import SecretClient

# Replace these values with your tenant/client details
tenant_id = ""
client_id = ""
client_secret = ""  # place the secret here (or read from a secure source)

credential = ClientSecretCredential(tenant_id, client_id, client_secret)
vault_url = "https://fabric-poc.vault.azure.net/"

secret_client = SecretClient(vault_url=vault_url, credential=credential)
secret = secret_client.get_secret("fabric-secret-name")

print("Fetched secret:", secret.value)        

Pros:

• Uses a “headless” identity (the SP) instead of your personal user credentials.
• Security can be centrally managed by controlling SP’s permissions in Key Vault.

Cons:

• You must manage a client secret. If you accidentally embed it in code, that can be a risk. Try to inject this secret at runtime (e.g., via environment variables or external orchestration) instead of hard-coding.

Method 2: Using User Credentials (Interactive)

Because Fabric notebooks by default run under your user identity, you can leverage that token to fetch secrets from Key Vault. The snippet below uses the notebookutils.credentials.getSecret approach (the exact name may differ slightly depending on your Fabric environment):

key_vault_uri = 'https://fabric-poc.vault.azure.net/'
secret_name = 'fabric-secret-name'

secret = notebookutils.credentials.getSecret(key_vault_uri, secret_name)
print("Fetched password from key vault is", secret)        

How it works:

• The Fabric session uses your interactive user token behind the scenes.
• If your user account has “Get Secret” permission on Key Vault, this call succeeds.

Pros:

• No separate service principal or client secret needed.
• Quick to set up if you already have user permissions in Key Vault.

Cons:

• Ties access to your personal credentials. If you leave the organization or lose permissions, the pipeline breaks.
• Not suitable for a production scenario where you want a “headless” identity.

Workspace Identity in Fabric: Current State & Limitations

In Azure, you could choose to run notebooks or pipelines as the system-assigned managed identity. However, Microsoft Fabric (in preview) does not yet expose that feature. Today:

• No “Run as Workspace Identity” toggle in pipelines or notebooks.
• Even if your Fabric workspace has a managed identity in Azure AD, there is no built-in method to use ManagedIdentityCredential() for notebook execution.
• Attempting ManagedIdentityCredential() in a Fabric notebook typically fails because the underlying environment does not expose the IMDS (Instance Metadata Service) endpoint.

Expected Future:

• Microsoft is working to bring feature parity from Azure Synapse to Fabric.
• Eventually, you should be able to run a notebook as the workspace identity, removing the need for client secrets entirely.

Final Thoughts and Best Practices

1. For Production: Use a Service Principal with strict permissions on Key Vault. Avoid hard-coding secrets in your notebook code. Instead, use environment variables or a secure parameter approach (once Fabric supports secure pipeline parameters) to pass in the client secret at runtime.
2. For Quick Tests or Demos: Using user credentials can be faster. Just ensure your user has Key Vault “Get Secret” permission. Keep in mind that you’re tying the solution to a single user account.
3. Looking Ahead: Workspace Managed Identity in Fabric will eventually simplify the setup. Once available, you won’t need a client secret or user-based authentication. Instead, you’ll just assign Key Vault permissions to the Fabric workspace identity. Currently, it supports only Storage accounts authentication.
4. Trade-offs: Service Principal: +Production-friendly, +headless, ?must manage secret. User Credentials: +No extra setup, ?tied to personal account. Future Workspace Identity: +Best of both worlds, +no secrets to manage, ?not yet supported in preview.

Let me know your thoughts on best approach :)