Two major Azure SDK Python classes for automating the azure resource deployments

#dataengineering #python #azure #azurecloud #azuredevops #infrastructureascode

Infrastructure as Code (Iac) is a major component of CI/CD?for any data engineering platforms.

Iac is nothing but automation code for deploying resources programmatically and thus creating a deployment pipeline which can create required resources?at runtime for specific application.

?

In Azure , the creation and life cycle of any resources are handled by Azure Resource Manager(ARM).ARM provides something called "ARM Templates"?for automating infrastructure deployments.

Azure Resource Manager (ARM) templates are a declarative syntax used to describe the infrastructure and configuration of Azure resources. ARM templates are written in JSON (JavaScript Object Notation) format and are used to deploy and manage Azure resources as a single, unified group.

?With ARM templates, you can define the infrastructure and configuration of your Azure resources, including virtual machines, storage accounts, networking resources, and other Azure services as a JSON file. You can also define dependencies between resources, set parameter values, and apply policies for governance and compliance.

One can deploy the templates using various tools, including the Azure portal, Azure CLI, Azure PowerShell and Azure SDK.

?When it comes to Azure SDK in Python for this purpose , the entry point is always creating a resource manager client with all privileges for creating and managing resources. For creating a resource manager client , you need to get authorization token from Active Directory (aka AD). And for getting authorization token , the client should either be a service principal or?an application with azure managed identity enabled

If you are new to these terms , please see my earlier post on the same: https://tinyurl.com/234xpj45

?The catch here is , authentication to access any resource by AD requires different inputs depending up on if the client is a managed identity , service principal or an AD group member.

So while implementing a resource manager client for all these use cases, one way is to place checks for all these use cases and write a verbose function or method so that the code will run on all environments based on client type.

?But azure-identity?package provides a special class called "DefaultAzureCredential"?that abstract all these complexities. Once you got the credential from AD , you can easily pass that to ResourceManagementClient class from azure-mgmt-resource package . And using this resource manager client , you can manage your infrastructure deployments using ARM templates.

DefaultAzureCredential

?Once you install azure-identity library from pip , you can get the token from AD using just one line of code?

from azure.identity import DefaultAzureCredentia
credential = DefaultAzureCredential()        

Wondering how this cover all the use cases we have discussed?

What happening is that It abstracts away the complexity of authentication and allows your code to work seamlessly across different environments (e.g., local development, cloud VMs, containers, etc.) without the need to modify your authentication code.

DefaultAzureCredential ?simplifies authentication by trying a series of authentication methods in a specific order. It first tries to use credentials from environment variables, then from a managed identity, and finally from an interactive login.

The order in which the authentication methods are tried is:

1. Environment variables: First the credential object looks for the following environment variables: AZURE_TENANT_ID, AZURE_CLIENT_ID, AZURE_CLIENT_SECRET, and AZURE_SUBSCRIPTION_ID. If these variables are set, the credential object uses them to authenticate --> So if you have a service principal created , then set these environmental variables using os.environ and then by just calling DefaultAzureCredential(), you will get the credential.
2. Managed identity: If no environment variables are set, the credential object looks for a managed identity associated with the resource. If a managed identity is found, the credential object uses it to authenticate.
3. Interactive login: If no environment variables are set and no managed identity is found, the credential object prompts the user to log in interactively. The user enters their credentials, and the credential object uses them to authenticate.

ResourceManagementClient

The ResourceManagementClient is a class provided by the Azure SDK for Python that allows you to manage Azure resources. Specifically, it is part of the azure-mgmt-resource package, which is a set of libraries that provide a simple and consistent way to manage Azure resources across various Azure services.

The ResourceManagementClient class provides methods for creating, updating, and deleting resources in an Azure subscription, as well as for listing and retrieving information about existing resources. You can use it to manage a wide variety of resources in Azure, including virtual machines, storage accounts, virtual networks, web apps, and more.

Below is an example where we are :

• Creating a resource manager client for a subscription using ResourceManagementClient with credentials from AD fetched via service principal credentials using DefaultAzureCredential
• Then listing existing resource groups under subscription
• Then creating new resource group using resource manager client

Creating a resource manager client for a subscription and Listing existing resource groups under subscription

Existing resource groups under subscription from portal UI:

Creating new resource group via resource manager client and listing the updated resource group list

Updated resource group list from portal UI: