Python for Azure: Data Lifecycle Management Policy [Automated Access Tiering & Purge]

Get notified and stay updated, click on the image below to Subscribe/Follow 'Python for Azure' Publication ???

For complete information and code based solution implementation, please click on the Full Article & Solution Implementation

Introduction

Consider a scenario where data is frequently accessed during the early stages of the lifecycle, but later occasionally only after two weeks. However, beyond the first month, the data set is rarely accessed. In this scenario, hot storage is best during the early stages. Cool storage is most appropriate for occasional access. Archive storage is the best tier option after the data ages over a month. By moving data to the appropriate storage tier based on its age with lifecycle management (LCM) policy rules, you can design the least expensive solution for your needs.

Automatic Storage Access-Tiering based on Rule Definition

Each rule definition within a policy includes a filter set and an action set. The filter set limits rule actions to a certain set of objects within a container or objects names. The action set applies the tier or delete actions to the filtered set of objects.

Lifecycle management (LCM) supports tiering and deletion of current versions, previous versions, and blob snapshots.

Filters limit rule actions to a subset of blobs or specific container within the storage account. If more than one filter is defined, a logical AND runs on all filters. Actions are applied to the filtered blobs or containers when the run condition is met.

Points to Remember:

With such Lifecycle management policy rules, it enables the following:

• Transition current versions of a blob, previous versions of a blob, or blob snapshots to a cooler storage tier if these objects haven’t been accessed or modified for a period of time, to optimize for cost.
• In this scenario, the lifecycle management policy can move objects from hot to cool, from hot to archive, or from cool to archive.
• Delete current versions of a blob, previous versions of a blob, or blob snapshots at the end of their lifecycles.
• Define rules to be run once per day at the storage account level.
• Apply rules to containers or to a subset of blobs, using name prefixes or blob index tags as filters
• Each rule can have up to 10 case-sensitive prefixes and up to 10 blob index tag conditions.
• Only 200 ‘prefix_match’ based filter rules are available for ADLS Gen2 with Hierarchical Namespace (HNS) enabled i.e., only 200 specific blob container or blobs can be targerted to have particular Data LCM rules.
• However, otherwise a specific Data LCM rule can be set at the scope of the whole storage-account which inherits the same LCM rule for containers and blobs in the storage account

Key Observations

• If a data set needs to be readable, do not set a policy to move blobs to the archive tier. Blobs in the archive tier cannot be read unless they are first rehydrated, a process which may be time-consuming and expensive.?Full Details Here
• If you define more than one action on the same blob, lifecycle management applies the least expensive action to the blob. For example, action delete is cheaper than action tierToArchive. Action tierToArchive is cheaper than action tierToCool.
• Lifecycle management policies are free of charge. Customers are billed for standard operation costs for the Set Blob Tier API calls. Delete operations are free.
• However, other Azure services and utilities such as Microsoft Defender for Storage may charge for operations that are managed through a lifecycle policy.
• If you enable firewall rules for your storage account, lifecycle management requests may be blocked. You can unblock these requests by providing exceptions for trusted Microsoft services.

For complete information and code based solution implementation, please click on the Full Article & Solution Implementation