How to Copy Azure Blobs from one Storage Account to Another Storage Account

Introduction

There are several ways of copying/move blobs from one storage account to another storage account but here i am going to discuss how we can copy blobs using AzCopy utility through DevOps pipeline.

1. Azure CLI
2. AzCopy utility
3. .NET Storage Client library

AzCopy Utility

The AzCopy utility was written specifically for transferring data into, out of, and between Azure Storage accounts.

A key strength of AzCopy over the Azure CLI is that all operations run asynchronously, and they're recoverable. The AzCopy command tracks the progress of copy operations, and if an operation fails, it can be restarted close to the point of failure. Additionally, you can tune the performance of the AzCopy command to match the processing power and bandwidth available to your local machine.

Steps what we have to perform

1. Generate SAS token on both source and destination storage account

Here is small caveat that Azure provides an option to create SAS token at Storage Account/Container/Blobs level. So, form where should we create SAS token?

We have to create SAS token at Storage Account level only then only we can provide proper permissions, without proper missions we can't move blobs.

Correct SAS token
https://mysourceaccount.blob.core.windows.net/mycontainer?sv=2018-03-28&ss=bfqt&srt=sco&sp=rwdlacup&se=2019-07-04T05:30:08Z&st=2019-07-03T21:30:08Z&spr=https&sig=CAfhgnc9gdGktvB=ska7bAiqIddM845yiyFwdMH481QA8%3D        

If you will create SAS token at Container/Blob level then you will not get required parameters in SAS token which is required.

Incorrect SAS token
https://mysourceaccount.blob.core.windows.net/failed?sp=r&st=2022-06-03T07:54:00Z&se=2022-06-03T15:54:00Z&spr=https&sv=2020-08-04&sr=c&sig=wURGbGEfoOpTBpskR7XK4KMsu6SzNuHr1fpXMHs4Poo%3D        

2. Create an empty YAML pipeline like below

trigger: none
?
pool:
??vmImage: 'ubuntu-latest'
?
steps: 
- task: AzureCLI@2 
  displayName: 'Archive blob-1' 
  inputs: 
    azureSubscription: my-dev-subscription
    scriptType: ps 
    scriptLocation: inlineScript 
    inlineScript: 'azcopy copy "https://mysourceaccount.blob.core.windows.net/sample?sv=2020-08-04&ss=bfqt&srt=co&sp=rltfx&se=2022-06-01T14:04:01Z&st=2022-06-01T06:04:01Z&spr=https&sig=WX%2F13QUJHuhYb6UxMl8BEuNRqKMly0%2FvNw8lGHyK41U%3D" "https://mydestinationaccount.blob.core.windows.net/sample?sv=2020-08-04&ss=bfqt&srt=co&sp=rwdlacupitfx&se=2022-06-01T14:05:10Z&st=2022-06-01T06:05:10Z&spr=https&sig=nHDVAGma29VYmh4fRqOgy0MZyZZzj1MnojcZ3o47ZCE%3D" --recursive' 
    enabled: false 
                                

3. Run pipeline as per your requirement and you can also add n number of task for n number of blobs in a single pipeline.