Steps for Migrating Data from Azure to AWS S3 using Rclone:
D.C. Teja ??
4x AWS Certified | Multi Cloud Engineer | Cloud Security | Migrations |Terraform | Cybersecurity | Ex-Rapyder Cloud solutions | Thread Detection | EDR | Email Security | Firewalls |
Step 1: Go to your Azure Account and run the following command in the Azure Cloud Shell. Change the scope of the command by replacing the placeholders with your actual Subscription ID, Resource Group Name, and Storage Account Name:
```bash
az ad sp create-for-rbac --name AWS-Rclone-Reader --role "Storage Blob Data Reader" --scopes /subscriptions/{Subscription ID}/resourceGroups/{Resource Group Name}/providers/Microsoft.Storage/storageAccounts/{Storage Account Name}
```
After running the command, you will receive credentials like the following:
```json
{
?"appId": "dffc217a-c2af-41b8-b120-f8819d8999a4",
?"displayName": "AWS-Rclone-Reader",
?"password": "B2E8Q~dAch55gCB5Rr4JafA.dcQwmfHbXW571ctI",
?"tenant": "ee873d89-a26e-47cf-8019-b65d6ee4d46c"
}
```
Copy these credentials as they will be needed later.
Step 2: Create an S3 bucket policy and attach it to the EC2 Instance Role. Edit the policy to replace "BUCKET_NAME" with your S3 bucket name.
Step 3: Launch an EC2 instance with a t2.medium instance type (public), and attach the previously created role to the EC2 instance. Install Rclone on the EC2 instance using the command `yum install rclone`. Check the Linux directory using the command `pwd`.
Step 4: Go to your home directory and create a file named `azure-principal.json` using the command `vi ~/azure-principal.json`. Paste the Azure credentials received in Step 1 into this file.
Step 5: Create a directory for Rclone configuration using the command `mkdir -p ~/.config/rclone/`.
Step 6: Create a file named `rclone.conf` in the `~/.config/rclone/` directory using the command `vi ~/.config/rclone/rclone.conf`. Paste the following commands into the file, replacing "AZStorageAccount" with your desired name for the Azure storage account:
```ini
[AZStorageAccount]
type = azureblob
领英推荐
account = dcmpod2
service_principal_file = azure-principal.json
[s3]
type = s3
provider = AWS
env_auth = true
region = us-east-1
```
Step 7: Check the connectivity between Azure and AWS using Rclone. Run the command `rclone listremotes` to see the available remotes. Use `rclone lsd s3` to list the AWS buckets.
Step 8: Copy the data from the Azure storage account to the AWS S3 bucket using the following command:
```bash
rclone copy AZStorageAccount:data-disk s3:data-disk
```
Step 9: Check the files in the AWS S3 bucket to ensure that the data has been successfully migrated.
Step 10: If any files are missing, use the sync command to ensure that the data in the destination matches the source:
```bash
rclone sync AZStorageAccount:blob-container s3:examplebucket-01
```
Important: When using the sync command, data that is not present in the source container will be deleted from the destination S3 bucket.
For more detailed information, you can refer to the AWS documentation at the following link: https://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/migrate-data-from-microsoft-azure-blob-to-amazon-s3-by-using-rclone.html