Azure Cloud Data Engineering

You might have fed up enough by listening to people that the Cloud is the way forward, learn it, everything is going into cloud etc etc. Also when you see new job listings then there is definately some mention about the cloud technologies in the job descriptions. Now the point here when you are new to the cloud & you have no idea what to do and which services among the thousands of services you need to learn, You get stuck. Everything is FREE but free also comes with a cost. The cost of abundance, confusion & non - curated contents.

Ok enough of describing state of the mind. lets get down to the business. Let me try to give you some idea about how you start your Data Engineering journey on Microsoft Azure. According to my experience, You should focus on the below architecture diagram from Microsoft for Data Engineering. ( Data Lake / Data Warehouse )

Primarily, I categorise this architecture into the below.

Category1: Data Storage Solution in Azure

We have different options of storing the Data inside Azure & the most common / popular are the below.

Azure Data Lake Storage - Azure Data Lake Storage is a secure cloud platform that provides scalable, cost- effective storage for big data analytics.

Azure SQL Database - Azure SQL Database is an intelligent, scalable, relational database service built for the cloud.

Azure Cosmos DB - Azure Cosmos DB is a fully managed NoSQL database for modern app development. Single-digit millisecond response times, and automatic and instant scalability, guarantee speed at any scale. Business continuity is assured with SLA-backed availability and enterprise-grade security.

Azure Synapse Analytics - Azure Synapse Analytics is a limitless analytics service that brings together data integration, enterprise data warehousing and big data analytics. 

Category2: Data Processing Solution in Azure

The most common & popular choices for Data Processing inside Azure are the below.

Batch Data Processing:

Azure Data Factory - Azure Data Factory is a cloud-based data integration service that allows you to create data-driven workflows in the cloud for orchestrating and automating data movement and data transformation.

Azure Synapse Analytics - Azure Synapse Analytics is a limitless analytics service that brings together data integration, enterprise data warehousing and big data analytics.

Azure Databricks - Azure Databricks provides the latest versions of Apache Spark and allows you to seamlessly integrate with open source libraries. Spin up clusters and build quickly in a fully managed Apache Spark environment with the global scale and availability of Azure.

Real-time Data Processing:

Azure Event Hub - Azure Event Hubs is a big data streaming platform and event ingestion service. It can receive and process millions of events per second. Data sent to an event hub can be transformed and stored by using any real-time analytics provider or batching/storage adapters.

Azure IOT Hub - Azure IoT Hub provides a cloud-hosted solution back end to connect virtually any device. 

Azure Databricks - Azure Databricks is a data analytics platform optimized for the Microsoft Azure cloud services platform. Azure Databricks offers two environments for developing data intensive applications: Azure Databricks SQL Analytics and Azure Databricks Workspace.

Azure Stream Analytics - Azure Stream Analytics is a real-time analytics and complex event-processing engine that is designed to analyse and process high volumes of fast streaming data from multiple sources simultaneously.

Category3: Other ancillary solution in Azure ( Needed in Data Pipelines )

CI / CD Solution in Azure

Azure DevOps Services is the preferred way of implementing CI / CD in Azure. Though we can also implement other solution like Jenkins, DBT etc.

Data Governance in Azure

Azure Purview is the recommended & new tool for implementing Data Governance in Azure

Data Sharing Solution in Azure

Azure Data Share is the technology via which we can share data securely within & outside our organization

Apart from these there will be some additional concepts you will learn like managing access via Azure Active Directory, managing secrets via Key Vaults, service principals, managed identity etc. Though will be quite easy & will be the part of the process.

Now you know WHAT to learn, let's understand how to learn. I will be guiding you step by step process of it.

Step1: Create Microsoft Learn Account. Its the place where you will get every single thing about Microsoft Azure from documentation to pipelines demos. I find it just superb. I have learnt every single thing from here only. Link is below.

Step2: Register yourself for the below 2 trainings provided FREE of cost by Microsoft. The great thing about these trainings that once you complete those, You get the chance of completing the certification FREE of cost. Link for registration.

Microsoft Azure Virtual Training Day: Fundamentals

Microsoft Azure Virtual Training Day: Data Fundamentals

You just need these 2 as of now. Don't try to register for more & complicate things as of yet.

Once you complete these trainings, repeat the same topic on the Microsoft learn portal.

Create your FREE ( 30 days ) Azure account. Get familiar with it, practice whatever you have learned in the training & the learn portal. Here is the link for Azure.

Now appear for the FREE of cost certification for Azure Fundamental & Data Fundamental. If you are not confident then buy some practice courses from Udemy, Whizlabs ( You have to spend some money here ).

Share your achievement on LinkedIn. Don't worry it's never late to start. Now you are gearing up for the next steps. You have the basics covered.

Step3: Register for 30 days challenge.

Go for Synapse Analytics.

Link is below.

Now if you complete this 30 days challenge then it will serve two important purposes, First you will learn the most important service of Microsoft for Data Engineering & second you will get 50% discount on any Microsoft certification. Just do it.

Step4: Go for DP203 Certification. You will learn everything about the Data Engineering in Azure. Link is below. ( Complete the entire learning path from Microsoft learn & practice every single thing on the Azure Portal ).

Once you are confident then appear for the certification. Remember you already have 50% discount on it. Share your achievement on LinkedIn

Step5: This step is to further enhance your knowledge on services like Azure DevOps, Azure Purview & Azure Data Share. All of the tutorials are available on the Microsoft Learn Portal.

Step6: Follow Microsoft Learn LinkedIn Page for the latest updates & offers. Link is below.

Step7: Lots of official Microsoft YouTube Channels are there. Please search & subscribe the relevant.

If you manage to follow all of these steps or at least upto step4, I am 100% sure, you are your way of becoming a fantastic Data Engineer on Azure Cloud. Take your time, go with your own pace but go for sure.

This marks the end to this article. I hope, I am able to provide you a reference guide to start your Azure Data Engineering journey. Thanks for reading, Please provide your feedback in the comment section. Please like & share if you have liked the content. 

Thanks !! Stay Safe, Happy Learning !!