A Complete Data Engineering Roadmap

I recommend you to bookmark this blog for your future references. As this blog contains all the resources you’ll need to learn about Data Engineering.

According to some reports, the number of data science interviews only grew by 10% from 2019 to 2020, while the number of data engineering interviews grew by 40% in the same period of time!

So, as the saying goes:

If you are in IT sector, you should constantly upskill yourself to keep up with the industry.

So, coming to the point: If you want to transition into the Data Engineering role, here’s a complete step by step process of what you should work upon:

Coding Basics

• Learn to write SQL queries.
• Work on your basic Python programming skills.
• Linux (Also, read about some server concepts like SFTP, firewall etc.)

Here are some resources for working on above skillsets:

Video lectures: Intro to SQL: Querying and managing data (Khan Academy)

Practice: Solve SQL (HackerRank)

Videos: Python Programming Beginner Tutorials (Corey Schafer)

Practice: Solve Python (HackerRank)

Videos: Linux Tutorial For Beginners (Edureka)

Practice: Linux Shell (HackerRank)

Build your first?project

Build a basic Flask API. If you want you can connect it to a front-end or just use it out for testing purposes.

Here’s an example of what you can build: Learn Flask for Python — Full Tutorial (freeCodeCamp.org)

Learn about data warehouses

Read: The Data Warehouse Toolkit, 3rd Edition — Kimball Group (Book)

Videos: Data Warehouse Tutorial For Beginners (Edureka!)

Build your second?project

1. Scrape an online source
2. Store encrypted data into SFTP
3. Create dimensional model
4. Pull data from SFTP and load into Data Warehouse (Don’t worry too much about workflows yet)

You can search some project ideas on github and their implementation. However, it’s good to start on your own.

Start learning about?testing

Unit testing, Integration Testing, etc.

Sources to learn: Check out some TDD courses on internet.

Learn about some workflow?tools

Launch Airflow with Docker. This will allow you to get a higher level understanding of Docker while learning Airflow.

Videos: Airflow tutorial 1: Introduction to Apache Airflow

Learn about Cloud and?NoSQL

Videos: Database Systems — Cornell University Course (SQL, NoSQL, Large-Scale Data Analysis)

Learn about streaming and distributed systems

Videos: Why You Need To Learn Apache Spark and Kafka

Blog: Data Engineering Project: Stream Edition

Start studying for interviews

Start practiceing some Data Structure and Algorithms as well as some SQL and Data Warehousing question.

Articles:

• Commonly Asked Data Structure Interview Questions (Geeksforgeeks)
• SQL Interview Questions (Interviewbit)
• SQL Interview Questions (Geeksforgeeks)
• Top 50 data warehousing interview questions and answers

Build your third?project

Use a cloud manages service like Kinesis to stream data and pair it with a batch ETL tool that pulls from a second source.

GitHub Repo: Udacity Data Engineering Projects

Learn enough UI/UX and dashboarding

Pick 1 dashboarding tool and learn how it function. Also learn a little bit about UI/UX.

Article: 10 rules for better dashboard design

Pick some of your?own

At this point, you probably have found some personal favorites in terms of tool types.

Why not learn a little more about them?

If you liked Docker, learn about Kubernetes, if you liked Dashboarding, pick a different tool, if you like the cloud, then why not learn a new cloud stack?

Last but not?least…

DOCUMENT YOUR WORK!!

Check out this blog for some great examples of Documentation.

I hope that’ll help you kickstart your journey with Data Engineering.

Learn slowly and have patience.

Have a great learning ahead!

What’s next for?you?

If you enjoyed this article, it would really help if you hit recommend below!

Follow me on Twitter, LinkedIn, and Medium

Read my all posts/articles here: Praveen Pareek