Engineering Grads' Path to Data Engineering: Why Data-Driven Decision Making is crucial these days?

As we are in the mid of 2024, the generative AI has started showing its effects. Imagine the fact that you hold the budget of your company. You have the purse strings to spend on projects. Suddenly somebody appeared and shouted statue. Then you are still for some time. This is what generative AI has done to all the technology spenders of the world. Businesses are thriving and even growing but everyone is in waiting. There is a chance that businesses have to spend more in future, but they are in a statue mode.

The impact of situation is coming down to availability of projects and in turn on the job market. We are seeing that job creation cycle is adversely impacted. This has directly reduced the number of jobs available to Engineering Grads. Similar to 2008 crisis it is not their fault. The business cycle will improve soon. Job market is going through a tremendous change. It is not going to be same again. I remember when I graduated the only things that were tested was aptitude and ability to speak English. We were hired and then trained rigorously.

What changed today are business want engineering grads to be business ready on from day one. For such a thing business is ready to pay more money as well. Point is while we wait for the business to comes back to an upcycle, we see there is a huge gap between expectation from business and availability of talent. I am writing these series of article to address this gap.

We will start with a simple concept today - Data Driven Decision

Why in 20th century the Data driven decision making is crucial?

Digital revolution took place in early 20th century. That is where world become tech savy. Everyone had a smart phone in their hand. Digital footprint of every person started getting recorded. This resulted in tremendous opportunity to collect data. On the other side the technology to store and analyze such a huge data also evolved. Data processing frameworks such as Hadoop, Spark etc. came up. On one side there is a huge collection of data, on another there is an evolution of data analytics frameworks. This led to a tremendous opportunity to personalize the services available to humanity.

What are a benefits of Data driven decision making?

Managers use to take the decision based on intuition. They would take decision based on their experience. World has evolved over time and data can give some crucial insights which might help businesses to take better decision.

In conclusion, Availability of Data, Evolution of Data processing framework has led to Data driven decision making. As a student who are trying to enter new into the job market it is crucial to know why we are seeing this trend.

What is the next step?

The beauty of Data driven decision making is it stretches across all fields of engineering. Data engineering is only for computer science is myth. New chemical experiment of a chemical engineer can be aided by the available data of past similar experiments. A civil engineer can design a better structure by analyzing data points of weather over last 100 years. Data space overall can be covered in two paths:

1. Data Engineering
2. Data science

Eventually we need to learn both data engineering and data science. To start with we must choose one. As both of them are vast in themselves.

How to make this choice? What path to follow, is the topic for the next article in this series.

Thanks for reading!! Happy Learning!!