Data Engineers: The Key to Unlocking Their True Potential

Data Engineers: The Key to Unlocking Their True Potential?

Data engineers are increasingly becoming a pivotal role in their organizations. According to a report by LinkedIn, data engineering is one of the fastest-growing job roles in Europe, with a 35% increase in job postings between 2018 and 2020. With higher demand for this specific role, the probability of an oversaturated market and not enough qualified professionals to fill it will grow.?

This scenario is not far ahead from reality. In a European Data Science Academy survey, 60% of respondents said they struggled to find qualified data engineering candidates. As well we have seen an increment in the number of data engineers who are unhappy due to the lack of expectations accomplished within an organization.?

Companies are Not as Organized as We Often Think??

Some organizations hire experienced data engineers, either on wishful thinking or unrealistic ideas. At hiring, they set enormous expectations for what is to come. Still, projects get delayed, and the reality is that data engineers end up cleaning data and doing low-level tasks without much challenge. Once they lose faith in those promises, they start looking for a new job.??

And you are left with an open vacancy for a big challenge that might or might not come soon.?

Managers get excited when a new project is about to start. It will require them to need extra resources. However, plans change. So, unless the plan is to have a need this week. Do not hire just yet. The argument that you must onboard and warm them up for the project is often exaggerated.??

Senior data engineers do not need much time to start. They will be productive from week 1 with good management and clear goals.?

What Can We Do to Hold on to the Top Data Engineers??

In my recent conversations with clients and guests of my podcast I have found some insights I would like to share with you:?

• Treat data as an asset: We know data is valuable but still is not treated as a true asset, like money. If we were on top of our data the way we were with our money, the value would flood cash and come in from every direction.?
• Many organizations work from quarter to quarter: Some companies use dashboards to get quick insights without much regard for how these fit in the data landscape. We end up with a big unmanageable, and unreliable data infrastructure. The short-term goal is more profit, and unless the data directly generates more money or cost reduction, it is not worth it to spend money on it right now.??

One CIO of a successful large digital retail company gave me a clever idea. He told me it is to have a clear business case behind every new initiative. However, at the end of the quarter, you had to decide to either kill it or make it sustainable. This rule changed the mindset of cutting corners and insisted that people follow up on doing this right, rather than just quickly. It did leave room for experimenting but also kept the data landscape manageable and future-proof.?

• Data Engineers struggle to make sustainable fresh solutions: Once we have this unmanageable landscape and if the basis is not reasonable, nothing built on top will be guaranteed to work in the future. And the data engineers know it by nature. However, skilled professionals want to develop solutions to be proud of. So, they will become unhappy and want to leave if their work is not used for long.?

Last recommendations:?

To be successful long-term you need support from the top of the organization. Not just with words and budget but with actual involvement. Management should be data literate by being curious and asking questions.??

They should understand how the reports and dashboards they read have become. If the CIO asks the right questions, this will drip down into the organization, following to answer. Now you are working on creating a culture of Data.?

• Only hire when you need somebody.?
• Make a realistic view of what is needed. Sometimes it helps to hire an external view to get a quick scan. You do not want to have a Rocket Scientist working on a bicycle. They will get bored.?
• Data sustainability should be on the calendar from the top. There should be curiosity about data. People need to talk about it and ask questions.?
• There is not always a need for a full-time internal employee. Many times, when you are heading towards an unknown territory it is good to start with a (part-time) contractor that has seen many different environments. He/she can help set up the course and even train your internal people.?

Bibliography:

• Emeritus, “Why Data Engineer Salary Levels are on the Growth Path globally”.? (2022)
• World Economic Forum, “How much data is generated each day”.
• Report “The Dice 2020 Tech Job Report”
• Medium, “How will the demand of data engineers be in 2023”.