There are 3 Types of Data Engineers

Then there were three. The final three. Only three.

It’s the truth; there are three types of Data Engineers only, and I promise you, you fall into one of the three groups. The truth can be hard; the truth can hurt, but the truth can also heal and lead to growth.

Aren’t you just bubbling over with excitement to know what kind of Data Engineer you are? Group 1, Group 2, or Group 3?

Maybe you’re not happy with where you are at. Maybe you want to move from Group 1 to Group 2 or from Group 2 to Group 3. Trust me, you can’t move from Group 1 to Group 3 just in a snap—it doesn’t work like that.

Group One is always jealous of Two; Two looks with longing at Three. Group Three looks down from their high towers onto the peons below them in One and Two. I don’t think any one group of Data Engineers is better than the other; they all have their place, and they are all needed.

We need people who love SQL, are wizards, and are adept at complex analytics. We need people who are good programmers, can build anything, lead projects, and architect Data Platforms. We need Data Engineering savants who are next-level programmers, building the next generation of data tools with Rust, etc.

Defining the 3 Types of Data Engineers

1. SQL and Analytics (Group 1)

These Data Engineers spend most of their time in SQL, producing analytics and dashboards. They work extremely closely with the business and may not write code every day, although some are good with Python.

• Characteristics: SQL experts who build data marts and dashboards.Strong business acumen and analytical skills.Often the easiest pathway into Data Engineering for those transitioning from business intelligence roles.

2. Senior Level Programmers and Architects (Group 2)

This second set of Engineers are the writers of code—complex code. They use a lot of Python but also Golang, Rust, Scala, and more. They can debug Spark pipelines, build CI/CD, and DevOps processes, and are comfortable working on a Linux server.

• Characteristics: More comfortable on the command line than the UI.Broad expertise across many tech stacks.Able to design and build a Data Platform from the ground up, understanding distributed systems.

3. Tooling Builder (Group 3)

This is the least common of the Data Engineering types—the Yodas of the data space. They build the tooling, whether private or open-source, that others use. They are the best of the best, Software Engineers who specialize in data.

• Characteristics: Building open-source tooling and contributing to the data community.Expert-level programmers, often at the Lead or Staff engineer level.Capable of innovating and creating the next big thing in data engineering tools.

Conclusion

I always thought I wanted to be a Group 3 Data Engineer, but I had to accept after years of programming that I just wasn’t able to make the switch. I suppose anyone can get there with enough work, and I wasn’t willing to put in that level of effort.

It's important to note that there can be some confusion between Group 2 and Group 3 Data Engineers. One could be a Staff Data Engineer without necessarily being a Group 3 member. You might just excel at building Data Platforms, leading teams, and have a vision that elevates you to the Staff+ level.

Extra Insight

Regardless of which group you belong to, it's essential to recognize the value you bring to your team and organization. Continuous learning and professional development can help you transition between these groups over time. Embrace challenges, seek mentorship, and stay curious. Whether you're focused on SQL analytics, coding complex data solutions, or developing new data tools, each path contributes to the evolving landscape of Data Engineering.

#DataEngineering #TechMistakes #SoftwareDevelopment #DataPlatforms #Coding #DevOps #Orchestration #DataPipelines #DataQuality #EngineeringBestPractices #DataOps #DataManagement #ContinuousLearning #danielbeach