What Does it Take to Become a Data Engineer?

Data engineering is one of the most sought-after careers today. In fact, it is ranked as one of the UK’s best jobs for 2022.?

Data engineers play such a vital role in ingesting data from its sources, creating complex pipelines, transforming the data, and loading it to data warehouses. These too are designed and built by data engineers.

Data engineers also ensure that there is a smooth, uninterrupted flow of data between servers and applications by improving existing procedures and integrating new data management technologies into the system. It is from here that data architects take over to build models and data scientists and analysts clean the data to derive useful business insights.

While most data engineers complete a degree in computer science, mathematics, engineering, or similar STEM subjects, many experienced ones started off by learning on the job, rather than following a pre-defined learning pathway. Some of my fellows at LEIT Data and I found our way here through the technical data engineering bootcamps. So, currently there seems to be no coherent or formal path available for data engineers.

If you ask around or read about what you need to become a data engineer, you’ll soon find out that you’d need to have a solid knowledge of database tools;?cloud platforms like Google Cloud Platform (GCP), Snowflake, Amazon Web Services (AWS), Azure; languages?like Python, SQL and Java; a good know-how of different operating systems, SQL and NoSQL databases, distributed systems like Hadoop, and basics of machine learning and data analytics, and so much more! But, in all honestly, there’s a lot more to data engineering than these qualifications and experience in the field.

I dug into some of the job descriptions for the role of a data engineer on Reed and found out that more than 90% of these don’t just ask you to have a good understanding and experience on the core subjects, but they also demand some specific personality traits and soft skills. Here are some of the requirements I picked from these job descriptions:

• Be an engineer at heart and enjoy the challenge of building reliable, efficient data applications systems, services and platforms.
• Ability to see the big picture and break it down into manageable chunks
• Experience of mentoring other data engineers
• Ability to thrive in a fast-paced SCRUM environment
• Humility and desire to learn from anyone
• Excellent verbal, written communication, and client engagement skills
• Great communication, teamwork, problem-solving, and organizational skills
• Strong collaboration skills, working both with internal and client stakeholders, ability to contribute thoughts to solutions and focus on common goals
• Strong communication and diligent listening skills
• Desire to learn and grow in a fast-paced and technology-driven environment

But there’s something more important that you need as a data engineer than all of the above. LEIT’s co-founder , Chris Tabb, in one of his popular posts on LinkedIn said:

“My worry now is too many people are looking to be a Data Engineer, as it is the latest cool role.?Like in the Data Science Boom we had that (most of those roles were just Data Analysts dressed up in Lab Coats). This will lead to people chasing the money and the title and not because the like it, love it or HAVE passion for the role. If you do not have a passion for data, but you want to chase the rainbow. DON’T do it. You may blag / bluff it for a bit.?But you can’t fake passion and you will be found out.?Especially if I interview you. Focus on something you do have passion for, as you have to work for many years before you retire.”

This clearly means the love and passion for what you do is the key to becoming a data engineer. I spoke to some of the most experienced data engineers at LEIT Data to find out more about what skills and requirements they think are important to be a data engineer, and they all seem to agree with Chris.

George:

To be truly good as a data engineer, you must:

• Find passion in whatever you do. All roles can have some boring bits about them, but you need to find something you like in them to stay motivated.
• Avoid just "doing tasks". Instead, look around and try to see the bigger picture and have a keen eye for detail.
• Stay up to date with the latest trends in data and learn new things every day.
• Find a "guru" among your colleagues - that one person who will point you in the right direction and help you get promoted quickly. It does not mean he or she will be telling you what to do but will point you in the right direction.
• Always think big but start small.
• Be on projects where you can both contribute with your experience but also find something new for yourself to learn from.
• Most importantly, build stuff like it is your own. It will change your mindset.

Dan:

• Enjoy and love what you do
• Be extremely good at problem solving, investigating, and experimenting
• Having a great attention to detail is something I think is super important too, mainly because I've lacked it and have had to discipline myself in this area.
• Always be open-minded to new ideas and concepts.
• One thing often over- looked is the soft skills, like being able to pick up and understand clients’ business processes and data quickly.

Andy:

?As the data world is moving quickly, roles definitions are getting blurred and new tech seems to be popping up daily, so for someone looking to get into data engineering,?

?The first thing I would advise to do is to try and get into a company with a mature data environment. It is at this point you will start learning in the real world from people with experience. Graduate roles are great way to get this experience, especially at consultancies.

Secondly, it’s not always about the tech, data engineers need to speak to people too, you need to gather requirements, discuss solutions, articulate yourself clearly in a technical and non-technical way and interact with people at all levels.?Practicing these soft skills will benefit you when you are in a role but also at the interview stage.

Thirdly, read data-related blogs from different companies or articles on Medium, follow influential data gurus on LinkedIn, join LinkedIn groups or sign up to Slack channels. It’s amazing how much good content is being put out there daily.

What should you concentrate on learning? That’s a tricky one! I would say SQL is a must, after that, Python is a pretty good language to have knowledge of. Job adverts are a great way of knowing what skills are in demand, sign up to some job sites and see what tools and technologies are being repeated in job adverts.

?There is so much new happening in data and so many ways to achieve the same thing that it’s impossible to have experience in everything. You need to be drawing from others around you with experience.?Be humble, nobody likes a know-it-all. Even after 20+ years in data, I am learning all the time, and you should be doing the same.

Jack:

• Data engineers need to be skilled programmers, using languages such as SQL and Python, as well as having a good grasp on different tools, such as Snowflake and Rivery.
• Just like any computing field, data engineers need to be keen problem solvers, but working in data you must also be able to grasp the big picture of the project you’re working on, for example not just being able to develop code in one area of a pipeline but understanding how this contribution affects the upcoming stages of the pipeline as well, ensuring it all flows together. This requires a good technical understanding as well as good communication with your team.
• On top of this, all data engineers need to have a passion for technology and data, using their skills to find useful insights for clients that would have otherwise been missed.

As you can tell, becoming a data engineer is not easy. The role expects you to combine many skills and do some intensive thinking and problem-solving tasks. It requires a deep understanding of tools, techniques, and a strong work ethic. Besides, you’ll need to be on top of everything you do, update your skills set regularly and keep learning new things all the time. However, your passion for data, confidence in what you do, and good communication, presentation and collaboration skills can help you succeed quickly.

If you have what it takes to be a data engineer, you will soon find it to be a rewarding career option for yourself.