Data Engineer

A data engineer is an IT professional whose primary job is to prepare data for analytical or operational uses. This occupation includes duties such as designing and building systems for collecting, storing and analyzing data. Data engineers are typically responsible for building data pipelines to bring together information from different source systems. These software engineers integrate, consolidate and cleanse data and structure it for use in analytics applications. They strive to make data easily accessible and to optimize their organization's big data ecosystem. The amount of data an engineer works with varies by organization, particularly with respect to its size. The bigger the company, the more complex the analytics architecture and the more data the engineer is responsible for maintaining. Certain industries are more data-intensive, including healthcare, retail and financial services.

The data engineer role

Data engineers focus on collecting and preparing data for use by data scientists and analysts. They take on the following three main roles:

Generalists. Data engineers with a general focus typically work on small teams, doing end-to-end data collection, intake and processing. They might have more skills than most data engineers, but less knowledge of systems architecture. A data scientist who wants to become a data engineer would fit well into the generalist role.

A generalist data engineer might undertake a project to create a dashboard for a small, metro-area food delivery service that displays the number of deliveries made each day for the past month and forecasts the delivery volume for the following month.

Pipeline-centric engineers. These data engineers typically work on a data analytics team with more complicated data science projects across distributed systems. Midsize and large companies are more likely to need this role.

A regional food delivery company might undertake a pipeline-centric project to create a tool for data scientists and analysts to search metadata for information about deliveries. They might look at distance driven and drive time required for deliveries in the past month, then use that data in a predictive algorithm to see what it means for the company's future business.

Database-centric engineers. These data engineers implement, maintain and populate analytics databases. This role typically exists at larger companies where data is distributed across several databases. These engineers work with pipelines, tune databases for efficient analysis and create table schemas using extract, transform and load (ETL) methods. The ETL process copies data from several sources into a single destination system.

A database-centric project at a large, national food delivery service would be to design an analytics database. In addition to creating the database, the data engineer would write the code to get data from where it's collected in the main application database into the analytics database.

Although exact responsibilities for data engineers differ by organization, other typical responsibilities include the following:

• Build, test and maintain database pipeline architectures.
• Create methods for data validation.
• Acquire data.
• Clean data.
• Develop data set processes.
• Improve data reliability and quality.
• Develop algorithms to make data usable.
• Prepare data for prescriptive and predictive modeling.