Supercharge Your ETL Pipeline with Docker: A Quick Guide for Data Engineers

In today’s fast-paced world of data engineering, efficiency and scalability are key. One tool that has revolutionized the way we handle data pipelines is Docker. In this article, I’ll walk you through how to get started with Docker by setting up a simple ETL (Extract, Transform, Load) pipeline. We’ll be using a Python script that reads data from SAP HANA, performs some operations, and writes the results to a MySQL database.

Why Docker?

Docker is a powerful platform that allows you to package your application and its dependencies into a container, ensuring it runs seamlessly across different environments. This portability, coupled with Docker’s ability to isolate environments, makes it ideal for ETL pipelines.

Getting Started with Docker: A Step-by-Step Example

Step 1: Setting Up Your Docker Environment

Before diving into the code, make sure Docker is installed on your machine. You can download it from Docker's official website.

Once installed, open your terminal and check if Docker is running by typing:

docker --version        

Step 2: Writing the Python Script

Write a simple Python script that connects to Source DB, performs a transformation, and writes the results to a Target database.

Step 3: Creating a Dockerfile

Now, let’s containerize this script. Create a Dockerfile in the same directory as your Python script:

# Use an official Python runtime as a parent image 
FROM python:3.9-slim 

# Set the working directory in the container 
WORKDIR /usr/src/app 

# Install any necessary dependencies 
RUN pip install --no-cache-dir hana-ml mysql-connector-python 

# Copy the current directory contents into the container at 
/usr/src/app COPY . . 

# Run the Python script when the container launches 
CMD ["python", "./your_script.py"]        

Step 4: Building and Running the Docker Container

With your Dockerfile in place, build your Docker image:

docker build -t etl_pipeline_image .        

Once the image is built, run the container:

docker run etl_pipeline_image        

This command starts a container that runs the Python script inside an isolated environment.