Automated Real-Time Data Streaming Pipeline using Apache Nifi, AWS, Snowpipe, Stream & Task

Unlocking the Power of Real-Time Data in the Cloud

In this project, I embarked on an exciting adventure in data engineering. I orchestrated the real-time flow of data using a fusion of modern technologies, aiming to build a smart system. This system generates random data and swiftly sends it to an AWS S3 storage space. Then, it instantly processes this data with Snowflake, all under the watchful eye of Apache Nifi, the conductor of this intricate data symphony.

Prerequisites

Before diving into the project, you'll need the following prerequisites:

• Access to AWS services.
• An EC2 instance with specific configurations: 8GB RAM, 32GB memory space.
• Docker container expertise.
• Proficiency in Python , including the 'Faker ' library.
• Familiarity with Snowflake for cloud data warehousing .
• Basic understanding of Apache ZooKeeper .
• Apache Nifi setup and configuration.
• Streaming & Task Processing .
• Understanding of Version Control (Git) .
• Basic understanding of SQL for Data Manipulation

Project Overview

This project demonstrates my ability to design and implement a real-time data streaming pipeline using Apache Nifi, AWS, Snowpipe, Stream, and Task. The pipeline continuously ingests data from a JupyterLab notebook, processes it using Apache Nifi, and loads it into a Snowflake data warehouse. The pipeline is designed to handle new and existing data updates.

Project Architecture

The architecture of our project involves several interconnected components:

• EC2 Instance: This serves as the project's foundation, where we deploy our Docker container and essential tools.
• Docker Container: Housed within the EC2 instance, this container contains Python, Apache Nifi, and Apache ZooKeeper, ensuring a consistent and easily replicable environment.
• JupyterLab and Apache Nifi: These are the workstations for data engineers. JupyterLab is accessible at 'IP/4888,' and Apache Nifi is reachable at 'IP/2080' (where IP is the EC2 machine IP ).
• Data Generation: Using Python in JupyterLab, we utilized the 'Faker' library to create random data records, including customer information.
• Data Streaming to AWS S3: Apache Nifi was used to establish connections for transferring the generated data to an AWS S3 bucket, providing scalable storage and real-time access.
• Real-Time Data Processing with Snowflake: Leveraging Snowpipe, we ensured that any new data or changes in existing data were automatically incorporated into the dataset. Slowly changing dimensions were used to track these changes.
• Target Table Creation: A task was executed to create target tables, ensuring the final dataset was always up-to-date and accurate.

Project Setup

Let's dive into the details of the setup, like taking a closer look at the instruments in our orchestra:

• EC2 Instance Configuration : I meticulously configured an AWS EC2 instance with 8GB RAM and 32GB memory space, choosing a t2.xlarge instance for optimal performance. Ports 4000-38888 were opened, and SSH access was set up. This is where we set up our concert hall.
• Docker Container Setup : Within the EC2 instance, a Docker container was created and populated with Python, Apache Nifi, and Apache ZooKeeper. Think of it as preparing our instruments, tuning them to perfection.
• JupyterLab and Apache Nifi Configuration : JupyterLab and Apache Nifi were configured to run on specific ports, making them accessible for data processing and orchestration. It's like setting up the conductor's podium, and the sheet music stands just right.
• Data Generation: In JupyterLab, Python code was crafted to generate random data, simulating customer information. This is where the composer writes the notes.
• Data Streaming: Apache Nifi was utilized to set up connections, ensuring the seamless transfer of generated data to an AWS S3 bucket. It's the conductor guiding the instruments.
• Real-Time Data Processing : Snowpipe, Snowflake streams, and tasks were set up to handle real-time data processing, enabling automatic updates as new data arrived. It's the conductor guiding the orchestra to play in harmony.

Key Components

The key components are like our orchestra members:

• AWS EC2: The Stage Where the Symphony is Performed The AWS EC2 instance serves as the stage for our data symphony.
• Docker: Docker played a crucial role in maintaining the tools and dependencies used in the project.
• Apache Nifi: Apache Nifi was the conductor of our data symphony, orchestrating the data flow.
• Snowflake, snow pipe, snowflake stream, and Task: In my workflow, this combination of Snowflake, Snowpipe, Streams, and Tasks enables the seamless automation of data loading and processing as new data arrives, eliminating manual intervention and ensuring up-to-the-minute data availability for your analytics and reporting needs.

The orchestration of these components is analogous to preparing an orchestra for a grand performance. The EC2 instance serves as the stage, Docker maintains our instruments, and Apache Nifi conducts the symphony of data, guiding it through the intricacies of real-time processing and streaming.

Reference:

• You can see my portfolio, where I have given all my data engineering projects: click here .
• For this project, please see my project portfolio at project link