Transforming User Insights: Real-Time Data Analysis with Kafka, Spark, PostgreSQL, Docker and Cassandra

Link repo: https://github.com/ntd284/streaming_realtime_data.git

?? Overview

In today's data-driven world, real-time data analysis is crucial for businesses to make informed decisions swiftly. Our processing system harness Kafka, Spark, PostgreSQL, Docker and Cassandra.

Components Involved

• Docker: Ensures a consistent and reliable environment.

• Apache Kafka: Distributed messaging system for real-time data streaming.
• Apache Spark: Processes and analyzes streaming data.
• Apache Airflow: This tool automates and schedules workflows.
• PostgreSQL: Stores processed data for further analysis.
• Apache Kafka's Confluent Control center: is a comprehensive management and monitoring tool for Kafka Clusters with user-friendly interface to track the health and performance of Kafka clusters, manage topics, configure alerts and analyze data streams.
• Schema registry: Manages and enforces data schemas for Kafka topics, ensuring that data producers and consumers adhere to predefined data structures.
• Cassandra: is an open source NoSQL distributed database, making it an excellent choice for real-time application.

The Challenge

Manually inputting data and updating systems later leads to outdated insights, as traditional systems struggle with increasing data volumes, resulting in performance bottlenecks and higher costs.

Integrating data from various sources often causes inconsistencies and complexities, compromising data quality and delaying analytics.

Without real-time monitoring and processing tools, issues may go undetected, leading to data loss or corruption. Inefficient storage solutions further exacerbate these challenges, causing high retrieval latency and increased costs.

Main Tasks

The project aims to establish a real-time data analysis system for capturing and processing user data efficiently by leveraging advanced technologies and a robust data pipeline:

• Enhanced Data Capture: Real-time ingestion to capture user interactions as they happen, reducing latency.
• Scalable Architecture: A distributed system to handle growing data volumes seamlessly.
• Improved Data Quality: Ensuring integrity and accuracy through schema management and real-time validation.
• Streamlined Operations: Automating workflows to minimize manual intervention and reduce errors.
• Comprehensive Monitoring: Tools to monitor data stream health and resolve issues promptly.
• Business Integration: Real-time insights integrated into business applications for immediate action.
• Data-Driven Decisions: Providing stakeholders with timely, actionable insights for informed decision-making.

Project Workflow:

1. Ingesting, formatting and producing messages:

To simulate user information, user data API will be used in the Kafka producer.

This data ingestion workflow utilizes an API to simulate user data, which is then fed into Apache Kafka for real-time processing. Apache Airflow orchestrates and automates the entire process, ensuring reliable data flow.

After processing, the enriched data is published back to Kafka for consumption by downstream applications. Kafka's Control Center and Schema Registry work together to monitor the pipeline's health and manage data consistency.

User data:

The message is encoded in JSON format:

{'id': '028750e7-ea42-4d3f-b86c-dba5fe996add', 'first_name': 'Tracey', 'last_name': 'Smith', 'gender': 'female', 'address': '6177 High StreetSwansea Bedfordshire United Kingdom', 'post_code': 'W8 9DY', 'email': '[email protected]', 'username': 'bigsnake836', 'dob': '1963-10-08T07:59:55.259Z', 'registered_date': '2002-10-28T07:42:18.018Z', 'phone': '019467 33243', 'picture': 'https://randomuser.me/api/portraits/med/women/38.jpg'}        

• id: Unique identifier for the user (UUID).
• first_name: User's given name.
• last_name: User's surname.
• gender: User's gender.

• post_code: Postal code for the user's address.
• email: User's email address.
• username: Unique name chosen by the user.
• dob: User's date of birth.
• registered_date: Date the user registered their account.
• phone: User's phone number.
• picture: URL to the user's profile picture.

Scheduler and automation in Apache Airflow:

Apache Airflow orchestrates this workflow, scheduling and managing the data pipeline. It ensures a smooth flow of data from the source, a user data API, to the data formatting functions.

Every minute, Airflow triggers a Python function that fetches user information and transforms it into a structured format.

This formatted data is then sent to it's destination with designated topic named 'user_created' within Kafka using a Kafka producer.

Essentially, Airflow acts as the conductor,ensuring timely data retrieval, processing, and delivery to Kafka for further utilization.

Control center is monitoring in real-time

Apache Kafka's Confluent Control center provides a centralized platform for monitoring the health and performance of kafka cluster in real-time.

• Topic health: This reveals information about specific topic, such as: messages, partitions and any replication issues.
• Procuder and consumer performance: monitor the rate at which messages are being produced and consumed across different topics.
• Offset: track the progress of producers and consumers within a topic. By analyzing offsets, we can understand how much data has been processed and identify potential lags.
• Cluster health: off an overall view of cluster's health, including information on brokers, Zookeeper nodes, and any errors that might be occuring.

2. Spark process data and transfer to temporary storage:

Apache Cassandra is an open-source distributed row-partitioned database management system (distributed DBMS) to handle large amounts of structured data across many commodity servers. It supports for only structured or semi-structured data and database consistency levels.

Apache Spark (PySpark) performs streaming data processing, recevied data from in Kafka, processing it, and writing the structured results into data storage with Cassandra before it lands in the Data warehouse.

User Data is stored in Cassandra storage:

?? Conclusion:

This project successfully establishes a robust real-time data analysis system by integrating advanced technologies such as Apache Kafka, Apache Spark, PostgreSQL, Docker, and Cassandra.

• By leveraging these tools, the system ensures efficient data capture, processing, and analysis, leading to timely and actionable insights.
• Key components like Apache Airflow automate and schedule workflows, while Kafka's Confluent Control Center and Schema Registry manage and monitor data streams, ensuring consistency and reliability.
• The architecture's scalability and comprehensive monitoring capabilities address traditional system challenges, such as performance bottlenecks, data inconsistencies, and high latency, thus enhancing operational efficiency and supporting data-driven decision-making for stakeholders.

Reference:

[1]. Realtime Data Streaming

[2]. Cassandra and pyspark