Introduction to Apache Kafka

By Victor Caceres Chian

In these days of modern data-driven enterprises, the ability to consume and analyze near real-time data is crucial. From monitoring customer interactions to optimizing supply chains, organizations rely on live insights for their operations. As this need arises, Apache Kafka increases in popularity for its efficiency and effectiveness.?

The publish-subscribe messaging system abstracts a lot of the complexity of handling data-streams with a high volume of data. It allows organizations to focus on applications instead of Infrastructure. Additionally, its API capabilities present a low entry barrier, as developers can code in multiple programming languages and ecosystems.?

Apache Kafka was originally conceived by LinkedIn and later open-sourced. It is mostly known for its scalability and fault tolerance but can also shine in ease of usage. This makes Apache Kafka attractive for organizations of all sizes.?

Furthermore, Apache Kafka also has a wide range of connectors and data sinks available, that provide production-ready solutions for integration with systems. Whether it be integration with databases, cloud services or on-premises systems, Apache Kafka’s community has a solution that enables rapid and effective deployment of pipelines.?

In this article we will outline Apache Kafka’s structure, use cases and ease of implementation in managing near real-time data processing. Whether it is a larger or smaller use case, Apache Kafka offers a suitable solution.?

??

Why use Apache Kafka??

The use of real-time data processing and streaming presents several key challenges that are addressed by Apache Kafka:?

• Scalability and Performance: As data volume and requirements grow, traditional processing systems struggle to maintain throughput and latency. Apache Kafka’s distributed nature is built for scalability and performance allowing it to handle millions of messages per second across the distributed cluster of brokers. By partitioning data, Apache Kafka ensures scalability and availability even under heavy load.?

• Data Loss and Durability: Hardware Issues, network issues or software bugs can occur in traditional processing systems. Apache Kafka provides fault-tolerance and durability by using replication across brokers and persisting the messages to disk. This ensures that messages are not lost and enables reliable data processing and analytics.?

• Real-Time Processing: Traditional Batch Systems are not suited for live processing. Apache Kafka facilitates real time processing by enabling ingestion and processing of data events as they occur. Extensions to Apache Kafka allow applications to react to real-time events, enabling use cases such as fraud detection, anomaly detection and real-time recommendations.?

• Data Driven Decision Making: Organizations rely on timely insights to make informed business decisions. Apache Kafka provides real-time access to data streams and enables real-time analytics. Apache Kafka empowers businesses to detect trends, identify opportunities, and respond to events as they happen, thereby gaining a competitive edge in the market.?

• Market Share: Apache Kafka has a dominant position in the market for real-time data streaming and messaging systems, with 39.8% for the market according to [6sense]. Due to its strong community support, active development, and widespread adoption, organizations consider Apache Kafka as a leading choice due to its proven capabilities and extensive ecosystem.?

What makes Apache Kafka work??

Apache Kafka is structured around several key components that work together to provide a robust and scalable platform for real-time data streaming:

• Brokers: The main component of Apache Kafka are brokers: individual servers or nodes within a cluster that are responsible for storing and managing the data. Each server stores a portion of the data and handles the relevant client requests, such as publishing or consuming messages. A cluster consists of multiple brokers for scalability and fault tolerance.?

• Topics: Data is organized inside Apache Kafka with the use of topics. A topic is a named data stream to which messages are published and later consumed. Can be thought of as a folder that provides a logical separation of data streams within a broker. Messages within a topic are ordered and immutable, enabling real time and historic processing of data.?

• Partitions: Topics in Apache Kafka are further divided into partitions. Each of them has an ordered sequence of messages. Together, they form the distributed and parallel storage of Apache Kafka. Partitions enable the horizontal scaling of topics, as they can be distributed across brokers and processed independently. The number of partitions of a topic determines its maximum parallelism and throughput for writing and consuming messages.?

• Producers: Messages are published to Apache Kafka topics by producers. They are client developed programs that send data to brokers, which then append the messages to the appropriate partitions within the topic. Producers can publish messages asynchronously and in batches, providing high throughput and low-latency data ingestion capabilities.?

• Consumers: Messages are retrieved and processed from Apache Kafka topics by consumers. They can be part of a consumer group, where each consumer within the group reads from some partitions, enabling parallel processing and load balancing across multiple instances of the consumer application. Apache Kafka consumers maintain their own offsets to track the progress of message consumption, ensuring reliable and at-least-once delivery semantics.?

In addition to this Apache Kafka structure, there are some useful extensions that enhance the capabilities Apache Kafka:?

1. Apache Kafka Connect: The integration of Apache Kafka with external data is simplified by Kafka Connect. It provides pre-built connectors for popular data sources such as databases, message queues, and file systems while supporting custom connector development using a simple plugin architecture.?
2. Kafka Streams: Apache Kafka includes a stream processing library called Kafka Streams. It allows developers to build stream processing applications using Java programming. Apache Kafka Streams enables stream processing operations such as filtering, mapping, aggregating, and joining data streams.?
3. Kafka Monitoring: Various monitoring and analytics tools are available for Apache Kafka to track the health, performance, and usage metrics of Apache Kafka cluster. They provide insights into Apache Kafka's throughput, latency, error rates, and resource utilization, enabling administrators to optimize cluster performance and troubleshoot issues.?
4. Kafka Security: Apache Kafka offers several security features to protect data both in transit and on disk. These include SSL/TLS encryption for secure communication, SASL authentication, and access control lists (ACLs) for authorization policies. Apache Kafka also supports integration with external authentication providers such as Kerberos for centralized user authentication.?

Who is using Kafka?

Many companies have implemented Apache Kafka for their use cases. Let’s review some of them:?

• LinkedIn: The original developer of Apache Kafka has one of the most extensive deployments of Apache Kafka. They use Apache Kafka for various use cases, including real-time monitoring, activity tracking, recommendation systems, and data integration.??

• Netflix: The Streaming Platform uses Apache Kafka as a core component of its business. They utilize Apache Kafka to process streaming data, to enable event-driven architectures, and for real-time analytics to improve user experience and content recommendations.?

• Uber: The Taxi Application relies on Apache Kafka for real-time data processing and analytics. They use Apache Kafka for tracking driver and rider interactions, monitoring system health, detecting anomalies, and optimizing dispatch algorithms.?

• Airbnb: Airbnb uses Apache Kafka for real-time data streaming and analytics to enhance its online marketplace. They utilize Apache Kafka to track user interactions, personalize search results, detect fraud, and optimize pricing algorithms.?

• Twitter: Twitter employs Apache Kafka as a key component of its real-time data infrastructure for processing and analyzing tweets, user interactions, and trends on its social media platform. Apache Kafka helps Twitter handle massive volumes of data and deliver real-time updates to users worldwide.?

??

Apache Kafka Deployments

The deployment of Apache Kafka involves setting up and managing Apache Kafka clusters. The preferred deployment depends on the organization's requirements, expertise, and infrastructure preferences. Whether the organization requires a small or large Apache Kafka implementation, there is an option that suits it.?

Apache Kafka as a Service: Sometimes referred to as a managed Apache Kafka. It is provided by cloud providers or third-party vendors. The provider manages the infrastructure, including provisioning, configuration, monitoring, and maintenance of Apache Kafka clusters.??

++ Pros:?

• Simplicity: Abstracts away the complexities of managing Apache Kafka clusters, making it easy for users to get started without much expertise.?

• Scalability: Services typically offer scalability features, allowing users to easily scale their clusters up or down based on changing requirements.?

• Reliability: Providers handle tasks such as monitoring, backups, and failover, ensuring high availability and reliability of Apache Kafka clusters.?

-- Cons:?

• Cost: Apache Kafka services often come with subscription fees and usage-based pricing, potentially leading to higher costs over time.?

• Vendor Lock-in: Users may become dependent on a specific Apache Kafka provider, making it challenging to switch providers or migrate to a self-managed deployment in the future.?

For starters or smaller Apache Kafka requirements, this is a preferred option.??

Getting Started with the Manged Service

?

Confluent Apache Kafka Service is a fully managed platform built on Apache Kafka, offered by Confluent, a company founded by the creators of Apache Kafka. It provides organizations with a solution for building and deploying real-time data pipelines, stream processing applications, and event-driven architectures.?

With Confluent Apache Kafka Service, organizations can run Apache Kafka without the operational problems of managing infrastructure. The service offers seamless integration with Apache Kafka and the broader Confluent Platform ecosystem, including Apache Kafka Connect and Apache Kafka Streams, along with security features. Whether you are a startup or an enterprise, Confluent Apache Kafka Service delivers scalability, reliability, and cost-effectiveness to leverage real-time data.?

Let’s explore an example of Apache Kafka using Confluent:?

We can access the Apache Kafka service via this website: https://www.confluent.io/get-started ?

The free trial of Confluent includes a budget that we can use to create a cluster for showcase purposes. To create a cluster, we must first select the cluster’s type, the cloud provider and the cluster’s name. As it is an Apache Kafka as a Service Platform, most of the configuration is taken care of by Confluent.?

We can select what type of Kafka cluster we want to create according to the use case requirements.??

We then later select the cloud it uses as infrastructure.?

Here we can insert the name of the cluster and see some more details on the pricing schema.?

When we are ready, we can launch the cluster.???

Once we have our cluster, we can view its details via the webpage portal. Cluster ID as well as the connection details can be seen here:

?

Via the Web interface: We can create new topics and configure some of their details. Confluent will then take care of most of the other technical settings.

To access our Apache Kafka data from our custom programs, we can create an API Key also via the web interface.?

With the API Key, we can connect to the Kafka Cluster, utilizing the libraries that Confluent provides. The Apache Kafka Python Client can be used if Python is our programming language of choice.?

In this example we create a Consumer and utilize the Cluster Details and the API Keys to send messages to our newly created Kafka demo topic.?

?

We can then view these messages in the web portal.?

Utilizing the same library and some other connection details, we can consume those messages by creating a Kafka Consumer. We could then process those messages the way our use case demands it.?

As for every demo, it is important to delete the cluster to avoid any unexpected charges, which can be done via the web portal:??

?

Self-Managed Apache Kafka: refers to Apache Kafka clusters deployed and maintained by organizations. In this setup, they have full control over the deployment, configuration, and management of the cluster, running on their infrastructure or in a private cloud environment. It is usually big companies that manage their own Apache Kafka infrastructure, as they have greater requirements and need appropriate teams to support it.?

?

++ Pros:?

• Control: Organizations have full control over their Apache Kafka clusters, allowing them to customize configurations, security policies, and integrations according to their specific requirements.?

• Flexibility: organizations can deploy clusters in their preferred environment, providing flexibility on infrastructure choices.?

• Cost: Lesser cost for organizations with large-scale deployments.?

-- Cons: ?

• Complexity:? Apache Kafka clusters require expertise in infrastructure provisioning, configuration, monitoring, and maintenance, which may be challenging for organizations without specialized skills or resources.?

• Scalability: Scaling managed Apache Kafka clusters requires manual intervention and coordination, leading to potential delays or downtime during periods of high demand.?

?

Self Hosted Solution

How to get started with a self-managed Apache Kafka solution? Here we sketch a small in-house Apache Kafka deployment and all its requirements that should be considered.?

• Zookeeper: A cluster of servers is required to manage cluster metadata, leader election, and synchronization among the Apache Kafka brokers. Each Zookeeper node should be on a separate server. Smaller deployments have 3 nodes for availability and resilience. Larger Apache Kafka deployments use 5 Zookeeper nodes, as they can sustain 2 server failures. Deployments with 7 or more nodes are rarely used and require expert tuning. All servers must communicate with each other. Thus, as the number of servers increases, the network overhead grows greatly: ?

?

?

• Apache Kafka Cluster: The deployment of at least 3 Brokers to ensure fault-tolerance and high availability. Each broker should run on a separate server or virtual machine. The servers act independently with minimal coordination during their normal operation. This decentralized architecture reduces the need for inter-broker communication and enables brokers to scale in number without significant networking issues:

• ?Security Configurations: Apache Kafka servers should also have security measures implemented. These include, for example, the addition of an Active Directory that serves as the central authentication provider for the Apache Kafka environment. The Kerberos-KDC inside the Active Directory is responsible for the issuing and validation of Kerberos tickets which are used for authentication between clients, brokers, and other Apache Kafka components. Authorization is defined by ACLs configured in Apache Kafka:?

• Monitoring and Alerting: The Apache Kafka cluster requires monitoring to track health, performance, and usage. It can be implemented with tools like Grafana and Prometheus. Configuring alerts enables administrators to react to any anomalies, errors, or performance degradation:

• Client Applications: Client Applications complete the Apache Kafka ecosystem. They interact with the Apache Kafka cluster and process data streams according to the Organization’s requirements. The Schema Registry is a service that manages schemas used by Applications and Apache Kafka topics, ensuring compatibility and consistency of data formats across producers and consumers.?

?

The deployment of a self managed Apache Kafka environment is usually a long-term project. Depending on the size and necessities of the organization, it may require an entire team for development and maintenance.?

Final Thoughts on Managed vs Self Hosted Solutions

To conclude this article, the decision between self-hosting Apache Kafka and opting for a managed Apache Kafka service depends on the organization requirements, expertise, and expectations. Self-hosting Apache Kafka offers greater control, flexibility, and potential cost savings for large implementations, but comes with the requirement of expertise in infrastructure, constant administration, and a longer implementation phase. On the other hand, managed Apache Kafka services provide simplicity and scalability with vendors handling the operation tasks but with lesser flexibility and potential higher cost in large projects. Whether self-hosted or managed, Apache Kafka has options that adjust to any organization’s requirements.?