Running Kafka on a Single Node in K8s Cluster

Introduction to Kafka:

Certainly! Apache Kafka is an open-source stream processing platform widely used for building real-time data pipelines and streaming applications. It was originally developed by LinkedIn and later open-sourced as an Apache project.

Here are some key points about Kafka that you might find useful for your blog:

1. Overview:

• Kafka is designed as a distributed, fault-tolerant, and scalable platform for handling real-time data streams.
• It provides a publish-subscribe messaging system and a fault-tolerant storage system for streams of records.
• Kafka is often used for building data pipelines, collecting and processing data in real-time, and integrating various data sources.

2. Architecture:

• Kafka follows a distributed architecture with brokers, producers, and consumers.
• Brokers are Kafka servers that store data and serve client requests.
• Producers publish messages to Kafka topics, and consumers subscribe to these topics to process the messages.

3. Topics and Partitions:

• Topics are logical channels for publishing and subscribing to messages.
• Partitions allow Kafka to parallelize processing, and each partition is stored on a different broker.
• Partitioning enables horizontal scaling and high throughput.

4. Producers and Consumers:

• Producers are responsible for sending messages to Kafka topics.
• Consumers subscribe to topics and process the messages.
• Kafka provides strong durability guarantees, and messages are persisted on disk.

5. Durability and Replication:

• Kafka ensures durability by persisting messages to disk, preventing data loss.
• Data replication is used to provide fault tolerance. Each partition has multiple replicas spread across different brokers.

6. Scalability:

• Kafka scales horizontally by adding more brokers to the cluster.
• It can handle large amounts of data and high message throughput.

Why Kafka Without ZooKeeper?

Kafka uses the Raft consensus algorithm for leader election in its Kafka Raft Metadata mode, which eliminates the dependency on ZooKeeper for managing cluster metadata. The Raft algorithm is a consensus protocol designed to ensure fault-tolerant replication of state machines in a distributed system.

Prerequisites:?

1. We need to pre-set up the Kubernetes cluster.

2. We need to install Helm.

3. We need to install Git.

Deployment Components:

1. Namespace:

?This is the namespace within which all components will be scoped.

2. Service Account:?

Service accounts control permissions and access to resources within the cluster.

3. Headless Service:?

It exposes ports 9092 (for Kafka clients) and 29093 (for Kafka Controller).

4.StatefulSet:?

It manages Kafka pods and ensures they have stable hostnames and storage.

Kafka cluster deployment without ZooKeeper

Step 1: After Helm & git installed, clone code from git-hub by the below command

Git URL: https://github.com/siva-pra/kafka-single-pod.git

?git clone https://github.com/siva-pra/kafka-single-pod.git        

Step 2: Once the clone completed, Change the directory and untar the package.

cd kafka-single-pod && tar xvzf kafka-0.1.0.tgz        

Step 3: Creating a namespace and deploying a kafka pod.

kubectl create namespace kafka        

Step 4: Install Helm package and deploy Kafka controller pod

helm install kafka kafka -n kafka        

Step 5: Now you can see that the Kafka pod is deployed successfully.

kubectl get all -n kafka        

Step 6: Now login to kafka pod and perform the test

kubectl exec -it kafka-0 -n kafka -- /bin/bash        

Note: While installing helm charts it creates a topic with the name “my-topic”

Step 7: Test Kafka Producer and Consumer

# Example producer command

kafka-console-producer.sh --bootstrap-server kafka-service:9092 --topic my-topic        

Note: Take a new terminal and log into Kafka pod and type below command to check in the Consumer terminal

# Example consumer command

kafka-console-consumer.sh --bootstrap-server kafka-service:9092 --topic my-topic --from-beginning        

Step 8: Cleanup

Provide instructions for uninstalling the Kafka Helm chart.

helm uninstall kafka -n kafka        

This is how Running Kafka on a Single Node in K8s Cluster created.

I trust our DevOps team's article offered valuable insights. Whether you're a newcomer or an experienced professional, the journey of learning never ends. Buildbot serves as our collaborative platform, nurturing a culture of continuous learning within our community. Excited to hear your thoughts as we embark on this exploration and learning journey together! ??

#Buildbot #DevOps

Written By,

DevOps Team

Buildbot Technologies Private Limited