Configuring MSK (Managed Streaming for Apache Kafka) on a Server

Amazon Managed Streaming for Apache Kafka (MSK) is a fully managed service that makes it easy to build and run applications that use Apache Kafka for streaming data. This article will guide you through the steps required to configure MSK on a server.

Amazon Managed Streaming for Apache Kafka (MSK) Reference: https://aws.amazon.com/msk/

Prerequisites

1. AWS Account: You must have an active AWS account or you can create one here.
2. AWS CLI: Ensure that the AWS Command Line Interface (CLI) is installed and configured with appropriate permissions.
3. Apache Kafka: Basic understanding of Apache Kafka.
4. Linux Server: A Linux server (e.g., EC2 instance) with sudo privileges.
5. Java: Install java-1.8.0

Step 1: Install and Configure AWS CLI

Install the AWS CLI on your Linux server using below command. If it is already installed, ensure it's up to date.

Install AWS CLI

sudo apt update
sudo apt install awscli -y        

Configure AWS CLI

After installation, configure the AWS CLI with your credentials.

aws configure        

You will be prompted to enter your AWS Access Key ID, Secret Access Key, region, and output format.

Step 2: Create an MSK Cluster

Define the Configuration

Create a configuration JSON file named msk-config.json with the following content:

{
  "ClusterName": "AkMSKCluster",
  "BrokerNodeGroupInfo": {
    "InstanceType": "kafka.m5.large",
    "ClientSubnets": [
      "subnet-xxxxxxxx",
      "subnet-xxxxxxxx"
    ],
    "SecurityGroups": [
      "sg-xxxxxxxx"
    ]
  },
  "NumberOfBrokerNodes": 3,
  "EncryptionInfo": {
    "EncryptionInTransit": {
      "ClientBroker": "TLS",
      "InCluster": true
    }
  },
  "KafkaVersion": "2.8.0",
  "LoggingInfo": {
    "BrokerLogs": {
      "CloudWatchLogs": {
        "Enabled": true,
        "LogGroup": "/aws/msk/AkMSKCluster"
      }
    }
  }
}        

Note: ClusterName, InstanceType and other details needs to be configured based on requirements and also make sure to replace subnet-xxxxxxxx and sg-xxxxxxxx with your actual subnet IDs and security group IDs.

Create the Cluster

Use the AWS CLI to create the MSK cluster with the configuration file, which might take some time.

aws kafka create-cluster --cli-input-json file://msk-config.json        

You can monitor the status of your cluster using:

aws kafka describe-cluster --cluster-arn <cluster-arn>        

Note: Replace <cluster-arn> with the ARN of your cluster, which you can obtain from the output of the previous command.

Step 3: Configure Kafka Clients

Install Kafka Client

Step-1: Download and extract Apache Kafka on your Linux server:

wget https://downloads.apache.org/kafka/2.8.0/kafka_2.12-2.8.0.tgz        

Step-2: Unzip the file at a specific location using below command.

tar -xzf kafka_2.12-2.8.0.tgz
cd kafka_2.12-2.8.0        

Step-3: On your client machine, create a JAAS configuration file named "users_jaas.conf" that contains the user credentials stored in your secret:

KafkaClient { 
	org.apache.kafka.common.security.scram.ScramLoginModule required 
	"username": "username",
	 "password": "pass"; 
	
};        

Step-4: Create a file named?kafka.client.truststore.jks?in a?./tmp?directory.

Use the following command to copy the JDK key store file from your JVM cacerts folder into the kafka.client.truststore.jks file that you created in the previous step.

cp /usr/lib/jvm/JDKFolder/jre/lib/security/cacerts /tmp/kafka.client.truststore.jks        

Note: Replace JDKFolder with the name of the JDK folder on your instance. For example, your JDK folder might be named java-1.8.0-openjdk-1.8.0.201.b09-0.amzn2.x86_64.

Step-5: Create a "client_sasl.properties" file inside? the bin directory of kafka installation with below details.

security.protocol=SASL_SSL
sasl.mechanism=SCRAM-SHA-512
ssl.truststore.location=<path-to-keystore-file>/kafka.client.truststore.jks        

Step-6: Use the following command to export your JAAS config file as KAFKA_OPTS environment parameter.

export KAFKA_OPTS=-Djava.security.auth.login.config=<path-to-jaas-file>/users_jaas.conf        

Set Up Kafka Producer and Consumer

Move to the bin directory of Kafka installation.

Usecase-1: Create a new topic:

./kafka-topics.sh --create --bootstrap-server <broker-1>:9096 --command-config client_sasl.properties --replication-factor 2 --partitions 1 --topic NewTopic        

Usecase-2: Start a Kafka producer:

./kafka-console-producer.sh --broker-list <broker-1>:9096 --producer.config client_sasl.properties --topic NewTopic        

Type some messages to produce to the topic.

Usecase-3: To consume these messages, start a Kafka consumer in another terminal:

./kafka-console-consumer.sh --bootstrap-server <broker-1>:9096 --consumer.config client_sasl.properties --topic NewTopic --from-beginning        

You should see the messages you produced in the consumer terminal.

Usecase-4: To see the list of topic in a broker, use below command:

./kafka-consumer-groups.sh --list --bootstrap-server <broker-1>:9096 --command-config client_sasl.properties        

Conclusion

By following these steps, you can configured MSK on your server using Linux commands. You have created an MSK cluster, configured Kafka clients, produced and consumed the topics. This setup allows you to utilize the power of Apache Kafka in a fully managed environment provided by AWS, ensuring scalability, security, and ease of management for your streaming data applications.