OpenShift 4.X Operators - Installing and playing with AMQ Streams operator

Red Hat AMQ Streams is a massively scalable, distributed, and high-performance data streaming platform based on the Apache Kafka? project. AMQ Streams provides an event streaming backbone that allows microservices and other application components to exchange data with extremely high throughput and low latency. This week, I walk you through the Operator Hub on OpenShift4 and play with AMQ Streams operator.

More on AMQ Streams

The core capabilities include:

• A pub/sub messaging model, similar to a traditional enterprise messaging system, in which application components publish and consume events to/from an ordered stream
• The long term, fault-tolerant storage of events
• The ability for a consumer to replay streams of events
• The ability to partition topics for horizontal scalability

Red Hat AMQ Streams provides a way to run an Apache Kafka cluster on OpenShift in various deployment configurations.

Installation

From your OpenShift Web Console under Admin perspective, go to Operator Hub tab and search for "AMQ". Find AMQ Streams from the search result and click "Install".

Rather than installing it on all namespaces, choose a specific namespace that you've created previously (oc new-project <name>). Under this specific project (project and namespace are used interchangeably), you'll see the newly installed AMQ Streams operator under "Installed Operators" tab.

Setting up a basic Kafka cluster

For the purpose of this article, I'll setup a basic Kafka cluster. Click on AMQ Streams and navigate to Kafka tab. Click on "Create Kafka" and use all default fields in the YAML.

Once created, click on "my-cluster" and notice how the type is "kafka" as opposed to a pod or deployment. One of the benefits of operators is, it uses custom resources. This enables us to talk in terms of the business logic of the specific service. Under "Resources" tab of my-cluster, a number of resources are being created:

oc get pods will also show the pods being created for this Kafka service:

Setting up a Kafka Topic

Head over to "Installed Operators" --> "AMQ Streams" and click on "Kafka Topic"

Click on "Create Kafka Topic" and on the YAML, change the default name from my-topic to hello-world. This will be deployed on my-cluster which you just created. Once you've created this topic, the operator receives an event that this new Kafka topic object has been created and it takes necessary steps inside the Kafka cluster itself.

rsh to view the Kafka Topic

Under your project, find the newly created Kafka pods running and we can rsh to a pod to ensure that the topic has been created. The following image shows that the hello-world topic was present inside the Kafka cluster:

oc rsh <name-of-pod>

cd bin

./kafka-topics.sh --list --zookeeper localhost:2181

You can also refer to the above code snippet. This concludes my article on AMQ Streams Operator. Next week, I'll see you with my final article on the OpenShift4.X weekly article series - an end-to-end project built and deployed on OpenShift.