Migrate Nifi standalone to cluster mode
Sunil Muniyal
Big data Operations (Cloudera) | Cloud Architecture (AWS / Azure) | Certified Azure Data Engineer
We do have articles and Apache Nifi documents that helps to setup a new Nifi cluster. However, I was unable to find any documentation that had straight steps mentioned to migrate a Standalone Nifi instance to Cluster mode (Please suggest if there is any). Instead, got reference from multiple articles and was able to write down plain and simple steps that are needed to be followed to perform this task. Please do share your views / thoughts towards the content of the article.
References:
https://pierrevillard.com/2016/08/13/apache-nifi-1-0-0-cluster-setup/
https://mintopsblog.com/2017/11/12/apache-nifi-cluster-configuration/
Configure ZK Nodes: "<nifi package>/conf/zookeeper.properties" on all nodes (preferred 3)
server.1=node-1:2888:3888;2181 server.2=node-2:2888:3888;2181
server.3=node-3:2888:3888;2181
Configure ZK data directory on each node:
Node 1:
mkdir ./state mkdir ./state/zookeeper
echo 1 > ./state/zookeeper/myid
Node 2:
mkdir ./state mkdir ./state/zookeeper
echo 2 > ./state/zookeeper/myid
Node 3:
mkdir ./state mkdir ./state/zookeeper
echo 3 > ./state/zookeeper/myid
Configure Nifi: ‘./conf/nifi.properties‘
####Same for all nodes####
nifi.state.management.embedded.zookeeper.start=true nifi.zookeeper.connect.string=node-1:2181,node-2:2181,node-3:2181 nifi.cluster.protocol.is.secure=false ####Set true to setup a secured cluster
On Each Node:
Node 1:
###exchanges between the nodes and the cluster coordinator,###
nifi.cluster.is.node=true nifi.cluster.node.address=node-1 nifi.cluster.node.protocol.port=9999 nifi.cluster.node.protocol.threads=10 nifi.cluster.node.event.history.size=25 nifi.cluster.node.connection.timeout=5 sec nifi.cluster.node.read.timeout=5 sec
nifi.cluster.firewall.file=
###exchanges between the nodes (to balance the data of the flows)###
nifi.remote.input.host=node-1 nifi.remote.input.secure=false nifi.remote.input.socket.port=9998 nifi.remote.input.http.enabled=true
nifi.remote.input.http.transaction.ttl=30 sec
###FQDN for the web server property###
nifi.web.http.host=node-1
Node 2:
###exchanges between the nodes and the cluster coordinator,###
nifi.cluster.is.node=true nifi.cluster.node.address=node-2 nifi.cluster.node.protocol.port=9999 nifi.cluster.node.protocol.threads=10 nifi.cluster.node.event.history.size=25 nifi.cluster.node.connection.timeout=5 sec nifi.cluster.node.read.timeout=5 sec nifi.cluster.firewall.file=
###exchanges between the nodes (to balance the data of the flows)###
nifi.remote.input.host=node-2 nifi.remote.input.secure=false nifi.remote.input.socket.port=9998 nifi.remote.input.http.enabled=true nifi.remote.input.http.transaction.ttl=30 sec
###FQDN for the web server property###
nifi.web.http.host=node-2
Node 3:
###exchanges between the nodes and the cluster coordinator,###
nifi.cluster.is.node=true nifi.cluster.node.address=node-3 nifi.cluster.node.protocol.port=9999 nifi.cluster.node.protocol.threads=10 nifi.cluster.node.event.history.size=25 nifi.cluster.node.connection.timeout=5 sec nifi.cluster.node.read.timeout=5 sec nifi.cluster.firewall.file=
###exchanges between the nodes (to balance the data of the flows)###
nifi.remote.input.host=node-3 nifi.remote.input.secure=false nifi.remote.input.socket.port=9998 nifi.remote.input.http.enabled=true nifi.remote.input.http.transaction.ttl=30 sec
###FQDN for the web server property###
nifi.web.http.host=node-3
Systems & Network Engineer
2 年it doesn't work
Deputy Manager at Vodafone BigData/Hadoop | Kafka Administrator
4 年Awesome!!!
Service Lead at Vodafone Intelligent Solutions (VOIS), HSW Champion at VOIS , CSR Volunteer for Brighter Futures@VOIS
4 年Thanks for sharing Sunil
Infrastructure Solution Architect | Cloud Specialist
4 年Thanks for sharing