Migrate Nifi standalone to cluster mode
https://upload.wikimedia.org/wikipedia/commons/thumb/f/ff/Apache-nifi-logo.svg/1200px-Apache-nifi-logo.svg.png

Migrate Nifi standalone to cluster mode

We do have articles and Apache Nifi documents that helps to setup a new Nifi cluster. However, I was unable to find any documentation that had straight steps mentioned to migrate a Standalone Nifi instance to Cluster mode (Please suggest if there is any). Instead, got reference from multiple articles and was able to write down plain and simple steps that are needed to be followed to perform this task. Please do share your views / thoughts towards the content of the article.


References:

https://pierrevillard.com/2016/08/13/apache-nifi-1-0-0-cluster-setup/

https://mintopsblog.com/2017/11/12/apache-nifi-cluster-configuration/


Configure ZK Nodes: "<nifi package>/conf/zookeeper.properties" on all nodes (preferred 3)

server.1=node-1:2888:3888;2181
server.2=node-2:2888:3888;2181

server.3=node-3:2888:3888;2181

Configure ZK data directory on each node:

Node 1:

mkdir ./state
mkdir ./state/zookeeper

echo 1 > ./state/zookeeper/myid

Node 2:

mkdir ./state
mkdir ./state/zookeeper

echo 2 > ./state/zookeeper/myid

Node 3:

mkdir ./state
mkdir ./state/zookeeper

echo 3 > ./state/zookeeper/myid

Configure Nifi: ‘./conf/nifi.properties‘

####Same for all nodes####

nifi.state.management.embedded.zookeeper.start=true
nifi.zookeeper.connect.string=node-1:2181,node-2:2181,node-3:2181
nifi.cluster.protocol.is.secure=false   ####Set true to setup a secured cluster

On Each Node:

Node 1:

###exchanges between the nodes and the cluster coordinator,###

nifi.cluster.is.node=true
nifi.cluster.node.address=node-1
nifi.cluster.node.protocol.port=9999
nifi.cluster.node.protocol.threads=10
nifi.cluster.node.event.history.size=25
nifi.cluster.node.connection.timeout=5 sec
nifi.cluster.node.read.timeout=5 sec

nifi.cluster.firewall.file=

###exchanges between the nodes (to balance the data of the flows)###

nifi.remote.input.host=node-1
nifi.remote.input.secure=false
nifi.remote.input.socket.port=9998
nifi.remote.input.http.enabled=true

nifi.remote.input.http.transaction.ttl=30 sec

###FQDN for the web server property###

nifi.web.http.host=node-1

Node 2:

###exchanges between the nodes and the cluster coordinator,###

nifi.cluster.is.node=true
nifi.cluster.node.address=node-2
nifi.cluster.node.protocol.port=9999
nifi.cluster.node.protocol.threads=10
nifi.cluster.node.event.history.size=25
nifi.cluster.node.connection.timeout=5 sec
nifi.cluster.node.read.timeout=5 sec

nifi.cluster.firewall.file=

###exchanges between the nodes (to balance the data of the flows)###

nifi.remote.input.host=node-2
nifi.remote.input.secure=false
nifi.remote.input.socket.port=9998
nifi.remote.input.http.enabled=true

nifi.remote.input.http.transaction.ttl=30 sec

###FQDN for the web server property###

nifi.web.http.host=node-2

Node 3:

###exchanges between the nodes and the cluster coordinator,###

nifi.cluster.is.node=true
nifi.cluster.node.address=node-3
nifi.cluster.node.protocol.port=9999
nifi.cluster.node.protocol.threads=10
nifi.cluster.node.event.history.size=25
nifi.cluster.node.connection.timeout=5 sec
nifi.cluster.node.read.timeout=5 sec

nifi.cluster.firewall.file=

###exchanges between the nodes (to balance the data of the flows)###

nifi.remote.input.host=node-3
nifi.remote.input.secure=false
nifi.remote.input.socket.port=9998
nifi.remote.input.http.enabled=true

nifi.remote.input.http.transaction.ttl=30 sec

###FQDN for the web server property###

nifi.web.http.host=node-3

Start Nifi on each node

Giovanni Pipitò

Systems & Network Engineer

2 年

it doesn't work

回复
shoeb ahmad shaikh

Deputy Manager at Vodafone BigData/Hadoop | Kafka Administrator

4 年

Awesome!!!

KHALID MULLA

Service Lead at Vodafone Intelligent Solutions (VOIS), HSW Champion at VOIS , CSR Volunteer for Brighter Futures@VOIS

4 年

Thanks for sharing Sunil

Manish Warang

Infrastructure Solution Architect | Cloud Specialist

4 年

Thanks for sharing

要查看或添加评论,请登录

Sunil Muniyal的更多文章

社区洞察

其他会员也浏览了