?A guide to installing Elastiflow

Elastiflow is some great software but has two problems. It is a hungry beast as you need to provide it some decent hardware. Secondly, the documentation and recipes are poor.

The minimum hardware required is a an i5 with 4 cores, 8 GB of RAM and a 1 TB hard disk. The recommended hardware is a xeon with 8 cores, 16 GB of RAM and hard disk on RAID 1+0.

I use the software for our SDWAN solution which supports netflow and IPFIX. It also has embedded ndpi.

Multiple attempts trying to install it using debian buster and ubuntu failed, I was eventually able to get it running using centos7 using this iso. I created a USB boot stick to load centos7 using Etcher. Once centos7 is installed we can login and start the Elastiflow install process.

• Update Centos7

sudo yum -y update
sudo yum install epel-release
sudo yum -y install nano git nginx
sudo reboot

• Install java. The ELK stack which Elastiflow uses WILL only work with version 8.

sudo yum -y install java-1.8.0-openjdk  java-1.8.0-openjdk-deve

• Add the repository for the ELK installation as follows:

sudo nano /etc/yum.repos.d/elasticsearch.repo

The contents should contain:

[elasticsearch-7.x]
name=Elasticsearch repository for 7.x packages
baseurl=https://artifacts.elastic.co/packages/oss-7.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md

• Update the packages.

sudo yum clean all
sudo yum makecache

• Install Elasticsearch 7

sudo yum -y install elasticsearch-oss

• Verify that Elasticsearch 7.x is installed correctly.

$ rpm -qi elasticsearch-oss

Name        : elasticsearch-oss
Epoch       : 0
Version     : 7.5.2
Release     : 1
Architecture: x86_64
Install Date: Wed 05 Feb 2020 18:31:26 SAST
Group       : Application/Internet
Size        : 392860309
License     : ASL 2.0
Signature   : RSA/SHA512, Wed 15 Jan 2020 15:53:50 SAST, Key ID d27d666cd88e42b4
Source RPM  : elasticsearch-oss-7.5.2-1-src.rpm
Build Date  : Wed 15 Jan 2020 14:21:45 SAST
Build Host  : packer-virtualbox-iso-1576086839
Relocations : /usr
Packager    : Elasticsearch
Vendor      : Elasticsearch
URL         : https://www.elastic.co/
Summary     : Distributed RESTful search engine built for the cloud
Description :
Reference documentation can be found at
https://www.elastic.co/guide/en/elasticsearch/reference/current/index.html
and the 'Elasticsearch: The Definitive Guide' book can be found at
https://www.elastic.co/guide/en/elasticsearch/guide/current/index.html

• Configure Elasticsearch 7.x.

$ sudo vi /etc/elasticsearch/jvm.options

Modify these settings.

-Xms4g
-Xmx4g

• Create a new data path for Elasticsearch

sudo mkdir /home/ES
sudo chown elasticsearch:elasticsearch /home/ES

$ sudo vi /etc/elasticsearch/elasticsearch.yml

Change the following line

path.data: /var/lib/elasticsearch to path.data: /home/ES

$ sudo vi /etc/elasticsearch/elasticsearch.yml

Add the following lines to the file.

indices.query.bool.max_clause_count: 8192
search.max_buckets: 100000

• Start Elasticsearch 7.x and ensure that it will start on boot.

sudo systemctl enable elasticsearch
sudo systemctl start elasticsearch

• To check that Elasticsearch 7.x is running.

sudo systemctl status elasticsearch

● elasticsearch.service - Elasticsearch

   Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: disabled)
   Active: active (running) since Sat 2020-02-08 16:11:35 SAST; 1h 32min ago
     Docs: https://www.elastic.co
 Main PID: 1794 (java)
   CGroup: /system.slice/elasticsearch.service
           └─1794 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.c...

Feb 08 16:11:22 w17-flow systemd[1]: Starting Elasticsearch...
Feb 08 16:11:27 w17-flow elasticsearch[1794]: OpenJDK 64-Bit Server VM warni....
Feb 08 16:11:35 w17-flow systemd[1]: Started Elasticsearch.
Hint: Some lines were ellipsized, use -l to show in full.

• Install Kibana 7.

sudo yum install kibana-oss

• Configure Kibana.

$ sudo vi /etc/kibana/kibana.yml
server.host: "0.0.0.0"
server.name: "elastiflow-fusion"

• Start Kibana and ensure that it will start on boot-up.

sudo systemctl enable kibana
sudo systemctl start kibana

• Configure the firewall rules.

sudo firewall-cmd --add-port=5601/tcp --permanent
sudo firewall-cmd --add-port=2055/tcp --permanent
sudo firewall-cmd --add-port=2055/udp --permanent
sudo firewall-cmd --add-port=6343/tcp --permanent
sudo firewall-cmd --add-port=6343/udp --permanent
sudo firewall-cmd --add-port=4739/tcp --permanent
sudo firewall-cmd --add-port=4739/udp --permanent
sudo firewall-cmd --add-port=19999/tcp --permanent
sudo firewall-cmd --permanent --zone=public --add-service=http 
sudo firewall-cmd --permanent --zone=public --add-service=https
sudo firewall-cmd --reload

• Test that Kibana is work by accessing https://ip-address:5601 You should see something like the below:

• Configure nginx

$ echo "fusion:`openssl passwd -apr1 usesdwan`" | sudo tee -a /etc/nginx/htpasswd.kibana
sudo setsebool httpd_can_network_connect 1 -P
sudo vi /etc/nginx/conf.d/elastiflow-fusion.conf


server {
    listen 80;

    server_name elastiflow-fusion;

    auth_basic "Restricted Access";
    auth_basic_user_file /etc/nginx/htpasswd.kibana;

    location / {
        proxy_pass https://localhost:5601;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection 'upgrade';
        proxy_set_header Host $host;
        proxy_cache_bypass $http_upgrade;
    }
    
}
 

sudo vi /etc/nginx/nginx.conf

Remove the server { } block in this file.  The last lines should be:

 include /etc/nginx/conf.d/*.conf;
}
      

• Edit the file sudo vi /etc/nginx/nginx.conf file:

Set in http block which affects all server blocks (virtual hosts).

http {
    ...
    client_max_body_size 100M;
}    

• Start nginx and ensure that it will start on boot-up.

sudo systemctl enable nginx
sudo systemctl start nginx

• You should now be able to logon using the credentials created on https://ip-address Use nginx as the port forwarded access instead of Kibana.
• Install Logstash.

sudo yum -y install logstash

• Configure Logstach 7.x.

$ sudo vi /etc/logstash/jvm.options

Modify these settings.

-Xms4g
-Xmx4g

• Update Logstash.

sudo /usr/share/logstash/bin/logstash-plugin install logstash-codec-sflow
sudo /usr/share/logstash/bin/logstash-plugin update logstash-codec-netflow
sudo /usr/share/logstash/bin/logstash-plugin update logstash-input-udp
sudo /usr/share/logstash/bin/logstash-plugin update logstash-input-tcp
sudo /usr/share/logstash/bin/logstash-plugin update logstash-filter-dns
sudo /usr/share/logstash/bin/logstash-plugin update logstash-filter-geoip

• Start Logstash and ensure that it will start on boot-up.

sudo systemctl enable logstash
sudo systemctl start logstash

• Verify that Logstash is working

sudo systemctl status logstash
logstash.service - logstash
   Loaded: loaded (/etc/systemd/system/logstash.service; enabled; vendor preset: disabled)
   Active: active (running) since Mon 2020-03-02 21:24:43 SAST; 2s ago
 Main PID: 25871 (java)
   CGroup: /system.slice/logstash.service
           └─25871 /bin/java -Xms4g -Xmx4g -XX:+UseConcMarkSweepGC -XX:CMSIni...

Mar 02 21:24:43 localhost.localdomain systemd[1]: Started logstash.

• Install Elastiflow.

cd $Home
mkdir flowtemp
cd flowtemp
sudo git clone https://github.com/robcowart/elastiflow.git
sudo cp -arv elastiflow/logstash/elastiflow/. /etc/logstash/elastiflow/
sudo cp -arv elastiflow/logstash.service.d/. /etc/systemd/system/logstash.service.d/

• Configure Logstack for Elastiflow by adding the lines below:

sudo nano /etc/logstash/pipelines.yml


- pipeline.id: elastiflow
  path.config: "/etc/logstash/elastiflow/conf.d/*.conf"

• Now

sudo reboot

• Verify that Logstash has started...

sudo tail /var/log/logstash/logstash-plain.log -f


[2020-02-08T16:13:00,860][INFO ][logstash.inputs.udp      ] Starting UDP listener {:address=>"0.0.0.0:6343"}
[2020-02-08T16:13:00,884][INFO ][logstash.inputs.udp      ] Starting UDP listener {:address=>"0.0.0.0:2055"}
[2020-02-08T16:13:00,968][INFO ][logstash.agent           ] Pipelines running {:count=>1, :running_pipelines=>[:elastiflow], :non_running_pipelines=>[]}
[2020-02-08T16:13:00,987][WARN ][logstash.inputs.udp      ] Unable to set receive_buffer_bytes to desired size. Requested 33554432 but obtained 212992 bytes.
[2020-02-08T16:13:00,988][WARN ][logstash.inputs.udp      ] Unable to set receive_buffer_bytes to desired size. Requested 33554432 but obtained 212992 bytes.
[2020-02-08T16:13:00,989][WARN ][logstash.inputs.udp      ] Unable to set receive_buffer_bytes to desired size. Requested 33554432 but obtained 212992 bytes.
[2020-02-08T16:13:01,002][INFO ][logstash.inputs.udp      ] UDP listener started {:address=>"0.0.0.0:4739", :receive_buffer_bytes=>"212992", :queue_size=>"4096"}
[2020-02-08T16:13:01,002][INFO ][logstash.inputs.udp      ] UDP listener started {:address=>"0.0.0.0:6343", :receive_buffer_bytes=>"212992", :queue_size=>"4096"}
[2020-02-08T16:13:01,002][INFO ][logstash.inputs.udp      ] UDP listener started {:address=>"0.0.0.0:2055", :receive_buffer_bytes=>"212992", :queue_size=>"4096"}
[2020-02-08T16:13:01,220][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}

• Download file for the dashboard from https://github.com/robcowart/elastiflow/blob/master/kibana/elastiflow.kibana.7.5.x.ndjson

Log on the server web page and navigate to Management/Saved Objects. Import the above file.

sudo reboot

Logon in again and go to the dashboard. You will be requested to create an index. Just use * and next and follow the prompts.

You can now go to Elastiflow - Overview and you should see something like this:

That is provided you have setup your networking kit to send data to Elastiflow! This setup is not documented here and is specific to each vendor's kit!

To save disk space you can change the setting for ELASTIFLOW_KEEP_ORIG_DATA to from the default of false to true.

You are able to adjust more settings by using the reference here. Because Elastiflow is resource intensive it is a good idea to install a system monitor like glances. On centos7 you need to use pip.

sudo yum install python-pip
sudo pip install glances

Another alternative is netdata. Run the install script as described on the web page and voila. You can now see what the Elastiflow is doing at https://ip-address:19999 It has detailed graphs including those for Elasticsearch as well as Logstash!

Ronald Bartels works at Fusion Broadband and is driving SDWAN adoption in South Africa.

Message me to find out more about our networking solutions!