Setup ELK Stack Architecture

Continuous Monitoring:

Continuous Monitoring is an important part of software development. It measures the health of software and improves the quality of the software, and this is based on the feedback we get from the insights gained from monitoring.

1. Uptime MonitoringICMP - ping --> To check whether the server is up or notHTTP - curl --> To check whether the application is up or notTCP - telnet -->To check the network packet response
2. Server MonitoringCPU usageMemory usageNetwork usageDisk usageSwap usageProcessBy using the command htop or top in Linux
3. Log MonitoringApplication logsSystem logs

/var/log/nginx        

In Centos/RedHat/Amazon

/var/log/httpd        

access logs -> /var/log/apache2/access.log

There details will be available in the access logs

1. Client IP
2. Path
3. User-agent (device and browser)
4. Event details
5. HTTP status code

error logs -> /var/log/apache2/error.log

1. Error Message

4. Security Monitoring

• Login Events

Ubuntu -> /var/log/auth.log

Centos/RedHat/Amazon -> /var/log/secure

• User Activity

Install Audit -> /var/log/audit

• Audit Logs

File/dir -> /home/ubuntu/data.txt

ELK Stack

ELK Stack is a open source tools that allow us to monitor, collect, process analyze & visualize data consisting of Elasticsearch, Logstash and Kibana.

Components of ELK Stack

Elasticsearch

1. No-SQL database
2. JSON document
3. Used for storing & searching collected data
4. Built-in Java

Logstash

1. Log ingestion (responsible to store logs into the Elasticsearch)
2. Log parsing (customize data before sending to the Elasticsearch)
3. Used for collecting & filtering the input data
4. Built-in Java

Kibana

1. Kibana is a graphical user interface that is used to display the data that was collected and stored in Elasticsearch.
2. Visualization tool (Web UI)

Beats

1. Multiple light weight data collectors

ELK-Stack

To set up the ELK-Stack architecture we need a good configuration machine of at least t2.large because ELK is a heavy software

Firstly we launched the EC2 instance

ELK Installation

$sudo apt update -y
$sudo apt install openjdk-11-jre -y        

$curl -fsSL https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -        

$echo "deb https://artifacts.elastic.co/packages/7.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-7.x.list        

Installing Elasticsearch

$sudo apt update -y
$sudo apt install elasticsearch -y
$sudo systemctl enable elasticsearch
$sudo systemctl start elasticsearch        

To check whether it's installed or not

$curl localhost:9200        

Installing Kibana with nginx

$sudo apt install nginx kibana -y
$sudo systemctl enable kibana
$sudo systemctl start kibana
$sudo nano /etc/kibana/kibana.yml        

To Verify whether Kibana is Installed or not

$curl localhost:5601        

After Installing Nginx, To check which ports are currently used by the server

 $ss -ntlup        

create the nginx config file

$cd /etc/nginx/sites-enabled 
$echo "" | sudo  tee default
$sudo nano /etc/nginx/sites-enabled/default        

Inside this file paste the nginx config and made the reverse proxy

server {
    listen 80;

    server_name server_ip;# replace server_ip with your ec2 instance ip

    location / {
        proxy_pass https://localhost:5601;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection 'upgrade';
        proxy_set_header Host $host;
        proxy_cache_bypass $http_upgrade;
    }
}        

$sudo systemctl restart nginx        

Installing Logstash

$sudo apt update -y
$sudo apt install logstash        

we need to open port 80 by server ip in the security group

By taking Port 80 in our browser, Now we can access Kibana

Install any beat by using the command

• Heartbeat - Uptime Monitoring
• Metricbeat - Server & Application
• Pocketbeat - Network Monitoring
• Filebeat - Log Monitoring
• Winlogbeat - Windows Monitoring
• Auditbeat - Security Monitoring

$sudo apt install metricbeat        

Type the command to get inside the path

$sudo nano /etc/metricbeat/metricbeat.yml        

Inside this file search Elasticsearch Output, If it is running on a separate server then we have to paste the IP in the place of localhost but in our case, it is in the same system.

To check the modules list in the metricbeat, use the command

$sudo metricbeat modules list        

If you want to allow any module then we have to use the command

$sudo metricbeat modules enable nginx        

$sudo systemctl start metricbeat
$sudo systemctl start nginx        

To Setup the default dashboard in Kibana

$sudo metricbeat setup
$sudo systemctl start metricbeat        

In the dashboard section, you'll find the multiple dashboards for multiple services

Stack Management > Index Management

Now we have to create an Index pattern without creating this we won't be able to visualize the data in Kibana

After adding the index pattern click to discover

Right now, we can see the data but it's in the raw format

System Overview

Host Overview

$sudo apt install filebeat        

$sudo systemctl start filebeat
$sudo systemctl start nginx        

Type the command to get inside the path

$sudo nano /etc/metricbeat/filebeat.yml        

Inside this file search Elasticsearch Output, If it is running on a separate server then we have to paste the IP in the place of localhost but in our case, it is in the same system.

To check the modules list in the metricbeat, use the command

$sudo filebeat modules list        

If you want to allow any module then we have to use the command

$sudo filebeat modules enable nginx        

In the dashboard section, you'll find the multiple dashboards for multiple services

Stack Management > Index Management

Dashboard in kibana

$sudo filebeat setup        

Now we have to create an Index pattern without creating this we won't be able to visualize the data in Kibana

Right now, we can see the data but it's in the raw format

Now we can see the data in the map format

Thank you for reading. I hope you found this article helpful.

Happy Learning :-)

Mounika Jilakari.