ELK and its importance

The success of a software product depends on how well we can maintain, enhance, and support a product you develop. There are multiple factors that aid in ensuring the product is up and running all the time. One such aspect is the resiliency: it is the ability to bounce back, and course-correct when any issue hits in production. The era of Cloud and Containerization has definitely taken the availability, resilience aspects to higher benchmarks. This article focuses on one other aspect, which is the efficient and timely management of application and system log data. 

Analyzing, monitoring logs, and sourcing system metrics from all systems running the product has been a key challenge. Imagine how many servers, networks, containers, datacenters you need to access to make sense of an issue in production. Given this complexity, a need arose for centralizing and managing the logs. The logs could be an application, operating system log, or container logs in their formats. Another key need was the ability to quickly detect the anomalies out of a plethora of logs. This is where companies like Elastic, Splunk to name a few are making a difference by fulfilling these needs. 

Let’s take the case of Elastic. Elastic has been preserving the ELK Stack, which has now evolved to BELK & SIEM solution. Too many acronyms to digest allow me to explain. ELK stands for Elastic Search, Log Stash, and Kibana. BELK- B stands for Beats. SIEM is another offering by Elastic, it stands for Security Information and Event Management System. This is for security teams to detect and respond to threats and collect information for security investigations.

Lets now understand what exactly ELK is all about.

• Elasticsearch is the distributed search and analytics engine at the heart of the Elastic Stack. It can index and search structured or unstructured text, numerical data, or geospatial data. Elastic Search has evolved from the Lucene search engine.
• Logstash and Beats facilitate collecting aggregating logs. It is an open-source data collection engine with real-time plumbing capabilities. Logstash can dynamically unify data from disparate sources and normalize the data into destinations of your choice. Beats are client installable plugins to source the data to Log stash.  There are plenty of plugins to understand logs of different formats starting from SYSLog files to IIS Logs. 
• Kibana is the Visualization engine. The data to Kibana is fired by the Elastic search engine indexes. Once you have the indexed data from Elastic search configured, you can play around with the queries and Charts. There is even a Machine learning component available to mine and automatically figure out anomalies in the logs. The sweet aspect is that you can view in one window App logs, CPU metrics, heat maps of your choice.

Elastic offers 3 versions of the product 

1. The traditional on-prem version 
2. Elastic in hosted cloud with BELK on AWS or Google cloud 
3. Elastic cloud, It has its own cloud service

The Elastic cloud just takes 5 minutes to spin off your BELK stack. Note, ELK is also offered by AWS, the AWS version of ELK is a basic version with fewer bells and whistles maintained by AWS, but for the best version of ELK Elastic is the place to go. The best place to learn more about elastic is of course https://www.elastic.co. They have very good documentation of each product offered. Try their elastic cloud it's free for use for 14 days and you will be surprised how fast you will be up and running with a Kibana dashboard.