Common ELK Stack Errors
Manmohan .
Sr.Executive DevOps Engineer | Ex-DevOps Engineer at Mediamelon.Inc | DevOps | AWS | Big Data | Hadoop | Kubernetes | Jenkins | Docker | AKS | EKS | Azure DevOps | CI-CD | RHCSA8 | RH294 | GitOps | DevSecOpsk
Issue: Kibana Unable to connect to Elasticsearch
When accessing Kibana via a web browser, you may encounter a page with this error:
Kibana error
Fatal Error
Kibana: Unable to connect to Elasticsearch
Error: Unable to connect to Elasticsearch
Error: Bad Gateway
:
This means that Kibana can’t connect to Elasticsearch. Elasticsearch may not be running, or Kibana may be configured to look for Elasticsearch on the wrong host and port.The reasons for this vary, but it is usually a matter of defining the Elasticsearch instance correctly in the Kibana configuration file.
Open the file at?/opt/kibana/config/kibana.yml?and verify that the server IP and host for ‘elasticsearch_url’ are configured correctly (both the URL and port):
# The Elasticsearch instance to use for all your queries.
elasticsearch_url: "https://localhost:9200":
now restart the kibana:
sudo service kibana restart
After Kibana has restarted, open Kibana in a web browser and verify that the error was resolved.If the problem persists, there may be an issue with Elasticsearch. Check out the Elasticsearch troubleshooting sections below.
Issue: Elasticsearch is Not Running
If Elasticsearch is not running, there are many potential causes. There are a number of indicators, and the most obvious one is that no no logs are appearing in Kibana. As specified above, the most reliable way to ping the Elasticsearch service is by doing CURL :
curl 'https://localhost:9200'
If all is well, you should see the following output in your terminal:
{
?"name" : "testing_demo",
?"cluster_name" : "elasticsearch",
?"version" : {
???"number" : "2.3.1",
???"build_hash" : "bd980929010aef404e7cb0843e61d0665269fc39",
???"build_timestamp" : "2016-04-04T12:25:05Z",
???"build_snapshot" : false,
???"lucene_version" : "5.5.0"
?},
If not, the output will look like this:
curl: (7) Failed to connect to localhost port 9200: Connection ref
Now where you have installed the Elasticsearch come over there and check the status of the Elasticsearch service:
sudo service elasticsearch status
If Elasticsearch is running, you will see this output:
Elasticsearch status (OK):
* elasticsearch is running
Otherwise, if the service is not running, you will see this message:
Elasticsearch status (Bad):
* elasticsearch is not running
In this case, you should follow the next few sections, which cover troubleshooting Elasticsearch.
Now, there are a number of possible reasons Elasticsearch is not running.
CAUSE: First, if you just installed Elasticsearch, you need to manually start the service because it is not started automatically upon installation:
sudo service elasticsearch start
* elasticsearch is running
If you still get a message that Elasticsearch is not running, you will have to dig in deeper. As with Logstash, the best place to try and debug the service is the log file:?/var/log/elasticsearch/elasticsearch.log.
Cause: Elasticsearch is Misconfigured
If you see log entries that indicate errors or exceptions (e.g.?ERROR,?Exception, or?error), try and find a line that indicates what caused the error. Here is an example of the error logs you will see if the Elasticsearch?network.host?is set to a hostname or IP address that is not resolvable:
..
[2022-08-04 15:24:43,495][INFO ][node ] [Shadrac] starting ...
[2020-08-04 15:24:43,626][ERROR][bootstrap ] [Shadrac] Exception
org.elasticsearch.transport.BindTransportException: Failed to resolve host [null]
at org.elasticsearch.transport.netty.NettyTransport.bindServerBootstrap(NettyTransport.java:402)
at org.elasticsearch.transport.netty.NettyTransport.doStart(NettyTransport.java:283)
at org.elasticsearch.common.component.AbstractLifecycleComponent.start(AbstractLifecycleComponent.java:85)
at org.elasticsearch.transport.TransportService.doStart(TransportService.java:153)
at org.elasticsearch.common.component.AbstractLifecycleComponent.start(AbstractLifecycleComponent.java:85)
at org.elasticsearch.node.internal.InternalNode.start(InternalNode.java:257)
at org.elasticsearch.bootstrap.Bootstrap.start(Bootstrap.java:160)
at org.elasticsearch.bootstrap.Bootstrap.main(Bootstrap.java:248)
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:32)
Caused by: java.net.UnknownHostException: incorrect_hostname: unknown error
....
Note that the last line of the line of the example logs indicates that?UnknownHostException:?incorrect_hostname?error has occurred. This particular example indicates that the?network.host?is set to?incorrect_hostname, which doesn’t resolve to anything. In a single-node Elasticsearch setup, this should be set to?localhost?or?127.0.0.1
To resolve this issue, edit the Elasticsearch configuration file.
sudo vi /etc/elasticsearch/elasticsearch.yml
The host configuration is located in the Network section of the Elasticsearch configuration file, and it should look like this:
//When Kibana and Elasticsearch are hosted on the same machine
network.host: localhost
http.port: 9200
//When Kibana and Elasticsearch are hosted on different machines
network.bind_host: 0.0.0.0
http.port: 9200
network.publish_host: <ServerIP>
Verify the configuration, and restart the service:
$ sudo service elasticsearch restart
If the issue is not the host definition, the log will give you an indication as to the cause of the error and will help you resolve it. Search the Elastic forums — the chances are that someone else has encountered the issue before.
And one last tip (on Ubuntu only): If you had Elasticsearch working properly and it suddenly it stopped, this might be due to a restart of your server as Elasticsearch is not configured to start on boot. To change this, you can use:
$ sudo update-rc.d elasticsearch defaults 95 10
Issue: Kibana No Default Index Pattern Warning or Kibana is Unable to Fetch Mapping
In this case, Kibana has established a connection with Elasticsearch but cannot fetch mapping for an index:
As the message displayed on the grey button at the bottom of the page indicates, Kibana cannot find any indices stored in Elasticsearch that match the default logstash-* pattern — the default pattern for data being fed into the system by Logstash (which is the method Kibana assumes you are using).
If you’re not using Logstash to forward the data into Elasticsearch or if you’re using a non-standard pattern in your Logstash configuration, enter the index pattern that matches the name of one or more of your Elasticsearch indices. If Kibana finds the index pattern, the grey button will turn into a pretty green one, allowing you to define the index into Kibana.
If you are using the conventional Logstash configuration to ship data, then there is most likely a communication issue.?Typically, this means that your logs are not being stored in Elasticsearch due to communication issues from Logstash to Elasticsearch, and/or from your log shipper (e.g. Filebeat) to Logstash. In other words, your logs aren’t making it through the chain from Filebeat, to Logstash, to Elasticsearch for some reason.
Data Flows to kibana in this way:
To resolve communication issues between Logstash and Elasticsearch, run through the?Logstash troubleshooting?sections. To resolve communication issues between Filebeat and Logstash, run through the?Filebeat troubleshooting?sections.
Issue: Logstash is Not Running
A common issue causing Logstash to fail is a bad configuration. Logstash configuration files, which are located in the?/etc/logstash/conf.d?directory, follow strict syntax rules that, if broken, will cause a Logstash error. The best way to validate your configurations is to use the configtest parameter in the service command:
$ ?sudo service logstash configtest
If there’s a configuration error, it’ll show up in the output. Fix the syntax and try to run Logstash again:
$ ?sudo service logstash restart
Check the status of the service with:
$ ?sudo service logstash status
If Logstash is still not running after you fix the issue, take a look at the Logstash logs at:?/var/log/logstash/logstash.log.
Read the log message and try to fix the issue as reported in the log. Here’s an example of a log message warning us of a deprecated host configuration:
{:timestamp=>"2022-08-04T08:10:42.303000+0000", :message=>"Error: The setting `host` in plugin `elasticsearch` is obsolete and is no longer available. Please use the 'hosts' setting instead. You can specify multiple entries separated by comma in 'host:port' format. If you have any questions about this, you are invited to visit https://discuss.elastic.co/c/logstash and ask.", :level=>:error}
As the message itself points out, use the Elastic forums to search for an answer to the particular issue you’ve encountered and as reported in the log.
Cause: SSL Files Do Not Exist
Another common cause for Logstash not running is problem with the SSL certificate and key files. For example, if they don’t exist where your Logstash configuration specifies them to, your logs will show an error like this:
Logstash logs (SSL key file does not exist):
{:timestamp=>"2022-08-04T16:51:31.656000+0000", :message=>"Invalid setting for beats input plugin:\n\n input {\n beats {\n # This setting must be a path\n # File does not exist or cannot be opened /etc/pki/tls/certs/logstash-forwarder.crt\n ssl_certificate => \"/etc/pki/tls/certs/logstash-forwarder.crt\"\n ...\n }\n }", :level=>:error}
{:timestamp=>"2022-08-04T16:51:31.671000+0000", :message=>"Invalid setting for beats input plugin:\n\n input {\n beats {\n # This setting must be a path\n # File does not exist or cannot be opened /etc/pki/tls/private/logstash-forwarder.key\n ssl_key => \"/etc/pki/tls/private/logstash-forwarder.key\"\n ...\n }\n }", :level=>:error}
{:timestamp=>"2022-08-04T16:51:31.685000+0000", :message=>"Error: Something is wrong with your configuration.", :level=>:error}
To resolve this particular issue, you need to make sure that you have an SSL key file (generate one?if you forgot to), and that it is placed in the proper location (/etc/pki/tls/private/logstash-forwarder.key, in the example). If you already do have a key file, make sure to move it to the proper location, and ensure that the Logstash configuration is pointing to it.
Now, start the Logstash service:
sudo service logstash start
If the issue has been resolved, there should be no new log entries. After several seconds, check the status of the Logstash service:
sudo service logstash status
If it’s running, you have resolved the issue.
Issue: Logstash is Not Shipping Data
You’ve got Logstash purring like a cat, but there is no data being shipped into Elasticsearch. The prime suspect in this case is Elasticsearch, which may not be running for some reason or other. You can verify this by running the following cURL:
$ curl 'https://localhost:9200'
You should see the following output in your terminal:
{
?"name" : "testing_demo",
?"cluster_name" : "elasticsearch",
?"version" : {
???"number" : "2.3.1",
???"build_hash" : "bd980929010aef404e7cb0843e61d0665269fc39",
???"build_timestamp" : "2016-04-04T12:25:05Z",
???"build_snapshot" : false,
???"lucene_version" : "5.5.0"
?},
?"tagline" : "You Know, for Search"
}
If Elasticsearch is still not shipping data, skip over to the Elasticsearch troubleshooting section below for more reasons why Elasticsearch might not be running properly.
Another common issue that may be causing this error is a bad output configuration in the Logstash configuration file. Open the configuration file at:?/etc/logstash/conf.d/xxx.conf?and verify that the Elasticsearch host is configured correctly:
output {
?elasticsearch {}
}
Restart Logstash:
$ sudo service logstash restart
Issue: Filebeat is Not Running
Filebeat runs on your?Client?machines, and ships logs to your ELK server. If Filebeat isn’t running, you won’t be able to send your various logs to Logstash. As a result, the logs will not get stored in Elasticsearch, and they will not appear in Kibana. This section will show you how to check if Filebeat is functioning normally.
Verify Logs Are Successfully Being Shipped
The easiest way to tell if Filebeat is properly shipping logs to Logstash is to check for Filebeat errors in the syslog log.
sudo tail /var/log/syslog | grep filebeat
If everything is set up properly, you should see some log entries when you stop or start the Filebeat process, but nothing else.
If you don’t see any log entries, you should verify that Filebeat is running.
Verify Service is Running
The most basic thing to check is the status of Filebeat:
sudo service filebeat status
If Filebeat is running, you will see this output:
Output
* filebeat is running
Otherwise, if the service is not running, you will see this message:
Output
* filebeat is not running
If Filebeat isn’t running, try starting it with this command:
sudo service filebeat start
Then check the status again. If this doesn’t resolve the problem, the following sections will help you troubleshoot your Filebeat problems. We’ll cover common Filebeat issues, and how to resolve them.
Cause: Configuration Contains a Syntax Error
If Filebeat has errors in its configuration file, which is located at?/etc/filebeat/filebeat.yml, the service will not be able to start properly. It will immediately exit with errors like the following:
Output
Loading config file error: YAML config parsing failed on /etc/filebeat/filebeat.yml: yaml: line 13: could not find expected ':'. Exiting.
In this case, there is a typo in the configuration file. To resolve this issue, edit the offending portion of the Filebeat configuration. For guidance, follow the?Configure Filebeat?subsection of the?Set Up Filebeat (Add Client Servers)) of the ELK stack tutorial.
After editing the Filebeat configuration, attempt to start the service again:
sudo service filebeat start
If you see no error output, the issue is resolved.
Cause: SSL Certificate is Missing or Invalid
Communications between Filebeat and Logstash require an SSL certificate for authentication and encryption. If Filebeat is not starting properly, you should check the syslog for errors similar to the following:
Output
Error Initialising publisher: open /etc/pki/tls/certs/logstash-forwarder.crt: no such file or directory
This indicates that the?logstash-forwarder.crt?file is not in the appropriate location. To resolve this issue, copy the SSL certificate from the ELK server to your client machine by following the appropriate subsections of the?Set Up Filebeat (Add Client Servers) section?of the ELK stack tutorial.
After placing the appropriate SSL certificate file in the proper location, try starting Filebeat again.
If the SSL certificate is invalid, the logs should look like this:
syslog (Certificate is invalid):
transport.go:125: SSL client failed to connect with: x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "elk.example.com")
Note that the error message indicates that the certificate exists, but is invalid. In this case, you need to follow the?Generate SSL Certificates section?of the ELK stack tutorial, then copy the SSL certificate to the client machine (Set Up Filebeat (Add Client Servers)).
After ensuring that the certificate is valid, and that it is in the proper location, you will need to restart Logstash (on the ELK server) to force it to use the new SSL key:
sudo service logstash restart
Then start Filebeat (on the client machine):
sudo service filebeat start
Check the logs again, to make sure the issue has been resolved.
Issue: Filebeat Can’t Connect to Logstash
If Logstash (on the ELK server) is not reachable by Filebeat (your client server), you will see error log entries like this:
syslog (Connection refused):
transport.go:125: SSL client failed to connect with: dial tcp 203.0.113.4:5044: getsockopt: connection refused
Common reasons for Logstash being unreachable include the following:
To resolve this issue, first verify that Logstash is running on the ELK server by following the Logstash troubleshooting sections of this guide. Second, verify that the firewall is not blocking the network traffic. Third, verify that Filebeat is configured with the correct IP address (or hostname) and port of the ELK server.
The Filebeat configuration can be edited with this command:
sudo vi /etc/filebeat/filebeat.yml
After verifying that the Logstash connection information is correct, try restarting Filebeat:
sudo service filebeat restart
Check the Filebeat logs again, to make sure the issue has been resolved.