Monitor your Docker containers with Prometheus and Grafana.
Grafana and Prometheus are one of the technology world's powerful monitoring and analytics duo.
So, what is Prometheus and Grafana?
Prometheus is primarily focused on collecting metrics from the services and storing them in a time-series database. While it's robust for this purpose, it doesn't provide extensive capabilities for visualization and advanced analytics out of the box. This is where Grafana comes in.
Grafana is a visualization and analytics platform that works exceptionally well with Prometheus (as well as other data sources). It allows you to create visually appealing dashboards and graphs based on the data collected by Prometheus. Grafana's strength lies in its ability to connect to multiple data sources, including Prometheus, and then create customizable, interactive dashboards to visualize the collected metrics in various formats like graphs, tables, and heatmaps.
In this article, we will setup a monitoring system for a docker containers to monitors all the actions of the containers.
Before starting to setup monitoring system, you will need to have a docker node that contains 2 or more containers in it.
If you have a node, let's start and dive into the article.
First of all, let's setup Prometheus to collecting metrics from docker node.
In this article we will use Ubuntu 22.04 LTS distro with average system requirements.
Installing and configuring Prometheus:
Create system user, group and service directories:
groupadd --system prometheus
useradd -s /sbin/nologin --system -g prometheus prometheus
mkdir /etc/prometheus
mkdir /var/lib/prometheus
Download the package and extract (You can get latest update from Prometheus download link):
wget https://github.com/prometheus/prometheus/releases/download/v2.49.0-rc.1/prometheus-2.49.0-rc.1.linux-amd64.tar.gz
tar vxf prometheus-2.49.0-rc.1.linux-amd64.tar.gz
Move binary files and configurations to related paths and set the ownership:
cd prometheus*/
mv prometheus /usr/local/bin && mv promtool /usr/local/bin
chown prometheus:prometheus /usr/local/bin/prometheus /usr/local/bin/promtool
mv consoles /etc/prometheus
mv console_libraries /etc/prometheus
mv prometheus.yml /etc/prometheus
chown -R prometheus:prometheus /etc/prometheus
chown -R prometheus:prometheus /var/lib/prometheus
Check service configutation:
Prometheus uses YAML files for its configurations. The main configuration file located in '/etc/prometheus/prometheus.yml'. The default configuration file explains itself. So we will leave as default in this article. If you want to add additional configurations, you can check the official documentation in the website.
Create systemd service for Prometheus:
nano /etc/systemd/system/prometheus.service
The given service configuration is suitable for us:
[Unit]
Description=Prometheus
Wants=network-online.target
After=network-online.target
[Service]
User=prometheus
Group=prometheus
Type=simple
ExecStart=/usr/local/bin/prometheus \
--config.file /etc/prometheus/prometheus.yml \
--storage.tsdb.path /var/lib/prometheus/ \
--web.console.templates=/etc/prometheus/consoles \
--web.console.libraries=/etc/prometheus/console_libraries
[Install]
WantedBy=multi-user.target
After the saving service file, we need to reload daemons by running:
systemctl daemon-reload
Last is starting, enabling and checking the status of the service:
systemctl enable --now prometheus && systemctl status prometheus
Based on default configuration, service will listen on localhost:9090. If you want to change to specific external ip, just change it on configuration file.
Example:
- targets: ["localhost:9090"] > - targets: ["your_external_ip:9090"]
In this article, we will leave on localhost and create nginx reverse proxy that proxies the requests and responses between external_ip:443 and localhost:9090.
Example configuration in nginx:
server {
listen your_ip:443 ssl;
server_name prometheus.contoso.org;
ssl_certificate /path/to/ssl/cert;
ssl_certificate_key /path/to/ssl/key;
access_log /path/to/access/log main;
error_log /path/to/error/cert;
location / {
proxy_pass https://127.0.0.1:9090; # Forward traffic to Prometheus
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_read_timeout 300;
proxy_connect_timeout 300;
proxy_send_timeout 300;
}
}
After the configuration file, restart your nginx service and access the endpoint in browser. You will see the dashboard of Prometheus.
Now let's integrate Docker to our Prometheus instance and collect the metrics.
In order for Docker to serve the metrics to the Prometheus we need to add below config in the '/etc/docker/daemon.json' file in the docker node.
{
"metrics-addr" : "127.0.0.1:9323",
"experimental" : true
}
In order for Prometheus to gather the metrics of the docker we need to add below config in the Prometheus configuration file under the scrape_configs block.
- job_name: docker
# metrics_path defaults to '/metrics'
# scheme defaults to 'http'.
static_configs:
- targets: ["localhost:9323"]
领英推荐
Note: Metrics-addr should be same as targets in Prometheus config.
After the restarting both Docker and Prometheus services. We can check the metrics in 'url_of_prometheus/metrics' url.
Now we will visualize these metrics via Grafana instance.
Let's install and configure Grafana:
Step 1:
Update and Upgrade server:
sudo apt update && sudo apt upgrade
Step 2:
Add Grafana GPG Key and repository
sudo mkdir -p /etc/apt/keyrings/
wget -q -O - https://apt.grafana.com/gpg.key | gpg --dearmor | sudo tee /etc/apt/keyrings/grafana.gpg > /dev/null
wget -q -O - https://apt.grafana.com/gpg.key | gpg --dearmor | sudo tee /etc/apt/keyrings/grafana.gpg > /dev/null
Step 3:
Install Grafana and start the service:
sudo apt update && apt install grafana
systemctl enable --now grafana-server && systemctl status grafana-server
By default, Grafana Web Interface listens on 3000. But you can change and set custom ports in '/usr/share/grafana/conf/defaults.ini' config file
Step 4:
Access the web interface with browser and login:
Initial credentials:
Username: Admin
Password: Admin
After the login, You need to change default password to custom.
So everything is installed. Let's integrate Prometheus with Grafana.
First of all, We need to add Prometheus as a data source to Grafana:
Go to: Grafana home->'+' sign at the top right->New Dashboard->Add visualization->Configure a new datasource->Prometheus
Fill the requested fields and click 'test and save'.
If test is succeed. Then go to Grafana home->'+' sign at the top right->New Dashboard->Add visualization->Prometheus(default)
Now in this window, We will configure the visualization of data based on Prometheus metrics.
In the 'select metric' dropdow section, We can choose which metric we want to visualize.
Just pick one of them and put into select metric section and click run queries button.
After the running script, graph will visualize the metric data. You can change the graph type in right panel. Stat will be suitable for out data.
Also in the right panel, you can setup the title of graph, description and others based on your preferences.
If you satisfied the final view, click apply button at the right top corner.
You can add multiple visuals in the 'add' section at the dashboard.
At the end you need to save the dashboard by clicking save icon at the dashboard section.
In conclusion, the combined use of Grafana and Prometheus presents a formidable solution for container monitoring. By leveraging Prometheus's prowess in collecting and storing time-series data and Grafana's visualization capabilities, users gain a comprehensive understanding of their containerized environments. This dynamic duo offers not only real-time insights into system performance but also the flexibility to create customizable dashboards, empowering users to proactively manage, analyze, and optimize their containerized infrastructure effectively.
Sabir Rajabov