How to setup Prometheus as a systemd service and why ?

For the longest time I was confused about systemd services, what were those? why do we need those? Here I will answer those questions and give you a fundamental overview of how we can apply it in terms of Prometheus.

The traditional way of installing Prometheus goes something like this,

• We download the Prometheus binary onto our server

wget https://github.com/prometheus/prometheus/releases/download/v2.40.1/prometheus-2.40.1.linux-amd64.tar.gz         

• We extract the contents of the tar file

tar xvf prometheus-2.40.1.linux-amd64.tar.gz         

• We move into our downloaded directory and run the Prometheus executable

cd prometheus-2.40.1.linux-amd64.tar.gz
./prometheus        

You should now see the Prometheus UI running on localhost:9090

Pretty simple, isn't it?

But there are some issues with this,

1) The moment you close your terminal, Prometheus stops.

2) You would have to manually start Prometheus after every boot.

Now, we do not want Prometheus which is a monitoring tool, to be unreliable like that.

So what is the solution?

What can allow us to run Prometheus automatically on boot, without being dependent on a terminal, and let us control it's lifecycle with a few handy commands?

systemd is the answer!

Going by definition,

A systemd service is a "background process" that is managed by systemd, a system and service manager for Linux operating systems.

Systemd manages your service using a configuration file called systemd unit file where you specify directives on how systemd must run the service.

Additionally using the systemctl command line utility you can start/stop or check the status, etc of your services easily.

Now that we have understood the benefits of leveraging systemd to manage our services, let's see how we can set up Prometheus as a systemd service

• Install the Prometheus binary and extract its contents, let's go over them for a bit

cd prometheus-2.40.1.linux-amd64
ls
LICENSE  console_libraries  prometheus      promtool
NOTICE   consoles           prometheus.yml  data        

prometheus : Prometheus executable

promtool : CLI tool to verify Prometheus setup

prometheus.yml: Configuration file for Prometheus

data: Time Series database storing metrics collected by Prometheus

console_libraries and consoles: components related to the Prometheus web interface and its configuration.

• Let's restructure our files now, we move the executables to usr/local/bin , our TSDB to var/lib/data and the rest of the files to /etc/prometheus

mv promtool /usr/local/bin
mv prometheus /usr/local/bin
mv data /var/lib/data
# Rest we move to the /etc/prometheus directory        

• We create a user to own the Prometheus service and make the user identity own the related service files

sudo useradd --no-create-home --shell /bin/false prometheus
sudo chown -R prometheus:prometheus /etc/prometheus
sudo chown prometheus:prometheus /usr/local/bin/promtool
sudo chown prometheus:prometheus /usr/local/bin/prometheus
sudo chown prometheus:prometheus /var/lib/data        

• All that's left do now is to create the systemd unit service file , by default it exists in the /etc/systemd/system directory

vi /etc/systemd/system/prometheus.service
# Copy paste the following 
[Unit]
Description=Prometheus
Wants=network-online.target #unit will to start after the network's online
After=network-online.target

[Service]
Type=simple
User=prometheus
Group=prometheus
ExecStart= /usr/local/bin/prometheus \
  --config.file= /etc/prometheus/prometheus.yml \
  --storage.tsdb.path=/var/lib/data

Restart=always

[Install]
WantedBy=multi-user.target # the service will start during boot sequence.
        

• We need to reload the systemd for it to read the new service, after which we can start it, check its status, and enable it to run on boot

sudo systemctl daemon-reload
sudo systemctl start prometheus
sudo systemctl status prometheus
sudo systemctl enable prometheus // enables it to run on boot        

Check localhost:9090 on your local machine and you should see the Prometheus UI !