Centralized logging - PLG

Centralized logging with PLG, Promtail Part 1

Why would you choose PLG

• lightweight and stores logs in S3 buckets
• can be easily integrated with Grafana default we will get single sign-on to see logs
• It's open source and effective resource usage

Datasirpi uses distributed centralized logging to enable log data to engineering teams.

Our system below brings logs from different systems?like VM, and Kubernetes clusters into a centralized cluster.

To check the latest version of Promtail, visit the Loki releases page.?https://github.com/grafana/loki/releases/

Steps to install Promtail

cd /usr/local/bin

curl -O -L "https://github.com/grafana/loki/releases/download/v2.4.1/promtail-linux-amd64.zip"

unzip "promtail-linux-amd64.zip"        

And allow the?execution?permission on the Promtail binary

sudo chmod a+x "promtail-linux-amd64"        

Create the Promtail config

Now we will create the Promtail config file.

config-promtail.yaml

server
  http_listen_port: 9080
  grpc_listen_port: 0

positions:
  filename: /tmp/positions.yaml

clients:
  - url: 'https://localhost:3100/loki/api/v1/push'

scrape_configs:
  - job_name: system
    static_configs:
      - targets:
          - localhost
        labels:
          job: varlogs
          __path__: /var/log/*log:        

Configure Promtail as a Service

Now we will configure Promtail as a service so that we can keep it running in the background.

Create a user specifically for the Promtail service

sudo useradd --system promtail        

Create a file called?promtail.service

sudo nano /etc/systemd/system/promtail.service        

And add this script,

[Unit]
Description=Promtail service
After=network.target

[Service]
Type=simple
User=promtail
ExecStart=/usr/local/bin/promtail-linux-amd64 -config.file /usr/local/bin/config-promtail.yml

[Install]
WantedBy=multi-user.target        

Now start and check the service is running.

sudo service promtail start
sudo service promtail status        

If you reboot your server, the Promtail Service may not restart automatically.

You can set the Promtail service to auto restart after reboot by entering,

sudo systemctl enable promtail.service        

Outcome:

Now we can fetch logs from the different instances and push the logs to Loki and store it distributed file system

#cloudcomputing #cloudnative #kubernetes #devops #datasirpi