Ansible Roles for Prometheus, Grafana and Nginx Reverse Proxy

Introduction:

In this article, I am going to explain how to create ansible roles for Prometheus, Grafana and use nginx reverse proxy to set up monitoring dashboard.

Imagine a busy office building with a single receptionist handling all visitor requests. This can quickly become a bottleneck. A reverse proxy acts as a similar intermediary in the world of web servers.

Instead of clients directly connecting to individual web servers, they connect to the reverse proxy. The reverse proxy then directs the request to the appropriate backend server based on pre-defined rules (like domain name or URL path). The reverse proxy acts as a first line of defense, hiding the location of backend servers and potentially filtering malicious traffic.

To achieve the goal of setting up Prometheus, Grafana, and Nginx reverse proxy using Ansible roles, we will follow these steps:

1. Create Ansible Roles for Prometheus, Grafana, and Nginx.
2. Define tasks, handlers, templates, and variables for each role.
3. Create an Ansible Playbook to execute the roles.
4. Create an inventory file to specify the target hosts.

Prerequisites:

Set Up An Ansible Lab

This involves the process of creating your servers, instilling ansible on the control nodes, generating SSH keys, give appropriate files and user permission, setting up your inventory files and testing the connectivity of your server. Below is the break down:

1. Preparing the Environment: This involves setting up the servers (manage and control nodes). This could involve provisioning virtual machines (VMs) using tools like VirtualBox, Vagrant, or cloud providers.
2. Generating SSH Keys (on Control Node): This is a crucial step for secure communication between the control node (where you run Ansible) and the managed nodes. You’ll generate an SSH key pair (private and public key) on your control node using: ssh-keygen
3. Adding Public Key to Managed Nodes: Securely copy the generated public key from the control node to the authorized_keys file on each managed node. There are various methods to do this, depending on your environment
4. Configuring Inventory File: Create an inventory file (e.g., inventory.txt) on your control node. This file specifies the managed nodes Ansible can connect to. It includes details like IP addresses, hostnames, and connection options (SSH user).
5. Setting File Permissions (on Managed Nodes): You might need to adjust file permissions on specific configuration files on the managed nodes to ensure Ansible has the necessary access to read, modify, or copy them.
6. Testing Connectivity: Once everything is set up, you can run a simple Ansible ping module (ansible -i inventory.txt all -m ping) to test if Ansible can connect and communicate with the managed nodes using SSH. you can click here to see how I set up ansible lab on other ansible projects

Step 1 Create Ansible Roles

First, create a root director and cd into it, then create directories for the roles using ansible-galaxy:

mkdir ansible_project && cd ansible project
ansible-galaxy init prometheus
ansible-galaxy init grafana
ansible-galaxy init nginx        

Step 2: Define the Prometheus Role Directories Structure

prometheus/
├── defaults
│   └── main.yml
├── files
│   └── prometheus.service
├── handlers
│   └── main.yml
├── tasks
│   ├── main.yml
│   ├── create_user.yml
│   ├── install.yml
│   ├── configure.yml
│   └── prometheus_status.yml
├── templates
└── vars
    └── main.yml        

follow the directory structure to write the needed Prometheus files prometheus/defaults/main.yml:

---
service: prometheus        

prometheus/files/prometheus.service:

[Unit]
Description=Prometheus Server
Documentation=https://prometheus.io/docs/introduction/overview/
After=network-online.target

[Service]
User=prometheus
Group=prometheus
Restart=on-failure
ExecStart=/opt/prometheus/prometheus \
  --config.file=/opt/prometheus/prometheus.yml \
  --storage.tsdb.path=/opt/prometheus/data \
  --storage.tsdb.retention.time=30d \
  --web.route-prefix=/prometheus \
  --web.external-url=/prometheus

[Install]
WantedBy=multi-user.target        

prometheus/handlers/main.yml:

# prometheus/handlers/main.yml
---
# handlers file for prometheus
- name: restart prometheus
  service:
    name: prometheus
    state: restarted        

prometheus/tasks/main.yml:

# prometheus/tasks/main.yml
---
- include: create_user.yml
- include: install.yml
- include: configure.yml
- include: prometheus_status.yml        

prometheus/tasks/create_user.yml:

---
- name: Create a dedicated prometheus user
  user:
    name: "{{ service }}"
    comment: "Prometheus user"
    createhome: false        

prometheus/tasks/prometheus_status.yml:

# prometheus/tasks/prometheus_status.yml
---
- name: Start prometheus service
  systemd:
    name: prometheus
    state: started
    enabled: yes
  notify:
    - restart prometheus        

prometheus/tasks/install.yml:

---
- name: Download prometheus package
  get_url:
    url: https://github.com/prometheus/prometheus/releases/download/v2.40.0-rc.0/prometheus-2.40.0-rc.0.linux-amd64.tar.gz
    dest: /tmp/prometheus.tar.gz

- name: Extract prometheus package
  unarchive:
    src: /tmp/prometheus.tar.gz
    dest: /opt/
    remote_src: yes

- name: Rename existing prometheus directory (if it exists)
  command: sudo rm -rf /opt/prometheus

- name: Rename prometheus folder
  command: sudo mv /opt/prometheus-2.40.0-rc.0.linux-amd64 /opt/prometheus        

prometheus/tasks/configure.yml:

---
- name: Set folder permissions for prometheus user
  file:
    path: /opt/prometheus
    state: directory
    owner: "{{ service }}"
    group: "{{ service }}"
    recurse: yes
    
- name: Copy systemd service for prometheus
  copy:
    src: prometheus.service
    dest: /etc/systemd/system/        

Step 3: Define the Grafana Role

Directory Structure:

grafana/
├── defaults
│   └── main.yml
├── tasks
│   └── main.yml
├── handlers
│   └── main.yml
└── templates        

follow the directory structure to write the needed Grafana files

grafana/defaults/main.yml:

---
grafana_admin_password: "your_admin_password"        

when all the directories and configuration files are set, go to step six (6) and run your playbook

click here to read full article

Step 6: Run the Playbook

Check Connectivity to Target Hosts:

ansible -i inventory.txt all -m ping        

Run the Playbook:

ansible-playbook -i inventory.txt runbook.yml        

This setup will install Prometheus, Grafana, and configure Nginx as a reverse proxy. Below is a snippet of most of the code run during this project: