Avoid the Blame Game: How Zabbix Monitors Configuration and Alerts Changes Like a Pro!

A Pro’s Guide to Zabbix Alerts for Sneaky Configuration Changes!

Imagine this:

You’re sipping your coffee on a typical Monday morning, feeling confident that your infrastructure is running smoothly. Suddenly, chaos ensues. Configuration files have mysteriously changed on one of your critical servers, causing service disruptions.

Who changed it? When did it happen? Most importantly, how do you fix it before your boss asks, “What’s going on?”

This is where Zabbix, your vigilant watchdog, comes to the rescue.

In this blog, we’ll dive into how you can configure Zabbix to track configuration changes and set up alerts to notify you instantly. By the end, you’ll not only save your Monday mornings but also prove yourself as the unsung hero of your DevOps team!

To monitor changes to the /etc/logstash/pipelines.yml file and configure Zabbix to send alerts to the dashboard with details of which user made the changes, you can use the following approach:

Solution Overview

1. Set Up AuditD to Track File Changes and User Details
2. Integrate AuditD with Zabbix
3. Configure Zabbix to Parse Audit Logs
4. Test and Validate the Setup

Requirements:

1. Zabbix Server
2. Linux Machine with zabbix agent

Config Changes in the Shadows?

Step 1: Set Up AuditD to Track File Changes and User Details

Install and Configure AuditD:

1. Install AuditD:

sudo apt install auditd -y  # For Debian/Ubuntu 
sudo yum install audit -y  # For CentOS/RHEL        

2. Add a Watch for /etc/logstash/pipelines.conf:

• Add the following rule to monitor the file for changes:

sudo auditctl -w /etc/logstash/pipelines.yml -p wa -k logstash_config_change        

• -w: Watch this file.
• -p wa: Monitor write and attribute changes.
• -k logstash_config_change: Assign a key for easier filtering in logs.

Note: Here I am using /etc/logstash/pipelines.yml , You can monitor any files.

3. Make the Rule Persistent:

• Add the rule to /etc/audit/rules.d/audit.rules:

-w /etc/logstash/pipelines.yml -p wa -k logstash_config_change        

4. Restart AuditD to Apply Changes:

sudo systemctl restart auditd        

5. Test the Configuration:

• Modify the file and check the audit logs:

echo "test change" | sudo tee -a /etc/logstash/pipelines.yml 
sudo ausearch -k logstash_config_change        

• The log should show details like the username and timestamp.

Using above solution we can achieve Caught in the Act: Tracks Configuration Shenanigans!.

Step 2: Integrate AuditD with Zabbix

Install the Zabbix Agent:

1. Install the Zabbix Agent on the system where AuditD is configured.

sudo apt install zabbix-agent -y  # For Debian/Ubuntu 
sudo yum install zabbix-agent -y  # For CentOS/RHEL        

2. Configure the Zabbix Agent:

• Edit the Zabbix agent configuration file (/etc/zabbix/zabbix_agentd.conf):

Server=<Zabbix_Server_IP> 
Hostname=<Unique_Host_Name>        

• Create a user parameter to monitor AuditD logs:

UserParameter=auditd.logstash_change,sudo ausearch -k logstash_config_change | tail -n 1        

3. Restart the Zabbix Agent:

sudo systemctl restart zabbix-agent        

Step 3: Configure Zabbix to Parse Audit Logs

Create a template or You can Import a template from GitHub

Creating Template and items:

Create template:

Go to Data Collection → Templates → Create template.

• Template Name: logstash config monitor by auditd
• Template Groups: Templates/Operating systems

Once the template is created, navigate to the item in the same template and create an item.

Create an Item in Zabbix:

1. Log in to the Zabbix Web UI.
2. Navigate to Configuration → Hosts → Items.
3. Create a new item:

• Name: get logstash config changes
• Type: Zabbix agent
• Key: auditd.logstash_change (same as created in userparameter in agent.)
• Type of Information: Log
• Update Interval: 60s (adjust as necessary).

We can get the data like below, but is not useful. But we can filter the fields what we required.

time->Wed Nov 27 16:10:17 2024
type=PROCTITLE msg=audit(1732723817.722:541): proctitle=746565002D61002F6574632F6C6F6773746173682F706970656C696E65732E636F6E66
type=PATH msg=audit(1732723817.722:541): item=1 name="/etc/logstash/pipelines.conf" inode=262503 dev=ca:01 mode=0100644 ouid=0 ogid=0 rdev=00:00 nametype=NORMAL cap_fp=0 cap_fi=0 cap_fe=0 cap_fver=0 cap_frootid=0
type=PATH msg=audit(1732723817.722:541): item=0 name="/etc/logstash/" inode=262501 dev=ca:01 mode=040755 ouid=0 ogid=0 rdev=00:00 nametype=PARENT cap_fp=0 cap_fi=0 cap_fe=0 cap_fver=0 cap_frootid=0
type=CWD msg=audit(1732723817.722:541): cwd="/home/karthick"
type=SYSCALL msg=audit(1732723817.722:541): arch=c000003e syscall=257 success=yes exit=3 a0=ffffff9c a1=7ffca95ca7a2 a2=441 a3=1b6 items=2 ppid=3580 pid=3581 auid=1000 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts4 ses=1 comm="tee" exe="/usr/bin/tee" subj=unconfined key="logstash_config_change"        

Here I am filtering only /home/karthick text.

For achieve above one, we can create dependant item then filter the logs or We can use same item and pre-processing.

Dependent item:

1. click 3 dot menu before the item name is called get logstash config changes. Then create the dependent item.

Name: logstash config modified user

Type: Dependent item

Key: modified.user.name

Type of Information: text

Master Item: leave as default

History: leave the default.

Preprocessing:

Name: Regular expression

Parameters: cwd=”([^”]+)”

Note: Here it's incorrect. Please change the “ Double quotation” correctly in the configuration Another column: \1

Create a Trigger for Alerts:

1. Navigate to Configuration → Hosts → Triggers.
2. Create a new trigger:

• Name: logstash config modified by {ITEM.VALUE} on {HOST.NAME}
• Opreational data: User: {ITEM.VALUE}
• Severity: High ( based on your wish)
• Expression:

last(/logstash config monitor by auditd/modified.user.name,#1)<>last(/logstash config monitor by auditd/modified.user.name,#2)        

• OK Event generation: None ( it will not solve automatically)

Do you Know:

• Expression — when the trigger expression is false, then the problem will be resolved.
• Recovery Expression: Add new condition for recovery. (Not required for this task)
• None ( it will not solve automatically)

This trigger checks if the logstash config modified user log has any content and generates an alert.

Step 4: Test and Validate the Setup

1. Test the Workflow:

• Modify /etc/logstash/pipelines.conf:

echo "change test" | sudo tee -a /etc/logstash/pipelines.yml        

• Check the Zabbix dashboard for the alert.

2. Verify the Alert:

• The Zabbix alert should display the username and other details (as captured by AuditD).

“Who’s Breaking Things at Midnight? Zabbix Has the Receipts! Now onwards”

Same As you can monitor Any configuration file on Linux.

Pro Tip: Centralize Logs for Better Analysis

If managing multiple systems, send AuditD logs to a centralized log server (e.g., ELK Stack, Graylog). Use Zabbix to parse the aggregated logs for broader insights.

Conclusion

By combining AuditD for detailed file change tracking and Zabbix for real-time monitoring and alerting, you can effectively track and respond to changes in critical configuration files like /etc/logstash/pipelines.yml.

DevSecOps — Community ??

Thank you for being a part of the DevSecOps — Community community! Before you go:

• Be sure to clap and follow ? Karthick Dkk????
• Follow: | LinkedIn | Newsletter | LinkedIn Groups | GitHub
• More content at DevSecOps — Community