What's up, home? part 25

Can you monitor your backups with Zabbix? Of course you can -- and you should! By day, I am a monitoring tech lead in a global cyber security company. By night, I monitor my home with Zabbix & Grafana and do some weird experiments with them. Welcome to my blog about this project.

Some time ago Zabbix blog had an excellent read about how to backup Zabbix. While I'm next time coming up with something Completely Else in this blog, this week I wanted to remind you about backups.

The topic came into my mind as my old trustworthy Apple Time Capsule from 2013 probably just said "No, I don't want to to do this job anymore". It's a good thing that I also regularly backup my Mac to an external USB drive which I only keep connected to Mac during backups and then put it in a safe place, and that I also do backup to iCloud and one other cloud service. Update one hour after publishing this blog post: Apple Time Capsule came back to life! But, I shall not trust it anymore.

But, for backing up my Raspberry Pi 4 and its Zabbix, I do something different. Let's move on to that, shall we?

My old and trustworthy friend

Long time ago -- 20 years ago long time -- I used BackupPC at work. Back then I liked it a lot, as it uses proven, field-tested components (Apache, Perl, rsync), it's fast, light and does its job with no questions asked.

Until lately I took my home Zabbix backups to an external USB drive connected to our home router by letting a cron job to dump the database with mysqldump and copying some custom script directories around. As this What's up, home thing is now world-famous and my home Zabbix has turned to an invisible family member, it was time to take backups a bit more seriously -- I don't want to lose my precious home Zabbix configuration!

So, I went to BackupPC's site and was surprised that it's still developed! Kind of -- the latest release is from 2020, and oh boy, it's much more polished than it was 20 years ago.

Faster than I can say "BA-NA-NA", I typed sudo apt install backuppc on my spare laptop, which nowadays runs Ubuntu 22.04 LTS.

Meet BackupPC

Here it is! Thanks to rsync and symbolic links, this thing did deduplication long before it became trendy.

And it lets me go back in time via a nice web interface from which I can easily restore individual files, directories or just everything. On October 24th I doing some testing, so don't scratch your head about that day in the screenshot.

Zabbix, meet BackupPC

At first I tried to use a Zabbix community template for BackupPC. It worked -- for the most parts. Unfortunately, at least with a combination of Zabbix 6.2 running on Raspberry Pi 4 and BackupPC 4.4.0 running on Ubuntu 22.04 LTS, the template was unable to fetch metrics, as Zabbix agent's web.page.get was inserting some garbage between HTTP headers and the expected JSON content -- I guess I did hit this issue.

To work around the problem, I just changed the templates master item type to be HTTP Agent instead of Zabbix agent web.page.get. The difference is that with the Zabbix agent based solution the agent can fetch the content locally from the BackupPC server so its web interface does not need to listen on anything else than localhost, and now I opened my BackupPC to listen on network and did let my Zabbix server to fetch the JSON.

If you want to try out my (very) slightly modified template, it's available on my GitHub.

After the initial hurdles the rest was a snap. Not every metric is working for me, as I understood that for them to work, I would need to use the bleeding edge version of BackupPC, but for now this is good enough. I can get all kinds of metrics about BackupPC, below is a custom dashboard I created for it.

... and even if my BackupPC is backing up my Zabbix perfectly now, it is not backing up itself as I did not yet configure it to do so, so here's how BackupPC would complain about my Zabbix backups too, should they be failing for any reason.

Do not trust your backups

What's worse than having no backups at all? Doing backups, but ONLY doing backups. Especially if you are just setting up the backup routine for something completely new, it's very likely that you miss something during the first try.

Backups are not a fire-and-forget thing, you better monitor them and regularly test them, or otherwise it's guaranteed that Mr. Murphy will hit you with a clue stick sooner or later.

So, here's my €0.02:

• Take regular backups
• Monitor them
• Test your backups
• Make sure you have more than one backup (my MacBook copies the Zabbix backup files from my Ubuntu daily, too, and from there my Zabbix backups are spread to everywhere my Mac backups are)

My future development in this area includes a script which would attempt to automatically restore the database and my scripts to a VM/container, install Zabbix and see if it works. If it would not, then my Zabbix would get alerted.

I have been working at Forcepoint since 2014 and during my long IT career have learned that you always have to have a guaranteed way to restore something.