What's up, home? part 25
Can you monitor your backups with Zabbix? Of course you can!

What's up, home? part 25

Can you monitor your backups with Zabbix? Of course you can -- and you should! By day, I am a monitoring tech lead in a global cyber security company. By night, I monitor my home with Zabbix & Grafana and do some weird experiments with them. Welcome to my blog about this project.

Some time ago Zabbix blog had an excellent read about how to backup Zabbix. While I'm next time coming up with something Completely Else in this blog, this week I wanted to remind you about backups.

The topic came into my mind as my old trustworthy Apple Time Capsule from 2013 probably just said "No, I don't want to to do this job anymore". It's a good thing that I also regularly backup my Mac to an external USB drive which I only keep connected to Mac during backups and then put it in a safe place, and that I also do backup to iCloud and one other cloud service. Update one hour after publishing this blog post: Apple Time Capsule came back to life! But, I shall not trust it anymore.

But, for backing up my Raspberry Pi 4 and its Zabbix, I do something different. Let's move on to that, shall we?

My old and trustworthy friend

Long time ago -- 20 years ago long time -- I used BackupPC at work. Back then I liked it a lot, as it uses proven, field-tested components (Apache, Perl, rsync), it's fast, light and does its job with no questions asked.

Until lately I took my home Zabbix backups to an external USB drive connected to our home router by letting a cron job to dump the database with mysqldump and copying some custom script directories around. As this What's up, home thing is now world-famous and my home Zabbix has turned to an invisible family member, it was time to take backups a bit more seriously -- I don't want to lose my precious home Zabbix configuration!

So, I went to BackupPC's site and was surprised that it's still developed! Kind of -- the latest release is from 2020, and oh boy, it's much more polished than it was 20 years ago.

Faster than I can say "BA-NA-NA", I typed sudo apt install backuppc on my spare laptop, which nowadays runs Ubuntu 22.04 LTS.

Meet BackupPC

Here it is! Thanks to rsync and symbolic links, this thing did deduplication long before it became trendy.

No alt text provided for this image

And it lets me go back in time via a nice web interface from which I can easily restore individual files, directories or just everything. On October 24th I doing some testing, so don't scratch your head about that day in the screenshot.

No alt text provided for this image

Zabbix, meet BackupPC

At first I tried to use a Zabbix community template for BackupPC. It worked -- for the most parts. Unfortunately, at least with a combination of Zabbix 6.2 running on Raspberry Pi 4 and BackupPC 4.4.0 running on Ubuntu 22.04 LTS, the template was unable to fetch metrics, as Zabbix agent's web.page.get was inserting some garbage between HTTP headers and the expected JSON content -- I guess I did hit this issue.

To work around the problem, I just changed the templates master item type to be HTTP Agent instead of Zabbix agent web.page.get. The difference is that with the Zabbix agent based solution the agent can fetch the content locally from the BackupPC server so its web interface does not need to listen on anything else than localhost, and now I opened my BackupPC to listen on network and did let my Zabbix server to fetch the JSON.

If you want to try out my (very) slightly modified template, it's available on my GitHub.

After the initial hurdles the rest was a snap. Not every metric is working for me, as I understood that for them to work, I would need to use the bleeding edge version of BackupPC, but for now this is good enough. I can get all kinds of metrics about BackupPC, below is a custom dashboard I created for it.

No alt text provided for this image

... and even if my BackupPC is backing up my Zabbix perfectly now, it is not backing up itself as I did not yet configure it to do so, so here's how BackupPC would complain about my Zabbix backups too, should they be failing for any reason.

No alt text provided for this image

Do not trust your backups

What's worse than having no backups at all? Doing backups, but ONLY doing backups. Especially if you are just setting up the backup routine for something completely new, it's very likely that you miss something during the first try.

Backups are not a fire-and-forget thing, you better monitor them and regularly test them, or otherwise it's guaranteed that Mr. Murphy will hit you with a clue stick sooner or later.

So, here's my €0.02:

  • Take regular backups
  • Monitor them
  • Test your backups
  • Make sure you have more than one backup (my MacBook copies the Zabbix backup files from my Ubuntu daily, too, and from there my Zabbix backups are spread to everywhere my Mac backups are)

My future development in this area includes a script which would attempt to automatically restore the database and my scripts to a VM/container, install Zabbix and see if it works. If it would not, then my Zabbix would get alerted.

I have been working at Forcepoint since 2014 and during my long IT career have learned that you always have to have a guaranteed way to restore something.

Eyal Klein

Director of IT @ SentinelOne

2 年
回复

要查看或添加评论,请登录

Janne Pikkarainen的更多文章

  • What's up, home? part 30

    What's up, home? part 30

    Can you monitor Northern Lights with Zabbix? Of course you can! By day, I am a monitoring tech lead in a global cyber…

    3 条评论
  • What's up, home? part 29

    What's up, home? part 29

    Can you detect a crying baby with Zabbix? Of course you can! By day, I am a monitoring tech lead in a global cyber…

    2 条评论
  • What's up, home? part 28

    What's up, home? part 28

    Can you monitor FlightGear flight simulator with Zabbix? Of course you can! By day, I am a monitoring tech lead in a…

    9 条评论
  • What's up, home? part 27

    What's up, home? part 27

    Can you have a Zabbix HA cluster at home? Of course you can! By day, I am a monitoring tech lead in a global cyber…

    4 条评论
  • What's up, home? part 26

    What's up, home? part 26

    Can you create fairy tales with Zabbix? Of course you can! By day, I am a monitoring technical lead in a global cyber…

  • What's up, home? part 24

    What's up, home? part 24

    Can you monitor Finland's total real-time power consumption with Zabbix? Of course you can! By day, I am a monitoring…

    4 条评论
  • What's up, home? part 23

    What's up, home? part 23

    Can you monitor a banana with Zabbix? Of course you can! By day, I am a monitoring tech lead in a global cyber security…

    4 条评论
  • What's up, home? part 22

    What's up, home? part 22

    Can you use Zabbix as a countdown timer? Of course you can! By day, I am a monitoring technical lead in a global cyber…

    9 条评论
  • What's up, home? part 21

    What's up, home? part 21

    Can you make sure your video streams are up with Zabbix? Of course you can! By day, I am a monitoring technical lead in…

  • What's up, home? part 20

    What's up, home? part 20

    Can you monitor your upcoming electricity bill with Zabbix? Of course you can! By day, I am a monitoring technical lead…

社区洞察

其他会员也浏览了