Google Cloud and Open Source Example - Low Cost Backup Solution with rclone

At NEXT19 Google Cloud announced their "Open Source" ethos in my view. They announced that you could now deploy over a dozen Open Source applications right from within the GCP Console thus making deployment and billing easier at the same time. I like to say that the Google Cloud Platform is one of the most "Open Source" aware cloud platforms available today. They just "get" what Open Source is all about.

You can never have enough backups! Never! Trust me on that one!

I like to say that one of my goals is always to get out of trouble faster than I got into it. And for me having good and reliable and usable backups has met that goal for both me personally but more importantly for my clients. In thirty years in this business I have literally seen two grown adults break down and cry when they could not use their backups to perform a system restore! Backups are just one of those things that you never need until you really need them and then it is far too late to pay attention to see if they have actually been working. It's kind of like not checking your UPS battery life status and only finding out that the backup battery has been dead for a few years just when you need it most. You know it's true!

Google Cloud Storage is a very low cost to solution to ensuring you have lots of backups you can go to. In fact I have already written about how it is the lowest cost static website solution you will find on the planet and the one with the lowest worldwide end user latency due to its global Content Delivery Network. But I digress.

Bring On the Rclones

rclone is an Open Source local to cloud or cloud to cloud synchronization tool. Project can be found here. Rclone is a Go program and comes as a single binary file. rclone support over two dozen files store protocols and destinations but today our focus is on Google Cloud Storage.

rclone features

• MD5/SHA1 hashes checked at all times for file integrity
• Timestamps preserved on files
• Partial syncs supported on a whole file basis
• Copy mode to just copy new/changed files
• Sync (one way) mode to make a directory identical
• Check mode to check for file hash equality
• Can sync to and from network, eg two different cloud accounts
• Encryption backend
• Cache backend
• Union backend
• Optional FUSE mount (rclone mount)
• Multi-threaded downloads to local disk
• Can serve local or remote files over HTTP/WebDav/FTP/SFTP/dlna

Simple and cost effective cloud backup solution

Now I know you are going to say that I'm crazy because we actually have the TropoCloudBackup managed backup services solution which we sell. That just means that when you sign up with us for this service that I lay awake at night making sure your backups are working instead of you. But in the spirit of Open Source and Google Cloud's billing philosophy of always giving the client the lowest cloud costs possible, and to show how well the Google Cloud Platform plays with Open Source software I thought I would share this low cost backup approach. And you can use Linux if you like locally so that there are no Windows Operating System costs. But you do have to do some of the work to set this up.

Use your local free backup tools!

Here is the concept. Setup a local server storage unit with enough disk space to handle the backups from your Windows or Linux or Mac servers or workstations. Then install rclone on a server or workstation that has access to your local backup storage unit and use it to sync its files to a Google Cloud Storage bucket. You can use builtin Windows Backup or your favorite TAR archive software to backup to the local storage facility. Then use rclone to sync files from the local storage facility to your Google Cloud Storage Bucket as in the above screen shot.

Bulk Stash Container version of rclone

Bulk Stash is a docker based version of rclone. This service allows you to copy files either to or from a remote storage service like Google Cloud Storage or locally from your laptop to remote storage. You can also use this for copying or syncing files locally to a remote SFTP server or between two remote SFTP servers.

rclone GUI Browser

Simple cross platfrom GUI for rclone command line tool. Supports Windows, macOS and GNU/Linux.

Features

Allows to browse and modify any rclone remote, including encrypted ones

Uses same configuration file as rclone, no extra configuration required

Supports custom location and encryption for .rclone.conf configuration file

Simultaneously navigate multiple repositories in separate tabs

Lists files hierarchically with file name, size and modify date

All rclone commands are executed asynchronously, no freezing GUI

File hierarchy is lazily cached in memory, for faster traversal of folders

Allows to upload, download, create new folders, rename or delete files and folders

Allows to calculate size of folder, export list of files and copy rclone copmmand to clipboard

Can process multiple upload or download jobs in background

Drag & drop support for dragging files from local file explorer for uploading

Streaming media files for playback in player like mpv or similar

Mount and unmount folders on macOS and GNU/Linux

Optionally minimizes to tray, with notifications when upload/download finishes

Supports portable mode (create .ini file next to executable with same name), rclone and .rclone.conf path now can be relative to executable

In conclusion rclone is a great tool for replicating local based backups into cloud based storage very efficiently and at the lowest costs possible. When needed you can retrieve a server backup image to your local storage facility if needed and restore it to the actual machine.

If you would like some assistance on implementing such a solution please don't hesitate to contact us. We can help you setup a do it yourself solution or we can also enroll you in our TropoCloudBackup managed services so you don't have to worry about having enough backups!

www.troposphere.tech