Stream it, don’t sync it! Using cloud object storage in 2020 and beyond.

After being in technology for a few decades, I like many of you have come across technology that at the time of discovery, it was evident that technology was going to make a splash ... a real big splash who's wake was going to continue for years to come. For me it was things like discovering Linux back in 1996 and coming across VMware on a SuSE Linux CD back in 1999/2000. Well, back in 2019 I came across one of these significant technologies ... LucidLink's Filespaces. I immediately saw LucidLink being so significant that after 9 years with a leading backup and disaster recovery company, I left and joined the LucidLink team back in October. I decided to write this article to share my thoughts and the compelling reasons why LucidLink's technology is ... (I hate to use this word, but it is appropriate here) revolutionary.

For years, I (along with many others) have wanted to use object storage … specifically, cloud-based object storage such as S3, Wasabi, Google, etc. … but never could use it the way I desired in an easy and efficient manner. 

Namely, I wanted cloud-based object storage to have the characteristics of a file system with global file locking that was addressable across multiple clients and operating systems. I wanted it to have the performance of local block storage, and I wanted the data to be secure. 

Seems like a simple wish list, right? 

Well, as many of us have experienced with the plethora of file-sync technologies, storage gateway appliances, caching appliances and the like out on the market; that wish list has not been met in its entirety … until I discovered LucidLink. 

LucidLink is a SaaS offering where the LucidLink software client resides on Windows, MacOS or Linux machines but only has the metadata for the data that resides on the object storage platform of choice. Keep in mind that the object storage platform can be cloud based such as S3, Wasabi, Google, etc. or it can be local object storage such as Cloudian, Scality, etc. LucidLink has a Bring-Your-Own-Storage model as well as a LucidLink storage option which is a LucidLink and Wasabi bundle. 

The LucidLink client presents the object storage via a folder to the OS, and only the data bits that are required at the present time are streamed to and from the object storage repository on demand over multiple parallel streams to the client. Never is an entire file downloaded/synced locally prior to use as with most other technologies on the market. Also of note is that unlike the typical put and get operations with object storage where 1 object equates to 1 file, LucidLink compresses and splits that file data into many small objects where only that required piece of data in a file is then streamed to the client. Add in caching (default 5GB, configurable between 0 and 1TB) and prefetching intelligence; the performance of accessing that data in the cloud becomes similar to that of local storage. 

As with watching a movie on Netflix, one doesn’t care how large the file is since you’re only streaming the bits you need on demand. The same holds true with LucidLink. One doesn’t care how large a file is since you’re only streaming the bits you need on demand. 

Makes sense, right?

The effect of this is apparent if you say create a VM locally on your hypervisor of choice but place the virtual disk(s) in the cloud storage via the LucidLink client. Booting that VM is instant and will run as if you had that virtual disk on say a local NAS appliance. Never is the virtual disk downloaded or synced locally. Your VM will be streaming the bits of data on demand while you use the VM!

But LucidLink goes much further than this … it is a log-structured distributed file system. In short, you can have multiple clients of different OSs all accessing and even modifying the same files concurrently. Gone are the days of downloading/uploading, syncing data as with old technologies found with vendors such as Dropbox, Box, etc.

The other component of LucidLink’s technology is security. LucidLink has built a “Zero Knowledge” security model. That means that only the client has the encryption key. Nobody but the client can see the data … not LucidLink or even the object storage provider such as AWS, Google, etc. has access to your data. Multiple users can be created and assigned RO or RW access to specific folders as well.

What makes LucidLink special is the combination of:

1. Providing local disk performance with cloud based object storage.
2. Log-structured distributed file system functionality.
3. Zero Knowledge security model. 

So, what about LucidLink use cases?

It's quite evident that the use cases are numerous ... LucidLink can provide storage for active archive data, video surveillance data, backup data, file server type applications, CAD file storage and much more.

I suggest you give LucidLink a try … you’ll be impressed.

Visit lucidlink.com and get started!