Would you delete all of your production data?

Companies spend millions of dollars to ensure data is not lost. But how many of them actually test it? Would you pull the plug on a server or delete all of the data on a production system? Recently, I was having a conversation with a client about our 3-node cluster. He asked me if I am so confident that our system is protected from data-loss, I should prove it. I pulled out my laptop connected to our primary data node and simply deleted all the data. I could see the clients heart drop to the floor. I must admit, my heart was beating faster than his, but I was trying to be calm.

All I had to do was run one simple command and had all the data restored from our secondary to primary cluster. Just like that! We both freaked out longer than it took to restore the data. That's right. In less than 30 mins, over 2TB of processed data was back as if it never happened.

This was the first time I have ever done anything so gutsy that it would involve mine and my company's reputation. We all design systems so that in case of a failure we can recover back. However, how many of us actually try to introduce a failure with full confidence? The FinanSeer, which is a state-of-the-art platform running HPCC Systems has been designed to recover from virtually any type of failure. It keeps multiple copies of data across the clusters and allows a very efficient restore of the systems. It would have been no different had I taken a flame thrower (thanks to Elon Musk's cute little invention) to the server. The result would have been no different with the exception of a couple of additional steps.

Data people are generally paranoid about losing data. That's probably why I backup my personal pictures on at-least 5 different encrypted backup devices. That's the exact mindset that went into designing the FinanSeer. Storage is cheap. Data loss? Not so much.