Disruptive technologies - both a smorgasbord and paradox of choice

Part of my job is to keep track of new and upcoming technologies. I actually think this should be part of the job description for anybody and everybody in IT, but that's a rant for a different day.

Having the luxury of researching new and upcoming technologies has made me realize that the selection, or supply if you will, of new and truly interesting software, systems and services is truly endless. The technology industry is a beautiful flower that has blossomed out of meritocracy and innovative ideas. With a steady flow of new ideas on how to improve things, emergence of open source, decent funding to tech start ups, has proved to be the right ingredients to keep me busy in my never ending endeavour of keeping up to speed in the IT landscape.

Having an increasing supply of interesting technologies available at a mere google search away is both a blessing and a curse. Choice is good, but at a certain point you will start to experience the paradox of choice. Keeping track of new and upcoming technologies, figuring out how to evaluate, and encouraging a culture in your organization where it is encouraged to go out and research, can all be a very daunting. Some companies tend to wait for vendors to show up and sell them on the latest and greatest. Others have trusted advisors and consultants to give them an impartial view. 

I could go on and on, but I thought I would give this article a slightly odd twist, by showcasing some of the more interesting companies and software I have come across recently. Don't take it as an endorsement, but more of a sample of all the interesting things happening in the industry, that might have gone under your radar. Food for thought that hopefully will spark your interest to think about how to deal with, and incorporate, new technology.

Concord

Concord is a high performance, real-time event processor on Apache Mesos. We have seen an increased interest in event and stream processing, and a lot of that can be attributed to the emergence of open source projects such as Spark and Storm. So what caught my interest in the newly founded Concord? Let me quote Martin Kleppman, Author of Designing Data-Intensive Applications, Contributor of Apache Samza:

There are a lot of stream processing frameworks, but Concord stands out from the crowd: native multi-language support, low latency and an operations-friendly design make it particularly compelling.

Very low latency, high throughput, and built with scale in mind. I'll definitely be keeping my eye on these guys. The one thing to note though is that it currently is at-most-once, best effort delivery. But let's be honest, that is fine for a lot of use cases, where speed and efficacy is preferred.

Who might they disrupt? Not as much displacing existing companies, but I think it will accelerate the adoption of real time stream processing, which in itself will be disruptive.

CockroachDB

Don't let the name put you off. Apparently the name comes from the resiliency of cockroaches, but really? To be honest, the name had me disinterested from the start, but I kept hearing the name every so often from people I truly respect. So what is CockroachDB? "The scalable, survivable, strongly consistent, SQL database."

Ok, that actually sounds very interesting. The emergence of NewSQL, NoSQL, whatever you call it, came as a response to the difficulty of scaling relational databases in a hyperscale world. You're not going to build Twitter on Oracle. However, NoSQL sacrifices a couple of things in the name of scalability and availability. If you are familiar with RDBMS, think of NoSQL as not being able to comply with ACID. Being able to scale up ACID complaint relational databases has long been reserved for very, very expensive enterprise grade databases such as Oracle. So reading this on the CockroachDB got me really excited.

"For the past decade, developers have struggled between the guarantees of traditional relational databases and the horizontal scalability of NoSQL systems. Virtually every major Silicon Valley success of the past 15 years has wrestled with its own version of RDBMS sharding, and just as many have experienced the frustration of trying to build increasingly complex apps without transactional guarantees. CockroachDB is a scale-out, SQL RDBMS with support for ACID transactions. It saves companies from endlessly reimplementing inefficient and wasteful workarounds, allowing them to focus on what matters most: their business."

That's seriously neat. The New Stack had a good and in depth article on it.

Who might they disrupt? I think this has the possibility to disrupt many traditional databases, and some NoSQL databases that were never really a good fit for non-ACID. Also, I believe there will be a small spillover from users who adopted (or wanted to adopt) AWS Aurora, but wanted to host themselves, or just plain didn't want to leave everything to AWS. 

Quobyte

I'll admit it, I'm a storage geek. So I'm both skeptical, and excited, when I read the description of Quobyte. It sounds almost too good to be true.

"Quobyte`s smart storage software offers a single storage system for block (Cinder), object (S3) and shared file system (Manila) out-of-the box with monitoring, automation and one-click-management ready to go in 1 hour on most Linux distributions. Quobyte`s shared storage solution means easy data sharing among VMs, proper file system backups such as private NAS without extra appliances or run multi-VM applications like LAMP-stacks with one shared storage."
https://wiki.openstack.org/wiki/Quobyte

So, it's block storage, object storage, and a shared file system....that is reliable, scalable and high performance? Shut up and take my money?

Who might they disrupt?  Any and all storage providers if they can deliver on their promises, because boy, does it ever sound promising.

DC/OS

There has been a common thread in the three technologies mentioned above, Concord, CockroachDB, and Quobyte. That is that they are all built on top of a new assumption, a new normal, that things should be built with a distributed system type mindset. Cloud native principles, such as scaling horizontally, start up fast and graceful shutdown etc etc.

Writing the next generation storage or databases means that you have to keep this new reality in mind, the problem is that this is really, really hard to actually implement correctly. If you had to think about everything from your database engine logic, all the way down to how you manage physical node failure and replacement in your cluster, you're going to have a bad time (tm). It's almost like writing an operating system from scratch. However, this operating system needs to be distributed, and work across a large cluster in your data center...enter DC/OS, the data center operating system.

DC/OS aims to abstract the underlying physical nodes, whether that be bare metal servers in your on premises, or a virtual machine like EC2 on AWS, and make it behave like a single machine, or OS if you'd like. It was created by a company called Mesosphere, which got its name from Apache Mesos, which in its turn acts as the kernel in this OS metaphor. 

Apache Mesos abstracts CPU, memory, storage, and other compute resources away from machines (physical or virtual), enabling fault-tolerant and elastic distributed systems to easily be built and run effectively.

DC/OS aims to make the goal of Apache Mesos, actually attainable, by making it easy to use, and adding a bunch of other things that you need in addition to bare bones Mesos.

Alright, why is this cool? I think this will power the next generation of technology, such as Concord, CockroachDB and Quobyte, since it solves many of the difficult parts of writing a distributed application. These apps, can then be packaged up into 1-click installers (called frameworks in Mesos/DCOS terminology). 

Summary

In this article I have just started to scratch the surface of the kind of technology that is rapidly entering the market. Some try to do things better, and some try to turn everything we know up on its head.

Did you already know of all the companies mentioned? If so, awesome, if not, maybe I gave you a little nugget to think about. How do you go about finding new things, how do you evaluate them, how do you know which ones will succeed? These are all hard questions, and unfortunately I don't have the answers. The only thing I do know for sure, is that I will be keeping a close eye on the development of all the aforementioned companies.

Finally, two honorable mentions that didn't make it due to space are Vamp.io, which deploys and manages microservices, and RethinkDB, which is an open source, scalable database that makes building realtime apps a lot easier.