Horse-driven Carriages and Real Time

Likely developed for the wealthy during ancient civilizations such as the Egyptians or Romans, horse-driven carriages became widespread in the Middle Ages for all sorts of uses including transportation, agriculture, and military logistics. In the modern era, they were used primarily for transportation in urban areas, and were popular among the wealthy.

As we know, later in the 19th and in the early 20th century, automobiles began to displace these primary forms of transportation. There were several reasons for this:

• Speed and Convenience: Automobiles are faster, more convenient, and great for longer distance travel as well.
• Pollution: Horse driven carriages produced waste, while automobiles were cleaner - especially in urban areas where horse waste was a problem.
• Safety: Automobiles were safer than horse-drawn carriages - both in terms of reduced chances of accidents, as well as providing better protections to the passengers in the event of an accident.
• Cost: While the initial outlay was higher, automobiles paid for themselves pretty quickly in terms of ongoing operational expenses. No need to have a stable, buy and stock feed, and deal with ongoing care of the horses.

When this transition was happening, it was probably unclear to many that such a transition would succeed. If you’re used to horses as the primary medium of transport for the last 100+ years, it would be difficult to imagine anything displacing it. It’s pretty obvious now that we look back at the stark differences between the two modes of transport.

There is a similar transition that’s beginning in the software world. End users need real-time convenience and updates as they interact with businesses. Businesses need to refactor their applications to provide real time access and information to their customers, or risk losing them to nimble startups that can meet this need.?

Yet, building real time applications is harder than ever - takes a long time to build, needs multiple products across multiple vendors, requires experts in distributed systems and cloud environments to develop and operate them, among other challenges. As a result, the backlog of these desired real time applications continues to grow.

The problems in building real time applications are the same as with horse-drawn carriages:

• Speed and Convenience: Idea to product - or time-to-market - for these applications in any enterprise is measured in years and needs experts in various technologies like message queues, data caches, databases, micro-services, several programming languages, and many others.
• Pollution: With today’s infrastructure one ends up with multiple copies of data, not synchronized with each other. Data durability and availability concerns abound, because one product’s durability SLAs might be quite different from other products.?While not quite like the waste from horses, it still stinks.
• Safety: Cloud infrastructure requires a “failure is standard” design ethos. Dealing with messages being lost or duplicated, processes being randomly killed, disks going bad is where much of the application development and operational time ends up going.?
• Cost: Not only do businesses need to pay high subscription / license fees for each of the many clustered products they would need, along with cloud resource costs, there are also the operational costs of watching, alerting, monitoring, and scaling this infrastructure - which like with horse carriages, is quite high.?

And like with horse-drawn carriages, because the current way has been in place for so long now, it is difficult for many to imagine that there could be a better way.

At Grainite (acquired by MongoDB) , we are developing the automobiles for these horse-drawn carriages. A system that no longer requires 5-6 different products to be pieced together to build a realtime application. Only one system holding a strongly consistent copy of the data. Developers don’t need to defensively program for failures; the platform provides strong guarantees like exactly-once message processing, data durability and resiliency SLAs that isolate the application from these failures. Develop your application entirely on your laptop, and then deploy the artifacts directly to a horizontally scaling cluster on each of the public clouds or within your own data center.

I would love to hear from you about your experiences and challenges with building real time applications, and if these problems resonate with you, show you what the future may look like.