What Do Hanoi Intersections And Water Puppets Have In Common With Distributed Cloud Systems?

Last week I presented at FOSSASIA which was held in Hanoi, Vietnam. During my time in Hanoi, I had two experiences that reminded me of aspects of Distributed Cloud Systems - Intersections, and Water Puppets!

1 Hanoi Street Intersections

How do you cross the street/intersections in Hanoi? As a foreigner, I initially had no clue and assumed it would be certain death. The traffic is basically "crazy" with millions of Mopeds (with up to 4 people riding at once, including kids), motorbikes, cars, buses, trucks, the occasional bicycle, Tuk Tuks and a few intrepid pedestrians flowing around each other miraculously - I didn't see a single accident while I was there. Here's proof that I survived the Hanoi pedestrians crossing the intersection problem (back at home again)!

Sure, there are traffic lights (which are really cool, they count down the time to the colour change), pedestrian crossings - yeah, not really any different to the road itself, no one stops or cares that you are "on" the crossing, and sometimes even traffic police blowing whistles - they are also universally ignored I think. However - it works! There are few actual traffic jams and I managed to walk safely across many intersections with the traffic flowing around me - as long as I walked steadily across and slowed down for mopeds etc. How does it work?

Well, I think it's fundamentally a distributed system with some synchronous signals (traffic lights, traffic police):

And some asynchronous behaviour - drivers of vehicles, once they are in the intersection, appear to have some 6th sense of how to progress and avoid each other - and pedestrians thankfully - there is a lot of tooting, but this is yet another signal mechanism (one that pedestrians are sadly lacking).

This is similar to my Uber Cadence + Apache Kafka "Ballet" pattern that I wrote about in this blog - integrating orchestrated slow, scheduled, stateful, synchronous sequences and choreographed fast asynchronous event flows could also be called the "Hanoi street intersection pattern"!

2 Vietnamese Water Puppets

The Vietnamese Water Puppets are a fantastic traditional entertainment originating in flooded rural Vietnamese rice paddies. The play is a series of short acts based on rural folk tales and includes a variety of characters such as humans (farmers and royalty), animals, fish, ducks, plants and mythical beasts, performing under, in and on/above the water. It's really mesmerising to watch and quite magical - the puppets - which are up to 1m tall - appear to be alive, and even though the narrative is in Vietnamese, it's pretty easy to work out what's going on and get caught up in the drama.

And at the end, all is revealed when the puppeteers step out from behind the bamboo screen.

But how does it work? The puppeteers control the puppets from behind the bamboo screen, standing in knee-high water, using a combination of poles and strings to control their actions. The puppets are made from wood, varnished so they are water-proof, and attached to floats so that it's possible to control them without having to hold them up at the same time. Very clever, but invisible to the audience. The water is the stage, but also hides the control mechanisms.

This is a bit like Cloud services - users and consumers of the services don't see anything that's happening behind the screen, typically all the operational aspects of cloud systems (e.g. provisioning, deployment, monitoring, security, performance, scalability, elasticity, failure handling, backups, upgrades, etc), and can focus on their application development and operations.

If you are ever in Hanoi I recommend this performance (someone said "If you haven't seen the water puppets you haven't been to Vietnam" and I agree!). I saw the "Thang Long Water Puppets":

And if you need open source cloud services I recommend Instaclustr's (we have a 30-day free trial).