Why Did You Pivot to the Cloud?

Sometimes I have the opportunity to interview candidates for Cloudreach (and btw, if you're interested in a cloud-focused career, check out our site).

Of course, technical questions form a good percentage of the questions I ask and the back and forth flow of the conversation. We all enjoy a good geek-out session about the latest innovations, the problems we've solved and the solutions we've architected.

Technical proficiency aside, I'm interested in a person's cloud journey. If, like me, you have broad and deep experience with how technology is done on-premises - the challenges of capacity planning, availability, disaster recovery and innovation (i.e., modifying your tech stack to meet and exceed customer needs) - you understand that the primary advantage of cloud methods is shrinking the distance between business value and the technology layer.

To illustrate this, prior to moving my career to the cloud, here's a rough approximation of where I was positioned, relative to the business (any business):

The diagram shown above is a simplification of a very complicated workflow; the gist is that the business (ideally in response to customer needs and requirements) directs technologists to provide and support services that help an organization accomplish its mission (whether that's profit, saving lives, education, etc).

Life in the blue silo, although it's the sharp end of the spear when it comes to customer experience (think of your banking website, for example) is typically hectic, high pressure and isolated from the business silo because of the brittleness of software (and to a lesser but still non-trivial extent, hardware). This brittleness is a factor in both the infrastructure and in the responsiveness of systems to the need for change. Brittleness is an isolating factor because so much effort is required to simply 'keep the lights on' that technologists find themselves in a race against obsolescence, failures and changing demands to maintain a solid level of service.

Which brings me back to how and why I pivoted my career to the cloud.

Quite a few years ago, I worked for a firm which provided testing materials, standards definitions and certifications for project management. The abstracted technology stack looked like what's shown above. One year, it just so happened that we performed an infrastructure refresh and an update of development tooling to meet customer needs for testing materials sold, unsurprisingly, via a website.

There was however, a problem. That particular year saw a significant increase in demand for testing materials and related items sold via the web site. This was a good problem to have but the infrastructure we'd so carefully engineered (and paid for!) could not keep up with the new level of demand. The over-provisioned VMware infrastructure couldn't accommodate more virtual machines to host web servers, the database servers were operating under strain and the customer experience was suffering as a result.

Needless to say, there was a lot of stress, frayed nerves and short tempers as the organization struggled to come up with a solution. Management demanded, technologists offered techy excuses (which, while true, fell on deaf ears) and the general mood became a mixture of dismay and panic.

In the midst of this drama, one of my outstanding and forward thinking colleagues offered a potential way out, one which none of us (except him) was familiar with to-date: AWS Elastic Beanstalk using AWS RDS as the database backend.

Here's a version of the whiteboard diagram he showed to explain the value:

The problems, he asserted, were three-fold:

1. Capacity planning is very hard because life is unpredictable
2. Software is often very brittle and does not respond well to exceptions (including unexpected demand) unless it's designed to do so and also, operating within an environment that elastically responds to need - whether it's horizontal, vertical, increase or decrease
3. The solution to the problems of availability, scalability and graceful failure will never be decisively addressed via our efforts within our data center or colocation partner

Then he concluded by stating that the (then new) technologies hosted by Amazon should be our preferred route.

We started a proof of concept which was quite successful. Despite this success, fear of the unknown and a good dose of toxic IT culture prevented an expansion of the PoC to the entire estate. It was too early in the cloud era and people were still being ridiculed for suggesting a cloud-first strategy.

Even so, that experience completely changed my outlook on my career: what it was and what it should be and although I was compelled for several more years to work in server-centric environments, the vision of a utility, closely aligned with customer and business needs, never left me. I worked to re-engineer my career for the cloud and have never looked back.

What's your cloud story?