On Cloud 9 - Declutter your IT to make room for organizational growth

Acknowledgement: My colleagues Himanshu Sahni and Vinod Gopinathan gave me valuable input for this article. Their experience implementing cloud programs was very insightful.

Decluttering is a powerful analogy to explain why organizations should embrace Cloud. Over the years organizations collect a mish mash of technology assets, driven by M&A, legacy technology which is difficult to sunset, unused capacity as one must design for peak demand, spaghetti code, desire to embrace the next best shiny toy, organizational structures and so on. This results in high total costs of ownership and in an increasingly fast paced business environment, this underlying complexity hurts further. It reduces speed and agility and distracts businesses from focusing on core competence

Most organizations I speak to today, have a cloud strategy and are at varying levels of adoption. However, if an organization is vacillating on adopting the cloud wholeheartedly, I would like to sound the alarm bell. The opportunity cost of not embracing the cloud is exponentially rising. In this article, I summarize the benefits of cloud adoption:

COST: Lets tackle the obvious one first. Adopting the cloud will reduce your TCO (which you can invest in growth initiatives if you choose). Here is why:

1.      Competitive Pricing: To lure firms away from On-Premise technology, cloud players have priced their offerings competitively. Competition between cloud players has further reduced this cost. For example, storage on AWS S3 (Simple Storage Service), a file based storage system, costs approximately 23 USD/TB/month. For AWS Glacier, which is a storage system for archival the cost is approximately 4 USD/TB/month. The cost of compute using EC2 (Elastic Compute Cloud) clusters is competitive too, though it varies based on the configuration and it is important to size correctly, to avoid over paying. These costs are lower than on-premise systems. And Google Cloud claims its cheaper than AWS.

2.      Support costs drop: Double clicking on TCO shows, reduced support costs contribute to overall reduction in costs. This is driven by the fact that cloud abstracts the underlying plumbing. In fact, AWS periodically keeps introducing tools that abstract functionality to higher levels of the technology stack. This enables firms to reduce support contract costs, support personnel costs and other associated costs like procurement

3.      Pay as you go: Moving from fixed costs to variable costs is a big advantage of moving to the cloud. Costs move in tandem with a firm’s business cycle. Importantly, firms don’t have to provision for peak demand, which eliminates unused capacity. Cloud scales to meet demand, for example, retailers during Black Friday or tax preparation companies during tax season. The cherry on the top is the availability of spot instances for compute, where a provider like AWS further discounts unused compute capacity by up to 90%, during lean demand periods. Organizations can avail of the same for tasks that are not time sensitive, for example, end of month sales productivity numbers. If these numbers get delayed by an hour or even a day, hopefully it won’t upend the world!

A few downsides to note. Not all cloud technology is as mature as it’s on premise counterparts. And to realize reduction in overall TCO, firms need to invest in rearchitecting their on premise applications to bring them closer to being cloud native. Just porting an Oracle instance as is, to save on storage and compute – will not realize the full potential of savings the cloud can enable. Finally, since cloud moves the cost model to rent vs buy, it is possible that monthly / yearly costs (like my credit card bill!) can run away. The onus of usage oversight however, is on the consumer and cloud providers give enough tools to enable monitoring.

COMPLEXITY: Cloud reduces IT complexity due to the following factors:

1.      Abstraction: Network, Hardware, Database and Application layers are often abstracted from users on the cloud, leaving firms to focus on their core competence, a point I discuss later in the article. Storage and compute resources auto scale to a firm’s needs, with minimal intervention.

2.      Interoperability: Cloud providers invest substantial effort in enabling the latest open source and proprietary technologies to work seamlessly on their cloud stack. For example, AWS ensures that Oracle DB, MS SQL, Apache tool set, Tensor Flow, Docker Containers etcetera all work seamlessly within their Cloud. Further AWS keeps enhancing its cloud stack to include more functionality natively. (Which often creates a direct competition with open source or proprietary technologies AWS hosts).

3.      Tools: To support abstraction, interoperability and to make it easy to use its platform, AWS regularly invests in building tools. For example, to make migration of data easy, it has built schema converters that ease data transfer from Oracle DB and MS SQL to its RDBMS, AWS Aurora. Across activities like provisioning, deployment, monitoring etcetera, AWS keeps building and enhancing tool sets that make the technologists job easy and allows them to focus on higher layers / functionalities of the cloud stack

A possible downside to reduced complexity and an over reliance on cloud technologies, is the loss of organizational muscle memory around the lower layers of the technology stack. But in many ways, cloud is like the Internet or the mobile phone, technologies that have created a profound impact and cannot be reversed. To ensure a firm is not held hostage by a cloud provider, one may consider using multiple providers. In fact, quite possibly in the future, industry forces will push interoperability between cloud providers and create one master cloud! As cloud becomes pervasive like electricity, and a systemic risk, it might also come under some form of regulatory oversight.  

SPEED and AGILITY: Speed, Agility and Complexity go hand in hand. To use a driving analogy, Speed is like acceleration, Agility is like maneuverability and (lack of) Complexity is like automatic gear shift. Along with reducing complexity, cloud enables both speed and agility in business. A compelling example is in integration of acquisitions. When Expedia acquired Travelocity, it could bring its applications on to its platform in about 90 days – that’s powerful! The following factors enable speed and agility:

1.      Scalability: A big benefit of cloud is that it scales up and down based on your business needs through its peaks and troughs. This allows firms to scale very quickly, for example, sign on new customers or bring a new product to market. And in a downturn, a firm is not left with unused technology infrastructure that increase business expenses.

2.      Made to measure: Cloud providers bring best of breed tools across storage, compute, ETL/ELT, deployment, monitoring, databases, ML libraries, analytics tool sets etcetera, all under one roof and ensure their interoperability. This allows firms to create architectures that are made to measure and allow them to evolve the same quickly, based on changing business imperatives.

3.      A fast Dream–to–Deploy cycle: Procurement, development, deployment and migration of software and data is fast, often reducing end to end timelines by 90%! Bringing a new product quickly to market, is one of the most fundamental competitive advantages, that cloud enables. And it creates a level playing field between incumbents and startups.

4.      Functionality as a Service: As mentioned earlier, cloud companies keep abstracting functionality to higher layers of the technology stack. Today a lot of the abstraction is all the way to business functionality, for example ML libraries for image recognition, speech recognition and video analysis. Firms can thus focus on the application part of the technology and are able to roll out better products faster.

A key point to note here is that moving to the cloud also requires a change in the mindset of the technology organization. Adopting automation, DevOps, Agile are important enablers for success on the cloud. Changing mindset and culture is hard and it is important organizations invest equal effort in evolving their technology development frameworks and mindset, as they do on building applications.

INFORMATION SECURITY, DATA PRIVACY and DISASTER RECOVERY: These topics are naturally a big concern for customers and are often a reason for highly regulated industries to resist moving to the cloud. I would however posit that the Cloud is more secure than on premise. The fact is, the opportunity cost of not adopting the cloud is exponentially rising and therefore, the focus should be on designing a cloud architecture that meets information security, data privacy and disaster recovery requirements.

1.      Information Security: Cloud companies provide state of the art tools to ensure information security like Virtual Private Cloud and encryption at rest. AWS also offers the AWS GovCloud, which is used by the US government, like the CIA. Use cases in business for GovCloud center around PII, sensitive patient medical records and financial data. Firms can also, request for single tenant servers, if their specific business requirement dictates such a configuration. Often though, information security is compromised due to lack of good architecture, lack of controls and employee carelessness and therefore firms must ensure they adhere to strong information security practices on the cloud, as they would with on premise systems

2.      Data Privacy: AWS provides firms full control of where their data resides from a geographic location perspective and allows encryption of data. Importantly, cloud providers enable their consumers to manage their own encryption keys. In this scenario, even if the law asks the cloud provider to share information, they cannot, as the data is encrypted and the keys are with the user. It is also quite possible that having your applications and data on the cloud enable you to comply to regulatory requirements faster. Cloud providers do a lot of the research and heavy lifting to comply to regulations, for example AWS has created a GDPR enablement toolkit

3.      Disaster Recovery: Organizations are using the cloud to create their disaster recovery sites, even for their on premise applications, allowing them to forgo the need for a physical DR site and the associated costs. Cloud providers supports multiple DR architectures and a cloud DR site can be made operational very fast.

FOCUS on CORE COMPETENCE: Embracing the cloud allows firms to focus organizational bandwidth on the application of technology, rather than building expertise on the underlying technology itself. For example, AWS offers out of the box algorithms for image analysis, video analysis, conversational chatbots and language services. All of this enables data scientists and developers, to accelerate business outcomes using machine learning. So, let’s explore how some of the leading companies have embraced cloud, in this instance AWS and driven business results:

Capital One: has been an early adopter of the cloud in financial services. Moving to AWS has enabled Capital One to reduce its data centers, projected to go down from 8 to 3 by 2018 end. Another example is the Capital One Mobile Banking app. Hosted on AWS, it allowed the firm to focus on creating the best app using design thinking, customer journey mapping and a customer centric mind-set, assured that the performance, reliability and scalability of the app would be taken care of by AWS. AWS also helps Capital One scale seamlessly to meet transactional peak volumes, for example credit card transactions during Black Friday.

Expedia: has some impressive stats. In a day, it approximately supports 750 MM searches, 9 B pricing and inventory updates, 3 B automated translations and deploys over 2000 code updates. This is made possible by the cloud which ensures resiliency, optimization and performance! Expedia’s competitive advantage is hyper-personalization, which requires vast amounts of data crunching. Using AWS, analysis that took them 3 hours to process now takes 30 seconds, that’s 360 times faster!! When Expedia.com experienced issues, they could roll out the entire front end onto AWS seamlessly, giving disaster recovery a new meaning of being always on! After acquiring Travelocity, they brought it onto the Expedia platform in 91 days, an unheard of speed and smoothness of acquisition integration

FINRA: is a financial regulator that oversees equity and options transactions to ensure market integrity and investor protection. With over 75 Billion market events per day at peak, they process more transactions in a day than Mastercard and Visa do in 6 months! By moving to the cloud, FINRA controlled rising storage and compute costs. Their focus has now moved from scaling to surveillance, running complex queries on over 20 petabytes of data. A perfect example was during the Brexit volatility, when AWS sprung tens of thousands of nodes to process the dramatic increase in volumes. The interesting part was, this was done automatically and FINRA only realized subsequently after analyzing the logs. Cloud makes such events seamless to manage!

Netflix: is perhaps AWS’ most famous tenant using their cloud in a very big way. One use case is AWS Kinesis, a tool for processing streaming data, which is being leveraged to analyze application logs in real time. This gives them insight into how applications are performing across availability zones and allows Netflix to make changes in real time, to enable customers to have the best viewing experience, for example, no lags in video streaming. This is a perfect example of how Netflix is using core cloud technology to improve user experience

Clutter it is said, is nothing more than postponed decisions. Hopefully, this article sheds some light on why organizations should leverage cloud to simplify their technology and create organizational bandwidth to focus on growth and encourages firms to start or accelerate their cloud journey!