The Convergence of Multiple Tech Revolutions. Building & Growing Startups in a Chaotic World! Part 2 - Technology (Agile, K8, ML & Security)

Welcome back to this blog series where we explore how the hardship of the past few years is impacting Founders, their Startups & how they're evolving to overcome the challenges. In Part 1, we discussed how the increase in the cost of capital is driving the supply of VC investment to shrink - forcing Founders to mature how they build their businesses & think deeply on how to allocate their capital.

In today's post, we'll be digging into the 4 revolutions that have been transforming technology departments, from People to Process & Technology. These revolutions are causing disruption; forcing engineers to rethink how they design, develop, launch, operate & secure systems. While painful, I'd argue that this is long overdue - particularly when we live in a world where the Digital world can have a meaningful impact on the Physical world. Whether it's targeting Hospitals, Gas Pipelines, Government Organizations or Consumers, malicious actors have been running rampant. Without further ado, I present the 4 revolutions; Agile Development, Distributed Computing, Data Maturity & Security.

Agile Development

Why are legacy systems so 'bad'? Why are they so challenging and costly to manage, update & upgrade? The answer is Monolithic Application Development. The way applications have historically been developed is by building a large code base, which is typically added to in order to make changes. While harmless for smaller, simpler applications, as you get into larger, complex applications - Monolithic apps become a big problem. Like a novel, the code base weaves in and out of different components or 'services', stitching all of the code together in ways that engineers don't always easily understand. Trying to make changes to Chapter 1 or 2 often has unforeseen circumstances on Chapter 5 or 6. Therefore, rather than fixing problems with those Chapters, you have to add more code to fix the issues. Not only is the code base a mess, it's also terrible difficult to manage and grow at scale from an Infrastructure perspective. As Monolithic applications grow, they require increasingly more hardware to operate and you run into issues impacting employee productivity, customer satisfaction & beyond.

Enter Agile & Service-based architecture! Rather than write code like a novel, Developers decided to take a different approach to building applications. They began breaking them down into smaller components, or 'Services', each of which is independent of the other parts and can be easily updated without breaking everything else. This new approach birthed new practices like DevOps (Developer Operations) & made it to where Developers were able to increase productivity, building more features faster and reduce launch cycles from months to days. This revolution in Development converged with a revolution in Computing, which took advantage of this new approach to building applications to optimize Infrastructure management & cost.

Distributed Computing

Given that Infrastructure teams no longer needed large, expensive & specialized machines to run the applications that their Developers were producing, they were able to take a new approach to data center architecture. As Virtualization began to give way to Distributed computing, the industry needed something else to truly catalyze Innovation. As applications became global, teams needed to manage thousands of servers globally and Operations struggled to keep up.

All of a sudden words like Docker & Containers began to pop up. Engineers developed a way to isolate an Application & Operating System from the underlying hardware. While not necessarily a new concept, it was adopted aggressively as the Enterprise began to understand the implications. Servers can fail but your App still runs! Companies like Google pioneered orchestration frameworks to manage these Container & Service based applications, with Borg their internal global cluster management system & Kubernetes, the open source version, being great examples. Organizations all over the world, across industries and geographies, large & small began to adopt Kubernetes.

While Containerization & Kubernetes were powerful, companies were running these technologies inside of their own data centers. This limited their ability to realize the advantages of dynamically scaling infrastructure based on need. They were still having to over-provision hardware & pay for servers to sit on shelves unused during off-peak hours. With the rise of Public Cloud, IT & Development teams were liberated from that hardware throttle. Applications could now scale dynamically, globally, based on load and the company was able to only pay for what it used. Not only that, the infrastructure could scale & manage itself!

Data Maturity

Historically, organizations have thought of their data as a liability. Although valuable for understanding the current state of the business, critical information was difficult to secure and subject to compliance regulation across industries. As such, many companies treated their data like treasure & buried it in a secret location. In the age of self-managed data centers, this meant employees can't work from home and everything is physically locked down. However, as people and technology evolved to demand the ability to work from anywhere, things changed. At the same time, the most valuable companies in the world were no longer oil companies! A new cohort of Digital technology companies rose to power and dethroned all the other industries by monetizing data.

Companies faced a conundrum; how do I make my data an asset while also ensuring its security? Folks were forced to both make data more available through applications for distributed work forces & pressured to monetize data while also being asked hard questions about HIPAA, PCI, GDPR, SOC2...

This Age of the Data company required new technology, new skills, new processes & a new mentality. Those who matured their data practices quickly started Digitizing workflows, implemented cloud-based productivity systems, built Data Lakes, Data Warehouses and started taking Governance seriously. By implementing these fundamental concepts and technologies, companies were able to unlock powerful new capabilities. One superpower in particular was Machine Learning. Before you knew it, ML became one of the most important competitive advantages not only to unlock new sources of revenue but also to accelerate the Digitization revolution.

Security

As computing evolved from Private data centers to other models, the jobs of security teams became harder and harder. With a secure physical location, you can lock down personal computers and servers with relative ease. Make sure everyone needs a badge to get into the office, lock all of the hardware down to the shelf & desks, enforce strong username & passwords.... not too bad! However, the needs of the organization and employees began to evolve. It wasn't feasible to lock data down physically given that teams in New York needed to collaborate with London & Tokyo! Also, the engineering team was building an API for external partners to be able to pull data related to certain projects...

Security teams were now required to secure data everywhere, whether on computers in employee homes or systems that needed to access it from the internet. Once opened, they struggled to secure Pandora's box by layering on additional systems. It seemed like they were now having to validate that every system, manufacturer, & partner had the appropriate controls/add-ons in place.

To address these new realities, new approaches to Security were also required. A shift away from managing everything themselves and towards offloading responsibility to appropriate parties began to take on. New concepts like Zero Trust, Single Sign-on & Multi-Factor Authentication took hold while Vendors were now evaluated not only on feature, function & price but also security capabilities. With the rise of Public Cloud, these teams were able to offload responsibilities to vendors where it made sense while enabling their teams on the nuances of securing Cloud & Hybrid environments.

Conclusion

The convergence of Agile, Distributed Computing, Data Maturity & Security has paved the way for Public Cloud while also facilitating the rise of Digital Native Startups that are already transforming the world. The best Digital Natives have Agile development teams who are deploying on Kubernetes or Serverless systems distributed around the world on Public Cloud providers; ensuring a high degree of scalability, reliability & security. They are building data companies that build Governance & Security in at the point of inception, knowing that customer Trust & Privacy are important principles upon which to build a business. They are developing Revolutionary applications for Machine Learning across use cases like cancer diagnosis, credit score optimization, product recommendations, anomaly detection & beyond!

Founders & teams who stand on the shoulders of giants will be able to achieve things that folks hardly thought possible, at a record pace. Whether it's being able to reach the scale of Pokemon Go or Shopify - Founders are more empowered today to build Technology companies on a solid technical foundation than ever before! I expect many traditional industries and companies to be disrupted as these Digital Native Startups reach maturity & scale, despite the macroeconomic headwinds of the past few years.