3x3 Principles of DevOps & IT Production Engineering

Introduction

The acceptance of Software-as-a-Service (SaaS) as the default software delivery method and the widespread usage of public cloud and virtualization technologies by the enterprises kickstarted the DevOps movement - a big tent of ideas and best practices that address the operations management challenges resulted in by this huge shift in software delivery and IT infrastructure usage.

A lot of the DevOps best practices were in place and widely practiced, such as Continuous Integration (CI), well before the keyword DevOps has become popular. Though the jury is still out there to determine what actually DevOps covers, it has been pretty clear to its practitioners that it is all about operationalizing the application. (See an earlier rhetoric of mine on that topic here https://www.dhirubhai.net/pulse/industrializing-code-alternate-definition-devops-theakanath/.) In IT production engineering, DevOps has become a standard practice now, but there have been few attempts to understand it fully and why it is a necessity.

The DevOps patterns are derived out of best practices that were tested in the field, and not set by some committee huddled in an Alpine resort. The requirement for a software vendor to deliver the service 24x7 with little or no downtime throws up its own operations management challenges and the result is a large set of best practices primarily focused on automating the production engineering tasks. DevOps has been maturing into an engineering discipline in that respect, moving beyond the spectacles of that being defined by buzzwords.

As the DevOps practice matures into an effective operations engineering discipline, it is possible to identify what kinds of problems it tries to solve and what sort of resources it requires. The 3x3 principles of DevOps are derived out of a matrix of 3 operational requirements - reliability, scalability and security, and 3 organizational resources - code, infrastructure and people that are needed to fulfil those requirements.  

Code

The application code is central to a SaaS based business. The business’s ability to industrialize the code by delivering the service reliably and securely to the customers at scale determines its long term success.  

Defect free code

The application needs to run with minimum defects and it should not break while running. For achieving both of those requirements the code needs to be tested thoroughly and frequently. Continuous Integration (CI) helps to test and deploy high quality code in production more often. If any issue is found with the code it can be found, fixed and verified quickly using the CI infrastructure.

Code that scales 

A multi-tenant SaaS deployment has to handle large volume of user traffic. Code should perform well under heavy load and also it should be flexible to respond to standard hardware scaling techniques such as using a load balancer. There are industry standard DevOps best practices available to address this. The microservices technologies such as Docker and Kubernetes provide more options for the code to be deployed in a highly scalable and elastic environment and the code needs to written to make use of such technologies available for scaling.

Unbreakable code 

Cyber security breaches and hacks usually exploit security vulnerabilities in the code. Hardening code has been a requirement ever since the software environment was exposed to the Internet and it predates DevOps. However, the widespread adoption of public cloud as the computing infrastructure exacerbated the problem. DevOps offers tools and methods to systematically find, fix and prevent security issues in code.

Infrastructure

The large scale adoption of enterprises to run their applications in the public cloud is a major shift from the classic data center or colocation based strategies. The evolution of DevOps best practices can be traced mainly to the responses to the shift of ground in infrastructure usage, from dedicated datacenter to public cloud.

Reliable infrastructure 

A reliable codebase needs to run on a reliable infrastructure stack to deliver highly available and reliable software service. With the availability of various virtualization technologies and public cloud services, the infrastructure can be provisioned and maintained as code (IaaC). A large set of DevOps tools and best practices are available in this area. 

Infrastructure that scales 

As with the application code, the infrastructure needs to withstand variable workload typical of a SaaS production environment. Public cloud platforms provide native tools to address this requirement. A large, matured set of DevOps best practices are available in this area. One of the focus areas of DevOps has been infrastructure provisioning right from its advent.

Hardened infrastructure 

The infrastructure layer of a SaaS production environment is at the frontline of cyber security attacks. While the infrastructure security best practices predate DevOps, their rollout and maintenance have become much simpler with the tools and automated processes brought in new by DevOps.

People

The most important asset of an organization is its human resource and DevOps has an impact on how that valuable asset is managed within a production engineering department.

Reliable people 

The SaaS applications need to be available 24x7. Irrespective of the highly automated processes in place, the availability and the capability of people who build automation and keep the lights on are very important. 

Scalable team 

The DevOps team needs to respond to operational requirements and issues that are highly dynamic in nature. For that the DevOps team has to be resourceful and flexible to address changing business needs and turn them around quickly. A highly scalable team, that is not necessarily a large team, needs to be set up, with diverse skills available in house. 

Secured team 

Security starts with the people and it has two levels in an organization. The DevOps team members must be committed to enforce security best practices as they normally have full access to infrastructure and tools. Them taking short cuts would introduce vulnerabilities to the production environment. The application developers and the rest of the staff would have access to a subset of resources and secrets. They can also introduce vulnerabilities depending on the level of access granted to them. While securing corporate resources typically falls into the hands of some corporate IT management group, the responsibility of DevOps team is to build guardrails around key infrastructure components to prevent any rogue employee from compromising the production infrastructure environment. 

To reiterate

The key IT production engineering objectives - reliability, scalability and security - can be achieved by implementing DevOps best practices to govern the major resources - code, infrastructure and people. The 3x3 principles of DevOps and production engineering constitute a framework to understand what DevOps does on the ground and how Devops practice aligns with the requirements of a SaaS based business. It also helps to audit an organization on where it stands in terms of rolling out the DevOps best practices.

About the author: Thomas K. Theakanath has over 25 years of experience in software development, system integration and IT production engineering. A hands-on DevOps practitioner, he led teams and projects in both large and small companies including Amdocs, Yahoo, SAP and multiple startups in Silicon Valley. A thought leader in DevOps he contributes regularly to publications such as DevOps.com and DZone, and speaks at conferences and meetups on DevOps topics. He has a bachelor's degree in Production Engineering from Calicut University and a masters degree in Industrial Management from Indian Institute of Technology, Bombay.