So What is DevOps: Origins, Key Concepts, Capabilities, Common Misconceptions and Way Forward

So what is DevOps

DevOps is a new term emerging from the collision of two major related trends. The first was also called “agile infrastructure” or “agile operations”; it sprang from applying Agile and Lean approaches to operations work. The second is a much-expanded understanding of the value of collaboration between development and operations staff throughout all stages of the development life cycle when creating and operating a service, and how important operations has become in our increasingly service-oriented world.

Where and Why does DevOps come from? It has its roots in Lean and Agile with The Agile Manifesto  in 2001.The core principles of the manifesto being to get away from the rigid, waterfall-oriented, documentation-heavy world of software development, which was resulting in most software development projects being late, over budget, or abject failures and move to an iterative approach where there was constant interaction with the customer, end-user, or a surrogate who represented them. Rigid milestones such as Requirements Documentation were shelved as it brought code no closer to being delivered from the day before and the goal was to use real running code as the true measure of progress; and to create requirements that did not need to be written in stone up front, but would evolve and be refined as the applications were being developed. In the end it It comes down to Business needing visibility into the status of what is being delivered by IT; Are they on time and on budget? Business needs the application delivery teams to provide feedback on how the clients and end users are utilizing the applications and services. Are they able to get the business value?

DevOps can be captured in the key Concepts of  Values, Principles, Methods, Practices, and Tools.

Values: Usually agreed upon to be embodied in the Agile Manifesto. These are the core values that empower agile. Individuals and interactions over processes and tools. Working software over comprehensive documentation. Customer collaboration over contract negotiation. Responding to change over following a plan

Agile Principles – Generally agreed upon strategic approaches that support these values and can be found in the the principles behind the agile manifesto.

Principles behind the Agile Manifesto

Our highest priority is to satisfy the customer through early and continuous delivery of valuable software. Welcome changing requirements, even late in development. Agile processes harness change for the customer's competitive advantage. Deliver working software frequently, from a couple of weeks to a couple of months, with a preference to the shorter timescale. Business people and developers must work together daily throughout the project. Build projects around motivated individuals. Give them the environment and support they need, and trust them to get the job done. The most efficient and effective method of conveying information to and within a development team is face-to-face conversation. Working software is the primary measure of progress. Agile processes promote sustainable development. The sponsors, developers, and users should be able to maintain a constant pace indefinitely. Continuous attention to technical excellence and good design enhances agility. Simplicity--the art of maximizing the amount of work not done--is essential. The best architectures, requirements, and designs emerge from self-organizing teams. At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behavior accordingly.

Agile Methods – Individual level process implementations of the principles. XP, Scrum, scaled-agile, your own custom process etc – this is where the philosophy gives way to operational playbooks of “how we intend to do this in real life.” None of them are mandatory, just possible implementations.

Practices - Highly specific tactical techniques used in conjunction with agile and DevOps. None are required to be agile but many agile implementations have seen value from adopting them. Standups, planning poker, backlogs, CI/CD all the specific artifacts a developer uses to perform their work.

Agile Tools – Specific technical implementations of these practices used by teams to facilitate doing their work according to these methods. In the DevOps world there’s been an explosion of tools in release (ex. Jenkins), configuration management (Puppet, Chef, Ansible), orchestration (ex. zookeeper), monitoring, virtualization and containerization (AWS, OpenStack, Kubernetes, docker etc.) and many more. While, as with Agile, it’s incorrect to say a tool is “a DevOps tool” in the sense that it will magically bring you DevOps, while there are tools being developed with the express goal of facilitating the above principles, methods, and practices, a holistic and EA based understanding of DevOps should incorporate this layer and principles and guidelines.

Key Capabilities

There are two key capabilities of DevOps at its core: continuous integration and continuous delivery. Without these two capabilities, there is no DevOps.

Continuous Integration: Delivering a software application or system today involves multiple teams of developers working on separate components of the application. Typically, the completed application also needs to interact with other applications / services to perform its functions creating an inherent need for developers to integrate their work with components built by other development teams and with other applications / services and hence making integration an essential and complex task in the software development lifecycle. The process of doing this at a regular cadence is referred to as continuous integration, and it is a key agile practice. Developers integrate their work with the rest of the development team regularly and test the integrated work. In the case of enterprise systems, which span multiple platforms, applications, or services, developers also integrate with other systems and services as often as possible allowing for early identification of risks.

Some of the best practices of Continuous Integration are: Only have single-source repository, build automation, builds are self-testing, everyone commits to the mainline every day, every commit builds the mainline on an integration machine, build should be fast, Test in a clone of production, anyone should be able to get the latest executable, everyone can see what is happening, and deployment should be automated.

Continuous Delivery: Continuous delivery simply involves taking the concept of continuous integration to the next step, when the application is built, at the end of every continuous integration build, it is delivered to the next stage in the application delivery life-cycle and delivered to the Quality Assurance (QA) team for testing and then to the ops team for delivery to the production system. The goal of continuous delivery is to get the new features that the developers are creating out to the customers and users as soon as possible. Not all builds that come out of a CI effort do not need to go to QA; only the “good” ones with functionality that is at a stage of development where it can be tested need to go to QA. Similarly, all the builds that go through QA do not need to go to production. Only the “good” production ones should be delivered to production. To test whether the builds coming out are production-ready, they should be delivered to a staging or test area that is a clone of production. This practice of regularly delivering the application being developed to QA and operations for validation and potential release to customers is referred to as continuous delivery and requires the creation of a delivery pipeline with the core capability that automates continuous delivery.

Other supporting practices to CI and CD to DevOps are

Automated Infrastructure as Code has to be done minimize cycle time and versioning environments using a software-defined environment.

Continuous Feedback: This is essentially getting feedback from each functional area of the delivery pipeline.

Continuous Testing: Is the capability for testing the application, the environment, and the delivery process at every stage of the delivery pipeline for the application being delivered and is  intertwined into the processes of continuous integration and continuous delivery. In continuous testing, the biggest challenge is that some of the applications, services, and data sources that are required to perform some tests may not be available or the cost is prohibitive and can be solved thru use of automated testing tools.

Continuous Monitoring: In production, the Ops team manages and ensures that an application is performing as desired and the environment is stable via continuous monitoring. Ops teams have their own tools to monitor their environments and running systems need to ensure that the applications are performing, from the process level down to levels that are lower than what system-monitoring tools would allow. This requires that Ops teams use tools that can monitor application performance and issues. It may also require that they work with Dev to incorporate self monitoring or analytics-gathering capabilities right into the applications that are being built. This would allow for true end-to-end continuously monitoring. As the technology in this space has grown, there has also been the emergence of tools and services that monitor application behavior and user sentiment, providing even finer-grained feedback that is useful to developers and the line of business. Continuous monitoring requires the capture and analysis of metrics in four areas:

• Application performance
• System performance
• Application user behavior
• User sentiment

Ops teams should not just gather this data but also run analytics on it and make their feedback consumable by their target audience, from deep technical Ops practitioners, like performance engineers, to non-technical line-of-business stakeholders. Data is of no value unless it is consumable. 

Continuous Business Planning: The DevOps practice of continuous business planning focuses on the lines of business and their planning processes. Businesses need to be agile and able to react quickly to customer feedback. To achieve this, many businesses today employ Lean thinking techniques. These techniques involve starting small by identifying the outcomes and resources needed to test the business vision or value and then continuously adapting and adjusting based on customer feedback. To achieve these goals, organizations measure the current baseline state, find out what customers really want, and then shift direction by updating their business plans accordingly, allowing them to make continuous trade-off decisions in a resource-constrained environment. The latest addition to the arsenal of capabilities is design thinking.

Collaborative Development: This is a given and is essentially in place to ensure that organizations with large, distributed teams enable visibility between, and collaboration among, cross-function practitioners and teams by access and visibility by practitioners not just to artifacts, work items, and metrics related to their functional area, but across all functional areas into which they need to have visibility and seamless hand-off of artifacts from one or team to another. Looking at DevOps as a cultural movement, where the fostering of communication, collaboration, and trust are the core tenets collaborative development might be viewed core capability of DevOps.

Architecture and Risk Mitigation: You cannot achieve continuous delivery with large, monolithic systems. While architectural refactoring was largely ignored in the early days of DevOps, it is going mainstream now, mainly thanks to the evolution of microservices and the application of 12Factor. 12FactorApp. The architectural transformation needed to achieve continuous delivery, irrespective of whether microservices are used, is to enable the delivery of changes in small batches—thus, reducing batch size. Reducing batch size is imperative Reduce risk, Improving quality and to enable faster delivery.

Continuous Improvement: At the core of CI is the fact that wherever you start, at any given maturity level, adopting DevOps is not a one-time project; it is an ongoing effort. The 3 focus areas to improve on are

• Application. Are the application changes that you just delivered functioning and performing as desired? What can you learn from the continuous feedback coming in to improve the app in the next iteration?
• Environment. Are the environments the application is running on performing and behaving as desired? Are the service level agreements (SLAs) being met? What can you learn from the continuous feedback that is coming in to improve the environments in the next iteration? Are your SLOs being met ie service level objectives ?
• Process. What can you learn from the experiences of the practitioners and stakeholders to improve the delivery processes themselves in the next iteration?
• While most organizations have efforts ongoing to continuously improve the application being delivered, fewer organizations have the same level of rigor for continuously improving the operational environments, based on real metrics

Metrics: To manage and consequently improve, you have to measure some critical metrics or Key Performance Indicators (KPIs). We need a baseline measurement of these KPIs, marking the starting point, and ongoing measurements to see if improvement is occurring. Not only do you need to measure that positive changes are occurring but also need to be understand cause and effect as to which actions result in improving KPIs. Making several changes to people and processes, and capturing which changes are actually resulting in positive improvement is a must. To know what metrics to capture you have to know the business drivers and business impacts you are capturing. Some example metrics might be

• Development time, Deployment time, Time in maintenance time, Time between releases, Backlogs, Build times, verification of the build time etc.

Culture factor in DevOps: DevOps is also Culture. Can you change culture? Even after all the process improvement and automation that can be introduced in an organization, the organization can only succeed at adopting the culture of DevOps if it is able to overcome the inherent cultural inertia. Organizations have inertia—an inherent resistance to change. Change is not easy, especially in large organizations where the cultural may have had years to develop and permeates across hundreds, if not thousands, of people. These practitioners or people, as individuals, may appreciate the value of adopting DevOps, but as a collective, they resist change and thus have inertia. Overcoming this inertia is key. Cultural inertia can be exhibited by the following statements:

“This is the way we do things here.”

“Yes, but changing X is not in my control.”

“Nothing is broken in our processes. Why should we change?”

“You will need to talk to Y about that; WE cannot change how

THEY work.”

“Management will never allow that.”

“Don’t you know we are in a regulated industry?”

“DevOps is the new flavor of the month. Let’s see how long this effort lasts.”

Over time, organizations develop behaviors; teams and groups divide up actions and responsibilities along organizational lines; checks and balances are established in the name of governance but are not related to true governance at all; processes exist, but no one knows why—they are just there; reports are produced that no one reads anymore, but no one is willing to do away with them; bad things happened in the past and resulted in approval requirements to ensure they never happen again; and so on. All of these behaviors build up inertia in an organization’s culture. Cultural inertia can be overcome by

1. Visibility: Shine the light into what each time and expose each team. There is no greater cause of mistrust than not having visibility into teams you have to engage with, and not knowing they did with the artifacts they are handing off to you.
2. Effective communication: Email and voicemail need to be done away with as sources of communication in a DevOps environment; so do project plan and status documents, slide decks, and spreadsheets. Communication needs to be live and peer-to-peer, not via email or tickets, or done through management. One practitioner should be able to communicate with any other practitioner she needs to, without having to go through a chain of command.
3. Common measurements: Lack of right measurements for practitioners and teams causes the most inertia among teams and practitioners. People will not change their behaviors, unless the way they are being measured matches the new, desired behaviors

Summary:

In the end it comes down to three primary practice areas that are usually discussed in context of DevOps.

• Infrastructure Automation – create your systems, OS configs, and app deployments as code.
• Continuous Delivery – build, test, deploy your apps in a fast and automated manner.
• Site Reliability Engineering – Wikipedia says it best here as a discipline that incorporates aspects of software engineering and applies them to infrastructure and operations problems. The main goals are to create ultra-scalable and highly reliable software systems.

There are also some things to remember such as DevOps is not just about NoOps, Tooling, Culture or Lack thereof, not just Dev and Ops, nor is it just a title or the be all and end off the whole organization and given a few points below.

• DevOps is just not about No operations with operations going away and development taking over as there is some misconception on this. Some parts of operations need to be automated, that means that either we ops people do some automation development, or developers are writing “operations” code, or both which is uncomfortable to some people.
• DevOps is also not simply implementing a set of tools. People will just implement the processes or tools of DevOps without the principles in mind, which is definitely an antipattern. Automation is just the exercise of power, and unwise automation can do as much damage as wise automation can bring benefit. Tools specifically designed to run systems in accordance with a DevOps mindset are valuable but DevOps is not just a sum of the tools used.
• Devops is not just Culture. Agile has not helped thousands of dev shops because the work on it stopped at “culture,”. DevOps consists of items at all the topics talked about, and is largely useless without the tangible body of practice that has emerged around it. Can figure out all those best practices yourself given the cultural direction and lots of time to experiment; Maybe, but sharing information is why we have the Internet.
• DevOps is not Devs and Ops and not an exclusionary Club. Does security come into play? Yes. Also Network admins and infrastructure teams etc. The point is that all the participants in creating a product or system should collaborate from the beginning; business, developers, and operations including security, network, and whoever else. There are =different kinds of business and developer stakeholders as well; just because everyone doesn’t get a specific call-out doesn’t mean that they aren’t included.  
• Lastly DevOps is not just a Job Title nor is it the Everything. Simply taking an existing ops team and calling them “The DevOps Team” does not make it DevOps Nor does changing a job title to “DevOps Engineer.” If you don’t adopt the values and principles above, which require change at an overall system level not simply within a given team, you won’t get all the benefits. Since DevOps plugs into the overall structure of a lot of lean and agile thinking, and there are opportunities for that kind of collaboration throughout an organization, but going and reengineering your business processes isn’t really DevOps . It is part of an overall, hopefully collaborative and agile corporate culture, but DevOps is specifically about how operations plugs into that. 

Last to Get started With DevOps there is not one path to DevOps just what works in your organization. Very successful DevOps initiatives have been originated from dev teams and from ops teams, top down and bottom up, from inside the company and from consultants etc. Anyone who tries to give a generic playbook for how you can get it implemented is not wise. Learn about the values, principles, methods, and practices of DevOps and trying to spread it via whatever channel is most effective.