Improving the software development process through DevOps

DevOps is one of the very popular terminologies especially in software development companies and there are many misunderstandings about what is DevOps, is it and new Software Development Life Cycle (SDLC), is it Agile or does it improve process agility, it is a role inside the team or what is the DevOps and how this can improve software delivery.

The big challenge

As we know that software after implementation and delivery, is used by the end-users inside the organization, it needs to be maintained and also operated, not only that it needs to be enhanced with the time until it is decommissioned or replaced with a new software powered by new technologies.

Most of the time, the team that manages the development, is different than who manages quality, and as well is different than who manages security, deployments, operations, and maintenance. Each team may have different objectives, leaders, and performance indicators. This led that the teams may compete more than collaborate and this often creates the blame culture and mindset.

This kind of siloed vertical disciplines inside the team, created a huge delay in the process, manual checks, and reviews, and even every team needs to deliver his work to the team after that in the process.

Whether it is Waterfall or Agile, software development contains almost the same steps, from analysis, design, development, testing, deployment, operations, and maintenance. Each SDLC deals with these steps in different ways according to the SDLC principles and planning. While most of the companies moved to Agile, they used the same team structures from Silos that every member is focusing on a specific area from these steps.

Although adopting Agile principles focus on enhancing the end-to-end development process by eliminating the issues in traditional software development lifecycles, the same challenges in managing the operations were still there and DevOps’ main purpose is to overcome those challenges.

What DevOps offered?

Many people ask what DevOps stands for, “DevOps” is short for “development” + “operations”. DevOps is not a separate Software Development Life Cycle, It consists of proven practices to enhance the overall development process including better interactions with the customer as well.

DevOps is integrating organization culture, People, Processes, and Technology to create new capabilities inside the organization to enable continuous delivery of value to the end-users who will benefit from the Software System. There is no widely approved definition, but most technology leaders company is using almost the same definition keywords and how DevOps is essential for continuous delivery of value to the end-users.

SDLC and DevOps are different concepts. They do not replace each other and neither do they compete. SDLC and DevOps perfectly complement each other.

DevOps is mainly used to replace the siloed development and operations to have multidisciplinary teams that work together to produce value to the end-users over time with shared practices and tools to effectively manage the process and ensure continuous collaboration between teams including the customers.

DevOps Capabilities

Microsoft as one of the leaders in DevOps defines 8 capabilities for DevOps which covers most of the important aspects for ensuring successful DevOps in practices, these capacities are:

• Continuous Planning
• Continuous Integration
• Continuous Delivery
• Continuous Operations
• Continuous Quality
• Continuous Security
• Continuous Collaboration
• Continuous Improvement

Each capability contains a set of practices published here in this link and I’m referencing the image used to summarize those capabilities and practices.

Other Companies also defined the main capabilities of DevOps, for example, Amazon, Google, and Others

Microsoft documented the required DevOps Capabilities and Practices perfectly to ensure the integration between culture, people, process, and technology.

I will briefly discuss each capability with the main example to illustrate how each capability contributes to the incremental created value for the end-users.

The main example that we will use here is about a retail company that wants to start online shopping by building an e-commerce platform.

Continuous Planning

This ensures that planning for software releases is not something fixed and can change by time according to the delivered value. During the planning, all team members are actually involved in planning the work that needs to be delivered, prioritizing what is important to be in the next release or sprint. This capability including its practices is the best fit with Agile methodology to ensure incremental value creation through sprints.

The value is actually defined here by what are the needed key results to be obtained from each sprint or release that contributes to the end business priorities and goal.

For example, imagine our example of a company that wants to build an e-commerce portal to sell its products, they need to define first what are the business priorities that they need to obtain, for example, “Becoming one of Top 3 in market share in online shopping leveraging our warehouse and our outreach to consumers to provide faster shipping and amazing experience”

To achieve that, the company started to set the objectives, such as, “building a strong state of the art scalable and reliable e-commerce platform.”

This is one of the key objectives to reach the defined business priority and to achieve that, they need to define a set of key results that they can achieve along the journey. Some of the key results are, “having the e-commerce MVP after 3 months”, “Integration with 3 payment gateways”, “Start shipping to 2 cities with full product list”, …etc.

These key results are driving the teams actually to plan their work together according to these priorities and key results. What has been explained was a framework called Objectives & Key Results (OKRs) which is a goal-setting framework designed to connect strategic goals set by leadership with the day-to-day activities that are executed by the teams.

Those OKRs are detailed in terms of features, requirements, and user stories that need to be delivered after that. OKRs can be reviewed quarterly and measured to track outcomes’ achievement.

Continuous Integration

The main idea of continuous integration is to minimize the time between code developed by one of the team members to be integrated with the main code repository for the whole software. So, each team member is integrating his/her work more frequently.

The benefits from that are huge, for example, ensuring that actually, the code is working with other components, nothing is broken, minimizing the bugs and errors that can be found during the code integration process, also minimizing the time to recover from issues as well, and other benefits too.

If we go back to our e-commerce example, each team member who is working on a specific user story or piece of requirements needs to push his work frequently and apply different practices to ensure the quality of the code as well. The continuous integration needs lots of automation to make sure that it works as a first quality gate and if everything works fine the code is integrated, and if not, it will reject the integration request.

The Image above which is captured from the Microsoft DevOps learning path differentiates between Continuous Integration, delivery, and deployment. In continuous integration, the goal mainly is to integrate the code frequently to the software product code, while In Continuous Delivery and Deployment, the focus is mainly to deliver the integrated code to users to be accepted and deployed to the production environment. I will explain more in continuous Delivery.

Continuous Delivery

It is mainly the frequency of software releases after the testing to be deployed to the production for usage by the end-users.

Most of us can relate this experience with massive daily updates in mobile applications, each day, I found at least 5 to 10 apps need an update. This implies how those applications’ owners are rapidly adding daily new features and enhancements to their applications.

Most of the legacy systems require manual deployment steps and if one step fails, it may require repeating the whole time-consuming process and it was a nightmare for the operations team and can take days and maintenance notice for customers which disrupts the business and harms companies revenue and in most cases produces huge losses.

When the final production deployment step needs to be triggered manually, we call this a Continuous Delivery, if the production deployment is automated without manual triggers and through automated pipelines, we call that a continuous deployment.

So, imagine this retail company found a security threat and they fixed the update and they need to deploy that update now. If this deployment activity is time-consuming, it will affect their reputation at each minute besides losing their customers and sales.

So, continuous delivery has a set of practices to ensure this faster-automated deployment, like infrastructure as a code, release pipeline, secure infrastructure deployment, configuration management, …etc.

Continuous Quality

This capability needs an entire culture shift in how the organization is managing the software quality, it is about instead of assuring quality to continuous quality by not waiting until the acceptance test to ensure quality but embedding the quality early in the process which is known as Shifting left quality. This left-shift does not mean that we will not perform the normal testing process but it is important to balance them and start early in the process by ensuring requirements quality, code quality, and test automation across all types of testing, for example, unit, integration, system, regression, …etc.

Using a framework like Test Driven Development can enhance the quality of written code and also ensure that the requirements and user stories are testable. This will require also shared quality responsibilities between teams not only the testing team, so the quality is becoming each team member’s job.

Continuous Operations

Continuous operations is a capability to make sure that the organization is avoiding system downtime and outages due to any issue, fault, or damage. How will the system be resilient and reliable to be used? And how to maintain the system without affecting its operations and usage from the end-user.

Updates from operating systems, security patches, and integration services are almost changing every day. Managing these kinds of updates with even new deployments are important to ensure system availability without interruption. Technologies like cloud computing helped in achieving that.

Some important practices to achieve this capability are auto-failover, applications monitoring, and auto-scaling. Blue/green deployment is one of the famous practices to prevent having any downtime while upgrading or maintaining the system.

Using the same example of the retail company, they can use application and system monitoring services to monitor the e-commerce platform traffic and this monitoring tool can also be integrated to trigger the automatic infrastructure scaling in case of unexpected traffic spikes.

Continuous Security

The secure development life cycle is more important than ever, similar to the quality the security requires also a mindset shift and to become each role responsibility. This also needs to shift these security procedures, standards, policies, and assurance to the left of the life cycle. So, not having the security testing and code scanning at the end of the development.

Now, many tools can detect vulnerabilities in code while developing to highlight any security risks and ensure that the security policies and standards are followed. Not only that but even how to design the system from architecture, used technology, infrastructure, and data to be secured.

We can see other terminologies like DevSecOps which stress integrating the security operations and mindset in the DevOps practices as well. As we illustrated here, Security is one of the core capabilities for following the full DevOps practices.

Continuous Collaboration

As we saw earlier that having all of these capabilities in place, will need strong team collaboration and alignment, from planning, testing, deployment, operations, …etc. to create this harmony. Most of the DevOps tools are now equipped with different collaboration tools and can integrate with other tools.

For example, Azure DevOps is providing integration with Microsoft Teams to provide an integration chatbot that can answer different questions, like what is the current backlog, what is the pipeline status, what are the current issues, as well as notifying the team with any issue directly to be fixed immediately.

Not only that but even the collaboration on planning, task assignment, providing wikis for teams for the faster onboarding process. This will enable the involved team members to respond to the required changes even faster and prompt the interactions between them.

Continuous Improvement

Improvements are always required and most of the time, the team is very busy to pause and take some time to think about what needs to be improved.

So, this capability is important for the team to look over all the processes and practices used and spot what needs to be improved. Without proper measurement, you will not know exactly what needs improvement, so Defining proper metrics is important for continuous improvement.

Going back to the retail company, imagine that they found that each new user story is consuming an average of 10 days from development until deployment to the end-users. This is impacting their productivity. They may be writing thousands of code lines, but this is not how productivity is measured now. Productivity needs to be aligned with the end key results in mind.

So, the team here should start analyzing the time to build, time to test, time to deploy, and time to communicate the new releases, and start investing what most consuming time in the process, why the build took more time than expected, do we need to enhance the user story and its tasks to be more clear. Why the test failed, did we miss defining the acceptance criteria for the feature, should we adopt Test Driven Development as a first quality gate for developers.

Takeaways

We talked about DevOps and how it can be embedded within the SDLC by adopting the main 8 capabilities and their practices. It is not easy and needs lots of work, learning, and mindset shift within the organization with leadership focus and support.

Many DevOps initiatives have failed due to a lack of culture focus, funding, and leadership support. DevOps practices introduce different changes in the organization from culture, resources, technology, and processes, as any change management project, needs sponsorship, putting the vision, proper planning, and proper management of the change process, through awareness, coaching, and overcoming challenges and resistance.

Thank you for reading and looking forward to knowing your thoughts.

Also published on melsatar.blog & Medium.