DevOps Measuring that makes sense!

At Bayer Crop Science, we have been practicing DevOps for few years now, with degree of focus varying from product group to product group and platform to platform. As a result, products and teams are at different levels of maturity when it comes to DevOps practices as in most organizations. We have been staring at these inconsistencies for quite some time and it has been one of the topics that triggered passionate discussions for many of our leadership meetings. Our conversation often looped around these questions: How can we consistently improve Dev Ops maturity, while delivering business value? How can we have an organic way of measuring maturity? How can we turn our teams into the driving force for improving maturity?  

What was clear to all of us was the obvious direct correlation between quality of our products and DevOps maturity. Product quality and the ability to diagnose problems proactively and to get solutions in the hands of users fast, plays a key role in having that superior user experience we all strive for. DevOps is truly an enabler for delivering quality products. When done correctly, DevOps maturity works as a true indicator of this quality and so it is extremely important to have an effective way to assess and improve maturity for the success of a product.

This brings up the obvious question; what’s the best way to figure out and improve DevOps maturity of our products?

There are three important basic steps involved in this process:

a.    Assess Maturity: A logical and important first step would be to see where the product is, in the maturity scale. It would be hard to improve on something if we don’t know where to start! Being able to effectively establish a maturity base line is critical.

b.    Define Target Maturity: Next step is to decide on an optimal target state for the product. Depending on the state and nature of the product, maturity target for optimal experience can vary. For example, maturity target can be much different for a product with small user base, minimal critical data vs a product with large user base providing complex business capabilities. Defining a practical target that is aligned with the product needs is important to identify the true gaps and the success of the maturity process.

c.    Strategize and Execute: Once gaps are identified, formulate a strategy to address the gaps and build a plan to meet the target maturity. Interestingly enough the success of this step depends primarily on the conviction and commitment of the product teams and therefore the trickiest of all the three steps.

The Human Factor

Our teams consist of many talented DevOps engineers, so skill was never a concern. As you probably know, there are plenty of tools out in the market to gather metrics on various aspects of DevOps practice and many of them do a good job in capturing those metrics. Our first approach was to license a few of these products, pump in all the metrics, share it everywhere in pretty slides and the magic will happen! Well, the magic didn’t happen for us. In spite of all these measuring, nothing much happened in the maturity of the products or DevOps practices.

It took some time for us to realize the root cause of this problem. There were two reasons for this behavior. a) While the metrics were valuable, it appeared foreign to teams and they struggled to take ownership. b) More importantly, these metrics failed to show a collective value statement for the product that teams could take as a goal and accomplish. As product teams are designed to work on value-based accomplishments, you can easily see the disconnect here. 

Our Journey

We finally addressed the DevOps Maturity problem for the products in our area with a good level of success, thanks to some of the innovative approaches we took. First realization was that the model will work only with active engagement from entire product teams in every step.

Faced with the task of assessing maturity, we soon realized the first task we need to complete is identifying a list of key DevOps areas that really matter to the platform. It just doesn’t matter what the world does if a metrics data adds little value to our products. In fact, it can be counter-productive due to the distraction it creates. All products are very different in terms of values they provide, operational needs and technical challenges. Being able to measure them in a way that makes sense for the product is crucial for proper assessment.

We formed a core focus group of developers across products, who are passionate about DevOps, for alignment on DevOps practices. After a lot of intense cross-team discussions, this team came up with a list of areas that really matters for the products. Obviously, these are the ones we need to pay attention to. Below table shows the DevOps areas we selected.

Now that we have a list, next task was establishing a scale that can be used across products to measure these key areas consistently. We evaluated the impact of these items to products and prepared a set of definitions in a simple scale of 1 to 5, 1 being the lowest and 5 the best. Below figure shows a sample definition for monitoring.

We used this framework to categorize and measure our products in a consistent fashion. DevOps Alignment team soon sprang into action brainstorming with product teams and assessing selected areas along the way. 

 Above is a sample Maturity Snapshot generated by the Alignment team.

This gave us a good baseline with key areas and scores. Our focus now shifted to addressing the questions around target maturity for these products and journey to that maturity. This process is somewhat complex and probably the most ambiguous primarily due to the fact that it’s something that needs to be figured out considering the operational needs and nature of capabilities provided by the product. Targeting a perfect 5 in all areas, while ideal, can be expensive and in some cases over-engineering. We considered many options and finally decided to live by the essence of Achieve Big Through Small by taking the approach of 3-month incremental planning. We set a short-term target maturity using the basic criteria, which is, tackle the most impactful areas for the product with least effort. Teams often arrived on these targets after many detailed discussions with product managers and business counterparts.

DevOps Alignment team soon became the body that brainstorms with product teams on a regular basis, assessing practices and having valuable conversations around practices. We realized the significance of these conversations as they gradually exposed gaps and challenges around DevOps practices to product teams. Finally, things started to make sense! 

As the Alignment team started identifying these gaps, we realized a need for collaboration across teams to address these gaps in a consistent fashion. It also emphasized the need for defining a guardrail on tools and practices to ensure consistency and efficiency of practices.

We solved this with the introduction of another team that focuses on adopting, building and standardizing tools and helping product teams to leverage these tools. We saw immediate value of this team with the range of contributions from standardization and championing of certain useful tools to custom developed tools that help improve maturity of our products. This team works very closely with the DevOps Alignment team that feeds in challenges and problem areas that are unique to products to help build solutions that improves product maturity. 

Summary

This model allowed teams to demonstrate maturity improvements to stakeholders as value added accomplishments for the product. This tremendously helped them to communicate accomplishments around product quality effectively and in turn help stakeholders understand how we are elevating the true customer experience. As part of continuing the maturity improvement, we constantly assess reality on the ground pertaining to each product and make adjustments to the model based on the experience, as needed.

To summarize, measuring DevOps maturity makes sense only when it’s done with keeping the nature and challenges of the product in mind. Product teams need to be clearly aligned on the value proposition so they can ‘deliver’ the value. Active and consistent collaboration of product teams is also critical to ensure that the learnings from the assessment are addressed and most importantly customers are experiencing the benefit!

I would love to hear your thoughts and comments here.