OKRs in Software Engineering

This is the latest issue of my newsletter. Each week I cover the latest research and perspectives on developer productivity. Subscribe here to get future issues.

This week I read Objectives and Key Results in Software Teams: Challenges, Opportunities and Impact on Development , a study by Microsoft researchers Jenna Butler, Thomas Zimmerman, and Christian Bird. The paper explores the challenges engineering teams face with the OKR framework and offers strategies for their successful adoption.?

The Objectives and Key Results (OKRs) framework is a goal setting methodology that aligns the objectives of an organization, team, and individual by establishing clear, measurable key results to measure progress towards.

My summary of the paper

The idea for this study originated from a request by a VP at a large multi-national company. The authors write, “the Vice President asked the first author [Jenna Butler] to analyze why decisions made at the top level of the organization often took a long time to yield changes in the code and why some cross-group projects were successful and quick, while others face resistance and eventually fail.”

To understand the challenges the organization was facing, the researchers conducted an exploratory study, and found that goal setting was one of the top challenges. Specifically, they identified that teams struggled to choose the right goals, measure them, and share them out with other teams—all problems which the researchers believed would be solved with the use of OKRs. This inspired the researchers to conduct a broader study on the use of OKRs in large software engineering teams.?

This study took place at an organization with around 4,000 engineers. It was conducted in two parts: first, a series of interviews with engineering leaders and individual contributors to understand what issues were getting in the way of moving quickly. Then, the researchers deployed a survey to surface insights about developer engagement, team culture and alignment with goals, maturity with OKRs and other engineering practices, and activities related to goal setting. Data was analyzed using several techniques, including thematic analysis, linear regressions, and hypothesis testing. All results in the paper are statistically significant.?

Here are the key findings:?

Characteristics of teams that successfully adopt OKRs

In the organization studied, management had mentioned there had been mixed success at adopting OKRs. Thus, one of the objectives of this study was to uncover the reasons why some teams were more successful. Specifically, the researchers explored whether there were any distinguishing characteristics or practices of successful teams that others could learn from.?

To investigate these aspects, researchers used linear regression to explore the correlations between different factors of OKR maturity and elements of agile software engineering. In this study, OKR maturity is a score for how effective an organization is at defining goals, measuring goals, communicating goals, reporting progress towards goals, reviewing progress towards goals, and adjusting goals or resourcing towards goals.?

OKR maturity was found to be positively correlated with the following:?

• Having a unified mission as a team. This is intuitive: a unified mission brings alignment, and this alignment may drive people to set better OKRs as it is clear what they’re working toward.?
• Overall happiness at the company. This aligns with the finding from previous research that progressing towards a meaningful goal is a top motivator for employees.?
• Working remotely. “We hypothesize this is because teams who do not have regular face time and in person meetings with leadership need to be more clear on what they are doing, how it relates to the org, and have better ability to share back their progress.”

This suggests that teams grappling with more challenging software may face greater difficulties in achieving OKR maturity.

Another interesting finding: engineering teams may be less aligned than leaders think. While managers are critical to successfully adopting OKRs—it’s their responsibility to break high-level ideas down into goals and metrics for their teams—the majority of managers studied reported that they are not very effective at this task. There is also a mismatch in how aware managers think their teams are about their goals: 60% of individual contributors say that goals are communicated to them monthly, but 65% of managers say that they communicate goals weekly or multiple times a week.

Challenges teams face with OKRs

As some teams were unsuccessful in adopting the OKR framework, the researchers sought to understand the top challenges teams face when working with OKRs:?

1. Creating and setting OKRs: The process of creating and setting OKRs was by far the most reported challenge teams face. Almost 1 in 5 respondents believe they are not effective at goal setting at all. Only 45% of teams think they’re effective at measuring goals. The researchers add: “Historically large software companies may have tracked the work they are doing vs the outcome that work drove. Measuring outcomes can be much harder than just measuring the work one did.”?

2. Data issues: Data issues were the next most reported challenge that teams face. This includes issues like getting trusted data and being able to connect to different technologies to collect actionable reports. For these reasons, it’s challenging for organizations to enable teams to meaningfully measure progress and improvements.?

3. Working with others: In theory, when all OKRs are shared broadly and transparently, teams can see dependencies or accidental duplication of work early and correct. However, this study identified this area as a top challenge: teams find it difficult to get aligned with each other on the outcomes they’re trying to achieve.?

4. Committing to priorities: In the organization studied, individuals felt leadership would sometimes select priorities and then change them at will. This makes it difficult to align teams to higher level priorities.?

Recommendations for improving goal setting

Based on the challenges reported, the correlations with high OKR maturity, and the open-text suggestions from the survey, the researchers provide some recommendations for improving the goal setting and OKR process in engineering teams:

1. Invest in the data pipeline. Setting and measuring OKRs is effectively impossible without easy-to-use data that’s compliant with the company’s data requirements. When there’s a lot of overhead involved in tracking progress, teams may decide not to measure at all. The researchers encourage leaders to set up a data pipeline in a way that makes it easy for teams to measure progress.?

2. Increase transparency. One of the top challenges faced with OKRs is understanding other team’s goals. If OKRs are instead shared transparently, teams can better understand what others are working on.?

3. Improve communication. The majority of managers surveyed in this study don’t feel effective at taking high level ideas and translating them into work their teams could do. Prior work has shown that facilitating external communication, driving alignment and guiding a team are key elements that make a great software engineering manager. These same skills need to be present for a manager to effectively lead goal setting and goal alignment. Managers need to especially focus on driving clarity about how the work of individuals and their team relates to both other teams and the overall goals of the organization.

4. Promote learning communities. Some teams do have high OKR maturity; by forming learning communities, leaders can pair teams having strong OKR processes with teams who are still new to the process. Ideally, the less mature team can observe how more mature team works. The researchers suggest appointing an “OKR Champion” to oversee the process of pairing teams.?

Note: we’ve seen DX customers run a similar process, where they pair engineering teams that are stronger in certain areas (e.g., their on-call process or time for deep work) with others who are looking to improve in those areas. If you’re interested in this, listen to my interview on how Toast facilitates learning opportunities like these internally.?

5. Guide the process

In this study, OKRs were not adopted formally—rather, teams organically adopted them at different times and using different tools and cadences for tracking. OKRs are intended to be used as an ongoing discipline to ensure teams work together and are focusing their efforts to make measurable contributions that drive the company forward. With that, the researchers advise leaders to thoughtfully roll out an OKR process to the organization.?

“Implementing an OKR framework with a large group of software engineers without careful rollout and tooling will let a thousand flowers bloom - teams will find their own, creative, unique way to use this process.”

Final thoughts

This paper may be helpful for leaders planning to implement OKRs, looking to improve alignment between teams, or help teams be more connected with their goals and direction overall. In particular, I found these two parts of the study most useful:

• The challenges teams face when adopting goals. This list might help leaders get in front of the problems they could face when rolling out a framework for setting and measuring goals. Having a reliable, compliant, and easy-to-use data pipeline, providing transparency into teams’ goals and progress, and leadership’s commitment to goals are all important.?
• The mismatch between how aligned managers think their teams are, and how aligned they really are. This study took place in one organization, so the specific challenges surfaced may be experienced differently in other organizations. However, it does point out the possibility that teams may feel less clear about their direction than managers believe they are. ?

That’s it for this week! If you’re interested in reading a guide for running an internal survey to identify problems impacting developer productivity, send me a connection request with the note “guide.”