Measuring Product Returns — using 'buckets’ [Part 1]

Tldr: Measuring the impact of product work has always been critical. However, it fell by the wayside in the past decade, fuelled by cheap capital and a focus on growth above all else. As a more rational approach has taken hold in the tech/start-up/software world, the gaze has shifted back to – very reasonably – quantifying impact. Here, in Part 1, I offer a model and an approach to help facilitate the conversation in your team that helps you get it roughly right and drive alignment.

Prefer to listen? Here’s a podcast version. Thanks NotebookLM.?

My background is in investing. I’ve spent thousands of hours reading annual reports, talking to companies, synthesising what I heard, and judging companies' prospects. The goal was to understand if a company’s stock was being priced fairly (mostly) or unfairly (rarely), considering the balanced probabilities of its future prospects. It was a fun role. It was deeply focused on maximising potential risk-adjusted returns – measured clearly – $ in, $ out. There are various approaches to calculating return on investment (ROI), but they all boil down to some version of “if I invest X $, what can I expect back and when?”?

When I shifted into product, I was struck by how un-ROI-focused it was. The focus and measures tended to be more squishy. More subjective. NPS. Usage. Adjectives like ‘delight’ are used a lot. However, any product endeavour involves spending real money, and any sustainable business needs to see a return on that investment to thrive.?

Over the past decade, I’ve come to appreciate why both ways of measuring impact have emerged — one more financially focused and one more qualitatively focused – I want to make the case that both need to be carefully considered and offer a way to do that.?

At the outset, let me confess: I find this tricky. I’ve tried a few different models. It’s been hard to get the nuance to land with everyone. When considering investment returns, boards, senior executives, and those on your team have varying motivations and literacy levels. Conflict often arises here as it draws out how different business people see the world. I certainly don’t have it all right. I aim to offer a few ideas to help you have a more productive conversation next time you’re asked.?

First, let’s zoom out.?

What game are you playing??

Product is a relative trade-off game. The game is to try and make the best allocation of resources to a basket of ideas that are likely to best execute on your stated strategy. You always have limited resources and limited information. Unless you have an immature or toxic team, it’s not a productivity game — squeezing 10% more out of a team, even if it’s possible, is only relevant if you are working on the highest-impact problems in the first place. Product teams fail not due to lack of productivity or skill but by building the ‘wrong things’.?

Having an open approach for choosing how to allocate your resources best, leads to more trust, better alignment across functions, and sets you up to give you the best chance of solving the right customer problems in the right order in the right way.?

Let’s use a working example to expand on this idea.?

Assume you have an operating software product with 1000s of customers; you are also growing, and you have cross-functional teams of product managers, designers, and engineers. You, probably along with your leadership partners in design and engineering, are tasked with driving the most impact with the resources you have. You are asked to regularly reporting to the CEO, other senior executives and the board about what you’re up to, why and what impact it’s driving. This is as common as it is reasonable.?

Thinking in bets

Live products require different types of product investment. This is a portfolio of bets on different types of work.?

I like to break the portfolio into different 'buckets' representing the different types of work.?

There’s no hard or fast rule on how many buckets are best. My preference is 3-5 – fewer than 3 and you don’t have the granularity you need to make different types of trade-offs. More than 5 and it tends to add too much complexity to dividing everything up.

Once talking in 'buckets', most folks will naturally grok that they should be measured differently.?

Buckets?

Product teams do three main types of work: maintain what you have, build things that solve existing customer problems or build things that solve new customer problems.?

Here are a few buckets you could consider:?

• Maintenance/bugs — small (<2 weeks) fixes to things that have been missed or go wrong.?
• Quality of life improvements — small (<2 weeks) improvements to existing capabilities. Think of all the small, important tweaks you make to what you have.?
• Platform — product work that enables future customer-facing improvements or doesn’t directly impact the customer but is required nonetheless?
• New — larger chunks of work that either expand how you serve customers or dramatically reimagines what you have. These are usually the new features you announce and market with a bigger splash.?

They all cost money but can be tracked in different ways. The goal should be to have the fewest metrics that give you the clearest signal, and you don’t have to wait a long time to assess. I’ll dive into suggested metrics in Part 2.?

Buckets are flexible

The bucket approach works regardless of organisation size and doesn’t involve any fancy solutions to implement.?

It’s also team set-up agnostic. You could have teams work on certain buckets or across buckets. You can add or change the team setup without having to reinvent how you measure impact.?

The benefit of having clear overarching buckets is that it helps facilitate a productive conversation with the wider business (what is important to us at the moment?), it helps set the context for the team (why are we doing this not this?) and gives you some guardrails for investment before discussing particular customer problems or ideas that crop up.?

Allocation across buckets for most companies should be treated as a target (e.g. +/-5%). This will avoid needless over-precision and wasted cycles arguing about what goes where and exactly how much you spend on each. The goal is to get the allocation roughly right, in line with what’s important to your business. For example, it’s not really important if your team spend 21% or 20% of their time on fixing bugs – the real debate is whether should it be 5% or 25%?

Trying it out

To operationalise, I suggest:?

• Host sessions with the team to [1] create draft buckets, then circulate a draft for input within a week, then finalise the buckets, next and [2] have a pass at allocation across the buckets.
• Create a way to tag work to buckets so it can be tracked
• Review quarterly and adjust as needed

Next up —?measuring impact

Now, I’ve introduced ‘buckets’ in Part 2 we’ll get into how best to track the different types of work.?

Wrap

Product is about driving impact with the resources and knowledge you have. It’s always limited. With measurement and ROI back in focus, now is a perfect time to show how the Product function can lead when it comes to delivering crazy good ROI, and buckets are a way of helping facilitate a productive, collaborative and more nuanced discussion across your business. Giddy up!

I’m curious. Have you ever used buckets or equivalent??