A better way to make decisions

I was in one of those meetings.

Five managers and a director surrounding a table with a spreadsheet projected on the screen, lukewarm coffee and a budding headache. We had done our homework: we’d gotten the pie in the sky ideas from our thought leaders, the pain points from our account executives, a laundry list of problems we ought to fix from the developers, all the reasons why our current models were inappropriate from the data scientists, and the latest rebrand from the product designers. Everything on the projector screen totals roughly three years worth of work. Now we had to pick what we would execute on for the next several months.

If you’ve been in a similar situation with a typical software development team, you probably know what came next. Half of the work was marked “high priority” and people felt the need to escalate some projects to “super-high priority”.

If you’re lucky, everyone in the room mostly agrees on the rough priorities, but it’s equally likely that tempers start to rise as someone can’t articulate why their team’s prized project is left on the cutting-room floor because it was merely a high priority. Other times, it’s not getting everyone aligned that’s difficult, it’s getting everyone to evolve in the same direction. What are these projects trying to achieve? What metrics should guide the decision beyond “gut feeling”?

In that specific meeting, the issue was lack of direction. We were all somewhat apathetic about the projects on display. We could agree that a few had to be prioritized to ensure the team’s survival, but the rest we thought was mostly down to preference as we doubted we could accurately measure the projects’ impact or compare them in any meaningful way.

As I was sitting there, I remember considering for the first time whether striving to be sitting around that very table in those meetings had been a mistake. We all had math-heavy backgrounds, why were we not using data to make those decisions? This was barely better than a decision made by seniority!

There had to be a better way.

Although I had a feeling we could do better, I couldn’t see past the key issues that emerge from trying to assess the impact of very immature projects, let alone the cost of some of those pie-in-the-sky ideas. The good news is that there is a superior way. I don’t know why it was so hard to find, why it isn’t more common practice for managers, why it’s been so difficult to find in my research.

Let’s rectify that.

What’s in a decision?

In my quest to discover how to be a better decision-maker, I had to start wrapping my head around what this meant. Research indicates that the average adult makes roughly 35,000 decisions daily. I will focus on a vastly reduced subset of worthwhile decisions.

? Worthwhile Decisions are defined by

• Impact: The outcome of the decision has enough impact for you to at least consider your choice for ten minutes.
• Non-dominant alternatives: There isn’t an obvious superior alternative.
• Non-trivial: You can’t pick randomly from the set of alternatives under consideration (after removing the obviously bad ones) because they aren’t equivalent in their outcome.

Even as a leader, you probably only go through a handful of those daily. By definition, those worthwhile decisions matter and likely deserve more attention than you typically give them to produce a formal decision.

?Formal Decisions

Given the information currently available (and the opportunity to gather more data if effective), pick the best alternative according to a well-defined decision under uncertainty strategy.

There is quite a bit to unpack there.

When you need to make a choice, there is an underlying question: should you gather more data before committing yourself? While experience is useful here, there is actually a scientific approach that answers this question rigorously called Expected Value of (new) Information.

Once in possession of all the information that is economical to gather, you then use a chosen strategy for making decisions under uncertainty (there exist many that we will explore depending on your context).

Note that even in hindsight, we need to evaluate our decision-making process based on the information that was available while making the call. If one really needs to include realized outcomes in the evaluation of the process, care must be taken to evaluate decisions by looking at aggregate trends rather than specific choices.

If done rigorously, this process is deterministic from the decision-maker’s point of view.

Importantly, a decision process can utilize, but not solely rely on, one’s own experience. Your gut feeling is a great tool (and you should develop it!), but when you are in uncharted territory, relying on your guts, your expertise, or your experience alone leads to worse outcomes.

When experience is used as the basis for making a judgment call, it depends on past experiences to map the current situation to what happened then and apply a similar solution to the problem at hand. That’s one reason why we count on senior leaders to make judgment calls hoping they can pull from a storied career to find a close fit from their experiences. This works great for recurring or familiar decisions, but is fundamentally flawed when used as the main tool for making decisions.

In other words, heuristics are a poor substitute to analyses when dealing with the unusual. In future articles, I’ll dig deeper into each step of the decision-making process.

Engineering principles are insufficient

I wondered for the longest time about why this skill set wasn’t already built into our education. I could easily set up a reinforcement learning area to teach AI about making decisions on partially observable stochastic processes, but never had I seen the same concepts or the same tools applied to my work as a manager.

From an admittedly simplistic viewpoint, engineers rely on tools that are more like recipes (e.g. if you need constant-time random access to elements, use a vector). Those recipes are based on principles that have been demonstrated formally or experimentally and have stood the test of time. Examples include the building code, the power grid standardization, or the continuous delivery framework. These codified work methods have been produced after an extensive exploration of the problem they aim to solve and provides baseline solutions for engineers to start from.

Leaders instead have to wrestle with the innovative ways things can break down and don’t have two years for an extensive study of the problem to find a solution. This novelty breaks the typical pattern of problem solving used in engineering, so we fall back on our experience and make use of our best judgment.

Crucially, it’s rarely when we need to make an important decision that we can afford to rethink our decision-making process.

In software development (and likely in most business contexts), the seniority of someone is positively correlated with the impact of the decisions that are demanded of them. In a way, this makes sense: the more senior someone is, the more professionally mature they likely are, the more they’ve seen, the more they can map the current situation to experiences to get a sense of how to address their current challenges.

The problem is that people who grow into a leadership position (This includes both managers and high-level ICs who have a lot of influence on a team’s direction.) are rarely trained to make effective decisions. This is doubly damning when leaders have all the mathematical background that would make learning how to make data-driven decisions an easy task, yet aren’t incentivized to, like me. If people learn at all about developing their ability to decide, it is through situational coaching from more senior leaders.

The problem with this approach is two-fold:

• It depends on you (or someone you trust) having experienced a similar situation in the past;
• It’s hard to generalize to a broader context.

If experience-based decision heuristics are inappropriate for the situation, then what is the better tool?

The tools already exist

Finance and insurance are two fields that routinely contend with making decisions under uncertainty. In both cases, the analysts in those fields (e.g., the quants or actuaries) have to make investments or underwrite policies in a way that won’t bankrupt their employer. To that end, they take a formal approach to estimate the risks associated with their decisions and use this process to guide their decision on resource allocation, pricing, etc. This is time consuming, so they consequently developed tools to be more efficient in reaching a conclusion.

The above is reassuring as it indicates that at least some people cared to develop a science-based approach to making decisions. I spent a lot of time trying to google “better ways to make decisions” or “decision science” and most of what came up, while interesting, was very far from a coherent exploration of the problems a leader has to contend with when dealing with uncertainty. I already knew about actuaries and quants, but didn’t make the link between their work and my job as a manager. I had to do a dreadful amount of digging into the issue before making the connection.

Obviously, I am not advocating that every manager gets a degree in actuarial sciences as they leave IC work behind. However, through this series of articles, I want to bring to leaders a set of tools based on what is used by quants and actuaries that can be used to build a formal process of decision making you can depend on in various situations.

In the literature, this is referred to as decision analysis, a discipline pioneered by Howard Raiffa in the 60s. It mixes mathematics, economics and psychology to formalize the decision-making process as it is encountered in the real world (as opposed to the simple games of game theory or the perfect information of mathematical optimization).

With a broader toolset, you’ll be able to either automate or be more deliberate in how you make decisions. You’ll adapt the tools you wield depending on your decision budget (what resources you can expend to make a better decision) and the time you have available to make a decision. As a leader, you need to be equipped to make rational judgments on when to say “yes”, when to say “no” and how to handle situations where new information affects your plans.

A key point worth reiterating is that building your arsenal is much harder to do reactively when you need to make a decision rather than preemptively when you realize that this is a set of tools that can be valuable to you, your team, your organization, your family, or your community. Equipping yourself with even basic decision tools will put you ahead of your peers that have never had to question how decisions are made unless your organization has a healthy culture of making data-driven decisions.

I eventually built a project prioritization tool for our teams. It used some insight I found reading “How to Measure Anything” by Douglas W. Hubbard to easily, but rigorously, estimate the return on investment of each potential project. It doesn’t give perfect forecasts, but is enough to make a rough ordering of projects to execute on without relying (solely) on the gut feeling of people around the table. It not only reduces planning timelines, it also offers a more robust justification for senior leadership as to why we decided to focus on those specific projects.

This, I found invaluable. I hope you will too.

Subscribe to the newsletter to stay up-to-date with the process and tools I will share over the coming weeks.

I'd also like to start collecting stories about important decisions (big and small!) you have made. If you want to share, either leave a comment or send me a DM!