Algorithms to Lead By – Decentralized Decision Making, Exploration, and Exploitation

“Leader” is a term that has undergone a paradigm shift. Historically, leadership has been associated with control, authority, and decision makers. But are any of those characteristics needed to be a leader? Are they even helpful? Leadership is about much more than control – it is about using situational behavior to exert influence in a manner that achieve beneficial outcomes. It is about getting the most out of the resources you have available to you. In a world that continues to value data more and more by the day, every manager has a powerful but underutilized tool at their disposal – algorithms.

Each team member is a unique algorithm capable of processing data in different ways to reach an outcome. Yet in a world where companies spend billions of dollars developing unique algorithms to handle data in hopes of finding a method or approach that works best; traditional “leaders” direct their teams to use a single method or algorithm to achieve a goal. We often find a single architect behind a solution because we are afraid of getting just a little bit messy. This ties into what is called the explore vs. exploit trade-off. Unfortunately, we often exploit mediocre methods because we fear just a little bit of exploration.

Explore vs. Exploit

Almost every decision in our lives comes down to the explore vs. exploit algorithm. Should we eat at a place we know we like? Or try a new restaurant? You might never discover your new favorite dish if you rely on exploiting your regular spot. Conversely, you might have wasted an opportunity for a perfectly good meal if your exploration goes bust. We even face this decision in trying to find a spouse! Move on from one date and you might find someone that was a better fit – or you might have just said goodbye to “the one”. The explore/exploit tradeoff is one that we are all familiar with.

When we are younger it is beneficial for us to try new things – after all our sample size of what is good and what is bad is still being formed. It is only much later that we can exploit those findings and consistently recognize quality. It’s why we don’t swallow quarters anymore or try eating anything we find on the floor (hopefully… seriously watch your babies). When it comes to team management, however, we seem to forget that we survived those quarters and extracted valuable lessons from them. Far too often management assigns roles and dictates how work should be done. Centralized decision making is the law of the land… and it is costing you!

Traditional Management is ok… Sometimes

Before coming off as biased – we should stress that there is a place for and benefits to centralized decision making. When the challenge you are facing is an easy and static target with a proven solution you should exploit that solution. If your project is on a short timeline that has no room for exploration, and a known method will get it done on time, you should implement the “guaranteed” structure. Centralized decision making in these cases is a risk reducer. This, however, only describes a small number of the projects we work on. Most projects we do are worth doing because they are complex. We are seeking an advantage, trying to something that has never been done before, or do it differently. So why do we still see an overreliance on structure and exploitation mode? There are many potential reasons, but 2 are anecdotally the most common:

1)     Loss Aversion – The first reason is simple and is the same reason most of us don’t explore as much as we should – “loss aversion”. We are wired to fear loss more than to be excited by success. This has been proven by various behavioral economic studies and is used to showcase that we act irrationally.

2)     Overconfidence – We should only enter exploit mode when we’ve built up good data on what works and what doesn’t. We think we’re experienced and know what will work because of our past projects. However, as we approach projects that are challenging and solving problems we once thought to be intractable – we are trying to fit data from irrelevant experiences to our new projects. Simply put, our experience curve for that particular project would be better served for exploration.

Decentralized Decision Making and Emergent Structure

Now that we’ve discussed the consequences of centralized decision making, we are still left with a conundrum. How do you implement decentralized decision making on a project, and how do you make it work? There are many variables at play but for a decentralized decision making to work you need the following: 

1)     Safety – failed attempts are viewed as learning experiences

2)     Trust – you need team members you trust to get the right thing done

3)     Clarity – everyone needs to know what they’re working towards. Tell the team “what” not “how”

4)     Self-forming teams – titles and roles are different things. Let team-members find and place themselves in the role that best benefits the team.

               It is important to note that this approach is not shunning structure. The idea is that you are letting the right structure emerge – as opposed to dictating an incompatible one. Placing team members that you trust to get the job done on a project without clear roles can seem like anarchy – but in the end people gravitate towards their competencies. Additionally, leadership is demonstrated – not assigned. Letting team members demonstrate knowledge and earn respect of their peers will empower them more than a title or assigned role. People naturally want to succeed – it is self-serving to think that a team can only do so if we tell them who will do what and how they are to do it.

               Letting teams explore early pays dividends. Every team and unique project should be thought of as challenges with small data sets. You need to let team members explore different approaches (and build your data set) if you want any shot at finding a differentiator. Let approaches compete, those that don’t work will eventually die, while the algorithms (approaches) with legs will have team members coalesce behind them. It is critical to note that during the exploration phase communication between team members needs to be unusually high. As team members prod at different approaches work must be done to both build compatibility and gather feedback.

You Found Something… Now What?

The entire point of your team’s exploration phase is to find an approach that works. Once you do have something that works (as in really works) it is time to transition to exploit mode. Exploration is great, but eventually we won’t find something better. So, when should a team make a leap of faith? This dives into another set of algorithms called “optimal stopping”. The short answer: grow and appetite for early exploration on each new project (the math says exploration should be high for the first 35% of the project’s time duration), and then exploit what works.

Old solutions aren’t optimized for new problems. Don’t be afraid to explore something that is.

Originally authored for Tactical Agile