Metrics that suck! What do you measure in business?

TL;DR:

• Using paired metrics can help you maintain good metrics that actually help the business grow.
• While building metrics is easy, designing?good?metrics is hard. Small mistakes, can and will result in hurtful and costly outcomes for your business.
• Give your Data/Business analysts the “amazing opportunity” to help you create some metrics that can boost your business and mark your objectives.
• More than tracking, metrics help you align stakeholders around one shared vision. Some call this the "North Star" of the business.
• One framework can help teams to make sure they are properly setting up metrics that leads them to their North Star, not the one in the south (Haha!). Use the input > output > outcome framework.

Simply put, metrics allow you to quantify what's important for you. when you define metrics, you are basically translating a strategy into a set of “quantities”. Well-defined metrics help you make sure you stay on track to achieve the goals of your business. But translation errors can get costly.

If you don’t set your metric properly, not only you will reach undesirable sometimes costly outputs and outcomes, but you will also have a damn hard job keeping the team aligned on the North Star.

You can read this episode on Substack as well. Check it out here and subscribe if you enjoy this.

Common challenges with metric definition

Defining metrics comes with a lot of challenges — but two of them really strike as ones to be extremely careful with whenever you following up on how your business is doing.

Challenge 1: The metrics incentivize the wrong behavior

Remember this line from Atomic Habits by James Clear ?

Imagine you are flying from Los Angeles to New York City. If a pilot leaving from LAX adjusts the heading just 3.5 degrees south, you will land in Washington, D.C., instead of New York. Such a small change is barely noticeable at takeoff — the nose of the airplane moves just a few feet — but when magnified across the entire United States, you end up hundreds of miles apart.

Metrics can create unintended consequences that may not align with the overall goals of the company. It is quite important to carefully consider the potential impact of metrics on behavior, and to ensure that they are designed in a way that incentivizes the right behaviors and outcomes. A company can easily lose sight of its strategy and instead focus strictly on the metrics that are meant to?represent?it.

Before moving forward to my example for incentivizing wrong behavior, and for the same of better understanding my example, let's take a look at this picture illustrating Goodhart's law:

The examples in the picture might be misleading you to think that goal-setting is bad, but the main idea of this law is to caution people against the side effects of BAD goal-setting.

Now let's get on with two examples:

Example 1:

About two years ago I took over as the CEO of a Fintech startup called "PhonePay" that provided an application with QR-payment and some value added services for passengers taking rides in yellow taxi cabs in the city of Tehran, Iran.

The team was using signups as a measure of success, and they had started to see some funny strategies emerging. Their Marketing and Sales team had came up with the idea of giving a free bonus to every new user to simply create an account. They indeed ended up with loads of signups, but not necessarily more customers! Users used the bonus once they signed up in their first ride with the taxi, but churned massively from that point on. In some scenarios, the users were using several phone numbers to get the bonus over and over again. Incentivizing bad behaviors in both the team and the users, right?

What they could've done instead was to use a paired metric. This would act as a guard rail. You keep the signups target but add a retention goal to control the quality of the new leads. Now, your teams will have a simple way to experiment with wild ideas while ensuring that they keep the business healthy. A paired metric for the Marketing and Sales team of PhonePay then would look something like this:

• Increase signups by 60%
• Keep week-4 retention of new users above 50%

Example 2:

I've been working with an Edtech startup called "Classeh" for the last two years. Our customer satisfaction department was burning funds massively, but the number of dissatisfied customers were growing by day!!

Let’s imagine reducing the number of dissatisfied customers who requested refund, is the number one priority for the customer success team. One solution someone on the team came up with, was to make it as hard as possible for the users to complain and by this, making it hard for the customers to request a refund. How? “Creating Agreement Forms” and making the customers print, sign, scan and send these forms, before granting them access to the system they had already paid for!

You see where this is all wrong? Most likely and in the long run, this strategy would increase churns of paying customers drastically in their next round of contract renewal. This also had other unintended consequences (e.g. increasing the number of ‘negative’ social media interactions, bad word of mouth, etc.).

This is not necessarily because people want to game the system — it is more because people don’t necessarily have a total understanding of the organization and everything that goes into it. As soon as someone’s performance is linked to a metric — such as reducing the number of complaints, it is fair game to expect them to try to move the metric, so, it's up to you, as a business leader to set these expectations, and rules of the game as clearly and comprehendible as possible.

Challenge 2: The deceptive metrics

Imagine the EdTech startup we mentioned before. Classeh offered an online learning platform to schools. After conducting an initial analysis, the company identified a strong correlation between the amount of time students spent using the platform and their overall academic performance. Based on this finding, the team decided to set a key objective and key result (OKR) focused on increasing the average time spent by students on the platform.

The various teams within the company initiated campaigns to boost student engagement on the platform, such as gamification elements, rewards for usage, and incentives for completing tasks. As a result, they successfully increased the average time students spent on the platform. However, a follow-up analysis revealed that despite the increase in usage, there was no substantial improvement in student learning outcomes or academic performance.

In this case, the chosen metric of time spent on the platform failed to capture the true measure of educational effectiveness. It overlooked the quality and efficacy of the learning experience and focused solely on the quantity of time spent. This oversight proved costly for the company, as significant resources were invested in driving student engagement without any significant impact on student learning outcomes.

A simple methodology to define metrics

Metrics play a crucial role in measuring success, but designing effective metrics can be challenging. One helpful framework is the "Input > Output > Outcome" framework, which I will explain in the following paragraph. To make the topic more engaging, let's use the example of an education program for the students designed by the schools that work with our EdTech platform. How can this framework be used to define metrics for a successful program?

Inputs are what you control

• Inputs is the “I” in ROI
• Inputs is what you bring to the table: the amount of time you spend on a task, the quantity of materials used to produce something, etc.
• Inputs should be fully in your control

In our example these inputs can include elements like the # of dedicated teachers, their expertise and experience, and the availability of funding by the school for developing these programs.

Outputs are moved by your inputs

• Outputs are the direct results that follow your inputs, like the next step in a funnel.
• Outputs can be moved via your inputs: if inputs increase (or decrease), outputs will change accordingly. They are very actionable.
• Outputs have a quick response rate to your activities. When the inputs increase or decrease, shortly after, the output changes accordingly. They are not fully under your control though.
• There is?a causal link?between your output and your outcome

Defining metrics that sit between inputs and outcomes can be challenging. These metrics are crucial because they bridge the gap between what you control and the ultimate goals you aim to achieve. Balancing actionability and causality becomes crucial in determining where exactly these metrics lie.

These metrics are often the trickiest to define because you want them to be actionable, meaning they can be influenced by your inputs and activities. At the same time, they should also have a causal link to the desired outcomes. Striking this balance is crucial to ensure that the metrics you choose are meaningful and can drive the desired results.

It requires careful consideration and analysis to determine where these "in-between" metrics should be placed. They should be actionable enough to guide your decision-making and allow you to track progress, while also demonstrating a clear cause-and-effect relationship with the ultimate outcomes you seek to achieve.

In our education program example, the grades of the students, the # of skills developed and mastered by the student, the improvement of the grades of students over time, etc. could be our outputs.

Outcomes are the Northstar

Outcomes hold the utmost importance in your business. They represent what you are striving to achieve through all your activities. Outcomes serve as the primary indicator of your business's overall health and success. They embody the fundamental "WHY" that drives both you and your team forward. However, moving outcomes is often more challenging than moving outputs.

Outcomes require the combined effort of multiple outputs working in harmony. Achieving outcomes takes time and patience. While outputs are the metrics you track on a day-to-day basis to monitor progress, outcomes are the ultimate goals you aim to accomplish over a specific period.

In summary, outcomes are the ultimate measures of success, reflecting the core purpose and aspirations of your business. While outputs provide immediate insights, it is the achievement of outcomes that truly signifies the long-term impact and fulfillment of your vision.

In our example, the # of students graduating high school with at least X skills that are useful for finding a suitable job could be the northstar.

Remember, You Make What You Measure!

Your business will optimize what it measures — regardless of the consequences!

Startup investor Paul Graham?argues?that ‘you make what you measure … so pick your measurement carefully’. The ride-hailing company Uber?measures?nearly everything they do by its impact on their Gross Merchandising Value — giving the entire company a single number around which to build all their growth and product initiatives.

If a company solely focuses on optimizing the metrics without considering the broader context and goals, it can lead to unintended consequences and negative outcomes by:

1. Neglecting other important aspects: By exclusively prioritizing the metrics, the company may disregard other critical factors that contribute to overall success. This narrow focus might overlook important elements such as customer satisfaction, long-term sustainability, ethical considerations, or employee well-being.
2. Misaligned priorities: Over-optimization of metrics can lead to a misalignment of priorities. The company might prioritize short-term gains or superficial improvements, neglecting the underlying factors that drive sustainable growth and customer value. This can ultimately undermine the long-term success and reputation of the company.
3. Gaming the system: Overemphasis on metrics can create incentives for employees to manipulate or game the system to achieve the desired numbers. This can result in unethical practices, compromised quality, or a disregard for customer needs and satisfaction.
4. Unintended consequences: Focusing solely on the selected metrics may overlook the potential unintended consequences of actions taken to optimize those metrics. For example, if the metrics prioritize speed of service, it may lead to rushed decision-making or compromised quality.
5. Ignoring innovation and adaptation: Hyper-optimization of metrics can discourage experimentation, innovation, and adaptation. Employees may hesitate to take risks or explore new approaches that may not immediately impact the metrics but could lead to long-term success and growth.

In the EdTech startup, these are actual scenarios that took place before we started making things work again:

• The sales team optimized the number of deals closed to the extent that they sacrificed retention. They closed bad-fit school managers who didn’t get value out of our product, by either lying to them or overstating our offerings. A large percentage of these customers churned a few months later.
• Our software engineering team optimized the number of features delivered in each sprint to the extent that they sacrificed code quality. This resulted in technical debt that hampered the product’s development speed so much, that we had to start redesigning and rewriting the entire code base.
• The customer support team optimized the number of support tickets handled per week at the expense of customer happiness by reducing quality time with school managers.

Why does this happen?

The reason this happens is because teams easily forget the purpose of a metric, and instead organize activities around the optimization of the metric itself. Over time, the original purpose of the measurement gets lost. This is especially true if the metric is tied to performance bonuses.

In his seminal book?"High Output Management", Andy Grove provides some tips on how to overcome the challenges with setting goals and metrics. There are a number of combinations of effect and counter-effect you can consider which works in every business:

• You could pair a short-term benefit against a long-term cost.
• You could pair a quantitative measurement with a qualitative one.
• You could pair a process metric (number of sales calls made) against an outcome-driven one (number of deals closed).
• Once you understand that metrics can be easily warped by institutional behaviour, you’ll be a lot more wary of single-metric measurements.

Good Companies and Great Business Leaders Fight Against This; You Should Too.

Best time to Start a Newsletter especially if you're interested in A.I. and futuristic technologies!

This section, I picked up from?Michael Spencer! He was the sole person who inspired me to start my own newsletter. So, I'm also inviting you to join. If you want to write a Newsletter, I think you should just do it! In 2023, Substack is the best discovery, growth and retention platform to gain new audiences and build valuable niche communities. There are also other platforms such as Beehiive. You can follow its founder?Tyler Denk ???if you'd like.

I’ll be thrilled if you started a Newsletter today, since we’re all learning together right?

Start a Substack Now!

Recommendation:

I promised to recommend and introduce similar newsletters, podcasts, or anything related to what I write and enjoy in every episode. Today, I want to introduce a great person, Terry Rice and his podcast. I've recently got to know and love his podcast called "Launch Your Business".

Check his profile here.

Resource:

(1) https://www.tability.io/odt-articles/on-the-importance-of-pairing-metrics

(2) https://hbr.org/2019/09/dont-let-metrics-undermine-your-business

(3) https://towardsdatascience.com/driving-operational-successes-through-careful-metric-design-ca55e3f84dad

(4) https://www.holistics.io/blog/beware-what-you-measure-the-principle-of-pairing-indicators/