Enterprise Vanity Metrics - Activities Are Not Outcomes

This article was authored by Craig Strong and Jana Werner

Are you measuring activities or outcomes?

Do you measure things like number of features added, support tickets closed, people trained, presentations conducted? The numbers might be impressive and create a feel-good dopamine hit from all the achievement, but often we find that these measures don’t correlate with actual customer outcomes and business value, like customer satisfaction, retention, and ... growth! Activities are not outcomes, and measuring activities is only helpful when it is clear how they link to outcomes through measurable cause-and-effect relationships. Worse, many customers we work with focus too heavily on what we call “vanity metrics” - metrics that make things look good on the surface, but don't actually indicate advancing their business.

As product, technology and business leaders, be wary of vanity metrics. Do more new features shipped really mean attracting more customers, or is this metric creating a negative customer experience from bloated products? These might end up confusing and frustrating your customers, wasting engineering time building things no one asked for or wants, inflating costs, and negatively impact customer experience. Ultimately the latter reduces innovation and agility in the long run and impact your ROI potential.

Similarly, many companies focus on activity metrics such as the number of customer support tickets closed as a metric for support team performance. But we find that this incentivizes the wrong behaviours - support reps will try to close tickets as quickly as possible, and issues don't actually get resolved, leaving the customer confused and disorientated. Customers get frustrated having to reopen tickets or escalate issues. We prefer to weight customer satisfaction scores and time to resolution instead. Response times are measured, but not in isolation or weighted equal to resolution. This elevates support teams to focuses on solving issues completely and delighting customers. Weighting measures on outcomes not activities, not only improves customer retention, but also encourages the team to innovate and share knowledge from a resolution standpoint, much of which feeds product and service teams for product performance.

How can you focus on metrics that actually help understand if you’re delivering value, instead of measuring busy work?

Although business model dependent, here are some examples of product and enterprise metrics that can provide useful progress measures:

1. Customer usage/adoption – This measures how much and how often customers are using the product. This indicates the product is solving a real need and provides value.
2. Time to market - Faster time to market allows a product to capture market share before competitors. Speed is critical.
3. Customer referrals - When customers recommend a product to others, it signals high satisfaction and loyalty. Referrals can fuel growth.
4. Error/defect rates - Lower error and defect rates demonstrate a high quality product that delivers on its promises. This can positively impact customer loyalty, trust and retention.
5. Net customer/gain and loss - Net change in total customers. Positive net customer gain shows product market fit and growth.

You may be wondering why Customer Satisfaction Scores (CSAT) and Net Promotor Scores (NPS) aren’t listed above. These metrics can provide useful directional feedback on customer satisfaction when applied correctly, but have limitations. CSAT surveys often use skewed response scales that compress dissatisfaction ratings, obscuring issues. NPS detractor feedback is not actionable. More granular, diagnostic metrics are needed to understand specific pain points and opportunities. So, although these numbers can be directionally useful, it’s recommended to dive deeper into their application and context to be applied to outcomes and learnings.

Examples of vanity metrics to avoid

It's all very well looking at what to measure, but in our experience it’s helpful to review your current metrics and challenge them to see if they are actually helping you learn and make an impact as opposed to track whether your simply being busy. Being busy without being productive is simply being wasteful and can be an easy trap to fall in as it’s often much easier to manage these metrics. To define useful metrics, you need to understand your business model, customers and growth levers. However some metrics are universally wasteful activity metrics and here are some examples of some common measures we would encourage you avoid or review:

1. Number of app downloads (doesn't measure engagement or retention)

2. Number of registered users (doesn't measure active usage)

3. Page views (doesn't measure meaningful actions/conversion)

4. Social media followers/likes not actual customers and engagement is low)

5. Bounce rate (doesn't provide insight into why users are bouncing)

6. Traffic sources (not actionable unless tied to outcomes)

7. Time on site/app (doesn't indicate value delivered)

8. Gross Merchandise Volume (inflated by returns/cancellations)

9. Number of workshops delivered (activity metric, not outcomes)

10. Number of presentations done (activity metric, not outcomes)

And if you’re a development team, watch out for Number of Features delivered and Story points as a metric. Number of features delivered is considered a vanity metric because it does not actually measure the value or impact those features have on users. Simply delivering a high quantity of features does not mean those features are useful or improving the product experience. More important metrics are around how much users engage with and benefit from new features. While velocity can be useful for development teams to gauge capacity, it does not actually measure if features are successful and valued by users. Impact is better measured through direct usage metrics and customer feedback.

Understand your customer and connect metrics to your business model

The key is tying activities and their metrics back to whether they deliver outcomes that matter most to customers. We want to incentivize the right behaviours, not just busy work. For example, measuring number of customer calls could be a vanity metric. But if we connect it to outcomes like first call resolution rate and CSAT, it becomes more meaningful. If you can then connect this to customer Life Time Value, this extends into your ROI. We want to optimize for fewer calls, but when customers do call we want to delight them. The point a customer calls for support is a signal that something has failed. Focusing on resolution steers support and product teams towards quality over quantity. Likewise, feature usage metrics are only useful if we connect them to things like faster onboarding, increased retention, higher NPS, etc.

And there is a context that needs to be applied. For instance, if you have budgeting software, you might expect feature usage in budget management to increase around the budgeting cycles. The time a customer spends on task for some business models is positive for instance advertising or content consumption, but in other domains this could be a negative indicator; consider the time it takes to complete your tax return. Understanding your customers’ behaviour and what your product and service intends to do so is a good place to start, when thinking about measuring outcomes.

Metrics should be part of an overall system focused on iterating, learning quickly, and delivering better outcomes over time. They help us learn which drivers really impact customer satisfaction and growth. With the right metrics we can course correct quickly when things aren't working. The path to great products and happy customers doesn't come from vanity metrics and blind activities. It comes from focusing on outcomes, connecting the dots between activities and experiences, and iterating based on what we learn. When product and operational metrics are not providing useful insights, you should re-evaluate and iteratively improve them by aligning metrics to key goals, incorporating qualitative feedback from analysing patterns against expected and intended behaviours and goals, and continuously reviewing these metrics as your organisation and data capabilities evolve. All metrics and measurements should be able to defend themselves in terms of the value they provide to learning and management, connecting to goals and objectives.

Summary

Activities are not outcomes, and measuring activities is only helpful when it is clear how they link to outcomes through measurable cause-and-effect

relationships. Worse, many organizations we work with focus too heavily on what we call “vanity metrics” - metrics that make things look good on the surface, but don't actually indicate advancing their business.

We propose companies should align their metrics to outcomes rather than activities. Connect metrics to real customer value and business model goals and invest time and capabilities into understanding what success looks like and identify metrics that indicate progress towards those goals. This requires a capability to continuously evaluate if current metrics provide actionable insights or just keep teams busy. Focusing on metrics that help teams learn, course correct quickly, and ultimately deliver better products that solve customer needs can transform your business and improve customer retention and ROI. With the right outcome-focused metrics, companies can build great products that grow their business.