The Limitations Of Data And Benchmarks

Numbers provide us a certain certainty. With their precision, they offer a sense of black and white, in or out. But, metrics alone aren’t enough. All the quantitative analysis in the world won’t lead me to the next great idea for startup. Those figures can’t create empathy, develop the right culture, or hire the right people. I’ve been thinking about this quite a bit because in both the recent Software Engineering Daily podcast I did with Jeff, and the presentation I gave at Launch Conference, the question of the limits of metrics surfaced.

In those conversations, we discussed two shortcomings of data. On Software Engineering Daily, Jeff asked whether metrics can lead us into a local maximum or minimum. And the answer is yes. Data is not a way to create new ideas. Pixar never ran linear regressions to create Woody the Cowboy. Rather, data is a way to optimize a funnel, whittle down a series of options, evaluate experiments. It is a filtering tool, not an ideation tool. Startup idea generation has always been closer to poetry (with a healthy addition of user research) than accounting.

At the Launch Conference, an audience member asked whether a single metric, even a proxy metric, is enough to determine the viability of an idea. The answer is no. Most metrics we evaluate are rear-view mirror metrics. And each metric only describes a facet of the business. To describe a publicly traded company, you might use five or six: market cap, revenue multiple, gross margin, cash flow, revenue growth rate, profitability. Even then, those figures provide only the foggiest outline of a company.

Like historians, investors use numbers to compare and contrast, to categorize and critique. We identify unusual companies, those with best in class sales efficiency or revenue growth. Management teams employ metrics to identify when a particular part of a company is performing in an unexpected way. Sagging quota attainment suggests sales recruiting and t practices are worth investigating. Often, data is a filter. 

We have shown in analysis on this blog how revenue growth is not correlated with series A pre-money valuation. And, at least one third of premium SaaS companies raise capital before generating a dollar of revenue. That means that the early stages, while we can look at metrics to evaluate companies, these numbers don’t tell the majority of the story.

It might be the case that as a company grows and matures and mechanizes its business model and its go to market strategy, that numbers capture more and more of the business. But even then, data is just one way to describe a business. 

I hope the metrics I publish inspire. They show what can be done, but not how to do it. They show that there are many different ways of building a company, whether it is the astronomical growth rate of Slack and Salesforce or the brick by brick execution of Atlassian or Concur. But they will never capture the entirety of the story. And for every one path trod by a business, there is another path less taken that a founding team will take to redefine all the rules and observations.

We can measure elephant’s height, the length of its tusks, its weight, how fast it runs, even sequence its genome. But like the six blind men who disagree on which animal stands before them, no one perspective, even a data driven one, is not sufficient to fully describe it.