5 Critiques of Lean Startup and Why They Don't Hold Up

After reading an article published on Harvard Business Review by Ted Ladd called "The Limits of the Lean Startup Method", I felt inspired to write this post to hopefully clarify some points that I think were overly simplified and to some extent misguided.

Don't get me wrong. I think that providing constructive criticism on any topic is great. Nothing should be taken for granted. Only through questioning, debating and constructive critique that leads to new insights can we improve. I hope that this post, and any comments that may come as a result, provides one small step in that direction.

But before we get started, for those who need to refresh their understanding of lean startup, here is a very brief introduction What is Lean Startup?.

1. "Having a strong strategy is more important than conducting lots of market tests."

This naturally begs the question, what is meant by a "strong strategy". Without defining generic statements, we have no clear foundation from which to even begin to tackle a problem. What is meant by "strong" and how can we know when it's "strong"?

Vision or purpose should guide direction, while strategy should adapt to learnings and changes in the environment. This means that we must be willing to question our strategy, which is in-line with a data-driven lean startup mind set. Our purpose, which should resonate with our personal beliefs and values, is the last thing that should change.

Although lean startup is much deeper than running experiments to in/validate assumptions, lean startup is able to inform not only tactics but strategy as well. So, it's not that "strategy" is more important than "lean startup", they should go hand-in-hand.

1. "We don't believe more validation is better. We tested it and it doesn't work for us."

This, and variations of the same, are classic examples of denouncing a method because it doesn't deliver the expected results, instead of questioning if you're doing a poor job of applying the method.

In lean startup, there are pitfalls to look out for, e.g. "vanity metrics" and "false validation". Letting yourself get off course by following non-actionable metrics, or thinking that you got validation when you didn't, are examples of poor application of the method.

More specifically, asking closed-ended questions such as "Do you like our product" with the customer responding "yes", doesn't equal validation. The customer may just want to get you out of his office and will say anything that achieves this goal.

The entrepreneur takes the "yes" as great news and extrapolates this even further when debriefing the team. Getting validation for things that the customer doesn't care about, or confusing what someone says with what they will actually do, is not the proper application of lean startup.

In lean startup we don't just go about testing everything that moves. We need to prioritize what we test for, be wary of untold bias, find smart ways to design our experiments to elicit the most valuable learnings with the least amount of resources -  and we need to decide what gets measured.

It's a tall order, but dismissing a method out of hand that generates tangible results for countless others, may not be the best approach.

1. "Doing customer interviews or customer development doesn't work for us".

Perhaps you haven't dug deep enough to understand the problems your customers are facing and the contextualized results or outcomes that they are trying to achieve. Also, you probably haven't rated the outcomes according to importance.

If you do a couple of shallow interviews and then go straight to building your MVP and running quantitative tests, I would not be surprised if customer development doesn't deliver the results that you had hoped for.

Another common mistake is to bundle problem and solution discovery interviews into one and the same interview, giving rise to a number of other problems.

1. "Too much feedback from customers might cause the entrepreneurs to change the idea so frequently that they become disheartened."

That's why the team has to prioritize what to take in and what to discard. Teams that jump around too much based on customer feedback, need help from experienced mentors to define their purpose and vision.

They also need to learn what metrics to focus on, which in turn is dependent on the business model and the stage that the startup is in. Alistair Croll and Benjamin Yoskovitz do a great job of explaining how this works in their book Lean Analytics.

Lean startup is not about "pivoting" all the time. Far from it. If you're pivoting all the time, you're not doing lean startup, you're doing the dance of the headless chicken.

Lean startup is about being data-driven, or data-informed, rather than letting opinions decide your next move. I agree with Tom Chi, former head of Experience at Google X when he says that way too many meetings are pure "guess-athons". Innovation comes from doing and iterating, not guessing.

It's about testing your most risky assumptions early on, and in essence focusing on finding something worth building, before investing resources into actually building and scaling. Or as Alberto Savoia, Innovation Agitator at Google likes to say "find the right 'it' to build, before you build 'it' right".

1. "The lean startup method might be producing “false negatives,” meaning good ideas are mistakenly rejected because the approach does not have a clear rule for when entrepreneurs and intrapreneurs should declare victory, stop testing, and                 begin scaling production."

There are no "off-the-shelf" clear rules to follow. These are created and decided on by the team. However, there are plenty of guidelines, tools and inspiration that other lean startup practitioners share. Just look to Sean Ellis or Ash Maurya as great examples of this.

Also, how do you know if it's a "good idea" unless you test it? Furthermore, entrepreneurs should never stop testing, even when scaling. You just enter a different phase of your company, where your metrics will change and the stuff that you test will as well.

If you're pivoting all the time you're simply not going about lean startup the right way. In today's fast moving world, the classic 5-year strategy is dead!

The essence of strategy is deciding what not to do. The great thing about lean startup is that if you begin to apply it in a disciplined fashion, and according to best practices, it will help you to figure out what your strategy should be.

Conclusion:

Getting great at lean startup takes practice. Expecting the results to automatically fall like manna from the heavens above after reading Eric Ries book "The Lean Startup", is not realistic.

I would encourage you to not only experiment when it comes to improving your products and business models, but also when it comes to the methods and tools that you apply when selecting, designing, running and measuring experiments in the first place.

Discarding a method such as lean startup because it didn't work out for you, instead of trying to identify where you went wrong and trying again, is akin to throwing out the baby with the bathwater.