The innovation delusion

If traditional planning is dead, then why do so many organizations still invest in planning techniques optimized for the Industrial Revolution?

One reason might be that we trick ourselves into thinking innovation is the kind of thing we can accomplish with a structured, linear process. When we do this, I think we’re confusing our stories about innovation with the process of innovation itself—and the two are very different.

The process of innovation is chaotic and unpredictable. It doesn't operate according to clean, regimented timelines. It's filled with iterative phases, sudden changes in direction, various starts, and stops, dead ends, (hopefully productive) failures, and unknowable variables. It's messy.

But the stories we tell ourselves about innovation, including the books and articles we read about great inventions and the tales we tell each other about our successes in the workplace, tidy that process up. Think about how many social media posts you’ve seen that feature nothing but the "high points."

That's the nature of good storytelling. It takes a naturally scattered collection of moments and puts them neatly into a beginning, middle, and end. It smoothes out all the rough patches and makes a result seem inevitable from the start, despite whatever moments of uncertainty, panic, even despair we experienced along the way.

We shouldn't confuse messy process with simplified story. When we do, we might mistakenly assume we can approach innovation challenges with the same practices we bring to neat and linear processes. In other words, we apply a set of management techniques appropriate for one set of activities (for more rote, mechanical, and prescriptive tasks) to a set of activities they aren't really suited for (more creative, non-linear work requiring autonomy and experimentation).

An innovation story

Here's one of my favorite examples of how this idea in action.

In the 1970s, the British motorcycle industry was desperately trying to figure out why its U.S. market share was plummeting while Honda's was skyrocketing. The company hired my former employer, the Boston Consulting Group, to help them figure out what was going wrong. BCG gathered some historical data, reviewed a two-decade sequence of events, and developed a neat, linear story explaining Honda's success.

Honda, BCG concluded, had executed an ingenious strategy: enter the U.S market with smaller motorcycles it could sell at lower cost, use the economies of scale it had developed in the Japanese market to set low prices and grow a market, then further leverage those economies of scale to grow their share in the States as demand grew. By all accounts, Honda had done it brilliantly, playing to its strengths while thoroughly and accurately assessing the new, target U.S. consumer. It had outsmarted, outflanked, and outperformed competitors with a well-executed plan.

It sounded great. But the reality was much less straightforward.

Yes, Honda did want to enter the U.S. motorcycle market. It initially attempted to copy its competitors there, building the larger bikes Americans seemed to favor. But bikes like that weren't one of Honda's strengths, and their versions had reliability issues. To make matters worse, their models didn't look much different than other offerings already in the market, so they weren't standing out. Suffice it to say, sales were not booming.

But in a happy coincidence, Honda's Japanese representatives visiting the States had brought their own motorcycles with them. Those bikes were different from the ones the company was attempting to sell to the American market. They were smaller, zippier, less bulky, more efficient, and generally less expensive. Sears took notice, contacted the reps, and the companies struck a deal that let Sears carry this new motorcycle—called the "Super Cub"—in its American stores.

And the rest, as they say, is history. The Super Cub would go on to become the best-selling motorized vehicle of all time, and Honda continues to produce it today.

In hindsight, the events that brought the Super Cub to the U.S. seem logical, almost boring. But Honda owed its success less to an ingenious master plan and much more to serendipity and happenstance than most people care to admit.

Open (and messy) innovation

Organizations (and especially leaders) like to think that success is always planned—that they've become masters of chaos and can almost predict the future. But they're often making those assessments with the benefit of hindsight, telling the stories of their haphazard journey in a way that organizes the chaos, essentially reflecting on a period of uncertainty and saying "we meant to do that."

But as I said, we shouldn't assume those stories are mirror reflections of the innovation process itself and build future initiatives or experiments on that mistaken assumption.

Imagine another motorcycle manufacturer looking to replicate Honda's success with the Super Cub by following BCG's narrative to the letter. Because the story of Honda's success seems so logical and linear, the new company might assume it could use similar processes and get the same results: plan objectives, prescribe behaviors, and execute against knowable outcomes. But we know that Honda didn't really win its market with that kind of "plan, prescribe, execute" mentality. It won through flexibility and a bit of blind luck—something more like "try, learn, modify."

When we're able to appreciate and accept that the innovation process is messy, we allow ourselves to think differently about approaching innovation in our organizations. We can begin building the kinds of open and agile organizations capable of responding to innovation as it happens instead of over-investing resources into pre-formed plans that try to force innovation into a linear timeline.

I saw this kind of approach several years ago when Red Hat released a new version of a product that included a major technology update. Version 5.4 of Red Hat Enterprise Linux was the first to include full support for a technology called the Kernel-based Virtual Machine (or "KVM"). For us, it was a significant innovation that promised to deliver immense value not only to customers and partners but also to open source software communities.

The technology was evolving quickly. Luckily, because we're an open organization, we were adaptable enough to respond to that innovation as it was happening and help our customers and partners take advantage of it. It was too important, and the competitive landscape too volatile, to justify withholding just so we could "save" it for a milestone moment like version 6.0.

When you go back and review the archived release notes for Red Hat Enterprise Linux, you'll see that it doesn't "read" like a typical software innovation tale. A game-changing development pops up at an unpredicted and unremarkable moment (version 5.4), rather than a pre-planned blockbuster milestone (version 6.0). In hindsight, we now know that KVM was the kind of "big bang" advancement that could have warranted a milestone release name like "6.0." But that's just not how the innovation process unfolded.

Don't get me wrong, organizations still need to maintain operational excellence and perform execution-oriented tasks well. But different kinds of challenges require different kinds of approaches, and we need to get better at building flexible organizations just as capable of responding to the unforeseen or unknowable.

An organization great at planning (and executing against that plan) will quite likely get the results it planned for. But when success depends on things we don't or can't predict, is getting exactly what you've planned for good enough?