What’s the Future of Market Research?

“To say that the research industry is in transition is an understatement. Industry leaders, both on the client and the supplier side, agree that a combination of forces is changing the sector at a speed not seen before,” according to ESOMAR’s recently published report Global Market Research 2018.

One of the important trends is that traditional research’s share of the insights industry is declining. According to ESOMAR “The results show how, for 2017, the broader insights and analytics category outpaced the growth of the traditionally defined market research sector, thus increasing its share within the industry – a trend that has been witnessed for the last few years. As of now, the traditionally defined sector accounted for just over 54% of the total Industry turnover in 2017, down from 61% in 2014.”

A time of transition

The world of market research is at a tipping point. It can’t continue as it is, and it is about to change dramatically. The critical question is will it evolve, or become extinct? Both are definite possibilities.

The old ways of conducting market research are increasingly ineffective, and a wide array of new information options is available.

For almost a century, survey research had a lock on insights. Want to know what people did? Do a survey. Wish to understand what people are thinking? Do a focus group to figure out what is going on, and then quantify it. There were few other reliable sources of specific information on people’s choices. But the world has changed, drastically.

The speed at which decisions need to be made has also increased dramatically. “We’ll have the results to you next quarter” just doesn’t cut it anymore. Information must be managed dynamically. As Finn Raben, Director General of ESOMAR, told me: “We can no longer afford to be the grey mice that live in a cupboard and say, ‘Ah, but you know, it took six months, and it is perfect research.’ Too late, pal. The company’s moved on.”

The world is awash with data on exactly what people do, when they do it, and whom they do it with. This flood of information is reshaping how we learn, what we can know, and the insights we can generate. The challenge now is piecing together the puzzle.

Nobody really knows exactly how much data there is, and how much data is being generated, because the volume is growing so fast. It’s commonly suggested that about 90% of all the data in the world today has been created in the past few years. But let’s just agree on this: we are drowning in data and desperately need people to transform the data into insights that shape decision making.

This need to tame and focus data is why market research professionals have an incredible opportunity. They are well positioned to harness this surge of information. But are they ready? Are they willing to make the changes necessary to move away from simply “doing surveys” to generating insights? I know that if we don’t evolve we will be left behind.

Is the end near?

In writing a book entitled The Insights Revolution: Questioning Everything, I had the privilege of talking to dozens of researchers, strategists and marketers around the globe. There was overwhelming consensus that change is coming, and quickly. “I think it’s going to be fascinating to see where things go, but I don’t think it’s going to continue the way it’s going for another 10 years,” said Shawn Henry of Camorra Research in New Zealand. Why? Because the traditional market research approach has lost its lock on knowledge.

“In the beginning we had almost a monopoly on insights,” says Antony R. Barton, Director of Product Innovation and Marketing Insights at Intel, “and that put us in a very special position, because nobody else had access to data. We had to do this survey, or we had to do this discrete choice, or this set of focus groups, because nobody else could get the data. And then, all of a sudden, there is this explosion of data. There are lots of ways to get at insights. I think the industry has struggled with that. Who needs a survey that takes nine weeks to complete? Or who needs focus groups where I will get back to you in six weeks?”

Elizabeth Moore, Director of Research, Insights and Analytics at Telstra in Australia, is one who has seen a big change. “I think the market research industry is really at a pivotal point,” she says. “Traditional models are being disrupted, and the way we buy insights is changing significantly. There’s a whole range of questions that, in the past, we would have gone to a market research house to help us answer, but we’re now actually using big data and analyzing our data.”

While there is widespread agreement that the future will have multiple streams of data, most see an ongoing role for survey-based research. The question is, what kind of change is coming?

While there is an infinite number of possible futures for the world of market research, there are three scenarios that I think are instructive. The first is the example of dinosaurs: mass extinction, but with a thriving set of descendants that do not resemble the mighty T. Rex. Another is shoemakers: once the sole source of footwear, they have been displaced by mass production driven by a relatively small number of designers and engineers. The third analog is video: once produced by a select few, it is now available to everyone, and is consumed more than ever.

Dinosaurs today

Dinosaurs are one of our society’s enduring fascinations—just ask any four-year-old. One reason they capture our imagination is that they are a powerful reminder that the world changes, sometimes in the blink of an eye. Dinosaurs grew to the size of jetliners and ruled the earth for more than 100 million years. Then everything changed.

It’s generally assumed dinosaurs became extinct. But the chickadee chirping outside your window and the seagull trying to steal your potato chips at the beach are reminders that, while T. Rex no longer rules the earth, the descendants of pterodactyls are everywhere.

Pterodactyls had the wingspan of an F-16 jet fighter. They were powerful, and terrifying. But they are long gone. So why did the dinosaurs that evolved into birds survive? Research suggests it was because they were smaller and nimbler.

What can we learn from dinosaurs? The nimble survive. And when behemoths fall, they fall hard.

This scenario suggests the demise of the market research industry as we know it. Large suppliers: gone. Big market research departments: so transformed as to be unrecognizable. But the insights function soars on, just in a radically different form.

The cobbler’s children

For most of human history, shoes were made by hand. Skilled workers practiced their craft to create individually crafted shoes—all very similar, but each one slightly different. The process was repetitive, but custom. Just like surveys today.

Then came the Napoleonic Wars, and the Industrial Revolution. The need for huge numbers of boots for the soldiers of the British Army was the impetus for an engineer named Marc Brunel to devise a system for automating shoemaking. Sir Richard Phillips visited Brunel’s factory and was greatly impressed by the speed and precision with which the boots were made. He also noted how the process eliminated the need for skilled labor and greatly reduced the cost.

In 1817, in Morning’s Walk from London to Kew, Phillips wrote, “In another building I was shown his manufactory of shoes, which, like the other, is full of ingenuity, and, in regard to subdivision of labour, brings this fabric on a level with the oft-admired manufactory of pins. Every step in it is effected by the most elegant and precise machinery; while, as each operation is performed by one hand, so each shoe passes through twenty-five hands, who complete from the hide, as supplied by the currier, a hundred pairs of strong and well-finished shoes per day. All the details are performed by the ingenious application of the mechanic powers; and all the parts are characterised by precision, uniformity, and accuracy. As each man performs but one step in the process, which implies no knowledge of what is done by those who go before or follow him, so the persons employed are not shoemakers, but wounded soldiers, who are able to learn their respective duties in a few hours. The contract at which these shoes are delivered to Government is 6s. 6d. per pair, being at least 2s. less than what was paid previously for an unequal and cobbled article.”

Under this model, the shoe market was revolutionized. Once a precious and singular possession, shoes are now designed by a handful of people, manufactured with little manual labor and sold for anything from a pittance to a pretty penny—depending upon the company’s ability to add value.

How many shoes do you have? I would wager it is many more than your great, great, great grandparents had at the dawn of the 19th century. And while each shoe is manufactured in the multi-thousands, it is rare to find someone wearing the exact same shoes as you.

In this scenario, survey research enjoys a much larger market than it does today, and is automated. This enables variety and scale. There would be just a handful of researchers designing studies, but there would be many more jobs for people whose role it is to sell the results by making them attractive and applicable.

Video everywhere

It is hard to imagine that not long ago video was rare and very expensive to produce. It is estimated that within the next few years, 80% of all internet traffic will be streaming video. As I take the subway to work, it seems half the people are gazing at their phones watching some form of video. The interesting thing is that most of them are watching completely different content. It is produced in great volume, but also in great variety. It wasn’t always this way.

I’ve been producing video, on and off, since high school. Back then, the only way I could get access to a video camera was to sweet-talk my art teacher into kindly asking the school’s audio-visual technician if I could use the one camera the school had. The picture was grainy, the camera needed a lot of light, and it was tethered to a very large, non-portable recorder. And there were no editing facilities, so everything needed to be shot in sequence.

After high school, I joined a local artists’ association, where they had a couple of video cameras, including one that was “portable”—if you consider an enormous shoulder-mounted camera and a 20-pound recorder portable. That gear was worth about $50,000 in today’s money.

To get access to editing equipment, I had to either talk the university into letting me use the film school’s very rudimentary two-machine set-up, or I had to produce material for the local cable station—which had a very simple analog mixer.

Today I carry in my pocket a video camera (some call it a phone) that is vastly superior to the artists’ association’s camera. And I can edit the results on my laptop with infinite flexibility. With so much video being produced, there has been an immense increase in the number of people who make their living producing an incredible diversity of content. Business and personal use of video is up dramatically.

In this scenario, market research becomes quick and easy to execute, and is much more widely used. But it remains a craft—the skill of the maker determines how valuable the content is.

The future is in our hands

It’s impossible to know which scenario will play out. It is quite probable that some combination of all these scenarios will transpire. But one thing these scenarios have in common is a radical change, creating new winners and losers. This change is not likely to be a single event. It is likely to be an ongoing process of continual, rapid evolution.

Disney CEO Bob Iger wrote in The Economist about the transforming world of media. His words are equally applicable to the domain of insights: “We once used the term ‘disruption’ to describe all of this [sweeping change], but the word now seems like a quaint relic of a bygone era, implying a paradigm shift with a beginning and an end, a before and after. What we are experiencing now is a state of perpetual permutation.”

Let’s get ready to embrace change, because as U.S. General Army General Eric Ken Shinseki said, “If you dislike change, you’re going to dislike irrelevance even more.”

This article was adapted from The Insights Revolution: Questioning Everything. The article was first published on the Maru/Matchbox blog, which you can subscribe to here.