How to Digitally Transform Market Research: 3 Lessons We Have Learned

Joon Nak Choi (JC) and Simon Lau

As an Insights Professional, you know that you need to come to terms with new tech. You will always appreciate the Sherlock Holmes-style adventure of old-school market research, intuitively mining and questioning to uncover hidden insights. Yet, you know that the fine craftsmanship of the industry is just a bit too manual, too slow, and perhaps too inefficient for today’s data-rich but cash-poor environment.

You may also know that you need more than just “AI”. You’re skeptical about the latest buzzwords; you’ve seen so many new tech initiatives coming from major market research agencies fail and you’ve seen so many startups fail to live up to their hype. Still, you can’t afford to ignore new tech…not only because you know it can revolutionize how well you know your consumer, but also because you simply cannot risk falling behind.

Knowing all of this, you are looking for something you can use – something that will get you closer to your consumer – without breaking your budget, taking years to integrate, or confusing your team and your stakeholders. You just want something built to your needs.

What should a forward-thinking market research professional do? Here are 3 lessons we’ve learned from our experience at Zectr, where we’ve worked with folks in the industry to bring market research automation to consumer-facing companies and agencies alike. Please read on for our best practices for digitally transforming market research quickly, easily, and effectively.

Lesson #1: Don’t Replace Humans – Enhance Them

Humans will always be part of the story because there are things that tech just can’t do. Don’t be fooled by hot takes like “…AI invented its own language because English was slow” or “Former Go champion…retires after declaring AI invincible”. The reality is that tech tends to be very, very good at highly structured tasks (we’ll talk about this more below)...but there are things that tech just can’t do. An algorithm lacks the intuition needed to understand nuance, put things into context and read between the lines. An algorithm also lacks the empathy needed to build long-term client relationships. And an algorithm isn’t flexible enough to deal with fickle executives or clients who change their demands at the last possible moment. So, you’ll always need humans as master storytellers, bizdev mavens, and responsible parties who make sure that the job gets done right.

Your mindset should be to liberate your humans from repetitive and boring tasks. Automate the monkey-work to give them more time to focus on what they do best. Do you really need your humans to put together bar charts – when you can get something like this by pressing one button? Automating stuff like this will not only enhance your humans’ capabilities by taking grunt-work off their hands, but also make them happier and more fulfilled.

We’re only scratching the surface here, though. What else can “empowering your humans” do for you in market research? And exactly how can you use tech to do this? Read on to find out…

Lesson #2: Attach Algorithms to Humans as Sidekicks

AI and algorithms shouldn’t be the heroes of the movie...but they should definitely be the hard-working sidekicks. Think about humans and algorithms as a superhero duo. There is always a face to the duo – the one who makes the intuitive leaps, builds the human connections, and tells the story. You also have a silent partner who does the harder and less interesting work behind the scenes. Humans and algorithms should work much the same way – if you want to transform your human into a superhero, give him or her an algorithm as a sidekick. Think of Iron Man and his trusty robotic sidekick, JARVIS. Sure, Iron Man gets to do the interesting stuff and receive all the credit. Yet, he couldn’t be out flying around, saving the universe, and getting so much attention if JARVIS weren’t doing the routine and boring, yet necessary stuff.

Keep your focus on the human, but don't forget about the algorithm! Source: Marvel Studios (2015), "Age of Ultron".

Where in market research could you put an algorithmic sidekick to work? Algorithms are not terribly intelligent – true “artificial intelligence” remains many decades away. So, you must tell them exactly what to do and how to do it. Think of the most unimaginative employee you’ve ever managed, someone who works really hard but needs clear guidance on everything. Algorithms are like this employee but are far smarter in at least one important way.

Like JARVIS, algorithms are incredible at finding patterns and bringing them to attention.

Consider an example you may already be familiar with. Machine learning algorithms are incredible at market segmentation. Back when we were management consultants, we used to build pivot table after pivot table in Excel to segment consumers by demographics, psychographics, and behavioral patterns. This process was painfully time-consuming…it took weeks to run a single segmentation. In recent years, market research professionals have started using some simple algorithms (e.g. k-means clustering) to help them find patterns. We’ve moved onto more advanced algorithms and automated more of the process; our more highly automated approach is not only faster and more efficient but also more granular and insightful. Please take a look at the screenshot below for an example:

You might be thinking “Wow! This looks really cool! But, it's so complicated,,,how can you understand it?” The trick is that algorithms can also help you simplify and summarize stuff. The screenshot below summarizes the same market segment:

We’ll post more on segmentation later – but for now, it’s a great example of the power of algorithms in market research.

We’ve found many other ways to use tech in market research. Algorithms have enabled us to become more efficient by automating boring, routine tasks like data processing. They’ve also helped us find subtle patterns hidden to the human eye. As you see in the diagram below, we’ve adopted tech in ways that make us faster and more efficient without compromising our human strengths:

We’ve found that putting humans and algorithms in a position to succeed – covering for each other’s weaknesses with their own strengths – is necessary but not sufficient. There is one more piece to this puzzle…

Lesson #3: Revolutionize…within Existing Processes

You’re now beginning to understand how to pair humans and algorithms. And you’re starting to think you can revolutionize how your organization runs market research. But then you come back to reality. Most likely, your organization is traditional, your people remain set in their ways, and your standard operating processes are inflexible. How can you digitally transform within such conservative corporate cultures?

The answer is simple. Introduce tech innovations without disrupting your organization, your people, and even your processes. Forget business school buzzwords like “disruptive innovation” and Silicon Valley mantras like “move fast and break things”. If your goal is to get buy-in from your stakeholders, you’re probably not going to get there by shell-shocking them with something radical and disruptive. But you might be able to get there by saying: “This is really the same great stuff. The tech just makes it better and faster”.

We learned this lesson the hard way. Coming in from outside market research, we thought we could force radical change on organizations and people to “disrupt” the industry. It took us years to realize that our clients really weren’t looking to shake things up. They were just looking for better ways to do what they were already doing. We recently increased our monthly sales by 10x after we refined our pitch and brought on board a twenty-year industry veteran. We’ve had far more success pitching “This is the same great stuff coming from the same great people, but the tech makes it faster and better” than we ever did pitching “If you change everything to revolve around our tech, you can be a leading-edge player”. People want tech and innovation. But they won’t go for anything that creates headaches. Under these circumstances, we’ve found that the trick is to make change quick and easy.

Another good example is our experience pitching our interactive reporting platform. Everyone knows that PowerPoint isn’t the most effective or efficient way to report market research insights. So, we built an interactive platform for reporting market research insights that combined the best features from PowerPoint decks, interactive dashboards, and data science visualizations (you’ve already seen some screenshots above). Yet, most market research projects still required an extensive PowerPoint report. How could we move our clients towards our next-generation platform?

Taking our clients’ wants and needs into consideration, we took an incremental approach. We spoke with our clients at various working levels. Listening to them, we felt the need to align with their existing processes instead of trying to offer them something completely different to what they’re used to. Rather than replacing the traditional PowerPoint deck, we decided to offer our interactive platform as a supplement. We felt that this would provide the security of a tried-and-true approach and the powerful new features we built into our interactive platform (e.g. interactive visualization, real-time filtering) at the same time. Our clients have come to prefer our interactive platform over time and we’ve recently started to deliver project final presentations via the platform instead of PowerPoint for the clients who are ready.

We have a general rule of thumb on creating revolutionary change from evolutionary efforts: progress from the familiar to the unfamiliar, especially with tech. People are used to things being done a certain way. Try to introduce new tech in familiar ways – describe them using tried-and-true words and emphasize that the tech performs a traditional functional role. The tech will just be faster, more efficient, or somehow better. Give people a chance to familiarize themselves with the tech instead of pushing it onto them. Take things forward one step at a time, let them migrate towards the tech on their own instead of forcing it upon them, build on wins by moving towards something more exotic. Before you know it, you will achieve a revolutionary transformation – and it might happen sooner than you think if you match the tech to your organization’s needs.

Conclusion: The Future Does Not Need to be Scary

There’s a lot of scary things out there in the world today, especially with consumer markets in turmoil. You don’t need to make digital transformation one of them. Remember that your goal is to empower humans by giving them algorithmic sidekicks – think of Tony Stark and his faithful robot companion, JARVIS. And remember that you can achieve revolutionary outcomes in an evolutionary way. Respect the algorithm, respect the organization, and above all, respect the humans. Before you know it, you’ll have transformed how your organization does market research.

Digital transformation doesn't have to be hard. Done right, it can really help grow your colleagues' productivity, happiness, and engagement. If you need tips or are interested in learning more, click here.