Building Brands with Better Data

Bottom line: Thoughtful survey design can enhance brand strategy, but poor design will alienate discerning stakeholders and simple register as noise to others.

Brands are in a relentless pursuit to stand out and resonate with their audiences.

Commissioning studies and whitepapers has become a popular strategy to position brands as thought leaders and innovators. How many times have you seen the words "a study shows..."?

However, many of these so-called studies fall short, often failing to provide new or actionable insights—or simply inaccurate and uninformative results. The crux of the issue lies in the design of these studies—small, non-representative samples, leading questions, and inadequate controls can render them ineffective. To build a resilient brand with a robust strategic backbone, it's imperative to prioritize quality over quantity in research endeavors.

The Pitfalls of Poorly Designed Surveys

Commissioning a study is not just about collecting data; it's about gathering meaningful insights that can inform brand strategy and consumer engagement. If your only goal is to generate buzz for the press, you're not going to have enough skin-in-the-game to do research right. Instead, you need to pursue a study to truly get to the right answer because it matters.

(Quick point: that means brands need to be more discerning about when to pursue a study!)

Unfortunately, many brands stumble into common pitfalls:

• Small, Non-Representative Samples: Relying on a limited number of respondents who don't reflect the broader target audience can lead to skewed results. This misalignment can cause brands to make decisions based on inaccurate representations of consumer behavior.
• Leading Questions: Crafting survey questions that nudge respondents toward a particular answer undermines the authenticity of the data. Leading questions can confirm existing biases rather than uncover genuine consumer sentiments.
• Lack of Adequate Controls: Without proper controls or by only conducting cross-sectional studies, brands miss out on understanding the nuances of consumer behavior over time. This approach limits the ability to draw causal inferences or track changes in attitudes and preferences.

Harnessing the Power of Thoughtful Survey Design

To extract valuable insights that can truly enhance brand strategy, consider the following approaches:

1. Ensure a Representative Sample

Invest in reaching a diverse and sizable sample that mirrors your target demographic. Utilize stratified sampling techniques to ensure all key segments of your audience are included. Gallup has been a leader here, but others, like Prolific , have been doing a great job too.

A representative sample means you can actually make statements about the population that you're interested in, rather than leaving it up to chance.

But careful, a large sample is not enough. Brands need to think about the selection process into the survey—that is, why some people opt in and whether it's truly random or close to it.

2. Craft Unbiased Survey Questions

Develop neutral questions that allow respondents to express their true opinions. Pilot test your survey with a small group to identify and eliminate any leading or loaded questions.

Unbiased questions yield genuine insights into consumer perceptions and needs, fostering strategies that align with actual market demands. Enough with the surveys yield meaningless results like, "99% of managers would like to have more data to make decisions."

3. Incorporate Longitudinal Studies and Controls

Design studies that track changes over time or at least include factors that can be used as controls to establish and/or gauge the degree to which results are causal. This could involve follow-up surveys or experimental designs that test specific variables.

Longitudinal data provides deeper insights into trends and the effectiveness of interventions because you can track the same unit over time, thereby helping rule out confounding effects. If that is not possible, then gather adequate controls (e.g., income, age, education if surveying individual consumers), which can be used to gauge potential causal forces.

Case Study from Uplevel

I recently read an interesting news story in CIO Magazine? (cc author Grant Gross ) about a study that they did surveying 800 of their developers about the efficacy of using Copilot to help with programming and development (see my post here).

Who: Uplevel is a Seattle-based company founded in 2018 by a group of tech veterans with experience from Microsoft and Tableau. They provide an engineering effectiveness platform designed to improve productivity and well-being for engineering teams. By analyzing data from tools like Slack, Jira, GitHub, and others, Uplevel generates insights on how engineers spend their time, whether they have enough focus time, and whether they are overloaded with tasks or meetings. The platform uses machine learning to help engineering leaders address challenges, predict work success, and identify bottlenecks to increase efficiency.

What: The analysis was based on a treatment-control design where 351 developers with access to Copilot (the test group) were compared to 434 developers without access (the control group). Both groups were similar in terms of role, working days, and PR volume. Metrics such as cycle time, pull request (PR) throughput, bug rates, and "Always On" time were tracked across two periods: January 9 to April 9, 2023 (before Copilot) and January 8 to April 7, 2024 (after Copilot). This approach enabled a direct comparison between developers with and without Copilot access, controlling for seasonal effects and avoiding self-reported data. The study's findings were nonetheless observational, making a full causal interpretation tougher.

Key findings include:

• Cycle Time and pull request (PR) Throughput: There was no significant difference in coding speed, cycle time, or PR throughput between developers with and without Copilot access. While some changes were statistically significant, they were minimal and inconsequential in practice (e.g., a reduction in cycle time of just 1.7 minutes).
• Bug Rate: Teams with Copilot access experienced a 41% increase in bug rates, suggesting a potential decline in code quality.
• Burnout (Always On Time): Both groups saw a decrease in extended working hours, a leading indicator of burnout. However, the decrease was less for developers with Copilot access (17%) compared to those without it (28%), i.e. Copilot did not mitigate burnout risks much.

They conclude that Copilot did not provide clear benefits in terms of productivity or code quality for this developer population. They nevertheless recommend a cautious interpretation, encouraging further experimentation, goal setting, and A/B testing to track improvements.

My Thoughts: Uplevel does a great job tackling a question that matters—the efficacy of GenAI, especially in technical work is a very hot and debated topic. They also found a counterintuitive result, which raises the salience and appeal. The team also did a great job setting up a treated and control group, but more could be done. For example, to test randomization, they could regress an indicator for the treatment on a set of control variables, showing that there is no systematic correlation. Uplevel also did a good job focusing on metrics that are objectively measured, rather than using potentially leading questions in a survey. Overall, an A!

The Strategic Advantage of Quality Research

By prioritizing thoughtful survey design, brands can uncover deep insights that drive meaningful engagement and loyalty. This approach transforms studies and whitepapers from mere marketing materials into powerful tools that inform strategy and foster growth.

Join the Conversation

Have you commissioned a study or used data analytics to enhance your brand strategy? We'd love to hear about your experiences and insights. Share your thoughts or questions, and let's explore how we can collectively strengthen our brands through better data.

Dr. Christos A. Makridis

PS, If you find this material useful and want to learn more through my online class, visit www.brand-backbone.com! You can also use the discount code "NEWSLETTER" to receive 30% off the class, or gift it to a colleague.