The Evolving Role of Data in Experimentation: Reflections from Superweek

Data and analytics have always been fundamental to AB Testing/Optimisation, but as the scale of Experimentation programmes continues to grow, it is becoming more important than ever.

At this year’s Superweek conference in Hungary, discussions with Analytics industry leaders really highlighted a crucial shift: data is no longer just a tool for designing and assessing experiments, it’s becoming the bridge between Experimentation and real business outcomes.

Connected data and actionable insights are essential for accurately measuring impact, enhancing product development efficiency, and driving broader business success.

This shift opens up possibilities to enable low-risk innovation, personalise more effectively, and automate more intelligently.

Reflecting on My Journey with Data and Analytics

I started my digital career in web analytics nearly 20 years ago, where my work revolved around tracking performance, understanding user behaviours, and reporting key metrics.

As MarTech evolved, my role expanded beyond analytics, incorporating a broader range of tools and strategies to drive insights and impact.

This eventually led me to conversion rate optimisation (CRO), where analytics was critical, but often misused. Many relied on funnel drop-offs to identify 'problem areas' and then turned to intuition for solutions. While this sometimes delivered quick wins, it rarely tackled deeper, systemic issues.

Learning to Focus on the Right Problems

Over recent years, my colleagues at Creative CX have helped me develop a much deeper understanding of the role data plays in problem discovery. I've learned that data isn't just about measurement, it’s a crucial tool in identifying the right problems to solve before considering potential solutions.

At CCX, we take a triangulated approach to research, combining qualitative and quantitative methods to gain a comprehensive understanding of the customer journey. Qualitative research, such as usability testing and interviews, uncovers the 'whys' behind user behaviours, while quantitative data, from surveys and analytics, helps us measure the prevalence and impact of these challenges. This approach mitigates cognitive bias, prevents blind spots, and ensures our experimentation efforts are based on reliable insights that drive meaningful improvements.

By embedding data-driven problem discovery into our work, we ensure that our solutions are more targeted, effective, and aligned with both customer needs and business goals. This shift from diagnosing problems to truly understanding them has been transformative.

Experimentation at Scale

Experimentation at Scale has broadened our remit beyond isolated touch-points. We’re looking at the entire customer journey (online & offline), collaborating with teams across organisations to connect insights from product, marketing, customer success, and beyond. One of the interesting developments in this evolution is our closer collaboration with data science teams. Many had already built separate models for back-end algorithm changes, feature flagging, and advanced data processing. Now they are working with others to achieve even more. Among many things, data science teams can provide us with models that help drive data-informed decisions and personalisation strategies. This integration, alongside other advancements, is helping us unlock more impactful experimentation opportunities.

Experimentation is now taking place across various environments, from front-end interactions to complex back-end systems - and offline environments too. To maximise the value of these efforts, we are focused on connecting experimentation activities in data warehouses and CDPs, ensuring that insights are unified and actionable across the business.

By embedding these capabilities into our processes and frameworks, we are making experimentation more rigorous, scalable, and impactful than ever before.

Enabling us to:

• Better understand the customer journey: both the initial sales journey and the ongoing relationship.
• Connect activities to outcomes: journey improvements, customer lifetime value, cost to serve, new services/features, and much more.
• Leverage advanced models: incorporate sophisticated models to enhance decision-making.
• Improve personalisation: create more tailored customer experiences that drive engagement and loyalty.
• Scale automation: automate processes to allow organisations to optimise at scale, with precision and efficiency.

A New Era for Analytics and Experimentation

Superweek was a reminder of how far we’ve come in the world of analytics and experimentation—and how much potential still lies ahead. The role of data has evolved from simply informing decisions to driving comprehensive, connected insights that span the entire business.

As we embrace these new opportunities, we can:

• Show value more clearly by linking experimentation to key business outcomes.
• Deliver personalised experiences that delight customers and meet their unique needs.
• Automate processes to help organisations scale their efforts more effectively.

The future isn’t just about running tests; it’s about leveraging experimentation to deliver meaningful value for both customers and businesses. By connecting data, outcomes, and teams, we’re unlocking a new era of innovation.

I’m excited to see where it takes us.