Burning Questions

Hey there,

Welcome to a free post of the databeats community newsletter tailored to help growth practitioners drive data-powered growth.

In last week’s post, Contextless Data Collection, I discussed the perils and consequences of collecting data without context. Today, we’ll discuss all the good things that happen in the presence of proper context.

So, when context exists, people start asking questions.

And asking a lot of questions without expecting immediate answers to every question is the only way to get better at asking the right questions – it’s a muscle that one needs to build.?

When I was leading Growth at Integromat (now Make.com – Zapier’s toughest competitor), and we were setting up our data infrastructure, I had a ton of questions.

In fact, we were acquiring hundreds of new users every day – sometimes even crossing a thousand in a day – and if we wanted to, we could have collected massive amounts of data to track everything every user did on the website and inside the app. However, as a bootstrapped company, Integromat operated lean, and thankfully so – it pushed me to think things through and question popular narratives like why it’s a good idea to collect as much data as you can.

I had been a power user of Integromat even before I joined the team in early 2018. And I’d already helped a ton of people with their questions about the product through a couple of communities I was active in. As a result, I intricately understood the product and the various personas it catered to, and by the time we started implementing our data stack in late 2019, I had enough context to start asking questions – lots of questions.

However, as I started listing them down – questions I thought I badly wanted answers to – a pattern emerged and it became crystal clear that every question of mine fell into one of two categories:

• Type 1 Questions: Whose answers I didn’t know what to do with – nice-to-have answers that would serve my curiosity but not much else. It wasn’t clear how to use the insights from those answers to drive growth.
• Type 2 Questions: Whose answers I knew exactly what to do with – answers that would give me, for instance, the insights to figure out what’s preventing users from hitting the activation milestone. I had clear use cases and the tools in place to use the insights to drive growth.

The Type 2 questions are essentially what I refer to as burning questions.

Here are two distinct characteristics of a burning question:?

1. It is very specific
2. It has context baked in

Let’s look at an example:

“We’re acquiring a ton of users every day but very few end up hitting the activation milestone; what’s preventing users from performing the actions leading to activation?”

It’s specific with context baked in.

The first part of the question, “We’re acquiring a ton of users every day but very few end up hitting the activation milestone” is the context to ask something specific like, “What’s preventing users from performing the actions leading to activation?”

On the contrary, a Type 1 question is super broad and lacks context. The answer tells what’s going on but it doesn’t help understand the cause and the fix.

Here are a few examples:

• “How many new users did we acquire in the last 30 days?”
• “How many users did we convert?
• “How many users did we lose?”

Answers to Type 1 questions like these provide a bird’s eye view of the health of the business – perfect for an executive dashboard powered by a BI tool.

But a bird’s eye view is not what you’re looking for as the person responsible for getting more users to use your product more often, is it? You’re not looking for numbers without any context for you to do something to increase or decrease those numbers.

You’re looking for specific, contextual information that you can act upon immediately and continuously.

For instance, you can identify the causes responsible for a low activation rate if you have context about the actions performed by users as they move through the onboarding process

But that’s not it, is it?

When you have context, you also have the exact data points you need to run campaigns and experiments to fix the problem.

As you start thinking through the solution, you are likely to have more questions and might need additional data points in the destinations where you intend to consume and act upon the data. At the very least, you’ll need to run through a series of steps to figure out whether the issue lies with how your product works, how and when a feature is presented, or something else altogether.

You might end up talking to inactivated users only to figure out that they had very different expectations from the product, indicating a misalignment between Marketing and Growth. By relaying the data to your marketing team, you might further learn that one of the ad campaigns being run by an external agency is driving visitors to an outdated landing page that talks about features no longer available on the free plan.

It’s good to keep in mind that context leads to burning questions, which lead to better collaboration and ultimately, better quality data. Let’s explore how.

Not just about insights??

It’s important to highlight that the goal of a burning question is not only to derive an insight but also to figure out how to run an experiment, how to do so effectively, and how to measure the impact of an experiment.?And experiments lead to additional insights which lead to more context and thereby, more burning questions.?

Growth practitioners, in particular, need to come up with a lot of burning questions that can’t always be answered using a predefined metric – questions that facilitate better collaboration between teams.

A scenario

Growth, in collaboration with Product, has decided to enable new ways for users to invite members to their workspace. While Engineering builds the feature, Growth wants to ensure that the feature is instrumented before it hits the production environment.

Growth also wants granular data to be made available in their activation tools to experiment with the messaging of the invitation emails and to send out reminders. As they think through all the specifics, they come up with the following questions:

• Can an invitee accept the invite using an existing account or they must create a new account using the email that was used in the invite?
• If an invitee uses their existing account, should we count them separately to ascertain the number of new users we acquire via the invitation flow? And if an invitee creates a new account, should we also allocate them their own workspace besides the one they join? How do we define the various metrics?
• Should invites expire? If so, should we send an email to the workspace owner to let them know? Should we also send a reminder to the invitee before the invite expires?

The growth team brings together folks from Product, Engineering, and Data so that everyone is on the same page sooner rather than later. Besides massive productivity gains and cross-team alignment, doing so enables fast, accurate answers to the burning questions that follow once the new invitation flow is live.

Closing thoughts

As growth professionals, asking a lot of questions is part of our job, and investing in the resources needed to answer our questions is the employer’s responsibility. Moreover, a number isn’t the answer to every question – we need our data and engineering counterparts to understand the problem we’re looking to solve and help us figure out the best way to solve that problem (using accurate data, of course).

Also, if internal teams are not equipped to answer our questions, there’s no shortage of external help. Organizations need to spend less on technology and more on equipping people like us to get expert, unbiased, actionable answers to our questions.?

Burning questions are highly contextual in nature so let me list down a couple more. You will notice that these are unlike the ones that can be answered by running a quick query or referencing an existing report:

• “We have improved the onboarding experience with detailed product tours to get more users to hit the activation milestone; how do we measure the impact of the product tours on our activation rate?”?
• “We’re running a series of email campaigns to drive users of inactivated accounts back into the app; how do we find out if users who open at least one email in the campaign come back to perform the actions leading to activation? And how do we compare the results of that cohort with the one where users become activated without opening a single email in the campaign?”

These are some of the burning questions I had during my time at Integromat and we’ll be digging deep into these in future posts. ?

Today’s exercise

List a few questions you’re trying to answer right now and then evaluate if they meet the criteria of burning questions:

• Do they have context baked in??
• Do you know what to do with the answers? In other words, do you have a clearly defined use case to put the answers into action? Have you listed the metrics?
• Most importantly, do your data and engineering counterparts know your expectations and understand your use cases?

This isn’t a quick exercise but it’s certainly one that you must initiate at the earliest.

And if you already have, well, amazing – I’d love to hear what worked and what didn’t, and maybe even collaborate on a guide based on your insights!

Coming up next

The next 3-4 subscriber-only posts will go deep into the process of getting answers to burning questions. In particular, they will help you figure these out:

• What data do you need to answer burning questions?
• Does the data originate in an internal source, external source, or both?
• How do the destinations where you consume data impact data collection?

If you found today’s post valuable, share it with a friend, and consider joining the databeats community if you haven’t already. For a mere $90/year (or $9/month), you will receive weekly guides like this AND you'll get access to our private Slack community. Join us?

Need social proof? Here’s what others are saying about databeats.