Talking Data: The Three Kinds of Questions that Managers should ask their Analytics Experts

Slowly but surely good practices in analytics are taking hold in organizations and most managers are becoming familiar with data-driven decision making: They are using customer and market data to better target their sales activities, operational data to streamline their processes, HR data to finetune recruiting and training, or sentiment analysis to better understand their social media impact, to name but a few examples.

To make such analytics efforts successful, high quality dialogues between managers and their analytics staff are crucial.

In our experience, asking the right questions about the data and its analysis is a key task for managers to get the maximum value out of their data and their analytics applications.

Asking the right questions is a key task for managers to get the maximum value out of their data.

To support managers and analytics teams in their collaboration, we have compiled particularly useful questions that managers should ask their data scientists whenever they are presented with new data or discuss analysis results together.

The questions that we found particularly helpful can be grouped into three areas:

     I.         Questions concerning data sources and data quality

   II.         Analysis-related questions

 III.         Application-oriented questions

While the first type of questions is instrumental to better assess the validity, reliability and generalizability of the data, the second set is helpful to understand what the data scientists have actually done with the data (and why). This prepares the ground for the third type of questions: application-oriented questions are probably the most important ones to turn insight into impact. They enable managers to actually use the data for their decision making. Still, you can only assess the answers to these questions if you have first asked the other two types of questions.

What are the benefits of asking these three kinds of questions whenever analysis results are presented? Let me just mention the main advantages of adding the questions below to your analytics meeting repertoire:

• They help analysts and managers build common ground and avoid misunderstandings.
• They improve the rigor and relevance of the data analysis, by providing focus, uncovering weak spots, and subsequently improving the data gathering and analysis processes.
• They assist you and your team in finding innovative new ways to exploit your data.

So how can you reap these benefits? Let’s look at the most important questions first, and then briefly describe the best way to ask them. Read the list below as a menu to choose from. You will never have time to go through all of them in a meeting, nor is that necessary. In our experience, using just one to two questions per type vastly improves most data discussions.

For every group, we have put the questions in a sequence that makes sense and that allows analysts who present data to gradually open up (and not shut done or become defensive).

I. Questions regarding Data Sources and Data Quality

Any analysis can only be as good as the underlying data. A manager must thus understand where the data came from and if it is fit for use. To assess the data sources and the quality of the data, a manager can ask his or her analytics staff the following five questions:

1.     Why are you focusing on this data? What questions does it answer and why are these questions crucial for us? In other words: What’s the value of this data?

A prudent question is one half of wisdom. Francis Bacon

2.     Can you tell me how, where (sources and their reliability) and when (time period) we have gathered this data? What biases, events, assumptions, or preferences may have affected this data gathering process?

3.     What do we know about the data’s reliability, consistency, completeness, and accuracy? Is the data trustworthy in your view and current enough?

4.     What data are we missing that you wish we whould have? Are there ways to get it? How have you dealt with missing values/data?

5.     How much is the gathering of this data costing us? Are there ways to make this cheaper, more automized (for example through AI)?

II. Questions related to the Data Analysis

Many data analysts love to talk about their techniques, tools, and how they generally conduct their work. It is thus important to focus the discussion on the analysis parts that really matter for the subsequent use of the data. The following questions can help you to steer the conversation towards useful data analysis aspects:

1.     What are the key terms and concepts that I really need to understand to make sense of this data? Please explain them to me as if I knew nothing about statistics.

2.     What were the analytical tools, procedures and assumptions that you have applied to this data and why those and not others? Is there anything else that I should know about these procedures?

3.     What surprised you when you analyzed this data?

4.     What was the greatest difficulty in analyzing this data? 

5.     What’s the finding you’re most sure about, and where do you have less certainty?

III. Application-related Questions

Data should be a catalyst for decisions and actions. To turn data into decisions, discuss the following questions with your analysts:

1.     What’s the key finding of your data analysis and why? What could we do in light of this data?

2.     What results could be easily misinterpreted or misused? How so? Where should we be careful regarding the validity or generalizability of our interpretations?

3.     Who else should hear about these insights? What should they do with this data (i.e., check it, combine it with other data, give their perspective on it, use it for their models or calculations)?

4.     Is there any other way we could exploit this data and get value out of it? Would visualizing (i.e. in a dashboard), personalizing, or pre-filtering it add value for certain user groups?

5.     If we could start this data analysis over, what would you do differently (to avoid errors, reduce cost, or answer additional questions)?

Not all analysts see it as their role to make suggestions based on their data analysis.

A word of caution here: not all analysts see it as their role to make suggestions based on their data analysis. Some see their job as mere data delivery and synthesis. This last set of questions should thus be asked carefully, iteratively and (most importantly) collaboratively. Try to establish a positive, collegial atmosphere when talking about the data’s uses. This brings us to the final section on how to ask analytics questions.

Asking Questions Constructively

Having outlined which questions managers should ask in analytics meetings, let’s now turn to how to best ask these questions. You certainly do not want your analytics staff (who may not always enjoy the communication side of their job) to become scared, defensive or worried every time they meet you to discuss data insights. So, finding the right tone, time, and tenacity for your questions is imperative. 

Your tone should be respectful, curious, and non-accusatory. The very advantage of the question format is that you are simply showing interest and want to know more, without casting a judgement or accusing somebody not to do their job properly. Hence use the power of the (open) question format.

The timing of your questions should follow the process outlined above. So, start with simple, fact-based questions and gradually move to more complex and opinion-based ones. Also make sure that you time your questions well in the sense that you do not interrupt your analysts when they are presenting an issue that is clearly very important to them.

Start with simple, fact-based questions and gradually move to more complex and opinion-based ones.

Regarding tenacity you should certainly challenge your analysts when they give you an evasive answer. At the same time, you should also show that you trust and respect them, for example when they have repeatedly given you the same kind of answer to a question.

Questions are of course not the only managerial tool to use in such situations. As important as probing for data sources, analysis aspects, and decision consequences, is acknowledging the work that has been done. So, don’t forget to give positive feedback to your analytics staff (and listen to their feedback to you) and thank them for their efforts. Frame data discussions as joint learning events and track how you can improve them continuously. In this way you will avoid the most common source of mistakes in management, according to its most revered guru, Peter Drucker, who said:

“The most common source of mistakes in management decisions is the emphasis on finding the right answer rather than the right question.”

Feel free to ask any question that you might have about our "Analytics Q&A Guide" in the comment section below.