3 Reasons Why Two Skilled Analysts Reach Different Conclusions With The Same Data

Nowadays, businesses produce a ton of product, financial and operational data. And, they hire smart, talented data engineers, analysts, and scientists.

Despite that, business leaders are often baffled that two analysts can come up with two entirely different numbers and conclusions when analyzing the same business question.

Why does this happen when both analysts have access to the same data warehouse? What does it cost the business? And, can businesses do something about it?

Let’s take an example. Let’s say you’re the VP of Customer Success at a growth-stage company. And you’re interested in knowing the answer to the following question:

Has our ticket resolution time for the last year gotten better?

Easy, right?

Not really.

Why do our analysts reach different conclusions?

Reason #1: The question isn’t specific enough

One of the primary reasons for coming up with a different answer is that you started with a different question.

When the business question isn’t specific enough and the metric definitions aren’t standardized, each data person comes up with their own interpretation of the question.

In this example, one could interpret any/all of the following differently:

• “Ticket”?— What constitutes a ticket? Are these tickets in ServiceNow or should we also include tickets from our internal ticketing system? Should we include all tickets that are created in our system? Or should we include only the customer-facing tickets? What about the seller-facing tickets?
• “Resolution time”?— Are we interested in mean or median? When do we consider the resolution to start and end? Should we exclude the time spent in the status of “waiting”?
• “Last year”?—This is one of my favorites. Sounds pretty straightforward right? How do we define last year? Is it the calendar year from Jan 1, 2021 to Dec 31, 2021? Or, is it the last financial year? Or, is it the last 365 days up until today?

Reason #2: The data source isn’t obvious enough

Once you have an agreed-upon definition of the business question, you would think that you can get the same number from both analysts because they have access to the same data.

No, because they have access to a ton of seemingly similar data, and it’s hard to tell which one is the “right source”.

Searching for “tickets” in your data warehouse might return a hundred tables (or more). This happens for a multitude of reasons but here are some:

• The data is from different sources?— One table could contain the raw dump of ServiceNow tickets. Another table could contain the raw dump of JIRA tickets. And, another could contain a combination of both.
• The data is at different levels of detail?— One table could contain all tickets. Another table could contain every single status change of all tickets. Another table could contain a daily aggregation of tickets.
• The data is managed by different owners?— In a growing company, the same data keeps getting duplicated really fast through both ad-hoc queries as well as scheduled jobs. Different teams (such as marketing, customer support, and engineering) may create their own version of the ticket data by excluding/including specific filters to serve their specific use cases.

And, the reason it’s harder is that more often than not people don’t have the time to name their tables in a hyper-descriptive way. And, it’s understandable — each individual is solving their analytic use case and not actively solving for the data discoverability for the rest of the company.

Confusing table names aren’t the only challenge. Figuring out the right data source can also be a nightmare because of:

• Cryptic column names?— Should you use?complete_ts?or?closed_ts?
• Vague data values?— What does?device_type = 3?even mean?
• Unclear expected filters?— Should we exclude tickets with?is_deleted = 1?
• Interchangeable synonyms?— What if ServiceNow is white-labeled as “TicketPortal” in your company? Should you search by both terms?

Reason #3: Their methods aren’t the same

Let’s say that we have an agreed-upon definition of the business question & metrics and that both analysts know how to determine “the right data source.”

At this point, both analysts have the same number.

But they could still reach different conclusions!

Why? Because they could go down different data exploration journeys through different drill-downs. They could be assigning different weights to different characteristics and could be compared against different benchmarks. As a result, come up with different narratives.

One analyst can compare the?ticket resolution time?Year-over-Year and say that our ticket resolution time has gotten worse. Another might dig deeper and find that, generally,?ticket resolution?has gotten?much faster?but the average is been dragged down by a specific type of ticket because that support team is severely understaffed.

Neither analysts are wrong. But the different narratives could influence different business decisions. ??

Why does it matter?

Imagine that you’re in charge of making any of the following decisions:

• Should we hire more customer support staff?
• Should we build a feature to address issues our customers are reporting?
• Should we cancel our contract with the SaaS ticketing tool?

All of these decisions involve significant money! ?? And, concluding an inaccurate or incomplete insight can cost your business!

Here’s how to prevent/reduce it…

1. Define & publicize the business glossary

The key to having a consistent interpretation of data questions is to?define?metrics/KPIs and business terms specifically and?socialize?universally.

For a successful glossary, we need to enable:

• Business people to easily create, define and modify glossary terms
• Technical analysts automatically propagate definitions from the source
• Group leaders to approve/reject/consolidate glossary terms & definitions

2. Make it easy to find the right data

Once we have an agreed-upon definition of the metrics, it is important to help analysts track down the data sources required to calculate those metrics and filter those data sources to the most valuable ones.

DataGalaxy’s “Share the data knowledge”?makes it easy for data consumers to find, select and understand data sources for metric calculations.

We do this with the help of:

• manual & automatic association of metrics & glossary to data tables
• advanced search, filtering, and browsing capabilities
• a customizable algorithm to rank data sources by relevance, usage & more
• curation tools for tables, dashboards, and glossaries

3. Create a culture of curiosity and openness

The same data can tell different stories. Diverse perspectives help your business think critically and eliminate blind spots.

Two analysts using different methods to tell contrasting stories is not necessarily a bad thing.

Creating a safe space and atmosphere where smart people can approach analytical problems with curiosity and challenge each other is great for your organization.