Data validity and reliability

Validity and reliability are two issues that I hear a lot about in my work - usually when people question the validity and/or the reliability of the information they've been given. These two terms are often bundled together (and sometimes even used interchangeably); however, they're very different issues, and therefore have different solutions. In the realm of data storytelling and analytics, trust is paramount... as without trust, the use of the data falls flat and people won't use the data to its full capacity. To build trust, we have to improve the validity and reliability of the data we have, so that people are motivated to use the data and take appropriate evidence-informed action.

Reliability relates to the consistency of the measurement. Think of it like stepping on a scale multiple times and getting the same reading each time - this means the scale is reliable. In our workplaces, we need to be able to ensure that if we were to replicate our data collection processes, and collect the data again, we'd yield similar results under similar circumstances.

Ensuring reliability isn't always straightforward, especially when dealing with qualitative data and the quirks of human subjectivity. Our brains are awesome, until they're not - we find it hard to do the same things, in exactly the same ways, especially when there are multiple people involved. To minimise the impact of reliability, we need to ensure that there is consistency in how the data is gathered (for example, asking the same questions, in the same way, or asking all customers the same questions).

On the other hand, validity relates to whether our data truly reflects what we aim to measure. It's about asking, "Does this data paint an accurate picture of our reality?" and "Is the data telling us what we THINK it's telling us?". Take, for instance, a company gauging customer satisfaction solely based on product reviews. While product reviews might offer insights into customer satisfaction, it would be invalid to extrapolate this data to make assumptions about the customer service they had experienced, from how they feel about the product.

Knowing these challenges, it worth considering how we address them, and enhance both the validity and reliability of our data. For starters, we need meticulous attention to detail, consistent data collection methods, clear criteria, and regular audits. But beyond that, it's also about fostering a culture of inquiry and transparency, so our staff understand the challenges, and know when to flag potential concerns.

I've seen firsthand how teams grapple with inherited data they can't trust, leading to wasted time and misplaced efforts. It's why fostering open dialogue around data quality is essential. When employees feel empowered to question and refine data practices, trust flourishes, and better data practices evolve.

As you navigate the world of data in your own world of work, remember to pause and reflect on the trustworthiness of your data sets. Are they reliable, and consistently reproducible? And are they valid, and do they accurately reflect the reality you seek to understand?

This newsletter started as a podcast; if you'd like to listen to this episode and/or follow the podcast, check it out here.

______________________________________________________________

I am a data storyteller and grounded researcher, and I help organisations use their data more effectively and help them tell great data stories. If you'd like a hand with data storytelling or strategy, I'd love to chat with you.

Keynote speaker | Author | Facilitator