Data in Education is Not Truth

Can we all please stop treating education data as is if it represents some infallible truth? I spend a great deal of time with schools and trusts talking about the importance of a sensible data strategy, and to understand how good quality data can be used appropriately to help make sound decisions to improve outcomes for young people.

However, we also have to recognise the pitfalls of data dependency that can occur at every level, from teachers, middle leaders, the senior team and school and trust executive. In particular, we need to be clear about the purpose of doing so, especially when we aggregate across schools, systems and geographies. Considering the purpose, does the data provide the insight we need?

'il meglio è nemico del bene' - Voltaire 1770

or, 'the best is the enemy of the good'. Large data projects can tie themselves in knots attempting to achieve 100% technical accuracy and my, perhaps unpopular, opinion is largely this involves wasting time that could be better spent elsewhere in improving outcomes. Given a raft of uncertainties that underpin data collection and aggregation, I'd simply ask is it good enough?

Here are just some on the questions that normally spring to mind: What do we know about how data is entered into source systems? Are the individuals that enter data motivated by data quality - does it impact the day to day work they perform? How do we know that data is collected with a consistent process across schools? How do we know that data represented by scales, categories, or other taxonomies mean the same thing to different people? What are the opportunities for data to change being exported from one system and imported to another? What about when system migration occurs? Do people really understand confidence intervals, and that 98 might actually be 103? Why do you need that number to two decimal places, are you really telling me that you are going to do something different based on some hundredths? Progress 8 is based on a prior cohort, predicting it for the future involves having at least one finger in the air. On looking at that dashboard or report, what are you now going to do differently? If there is nothing different to do, the collection and analysis has just been a waste of time.

I'm an advocate of data for actionable insight. If we can see a trend in attendance that may lead to persistent absenteeism, identify how good teaching practice in one area may be extended to others, determine a group of students at risk and put in place the right intervention in time to improve outcomes, then we are using data, that is good enough, for the purpose we need.

As communities in the UK and around the world build systems, such as those coordinated by Microsoft's Open Education Analytics, we will continue to discuss the challenges around standards and schemas, joining disparate systems in meaningful ways, but above all come to a common consensus on the 'use cases', or purpose, of doing this in the first place. We'd love to get your involvement and input as we do this; to help ensure that we do not fail to continue to make good progress improving students outcomes for the sake of waiting for perfect to waltz along.

Is your data good enough? What does it mean? Now, what are you going to do differently?