A new national, real-time, Big Data framework for our schools

Big Data has the power to transform our school system, deliver significant improvements in student achievement and close the attainment gap. I recently commented on how I thought the UK government’s new EdTech strategy could be improved, but I left out one key and vital area that the strategy is missing — the use of Big Data.

I’ve long been a big fan of the work that the Education Endowment Foundation (EEF) has been doing to get hard statistical evidence behind the efficacy of many different teaching initiatives in our schools. Most of these initiatives have nothing to do with technology,  but in the last year or so they have started to investigate the impact of EdTech too. Whilst I am a fan of the EEF, I think there’s a much better way based on real-time Big Data.

Let me explain …

Running large scale trials is expensive and slow

Three years ago I met with the EEF to see if it would be possible to run a trial of our software in schools to get evidence of impact. They told me that they didn’t run trials of EdTech because (and I paraphrase) “there are so many different EdTech products and it costs so much to run a trial, how do we choose which one to test?” Running a single trial costs a minimum of ￡60k, but some trials cost a lot more. Indeed, the grant for evaluating EdTech product Diagnostic Questions is a staggering ￡2m.

I’m glad they are now running a number of trials of EdTech products. However the core problem still remains, that the EEF are only able to run a small number of trials due to the high expense, yet there are literally thousands of EdTech products out there. This is a problem not just limited to EdTech; there are thousands of teaching initiatives that have the potential to deliver improvements in student learning, but most never get tested at all, let alone at a level that delivers robust statistical significance.

The reason why the EEF trials cost so much is because, in order to detect statistically significant improvements of one or two percent, they need to A/B test impact over typically 20 schools and a hundred classes over a year or two. This is to make sure that any difference in demographic background, teaching quality, resources and facilities, or any other unknown bias, is averaged out over a large range of different classes in different schools. However, even with a hundred classes it is difficult to remove all biases. With a large and diverse sample the EEF can however deliver statistically significant results.

A better way

There is a better way ... suppose we actually tracked the progress of every class in the country, every day. And suppose that we also tracked exactly what teaching methodologies and EdTech products were being used by teachers in each and every class every day. Suppose we did this automatically (we certainly don’t want to increase teacher workload). And suppose we then used modern Big Data techniques to identify and quantify the impact that every methodology and EdTech product is really having on student attainment, automatically, even when they are used on a small scale.

Identifying this impact is possible with such a system for two reasons. First, when things are making a real difference, it is safe to assume that the consequent improvement in student attainment will be much greater than one or two percent. Our own EdTech product Quizalize has been shown to improve student scores in end of year exams by 8-10% in tests by individual schools in the United States, for example. When the differences are this big you need a lot less data for the improvements to be statistically significant and meaningful.

Second, by recording all data in all schools, you are able to model and estimate hidden variables that are not directly measurable, for example teaching quality. You can then calculate and account for the contribution to improvements in student attainment due to each factor, hidden or otherwise. This makes it possible to gather more accurate measurements of the improvement resulting from a specific methodology or product since unknown biases can be estimated and removed.

Moreover, by tracking data every day and against micro learning objectives you can observe improvements in days rather than years.

The result is that you can quite literally test thousands of different methodologies and products simultaneously and get results far faster.  If you are a physics teacher with a new idea for teaching atomic structures, you can simply start to apply it yourself and perhaps get a handful of other teachers around the country to apply it too. If it is effective the system will observe a resulting improvement in student attainment automatically. If you are a school head and want to see the impact of free breakfasts for students, you can do the same. Or if you are an EdTech company with a new Virtual Reality EdTech product for teaching World War I History, just get half a dozen teachers to trial it and initial data on efficacy will be there.

You may still want to do a big multiyear multi-school trial afterwards, as this really is the gold standard and tools that provide short term wins can even have a negative impact in the long term, but you would have much better candidate methodologies and products in the first place.

Such a rich data set will also provide new and valuable insights into one of the nation’s most pressing education problems, the attainment gap. This is the gap whereby children from certain sociodemographic backgrounds achieve worse academic results than children of equal natural ability from other sociodemographic backgrounds. As a country, we understand in great detail that it happens, but we struggle to identify the real root causes and implement effective remedies. Unfortunately, current measurement data is limited to that from end-of-year and mid-year tests. Perhaps initiatives like free school breakfasts really would make a difference on closing the attainment gap, as it’s hard for a student to focus when they are hungry, but this is just one of thousands of potential remedies that go unmeasured and untested.

Whilst overall impact can give you valuable insight into which methodologies and products are effective in general, cohort analysis of real-time daily data may provide key insight to identify remedies for closing the attainment gap.

Technology suppliers and school buyers will both benefit from real-time data on how their product improves attainment too. If a startup launches a new mobile app for students to learn Spanish, for example, they typically then get a couple of local schools to test it. However, getting any hard data on the real impact of the app is incredibly challenging for small startups. With a real-time national data framework tracking resource efficacy on a daily basis, that startup would get hard, independently validated data immediately and at no cost. The startup would even be able to A/B test different features for efficacy to help inform product improvement. School buyers would no longer need to rely on suppliers’ claims of efficacy too, as they would have automatic independent data available by default. Incumbent, branded suppliers with low efficacy and usage would rapidly be replaced by smaller, more innovative suppliers with products that really do make a difference.

The cost of Big Data

How much would it cost to build such a real-time Big Data system for our schools? Well, the good news is that we’ve already built a first version with our ￡1.5m investment into our brand new Zzish Insights real-time Big Data platform for school systems. Built on a cutting edge Big Data engine it allows us to run queries in seconds across billions of formative data points, queries that would take minutes or hours to run on legacy systems. Zzish Insights aggregates the data from the hundreds of formative assessments done every day, either in class or for homework, across hundreds of schools in a network to give school leaders and government a daily update on how every class is progressing against every learning objective in every grade and curriculum. It’s a unique and incredibly powerful product and we are committed to investing at least a further ￡5m to make it even more helpful to school, network and government leaders over the next three years.

We designed Zzish Insights initially for the large US school districts that have more than 100,000 students spread across 200 or more schools. But we are also deploying it for the Philippines government who want to track the academic progress of a million students In 50,000 classes and 1,000 schools across two provinces. On a smaller scale, Zzish Insights is also a valuable tool for individual school, year and department heads too.

Additional Benefits

Indeed a real-time Big Data system for our schools, has a number of other valuable benefits for all stakeholders across the school ecosystem.

Enabling timely, positive interventions

A first additional benefit of such a system is that it can be used to detect emerging class and curriculum level issues much earlier than would otherwise be possible. This allows school, network leaders and government leaders to make positive interventions to fix these issues before they become even bigger problems.

For example, if, on average, students across a region are struggling with a particular learning objective much more than others, this can be detected early and targeted investment in new teaching resources supporting that learning objective can be made. If a particular class is making slower progress than average, then perhaps the teacher can be matched with another teacher whose class is making above average progress for mentoring. Alternatively the teacher might be given targeted CPD to help develop their teaching skills.

Indeed, in the US market, Instructional Coaches are finding Zzish Insights an invaluable tool for them to identify potential issues and plan exactly where to focus their training efforts on a daily basis. It’s powerful and exciting stuff that we expect to deliver real impact.

Existing district wide data platforms are already big business in the US. Market leader Illuminate Education will generate $100m in revenue this year, enough to make it a new EdTech unicorn. But whereas such current market leaders provide valuable insight to disrict leaders they are fundamentally limited in two ways. First they are built on legacy database technologies that limit the analysis that can be done. Second they work primarily on end of term summative assessment data and identifying an issue at the end of term means that it will already have become a problem. Such existing systems are fundamentally summative in nature.

Indeed, you can think of a national real-time Big Data system for schools as a formative assessment system at the school and class level. Our formative assessment tool for teachers, Quizalize, is designed to answer three questions for teachers as quickly as possible, “Which students need help?”, “What does each need help with?” and “How can I help each best?”. Our Zzish Insights product for school leaders and government is designed to answer essentially the same questions, but at a higher level, “Which schools and classes need help?”, “What does each need help with?” and “How can I help each best?”

This final question, “How can I help each best” is a super interesting problem and our real-time Big Data platform aims to directly answer this question for school leaders in the future too. We currently answer this question for teachers in Quizalize by measuring the effectiveness of differentiated follow-up resources that teachers assign to help students master specific learning objectives. These measurements enable us to make even better recommendations for teachers, as our algorithms can predict will resource will help each individual student improve best.

We want to do exactly the same thing at the school and class level with teacher training resources. For example, if many teachers are struggling with teaching Bhor’s atomic model to students and the intervention chosen by one school leader is to provide a CPD video on how to teach this learning objective well, we can immediately measure the impact of this CPD video on class progress and then automatically recommend it for other teachers when our algorithms predict it will help them teach their class better too. We can further identify the teachers who are excelling at teaching Bohr’s model and automatically ask them to share their expertise and knowledge on how to teach it with the rest of the community, perhaps by sharing lesson plans or by creating a CPD video on YouTube themselves.

Measuring resource usage

Second, a real-time data system for our schools that tracks resource usage every day would also be very useful in simply measuring how much specific teaching resources are used. Schools invest tens of millions in aggregate on resources, but school leaders and government have little idea as to whether these resources are actually used, let alone whether they are effective.

It can even provide the basis for new methods of rewarding resource suppliers based on usage and efficacy, not just on whether a fixed price three year contract is signed. Music systems such as Spotify reward artists based on how much their music is listened to. Why shouldn’t we reward resource suppliers on this basis too? Indeed one of our goals at Zzish is to enable schools to reward resource suppliers in this way.

Insight to improve learning

A final benefit is in the deep data analysis that can be done to understand individual student progress through a curriculum. For example, such data would provide invaluable insight to teaching and student learning and can be used to automate the creation of optimal, differentiated and personalised lesson plans. This kind of data set is the perfect material for AI algorithms and at Zzish we are already starting to apply such techniques to help teachers by analysing the daily formative data across hundreds of classes in Texas.

Summary

In summary a new national, real-time data framework for our schools would have some powerful benefits:

1. Identifying teaching methodologies and EdTech products that have a real impact on student attainment and closing the attainment gap much faster and more cost effectively than is currently possible
2. Helping technology suppliers get data on their product efficacy that they can use to improve and enhance their products
3. Identifying emerging issues at the class and school level and making positive interventions to address these issues before they become a problem
4. Measuring the impact of class and school level interventions to address issues and automating recommendations to address similar emerging issues elsewhere
5. Measuring the usage and efficacy of purchased resources and ensuring value for money
6. Gaining a rich data set for analysis to provide insight into learning and the development of more optimal, differentiated and personalised lesson plans.

Our products for teachers give them superpowers by enabling them to personalise their teaching and improve student attainment. We hope that a national real-time data system will give school, network and government leaders similar superpowers, allowing them to give far more timely, targeted and impactful help to their schools and teachers. It will help improve student attainment too.

Of course, some heads and teachers will be wary of a system that effectively monitors teachers on a daily basis. Indeed, it is critical that such a real-time system is used only to provide heads and teachers with help and never for evaluation. The current SATs, progress 8 and attainment 8 metrics based on summative student tests are an acceptable framework for summative evaluation of schools. A national, real-time data framework for our schools must be a formative tool and must be used only for identifying where schools and classes need help and then for giving that help through positive interventions.

Fortunately, teachers really do understand the value and benefits of such a real-time Big Data system and so far have been very welcoming of the benefits that it brings. The reality is that teachers love tools that make it easier for them to have an impact on student learning. We have had many teachers thank us for creating classroom tools that are making their jobs easier and more rewarding and some have even told us that we have reignited their love for the teaching profession. On a personal level, this feedback is one of the most rewarding aspects of building these tools.

I also believe that such data should be (as far as possible while ensuring the privacy of students and teachers) made available for any approved company or institution to analyse and build upon. Individual, class and school data can be anonymised, for example, and the data from such a system would provide an unparalleled data set for others to discover new insights into learning and to develop new methodologies and products to improve student attainment. It always pains me that the current periodical data that the government collects is kept behind lock and key and that companies like mine are unable to use it to learn from ourselves and so help improve student learning.

The good news is that individual schools and MAT leaders don’t need to wait for the government to deploy such a national system on their behalf in order to reap the benefits, they can take the lead and start using Zzish Insights today. Hopefully other companies will invest and create competing products soon so that this becomes a vibrant, new, competitive global market segment led by UK companies. However, it would be truly amazing if the UK government took the lead and showed how real time Big Data at a national scale can transform teaching and learning.

It is one of the great opportunities of our age.