The jig is up
It's counterintuitive that K12 education has numerous studies demonstrating programs and products with strong positive effects (see Johns Hopkins University School of Education Evidence for ESSA and U.S. Department of Education What Works Clearinghouse) and yet NAEP (national assessment) scores have been virtually flat for five decades. In medicine, drugs with strong efficacy change health outcomes at scale. Why don't we see this in education?
If you're starting from first principles, you might wonder:
- Perhaps education studies just aren't that replicable. A product or intervention with a strong effect in one place might not produce the same effect at scale with uneven implementations.
- Perhaps there's an information problem in the education market and despite best efforts of initiatives like Evidence for ESSA, the products and services with the best results don't spread and instead schools spend limited budget on slickly marketed products with inferior results.
- Perhaps the most effective education products and services are so expensive that they can't scale even if they work. In education there's nothing like the US health insurance system that can scale extremely expensive solutions.
All three scenarios are pretty dispiriting. But, Laurence Holt just pointed out and Dan Meyer amplified (pun intended) a new scenario that I hadn't considered, The 5 Percent Problem.
Laurence Holt asks the same question I did:
Those gains, and many others like them reported each year, are impressive. Since use of these tools is widespread, one could be forgiven for asking why American students are not making impressive gains in math achievement. John Gabrieli, an MIT neuroscientist, declares himself “impressed how education technology has had no effect on . . . outcomes.â€
He digs into a Khan Academy study with an impressive 0.26 standard deviation effect that only applied to the 4.7% of students who used the program 30 minutes per week as recommended, and goes on to cite several other similar studies.
领英推è
If education studies sometimes feel like going down the rabbit hole, Laurence Holt 's discovery feels like we've reached the bottom, and now we might know how to go back up. When education providers were gaming uncontrolled studies, we placed a premium on controlled trials. But it turns out you can game controlled trials too by excluding nearly all the students from the results.
Districts want interventions that significantly impact all of their targeted students, and if students aren't motivated enough to engage with the intervention in the first place, districts shouldn't be persuaded that the intervention is effective.
The school year is ending, which means it's results season! As researchers and companies publish studies let's keep a close eye on how much of the treatment population is included in the results.
ML/AI, Data & Product
9 个月Nice post! If you haven't listened to John List I think you would enjoy his discussion of the Voltage Effect https://podcasts.apple.com/us/podcast/john-list-on-scale-uber-and-the-voltage-effect/id135066958?i=1000571078949
Data Scientist | Quantitative & Mixed-Methods Researcher | Research Manager | Linguist | PhD
9 个月This is a nice summary of recent discussions, and it highlights why the WWC prefers studies that employ an intent-to-treat (ITT) analysis. ITT analysis is important because it helps to avoid overestimating the effectiveness of an intervention, and it accounts for non-compliance and non-adherence that often occur in real-world situations. This is also known as “Once randomized, always analyzed."
Founder of Learning Experience Design (LXD) Research
9 个月There are all types of product research. Some tell you the recipe for success (hopefully replicable) and others let you know if something new is better than what is happening with typical materials. Each have their value - and we need to keep the sample sizes clear, so the reader knows who is being cut and why.
Founding Partner at Project ARC, PBC | Transforming Educator Professional Learning
9 个月Most EdTech is junk and falls into the substitution level of SAMR. I blame developers for not understanding the underlying pedagogical foundations that have the power to transform education. However, I also blame organizations such as ISTE for allowing these companies to push their crap products at their conference. 95% of what is on that conference floor is not worth a fraction of the companies' prices. A few years ago, I participated in a famous study that I dropped out of because the researchers told the teachers to do something I completely disagreed with. That action completely negated the purpose of the study and introduced a variable that was NOT reported in the final results. Yet, that study receives a ton of press as an exemplar, making my blood boil. While not explicitly related to EdTech, I also conducted a systematic literature review of 747 studies and found that 79% of the researchers had zero clue about what they were conducting research about. The results were all published and, in many cases, highly touted. Yet, the underlying pedagogical foundations were utterly inaccurate. Intuitively, I believe there is a correlation between a lack of pedagogical understanding and EdTech product development.
Founder/CEO Educated AI ~ School Principal (Retired) ~ LearningGarden.ai
9 个月Technology is a tool and not a pill to swallow. Edtech is only as good as the educator makes it. Tutorbots and all the magic of AI won't change this as humans are both predictable and unpredictable at the same time. Good luck figuring us out AI. ??