I don't think we need to blow up how we measure schools...yet

Back in the spring I asked whether assessment is holding back the Science of Reading. I picked on DIBELS, writing:

In Kindergarten, DIBELS tests (1) letter naming, (2) phonemic segmentation (if I say 'am' you would say /a/ /m/), (3) nonsense word fluency (/h/ /a/ /p/ is pronounced 'hap'), (4) and word reading fluency (which includes lots of sight words).

I'm not a psychometrician. My understanding, though, is that letter naming may be less important initially than pronouncing phonemes from letters, segmentation may be less important than blending, nonsense word fluency may be less important than blending real words, and sight words are not a key indicator of early word reading fluency.

I pointed out that a large assessment company might have an incentive not to follow the latest research if that meant rapidly changing the nature of the assessments that company produces, since if the company did so, the company's customers might see surprising decreases in student performance and choose a competing product.

I got some pushback on that from folks I trust. Again, I'll emphasize, I'm not a psychometrician. But I'll certainly be curious if real psychometrician-economist double majors choose to pull on this thread.

Regardless, if, in that article, I was pushing on a small brick at the bottom of the education assessment wall, in his latest post, "We Measure Our School Systems the Wrong Way" Tim Daly advocates knocking down the whole wall.

Tim writes:

It's time for a major correction. We should define the success of our education system based on outcomes for 25 year-olds, not 18 year-olds...By capping the responsibility of K-12 systems at the end of high school, we give them a free pass for the dismal life outcomes many students - particularly those from less privileged backgrounds - are experiencing in their 20s.

Instead, Tim argues, we should measure (1) educational attainment, employment with a living wage, civic engagement (ie. voter registration), safety and lawfulness (ie. criminal record).

I disagree. Well not entirely. I do think longitudinal outcome measurement is an important frame for public policy, but I don't think this is the most important problem we face. Not by a long shot.

Implicit in Tim's argument is the assumption that if we had all the data on the outcomes we care about and the political consensus to manage to those outcomes we could work backwards and design our schools to meet those outcomes. I'm skeptical, particularly since we have struggled massively to design our schools to meet the far more concrete outcomes that Tim criticizes--test scores and high school graduation.

Let's say we agreed that the longitudinal outcome we cared about was voter registration (to pick one that Tim suggested). We wouldn't optimize for it by making kindergarteners recite the Pledge to Register as a Voter every morning during circle time. Instead, I think most folks would agree that we'd teach them how to read, since that is a precursor skill to voting that is also age-appropriate.

Tim wants to move the goal line with his suggested outcomes. But we have not proven that we know how to get to the ten yard line, since national 4th grade reading scores are exactly where they were 30 years ago. And we called it a crisis back then.

And if we can't predictably, efficiently and scalably get to the ten yard line, we'll never make it to the goal line, no matter where we choose to draw that ultimate goal.

The good news is that everyone--parents, voters, policy experts, governors believe that we need to get all students reading by third grade. There is no argument about where to draw that line.

We also know, based on decades of research, that beyond first and second grade, learning how to read becomes much harder and more expensive than it is in kindergarten and first grade.

So we should direct every available resource toward teaching children to read in kindergarten and first grade and be open to redesigning staffing, space, and schedules (hello, paraprofessionals!) to making that happen. And then, at long last, our time will become well spent debating whether to measure school districts by life and academic outcomes at 18 or at 25.