Why the big ed reformers are wrong

(Originally published on edcontrarian at https://edcontrarian.blogspot.com/2022/08/why-big-reformers-however-well.html)

A good friend recently sent me an article that outlined the three big ideas in the idea of education reform as championed by Chester Finn, Frederick Hess, and Michael Petrelli. Petrelli Sums up their goals this way: “First, that the nation’s foremost education objective should be closing racial and economic achievement gaps. Second, that excellent schools can overcome the challenges of poverty. And third, that external pressure and tough accountability are critical components of helping school systems improve.”

That three-legged stool has been the basis for both federal and state policy since at least 1990, and possibly earlier.?

The first two legs are of course entirely reasonable. The future of our country will be much brighter when both racial and economic achievement gaps are closed, and poverty remains the greatest barrier to the American dream.?

But that third leg, the idea that external pressures and tough accountability are how the first two legs may be put in place, that I take issue with. So much so I would argue that the existence of that third leg as they define it will prevent the first two from ever occurring.

Finn, Hess, and Petrelli readily admit that the first two legs of the stool have not yet come into view, but over the past two decades have continued to act as if their theory of action for the third leg will one day prevail. That's a remarkable amount of stubbornness that doesn't exist in any other field or profession. There is no such thing in any other profession as a decades-old theory that failed to produce results and was still being defended for the simple fact that logic and practical thinking would have long ago dismissed it. And yet here we remain.

A big part of their theory includes the accurate assertion that prior to the existence of their third leg education was largely measured on inputs, which failed to signal anything about the effectiveness of the educational effort. That was appropriately seen as an issue given the importance of the educational process and the investment it required.?

But the reaction of Finn et. al. to this day leaves me a bit stumped given the thoughtless way they swung the pendulum from accounting for inputs to accounting for outputs. I find it thoughtless because multiple professions and organizations faced with the same problem had already produced models and examples that could have informed the process, models that moved to focus on the effects organizations were having and how best to observe them.

Those organizations focused on what we now in hindsight call stakeholder benefit. Their discovery was that the effectiveness of an organization is best assessed through the degree of stakeholder benefit it provides and the degree of alignment those benefits have with stakeholder needs—a worthwhile output if ever there was one.

But organizations that were interested in addressing their organizational accountability through the lens of stakeholder benefit occupied a very different place in the psyche of Americans. Businesses, hospitals, law offices, accountants, engineers, nonprofits, etc., all had two things that education did not: the public’s trust and an identified list of benefits that those engaging with each profession or institution knew and understood.?

The reasons behind the lack of public trust have filled multiple volumes already and could fill many more, but regardless their existence and history are now generations old. Here I only need to deal with the effect and not the cause, and that effect is clear: “A Nation at Risk,” “Why Johnny Can't Read,” “a rising tide of mediocrity,” “NAEP scores flat,” “American students behind their international peers,” etc. Whatever the reason behind the headlines the assumption has long been that the American public education system is always and forever on the cusp of failure, if it hasn't failed already. “Trust” and “public education” are rarely seen in the same sentence unless the word “lack” is too.

The reasons behind the failure to understand the stakeholder benefit for schools seem much simpler and much cruder: no one seems to have looked. Perhaps it was the lack of trust in public schools or some other reason, but the question as to whether a benefits-based model would work in schools (it does—we’re doing it today in lots of places) was never answered because it was never asked.?

Instead, what is arguably the strangest and most anomalous of all the decisions made in the history of organizational accountability came to be: the selection of a narrow, esoteric research instrument as the basis for an outcome metric, or as you know it, a standardized test score, as a proxy for the entire educational effort.

What, it must be asked, is the benefit to a student in a standardized test score? The answer is quite simple: nothing. Standardized test scores provide an estimate as to where a student is at a moment in time but have no capacity or ability to indicate the causes that led to that moment. Great instruction would be a student benefit, but a test score, no matter how high or how low, on its own has no ability to signal whether the benefit occurred. Engaging content in a deep way, getting ready for life after school, and having a solid foundation in academic basics are also benefits that are critical to student success and none of them are in a test score.

To repeat myself, all that is in a standardized test score is an estimate of where a student is in an academic area lifetime to date without any reference as to what brought a student to that point. As part of a larger data set this information can be useful to researchers, but as it signals nothing about the effectiveness of a school it would be pointless to include it in an accountability system where the point was to understand effectiveness.

That least logical thing that could ever be included in any accountability system is an outcome indicator that can’t tell you anything about the effectiveness of the system.

Not only did the Finn, Hess, Petrelli, and their counterparts select as their primary effectiveness metric an instrument that can't say anything about effectiveness, but they then built the third leg of their stool by assigning sanctions to schools that did not comply with some version of what the test results should look like. I need to say that again to make clear just how ridiculous that was: the third leg of their stool punishes schools whose test scores do not match a preconceived notion of what the test scores should be, even though those test scores can say nothing about the effectiveness of the schools that students attend.

You may need to read that several times to catch the absurdity of it. I've written something similar a hundred times and every time I do I have to reread it because the ridiculousness of it amazes me. It means that the billions invested in the accountability systems across the country leave us blind as to whether schools are or are not effective. That is stunning to say the least.

But Finn et. al. took it a step further. They and many like them didn’t trust schools and so didn’t trust educators with the organizational accountability function. Rather, they wanted it imposed from outside: they wanted to layer onto schools and educators an authoritative accountability system on schools demand that schools comply with it.

That word compliance is important, because therein lies the second (third?) part of their ridiculous error: compliance has never, can never, and will never signal effectiveness, nor can or will it drive organizations to be more effective than they were in the past. Period.

If you are an educator you work in a school or a central office where you and all your colleagues have complied with the requirements for obtaining an educator certification. Does perfect compliance also mean you are an effective school or organization? Of course not. It means you are all certified. Effectiveness is determined elsewhere.

Or perhaps a bit more to the point, virtually every driver you encountered on your way to work this morning has complied with the requirements to obtain a driver’s license. Does that also mean that everyone you encountered this morning was an effective driver? I’ll let you answer that.

Thus, the third leg of the stool is a mess. It substitutes compliance for effectiveness, which is always a mistake as they are not the same thing—not even close. That on its own is enough to doom their system. But then those that advocate for such a thing insist on pretending standardized test scores can signal effectiveness, even though they cannot. That too on its own is enough to doom the system.

If two wrongs could make a right we’d be in great shape, but alas, they cannot.

Here’s where that leaves us: the goals represented in the first two legs of the stool, which are critically important, will never be met so long as the third leg of the stool is as screwed up as it is. Compliance and a standardized test score are equally wrong methodologies for understanding and promoting effectiveness, and so much worse when combined. We prolong the day of getting it right so long as we allow the thinking of Finn et. al. to win. It's wrong, more people should say so, and every school leader should do something about it.