A-Level results in England – the pitfalls of an inconsistent approach

In the absence of real exams, awarding A-level results this summer was always going to be a challenge. In their hearts, most would feel that the moderation of teacher assessed grades is a tough but necessary step to ensure fairness. On reflection, most would see doing this via a data led algorithm a useful step towards transparency. Unfortunately the inconsistent application of the selected algorithm has led to sense of lingering unfairness in the overall results.   

Teachers will often be overoptimistic about their student's grade potential for a variety of good (and human) reasons. Moderation downwards of teacher assessed grades is clearly necessary in the interests of equity between schools and to prevent overall grade inflation. In normal times we use public examinations to settle these questions.

In the unique circumstances of the coronavirus crisis, an alternative moderation process has been required. Inevitably it has to be tough and even blunt, so it’s doubly important that it be seen to be evenly applied.

Unfortunately the result of the process adopted in England this summer has been a sharp increase in the number of students from fee-paying independent schools achieving top grades. The equivalent increase at state schools is much smaller.

Ofqual’s algorithm

The grade moderation process adopted by Ofqual is called the Direct Centre Performance model (DCP). Many would see it as a very plausible, logical approach:

• Look at the past grades achieved in each subject by each school (the historic distribution)
• Adjust this for the ability of the current generation of students based on how well they have done in previous GCSE exams compared to their predecessors (the new predicted distribution)
• Ask teachers to rank their pupils in order of ability (a ‘force rank’ of students)
• Combine the teacher's ranking with the predicted distribution to create a simulated mark for each student (an ‘imputed’ mark for each exam)
• Set final grade boundaries at a national level (similar to a conventional year)

Ofqual has used this approach to produce ‘model predicted grades’ (MPGs) as an alternative to the raw ‘centre assessed grades’ (CAGs) submitted by teachers. A high emphasis on statistical rigour has been in place throughout the process. The need for a ‘force rank’ input is a heart rending requirement on teachers, but probably an unavoidable one that we should thank them for producing.

If this approach was applied across all students, the results might indeed be described as tough but transparent and at least arguably fair. It’s still to be hoped that individual high potential candidates impacted by the blunt edges of the moderation can be accommodated by the appeals process or flexibility in university entrance criteria.

A problem of statistics

But of course this process could not be so simply applied. The obvious question is what happens when the number of students in a single subject/school grouping is so small that the proposed modelling process is no longer statistically valid?

Anyone with knowledge of the UK education system will immediately see that this question also lies right across a politically sensitive fault line. State schools teach pupils in relatively large class sizes (where the statistics of MPGs would be defensible). Independent schools often boast of much smaller numbers by subject (where this form of MPGs would be problematic).

Ofqual has chosen to square this circle by the simple expedient of falling back on teacher produced CAGs for small cohorts: above 15 students per school/subject group MPGs are used; below 5 CAGs are used, with a tapered transition in between. This anomaly in approach seems the most likely explanation for the inflation in grades awarded to students from independent schools. Ofqual even discusses the likely problems of this ‘leniency’ in their technical report on the awards process (page 105), but fails to consider any realistic remedies.

A problem of accountability

Inexplicably, the officials and politicians overseeing this do not seem to have spotted in advance the breach of the principle of fairness that the Ofqual process constitutes. Given the possible political damage, this seems more likely due to ineptitude in overseeing a large statistically driven process rather than deliberate calculation.

Unfortunately the mainstream media also seem incapable of asking questions that get to the heart of what has gone wrong. This issue is not that it is unfair to revise teacher assessed grades downwards. The issue is that many independent school assessed grades were not equivalently moderated.

Prime Minister Boris Johnson has insisted that “The exam results we've got today are robust, they're good, they're dependable for employers”. Unfortunately, it’s harder to say that they are fair.

Update and prospects...

Since this article was initially published, the government has announced a U-turn and decided to issue A-level and GCSE grades based on whichever is higher, the CAG or the MPG. At least this removes the injustice created by the failure to equivalently moderate all grades.

Unfortunately the new approach leaves wreckage strewn across the wider educational scene that it will take years to clear away. 40% of A-level results issued this year will now be seen as inflated compared to previous years (though of course we don't know which ones these are). This makes it a great challenge to get the class-of-2020 into the right place for the next steps in their development. Conflicting pressures will be felt across the full spectrum of university, further educational and employment opportunities. The worse outcome is to end-up on a course, or in a job, where you will struggle!

Coming down the line, the class-of-2021 will argue that it's set to have an even rougher deal. More of its actual education has been disrupted rather than just the final exams. Given the great differences in how effectively individual schools have responded, we should probably expect a very unevenly spread 'dip' in actual exam performance in 2021. Worse, these students will have to compete with some of those benefiting from the unmoderated grades of 2020. Trying to define what is 'fair' for next year is an even harder challenge.