An Argument in Favour of Results by Algorithm

By Karen Lancaster

***

 

This is the first post in a PESGB Blog series focussed on educational assessment and the UK Government’s handling of England’s 2020 national exams (GCSEs and A Levels) in the wake of Covid-19.

 

***

 

In 2020, the UK government cancelled all A-Level exams due to the coronavirus pandemic. Students’ results were then calculated using teacher-predicted grades and altered by an algorithm. The use of the algorithm meant that 40% of students received results lower than their teacher-predicted grade. This was met with huge protests and resulted in a government U-turn a few days after Results Day, with students instead being awarded the grade predicted by their teacher. The consensus seemed to be that this was the fairest outcome.

I challenge this idea. Awarding teacher-predicted grades is unfair to the non-2020 cohorts. Altering predictions with the use of an algorithm – but not the algorithm that was actually used – would have been fairer.

When it became clear that exams would not take place in 2020, teachers were asked to provide predicted grades for their students. Having had 13 years’ experience of teaching A-Levels (and predicting A-Level grades) I can say with confidence: predicted grades consist of optimism and guesswork. This is acceptable in normal years, but 2020 was not a normal year, because these optimistic guesses ended up actually being awarded to students!

Optimism is endemic to the A-Level grade-predicting system. It is common practice among A-Level teachers to err on the side of generosity. If there’s a student who generally gets Cs on her work, but occasionally gets Bs and one time she got an A, then her predicted grade is a B or even an A: certainly not a C. Of course, in the exam, she is most likely to achieve a C.

Based on anecdotal evidence from teacher-friends, and my own experience, I can say that generosity in predicted grades is encouraged and sometimes even enforced by team leaders and headteachers. There is often huge pressure to be generous in grade predictions, lest teachers cause instant rejections from a student’s preferred university.

My experience is supported by UCAS statistics, which show that it is perfectly normal for students’ actual grades to be lower – sometimes significantly lower – than their teacher-predicted grades. In fact, in 2019, just 21% of 18-year-olds met or exceeded their predicted A-Level grades (UCAS 2019); indeed, 43% of students achieved grades which were three or more grades lower than their predicted grades (UCAS 2019, 4). The year 2019 was not an anomaly; similar patterns have occurred every year in the past decade (see UCAS’s published data here).

This over-prediction of grades doesn’t matter much in normal years, because it is actual grades which count, not predicted grades. In every year prior to 2020, students’ predicted grades gave a very rough idea of the sorts of grades a student might be able to achieve on a good day, but it was their actual performance in the examinations which determined their A-Level grades. A system where over-prediction is endemic is OK for normal years, but awarding these overly-ambitious predictions to students simply because there’s a pandemic happening is absolutely ludicrous and fundamentally unfair to all previous cohorts who had to sit exams for their A-Levels. In every year-group prior to 2020, only around a fifth of students achieved as highly as their teachers had predicted, but after the government U-turn in 2020, all students ‘achieved’ their predicted grades (or higher). This resulted in a huge leap in the A-Level grades awarded in 2020 compared to those awarded in previous years. It is highly unlikely that this leap in results is due to the class of 2020 simply being so much better than all other previous cohorts: it seems far more probable that the awarding of overly-generous predicted grades is the cause of this phenomenon.

(Image reproduced from Nuffield Foundation’s Education Data Lab)

 

To correct for this phenomenon of overly-ambitious teacher predictions, an algorithm should have been used – but not the algorithm which was used in 2020, which disadvantaged those in large class sizes or historically poor-performing schools; clearly that was an inherently unfair algorithm. But abandoning algorithms altogether amounts to throwing the baby out with the bathwater.

A more appropriate algorithm should have accounted for the fact that teachers tend to over-predict grades, and reduced grades down by the same amount as previous years. In other words, if A-Level results are generally 25% lower than predicted grades, then predicted grades needed to be reduced by 25% in 2020. This would have been fairest to conduct at the level of individual teachers, but would also have been reasonably fair to conduct it at an institutional level, since it is generally school leaders who instruct individual teachers on how generous to be with their predictions. The algorithm could also have corrected for the sorts of students who tend to achieve or exceed their predicted grades.

Perhaps the algorithm I suggest above is not nuanced enough, but it should have been feasible for data analysts and educational experts to work together to sort out some algorithm which ensured that students’ results reflected the fact that teachers almost always over-predict A-Level results.

A further benefit for the class of 2020 is that if the original algorithm used by the government had generated a result higher than the teacher-predicted grade, those students were allowed to keep their algorithm-generated grade even after the U-turn. This resulted in maximal inflation, with every single student in the country achieving or bettering their teacher-predicted grades.

In the end, the class of 2020 have turned out to be a very privileged few, being awarded the optimistic predictions of their teachers, or having that prediction inflated by an algorithm. It is painfully unfair for previous cohorts and future cohorts alike that in 2020, A-Levels were dished out so lavishly, while all other cohorts must sit exams which, 80% of the time, turn out results lower than teacher-predicted grades.

 

References

UCAS (2019) ‘Chapter 8: Qualification,’ in 2019 End of Cycle Report. All chapters available online here. References in text linked to Chapter 8.

 

This post can be downloaded as a PDF.

About the Author

Karen Lancaster

PhD student, University of Nottingham

Karen is a PhD Philosophy student at the University of Nottingham. Her thesis examines some of the ethical issues surrounding the use of care robots in residential homes for the elderly, including whether a robot can care, and how robots should conceive of consent. She is interested more broadly in social, political, and moral philosophy, and philosophical issues arising from our use of emerging technologies. Prior to PhD study, Karen taught A-Level Philosophy and Sociology for 13 years.

Website: https://karenlancaster.weebly.com/


By this Author