Rendering transparency to ranking in educational assessment via Bayesian comparative judgement

Gray, A ORCID: 0000-0002-1150-2052, Rahat, A, Lindsay, S, Pearson, J and Crick, T (2026) 'Rendering transparency to ranking in educational assessment via Bayesian comparative judgement.' Review of Education. (Forthcoming)

Text 17675.pdf Restricted to Repository staff only | Request more information

Abstract

Transparency in educational assessment has become an increasingly pressing concern, particularly in the aftermath of the pandemic, as institutions seek more equitable, robust, and defensible methods of evaluating student work. Comparative Judgement (CJ) has gained traction as a promising alternative to traditional rubric-based marking. However, despite its potential, CJ has been criticised for its perceived opacity, particularly in high-stakes contexts where fairness, auditability, and trust are paramount. This paper investigates whether Bayesian Comparative Judgement (BCJ), which applies Bayesian statistical methods to CJ, can enhance transparency by making the judgement process more structured, interpretable, and accountable. BCJ introduces a probabilistic framework that incorporates prior knowledge and updates beliefs based on new evidence, allowing for quantification of uncertainty and clearer justification of ranking decisions. It enables greater insight into the consistency of judgements and highlights areas of disagreement among assessors. We also evaluate a recent multi-criteria extension of BCJ that models each learning outcome (LO) separately, mirroring the structure of rubric-based assessment while retaining the efficiency and comparative strengths of CJ. This approach supports the generation of both outcome-specific and holistic rankings, offering detailed feedback without sacrificing the coherence of the overall evaluation. Using real-world assessment data from a UK higher education course involving experienced professional markers, we demonstrate the application of BCJ and multi-criteria BCJ in practice. Our analysis highlights how these models can provide rigorous, transparent insights into the reasoning behind both individual and collective rankings. We also discuss how BCJ supports external validation of assessment outcomes. Finally, through semi-structured discussions with participant markers and expert CJ practitioners, we qualitatively assess the perceived transparency and usefulness of BCJ in authentic settings, particularly where high-stakes decisions are made. We conclude by outlining the benefits and limitations of BCJ and its relevance across varied educational contexts.

Item Type:	Article
UN SDGs:	Goal 4: Quality Education
Keywords:	comparative judgement, assessment, transparency, Bayesian statistics, machine learning, active learning, higher education
Subjects:	L Education > L Education (General) T Technology > T Technology (General)
Divisions:	Bath School of Design
Date Deposited:	02 Apr 2026 09:58
Last Modified:	02 Apr 2026 09:58
ISSN:	2049-6613
URN:	https://researchspace.bathspa.ac.uk/id/eprint/17675

	Request a change to this item or report an issue
	Update item (repository staff only)