Workplace-based assessment in clinical radiology in the UK - a validity study

Abstract

In 2010, the Royal College of Radiologists introduced workplace-based assessments to the postgraduate training pathway for clinical radiologists in the UK. Whilst the system served the purpose of contributing to high-stakes annual judgements about radiology trainees’ progression into subsequent years of training, it was primarily intended to be formative. This study was prompted by an interest in whether the new system fulfilled this formative role. Data collection and analysis spanned the first three years of the new system and followed a multi-methods approach. Descriptive statistical analysis was used to explore important parameters such as the timing and number of assessments undertaken by trainees and assessors. Using the literature and an iterative analysis of a large sample of trainee data, a coding framework for categories of feedback quality enabled assessors’ written comments to be explored using deductive and inductive qualitative analysis, with inferential statistical analysis of coded assessor feedback statements. For example, Ragin’s (1987, 2000, 2008) qualitative comparative analysis, QCA, was used to explore whether the assessments met necessary and/or sufficient conditions for high quality feedback. Pairs of assessor-trainee feedback comments were also analysed to establish whether any dialogic feedback interactions occurred. The study presents evidence that despite its intentions, the new system is generally failing to meet its primary, formative aim. As a consequence, the influence of negative washback on assessment practice was reflected in a number of findings. For example, there was evidence of trainees taking an instrumental approach to the assessments, undertaking only the prescribed minimum of assessments or completing assessments in the later stages of placements. Combined with evidence of retrospective assessment, i.e. after completion of the placements, the observed patterns of assessment over the three years are consistent with a box-ticking approach. This study explores the contextual policy and practice dimensions underpinning these and related findings and discusses the implications and recommendations for future arrangements.