The Royal College of Pathologists – the overall

assessment system

Dr. Trevor GrayDirector of Examinations and

Assessment

The Assessment System

Overview of PMETB and assessment strategy

How WBA fits with other assessments Advantages and disadvantages of WBA

compared with other assessments Local organisation of WBA and how it is

administered by the College Future prospects

Overview PMETB (and the MMC process) has laid down

significant challenges to medical postgraduate training (elegantly dissected in the Tooke report)

PMETB principle is that training and assessment should be much more integrated – an assessment system

Workplace-based assessment is an essential element of the assessment of trainees undertaking specialist medical training

The specialty curriculum, training, exams and assessments are core College business (in the charter)

The College’s aim is to develop a validated assessment system which is fit for purpose

PMETB strategy on assessment

PMETB principles for assessment 2004 Nine principles for assessment systems Now backed up with technical documentation

Principles based on “overarching strategy” Workplace-based assessment Examinations of knowledge Examinations of Clinical Skills(“Workplace-based assessment” PMETB, Jan 2005)

Specialty curriculum and assessment system now approved by PMETB

(i.e. match strategy)

Workplace-based Assessment

Assessment based on performance in the workplace and measured there

Assessed locally by supervisors as well as staff of all grades in contact with the trainee

Assessment occurs during normal work immediately followed by feedback so trainee receives continuous feedback on their routine performance

Results put into trainee portfolio for their annual review and hence important for trainee development and ultimately their employment contract

College Assessment Systemapproved by PMETB Workplace-based assessment Year 1 assessment MRCPath Part 1 (test of knowledge) MRCPath Part 2 (test of skills and applied

knowledge)

Feedback on regular basis (WBA) Educational supervisor’s report ARCP (Annual Review of Competency

Progression)

The College Assessment system: Miller’s pyramid

Knows

Knows how

Shows

Does

MRCPath Part 1

MRCPath Part 2 Practical

WBA e.g. Mini-CEX, DOPS

WBA - MSF

Value of multiple assessments Key is several

modes of assessment – triangulation

A single exam system (MRCPath) does not provide adequate coverage of clinical practice

WBA is also not sufficient in itself

(Assessment System Guidance – PMETB)

Examinations and Assessment

Examinations are simply one particular form of assessment

Assessments are: formative (involve feedback and are intended

to guide training) or summative (assess whether reach given

standard) e.g. driving test, MRCPath All assessments should be “blueprinted”

to the curriculum to demonstrate that the most appropriate method is being used

How Workplace-based Assessment fits with MRCPath

Workplace-based assessment depends on assessment by local staff, examinations rely on assessment by those who have had no contact with the trainee

Examinations provide an independent assessment of knowledge and practical skills to national standards

Workplace-based assessment provides a local insight into candidates’ performance in routine tasks and interaction with staff and patients

How Workplace-based Assessment fits with MRCPath

Workplace-based assessment is a local process governed by deaneries albeit to PMETB standards

Examinations are organised nationally by the College

Comments, complaints and appeals for WBA go to deanery, for exams go to College

Workplace-based Assessment – Advantages

Measurement occurs in more authentic environment than the simulated one of the examination room

Trainees can take assessment in their own time and are not limited to exam timetable

The assessment tools are set up to provide extensive feedback to the candidate (formative)

Use of computerised forms makes them easy to record

Workplace-based Assessment

– Disadvantages To avoid time-consuming “busy”

assessment forms, the assessments are often simplified if not simplistic

A robust rating can only occur if there are a number of independent observers – which can be difficult in small specialties

They have only been validated in a number of clinical settings but not in all pathology settings

Computerised forms can over-simplify complex judgements and may not reward excellence appropriately

Year 1 Assessment (OSPE)

Developed to provide evidence of progression in run-through training in the absence of further recruitment and selection

Designed to assess progress in first year and be taken after 7 months training

It is a test to “indicate the capability and potential of a trainee through a test of applied knowledge and skills relevant to the year 1 curriculum”

F1 F2 ST1 ST2 ST3 ST4 ST5

WBA

Y1A

MRCPath

CCT Part 1 Part 2

MSF

Workplace-based assessment

ARCP

Foundation Stage A Stage B

Stage C

Stage D

Training and Assessment Plan

The training environment Workplace-based assessment utilises the

skills of a wide variety of professional groups as assessors: medical, nursing, BMS, clinical scientist, ancillary staff

The College cannot train or quality assure all assessors

Training centres are recommended to develop a training environment where those with the skills needed for assessment are recognised and training is cascaded from those attending events such as this

Workplace-based Assessments and the College

WBA is part of assessment system for Specialist Trainees from 1st August 2007

Assessments need to be undertaken from early in training and recorded for ARCP

2007 – Launch of tools and paper-based assessment forms

College is researching web-based systems similar to the ones used in foundation

Workplace-based Assessments and the College

April 2008 – pilot web-based system By August 2008 – launch of web-based

system Use data generated by all assessments

including MSF and OSPE for evaluation of assessment strategy

Ensure compliance with MMC Gold Guide and planned ARCP

Vision for College Assessment system College will have integrated specialty

curriculum and assessment system endorsed by PMETB

College will have fully validated workplace-based assessment tools delivering feedback on progress to trainees throughout course

Valid and reliable tests will underpin MRCPath examination – Part 1 test of knowledge, Part 2 test of applied knowledge and skills

Factors affecting future assessment strategy

Selection into specialty may not be an annual process in future compromising ability to deliver year 1 assessment (except maybe in Histopathology)

Multiple recruitment may require more flexibility in arrangements for WBA

Selection into specialty may involve a national set of tests (Tooke and Academy) displacing Year 1 assessment

Factors affecting future assessment strategy

Workplace-based assessment is likely to continue but its use must be evidence-based

Workplace-based assessment is not cast in stone – work in progress and subject to improvement following evaluation

Place of individual components of assessment in MRCPath, Year 1 assessment and workplace-based assessment may change

Summary The College’s education and training, assessment

and examinations departments have worked hard to produce curricula and assessments acceptable to PMETB

This training day is the first for workplace-based assessment and is intended to assist training centres to develop assessment locally

Workplace-based assessment will be subject to a full programme of validation and optimisation to ensure fitness for purpose

The College is looking to source a web-based system to support workplace-based assessment

Glossary

ARCP Annual Review of Competence ProgressionCbD Case-based Discussion DOPS Direct observation of practical skillsECE Evaluation of clinical events MSF Multi-source feedback (ePATH-SPRAT)Mini-CEX Mini clinical evaluation exercise OSPE Objective structured practical examination PMETB Postgraduate medical education and training boardWBA Workplace-based assessment