Assessment of Professionals

M. Schürch

How do you assess performance?

• How do you currently assess the performance of your residents?

• What standards do you apply?

• What evidence can you provide?

• Is the evidence robust enough?

• Could the resident challenge your assessment?

Outline

• Principles of assessment

• Methods of assessment

• What standards do apply

• Assessment in the new curriculum

• Conclusion

Why assess?

• Provide direction and motivation for future learning: assessment drives learning …

• Screening out those who are incompetent

• Meet public expectations

• Choose applicants for advanced training

What to assess?

• Knowledge

• Skills– technical– non-technical

• Attitude

How to assess?• Here is the problem• Depends on why and what!

Formative vs summative assessment• Formative assessment

– guides future learning– provides reassurance– promotes reflection– shapes values

• Summative assessment– overall jugement of

competence– fitness to practice– qualification for advancement

can: reinforce intrinsic motivationinspire to set higher standards

Formative vs summative assessment• Formative assessment

– guides future learning– provides reassurance– promotes reflection– shapes values

• Summative assessment– overall jugement of

competence– fitness to practice– qualification for advancement

can: reinforce intrinsic motivationinspire to set higher standards

Methods of assessment

• Simulation– standardized patients– computer based simulation

computer programms, high fidelty simulators

• Work-based assessment– case based discussion– mini-clinical evaluation exercise (mCEX)– direct observation of … (DOPS, DOCE)– multisource (360°) assessments

Hierarchy of professionalism

Competent Knowledge

Technical Skills

Non-Technical Skills

Pro-fessional Behaviour

MCQ

VIVA

OSCE

VIVA

MCQ: multiple choice questionsOSCE: objective structured

clinical examinationVIVA: oral exam

Hierarchy of professionalism

Competent Knowledge

Technical Skills

Non-Technical Skills

Pro-fessional Behaviour

mini-CEX

CBD

DOCE

MSF (RET?)

mini-CEX: clinical evaluation exerciseCBD: case based discussionDOCE: direct observation of

clinical encounterMSF: multi source feedback

Criteria to choose methods

• Reliability accurate, reproducible

• Validity measures what it claims to

• Impact on future learning and practice

• Acceptability to learners and facultyfair, feasible

• Costs trainee, institution, society

What standards do apply?

• Limen referenced– Written exam, MCQ– Standardized patients, simulation

• Relying on assessors judgement of competency against expectation at a particular level of training– Workplace based assessment

• Patient difficulty• Faculty stringency• Equivalence of assessments?

Challanges of work based methods• No absolute standards

– most trainees will be satisfactory

• Alternative assessments for those who are unsatisfactory

• Importance of selection of assessors• Equivalence

– not useful for ranking TraineesMultiple assessments by multiple

assessors

Who should assess?

„ … one of the capabilities of a competent individual is the skill of recognising competence in

others.“

Ronald Burnett: The Limits of Competence 1991

Assessment in the new Curriculum

www.sgar-ssar.ch

defines competencies and standards

Assessments in the curriculum Fo

rmati

ve

Su

mm

ati

ve

Formative assessments

DOCE

RET (Resident Evaluation Tool)

• the Resident Evaluation Tool RET (Anästhesie Basel, Abbott) can be used to give Multisource-Feedback

• RET combines items of mini-CEX and ANTS (non technical Skills)

Example of a rating, that needs an action plan

Example of an excellent rating

Conclusion

• Assessment is essential for learning• Good assessment is a challenge

„ … one of the capabilities of a competent individual is the skill of recognising competence in others.“

Ronald Burnett, 1991