27
Slide 1 Authentic Performance Authentic Performance Assessment Assessment Ellen J. Lehning, Ph.D. Ellen J. Lehning, Ph.D. Department of Anesthesiology Department of Anesthesiology

Authentic Performance Assessment

Embed Size (px)

DESCRIPTION

Authentic Performance Assessment. Ellen J. Lehning, Ph.D. Department of Anesthesiology. Objectives. Define clinical competence and authentic performance Design an authentic performance assessment Implement an authentic performance assessment. What is the First Principle of Education?. - PowerPoint PPT Presentation

Citation preview

Page 1: Authentic Performance Assessment

Slide 1

Authentic Authentic Performance Performance AssessmentAssessment

Ellen J. Lehning, Ph.D.Ellen J. Lehning, Ph.D.Department of AnesthesiologyDepartment of Anesthesiology

Page 2: Authentic Performance Assessment

Slide 2

ObjectivesObjectives

Define clinical competence and Define clinical competence and authentic performanceauthentic performance

Design an authentic performance Design an authentic performance assessmentassessment

Implement an authentic Implement an authentic performance assessmentperformance assessment

Page 3: Authentic Performance Assessment

Slide 3

Teach to and Test Desired Teach to and Test Desired Endpoints:Endpoints:

What is the First Principle of What is the First Principle of Education?Education?

Educational OutcomesEducational Outcomes

Page 4: Authentic Performance Assessment

Slide 4

Pass Oral BoardsPass Oral Boards

Pass Written Pass Written BoardsBoards

Achieve Clinical Achieve Clinical CompetenceCompetence

Desired Outcome: Desired Outcome: Assess: Assess:

Page 5: Authentic Performance Assessment

Slide 5

What is Clinical Competence?What is Clinical Competence?

““IIntegrationntegration of knowledge, skills of knowledge, skills and attitudes.”and attitudes.”

Rice et al., Competency-based objectives for clinical training. Can. J. Med. Technol. 57:136, 1995

Page 6: Authentic Performance Assessment

Slide 6

What is Clinical Competence?What is Clinical Competence?

““The habitual and judicious use The habitual and judicious use of communication, knowledge, of communication, knowledge,

technical skills, clinical technical skills, clinical reasoning, emotions, values, reasoning, emotions, values,

and reflection in daily practice and reflection in daily practice for the benefit of the individual for the benefit of the individual and community being servedand community being served.”.”

Epstein and Hundert, Designing and assessing professional competence. JAMA 287:226, 2002

Page 7: Authentic Performance Assessment

Slide 7

What is Clinical Competence?What is Clinical Competence?

Miller, The assessment of clinical skills/competencies/performance. Acad. Med. 65:563-567, 1990.

Knows

Knows how

Shows how

Does

Miller’s Pyramid of Competence

Page 8: Authentic Performance Assessment

Slide 8

What is Clinical Competence?What is Clinical Competence?

““Competence . . . means being Competence . . . means being able to function in able to function in contextcontext.”.”

Chambers and Gerrow, Manual for Developing and Formatting Competency Statements. J. Dental Educ. 58:361, 1994.

Page 9: Authentic Performance Assessment

Slide 9

How is Clinical Competence How is Clinical Competence Assessed?Assessed?

““Competence is concerned with what Competence is concerned with what people people cancan do . . ., that is, potential.” do . . ., that is, potential.”

While, Competence versus performance: which is more important? J. Adv. Nurs. 20:525, 1994.

““Performance is concerned with what Performance is concerned with what people people dodo . . ., that is, what is actually . . ., that is, what is actually

done in the real-life context.”done in the real-life context.”

Competence is inferred from performance!Competence is inferred from performance!

Page 10: Authentic Performance Assessment

Slide 10

What is Authentic Performance What is Authentic Performance Assessment?Assessment?

Performance can be Performance can be simulatedsimulated or or authenticauthentic

SimulatedSimulated performance assessment uses performance assessment uses an an artificialartificial context: context:

SPs, OSCEs, Long Case - Standardized or SPs, OSCEs, Long Case - Standardized or UnstandardizedUnstandardized

AuthenticAuthentic performance assessment uses performance assessment uses a a realreal context: context:

Direct observation of a resident caring for Direct observation of a resident caring for a real patient with a real problem in a a real patient with a real problem in a

real setting – Unstandardized!real setting – Unstandardized!

Page 11: Authentic Performance Assessment

Slide 11

Why Use Authentic Why Use Authentic Performance?Performance?““Testing should be as close as Testing should be as close as

possible to the situation in which possible to the situation in which one attacks the problem.”one attacks the problem.”

““Ill-structured problems are not found Ill-structured problems are not found in simulated and/or standardized in simulated and/or standardized

tests.”tests.”

““The variation inherent in The variation inherent in professional practice will always professional practice will always elude capture by a set of rules.”elude capture by a set of rules.”

Wiggins, Assessing Student Performance: Exploring the Purpose and Limits of Testing, Jossey-Bass, Inc. 1993

Page 12: Authentic Performance Assessment

Slide 12

How is it Done?How is it Done?

““There are few validated strategies to There are few validated strategies to assess actual clinical practice.”assess actual clinical practice.”

““Assessment at the apex of Miller’s Assessment at the apex of Miller’s pyramid, the does, is the international pyramid, the does, is the international

challenge of the century for all challenge of the century for all involved in clinical competence involved in clinical competence

testing.”testing.”

- Ronald Epstein

- Val Wass

Page 13: Authentic Performance Assessment

Slide 13

Design Step 1 – Select Design Step 1 – Select CompetenciesCompetencies

Breakdown ACGME’s six global Breakdown ACGME’s six global competencies into specific competenciescompetencies into specific competencies

Technical skills, case management Technical skills, case management skills, clinical decision-making skills, skills, clinical decision-making skills, etc.etc.

Avoid too broadAvoid too broad

Avoid too specificAvoid too specific

Appropriate for the resident’s training Appropriate for the resident’s training levellevel

Sequenced appropriately over the Sequenced appropriately over the training periodtraining period

Page 14: Authentic Performance Assessment

Slide 14

Design Step 1 – Select Design Step 1 – Select CompetenciesCompetencies

Write a competency statement:Write a competency statement:

Chambers and Gerrow, Manual for developing and formatting competency statements. J. Dental Educ. 58:361, 1994.

Verb Direct Object Qualifying conditions

Page 15: Authentic Performance Assessment

Slide 15

Example Competency StatementExample Competency Statement

Induce anesthesia for an ASA I-II Induce anesthesia for an ASA I-II patient by the end of the three patient by the end of the three

month periodmonth period

Page 16: Authentic Performance Assessment

Slide 16

Example Competency Example Competency StatementsStatements The Global Communication Skills CompetencyThe Global Communication Skills Competency

Communicate accurately, efficiently and supportively Communicate accurately, efficiently and supportively with the patient, patient’s family and the health care with the patient, patient’s family and the health care teamteam

Specific Communication Skill CompetencySpecific Communication Skill Competency

Build a Relationship (Rapport Building)

Component Skills used to Build a Relationship

Establish initial rapportEstablish initial rapport Support SupportEmpathy PartnershipReflection Legitimation

Respect Apology

Page 17: Authentic Performance Assessment

Slide 17

The Performance Rating ProcessThe Performance Rating Process

Specific performances are judged by expert Specific performances are judged by expert

rater(s) who synthesize multiple impressions in rater(s) who synthesize multiple impressions in

comparison to criteria given in a rating task comparison to criteria given in a rating task

and filtered through the experience of the and filtered through the experience of the

rater.rater.

Chambers, A primer on competency-based evaluation. J. Dent. Educ. 61:651, 1997.

Specific performance(s)Specific performance(s)

JudgmentJudgment

ExpertExpert

Criteria (Criterion referencing)Criteria (Criterion referencing)

Page 18: Authentic Performance Assessment

Slide 18

Design Step 2 – Select SamplingDesign Step 2 – Select Sampling

Which performances or tasks will be observed?

How many observations?

When?

Page 19: Authentic Performance Assessment

Slide 19

Design Step 3 – Select RatersDesign Step 3 – Select Raters

Raters are qualified individuals who have an Raters are qualified individuals who have an

opportunity to observe and draw conclusions opportunity to observe and draw conclusions

about residents – they judge the presence and about residents – they judge the presence and

the quality of the competency to be assessedthe quality of the competency to be assessed

How many ratersHow many raters

AttendingsAttendings

Other professional staffOther professional staff

PatientsPatients

SelfSelf

Page 20: Authentic Performance Assessment

Slide 20

Design Step 4 – Select Rating Design Step 4 – Select Rating CriteriaCriteria

Rating criteria can either be Rating criteria can either be checklistschecklists or or rubricsrubrics

ChecklistsChecklists Presence or absence of a competency or Presence or absence of a competency or

of the components of a competencyof the components of a competency Do not judge qualityDo not judge quality May be appropriate for technical May be appropriate for technical

procedures or beginning level procedures or beginning level

competenciescompetencies May not capture sophistication of May not capture sophistication of

complex performances:complex performances:Hodges, B., Regehr, G., McNaughton, N., Tiberius, R., and Hodges, B., Regehr, G., McNaughton, N., Tiberius, R., and Hanson, M. 1999. OSCE checklists do not capture increasing Hanson, M. 1999. OSCE checklists do not capture increasing levels of expertise. Acad. Med. 74:1129-1134.levels of expertise. Acad. Med. 74:1129-1134.

Page 21: Authentic Performance Assessment

Slide 21

Example ChecklistExample Checklist

Yes NoCompetency:Built a relationship?

Specific Skills:Established initial rapport? Employed Legitimation? Provided support? Demonstrated empathy? Demonstrated legitimation? Demonstrated apology? Showed respect?

Page 22: Authentic Performance Assessment

Slide 22

Design Step 4 – Select Rating Design Step 4 – Select Rating CriteriaCriteria

Rubrics Rubrics Guidelines, rules, or principles by Guidelines, rules, or principles by

which performances are judgedwhich performances are judged Reflect best thinking as to what Reflect best thinking as to what

constitutes a good performanceconstitutes a good performance Contain multiple quality levelsContain multiple quality levels All rating levels must be written-down, All rating levels must be written-down,

defined and describeddefined and described Best rubrics contain anchor(s) to Best rubrics contain anchor(s) to

illustrate the different points on the illustrate the different points on the

quality scalequality scaleArter & McTighe, Scoring Rubrics in the Classroom, Corwin Press, Inc. 2001

Page 23: Authentic Performance Assessment

Slide 23

Example RubricExample RubricExcellent – Smooth and efficient. Able to use knowledge, judgment and skills to adjust management appropriately to the specific patient and operative procedure.Competent – Lacks smoothness and efficiency but is able to use knowledge, judgment and skills to adjust management appropriately to the specific patient and operative procedure.Beginner – Lacks smoothness and efficiency. Able to manage the case but exhibits limited use of personal judgment and responsiveness to the specifics of the patient and operative procedure. Requires some limited coaching or attending intervention.Novice – Can only manage the case with extensive coaching and attending intervention.

Page 24: Authentic Performance Assessment

Slide 24

What are the Desirable What are the Desirable Characteristics of Performance Characteristics of Performance

Assessment?Assessment?Face validityFace validity

Low CostLow Cost

FeasibleFeasible

Non-reactiveNon-reactive

GeneralizableGeneralizable

ValidValid ReliableReliable

Page 25: Authentic Performance Assessment

Slide 25

Psychometric IssuesPsychometric Issues Write good rubricsWrite good rubrics

Train the ratersTrain the raters– Only need 1 rater/performanceOnly need 1 rater/performance

Sample over a broad array of cases/patientsSample over a broad array of cases/patients

Our intercase agreements range from 0.2 – 0.9Our intercase agreements range from 0.2 – 0.9

Wass et al., Generalizability in range of the Wass et al., Generalizability in range of the

OSCEOSCE

ACGME preliminary data shows adequate ACGME preliminary data shows adequate

generalizability, validity and reliabilitygeneralizability, validity and reliability

Page 26: Authentic Performance Assessment

Slide 26

Authentic Performances vs. Authentic Performances vs. Authentic ProductsAuthentic Products

Authentic Performances

Patient Care

Professionalism

Communication Skills

Authentic Products

Patient Care

Ethics

PBLI

Systems-Based

Assessing an:

Authentic Performance

versus the

Product of an Authentic Performance

Page 27: Authentic Performance Assessment

Slide 27

Why Focus on Authentic Why Focus on Authentic Performance Assessment?Performance Assessment?

ACGME outcomes mandateACGME outcomes mandate

Current assessment is rudimentaryCurrent assessment is rudimentary

AccountabilityAccountability

Murray et al., The accountability of clinical education: its definition and assessment. Med. Educ. 34:871, 2000

Bordage et al., Education in ambulatory settings: Developing valid measures of educational outcomes, and other research priorities. Acad. Med. 73:743, 1998