Resident Self-Assessment in the Milestones Era: A Win-Win Approach to Resident Professional

Development and Program Improvement

APPD Meeting March 28, 2015

Presenters Kim Gifford Children’s Hospital at Dartmouth Franklin Trimm University of Southern Alabama Bridget Oliveri Children’s Hospital at Dartmouth Susan Guralnick Winthrop University Hospital Ann Burke Wright State University No conflicts of interest to report

Objectives Participants will be able to:

• Prepare residents to perform milestones-based self-assessment

• Integrate resident self-assessment into the process of clinical competence assessment

• Use resident self-assessment and clinical competency committee assessments to promote learner development

• Use resident self-assessment and clinical competency committee assessment for program improvement

• Create a plan to utilize milestones-based assessments for learner development and/or program improvement for use at their home institution

Self-assessment Literature Review

Self-assessment does not correlate well with external absolute measures of performance (Davis et al 2006)

Slowing Down When You Should

Self-assessment of strengths and weaknesses may be more accurate (Lynn et al 2006)

Learners often “know when to look it up” (Eva & Regehr 2007)

Self-Assessment Can be Calibrated

Martin et al 1998 Ward et al 2003 Colthart et al 2008

Milestones as a Roadmap (Schumacher et al 2013)

Self-Assessment Exercise Assess your own level on the Milestones self-assessment sheet provided.

Discuss the self-assessment experience with your table

• Which competencies are easier/harder to self-assess?

• What external data could you seek to improve the accuracy of your own self-assessment?

• What guidance should (or do) you give your residents to complete a milestones-based self-assessment?

Collection and Synthesis of Data to Determine Clinical Competence

Synthesize into Milestone Levels

-Portfolios -OSCEs

-Direct Observation -Case Logs

-Global -3600

Triangulation improves clarity

Self-assessment recommended for inclusion in CCC review AHME Teleconference “Clinical Competency Committee: The ABC’s of CCC’s”- 10/7/14

Validity of Resident Self-Assessment Using the Pediatric Milestones:

How Well Does Pediatric Resident Self-Assessment Match External Assessment?

Su-Ting Li, MD, MPH; Kimberly Gifford, MD; Daniel Tancredi, PhD; Alan Schwartz, PhD; Ann Burke, MD; Ann Guillot, MD; Susan Guralnick, MD; John Mahan, MD;

R. Franklin Trimm, MD For APPD LEARN validity of resident self-assessment using

pediatric milestones group

Method Prospective multi-institutional study Association of Pediatric Program Directors Longitudinal Educational Assessment Research Network (APPD-LEARN)

IRB approval through UC Davis, AAP, and each individual study site.

Comparison of: • Residents self-assessed milestones • CCCs determined resident milestones

Analysis

Univariate and bivariate analyses for mean milestones levels by year and competency Pearson correlation coefficients (PCC) calculated between self-assessment and CCC assessment scores for each learner

Results

47 pediatric residency programs (25% of all pediatric programs)

1741 residents

Resident and CCC Assessment of Competencies by Level of Training

Results • Highest PCC for patient care • Lowest PCC for professionalism and PBLI 4

PCC improved from 1st to 2nd observation period Higher # missing CCC-assessments for: • Practice-based learning and improvement • Professionalism • Systems-based practice

Individual Activity

Using the index cards on your table

On side 1 - How could you use learner self- assessment and CCC assessment to promote learner development ? On side 2– How you might use this data for program improvement?

Learner Development Discussion

Identify ways in which resident self-assessment, when compared with clinical competency committee assessments, can be used to promote learner development Each table has specialty-specific milestones profiles to help stimulate discussion.

Sharing of ideas

Using milestones-based self-assessment for learner development

From the resident perspective

Suggestions for Using Self-assessment for Learner Development

Inputs: Specific goals

Milestones Questions (ITE/quiz) Recalled experience

Recorded performance

Outputs: Career development

Learning needs Study plan

Goal setting Curricular choices

External comparison and/or coach facilitation Context: Didactics/retreats, coaching (PD/advisor/supervisor), remediation

Program Improvement Discussion

Identify ways in which resident self-assessment and clinical competency committee assessments can be used to target areas for Program Improvement. Each table has specialty-specific milestones summaries to help stimulate discussion.

Sharing of ideas

Using milestones-based self-assessment for program improvement

Other thoughts?

Suggestions for Using Self-assessment for Program Improvement

Program Components: Curricular quality

Gaps/redundancies Individual resident needs

Improvements: Assessments

Curriculum structure Rotation content

Faculty development

Program Evaluation Committee, retreats, housestaff meetings, rotation-specific

Overall Value of Self-assessment

Residents understand milestones

Teach/assess Practice-Based Learning and Improvement and Professionalism • Clarifies learner perspective and self-awareness

– Creates insight and motivation for learner – Allows resident to drive improvement (better goals)

• Guides coaching – Maximizes experiences by focusing efforts and

allowing for more ah-ha moments – Helps remediation planning

Commitment to Action

Given the ideas you heard today, how will you plan to utilize milestones-based self-assessment for learner development and/or program improvement? Write a note to yourself to commit to taking a first step toward your goal! Please leave your notecards on the table we will synthesize the suggestions and email to the full group