ACGME

2016 Annual Education Conference

SES052

How to Use Milestones to Provide an Evidence-based

Early Warning System for Identifying the

Academically Vulnerable Resident

February 26, 2016

Ann Dohn, MA, DIO and Nancy Piro, PhD

Conflict of Interest

• Ann Dohn, MA - No Conflicts of Interest to Declare

• Nancy Piro, PhD - No Conflicts of Interest to Declare

Agenda - Criticality of Early Identification

• Results of a national survey of DIOs and program directors on Milestones

• Milestones as an early identification of resident academic vulnerability – Evaluations, Safety (SAFE) reports, in service exam scores– Interview behaviors, letters of reference, and summative evaluations

from prior training programs.

• Mining the Milestone Data and Other Data to Identify at Risk Residents– Making it easy

• Tools which summarize resident Milestones as well as other data to facilitate collaborative discussion and remediation planning

• Using learning resources including educational testing and consultation with appropriate specialists

• Electronic resource toolkit to “Take Home”

Results of a 2015 National Survey of Program

Directors on Milestones

5.34%

24.43%

40.46%

18.32%

9.16%

2.29%

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

Very StronglyAgree

Strong Agree

Agree

Disagree

Strongly Disagree

Very StronglyDisagree

“Milestone evaluations increase our (PDs) documentation of traineeweaknesses/areas for improvement”

Results of a 2015 National Survey of DIOs on Milestones

33.33%

25%

41.67%

“Milestone evaluations increase our (DIOs) documentation of trainee weaknesses/areasfor improvement”

All Agreed

This would never happen…

“What do you mean I have to put my resident on probation? I want to terminate them at the end of the month!”

Evaluations

• Competency based faculty evaluations of residents (rotations)

• 360 evaluations (Techs, Allied health Professionals, nurses, patients, families, coordinators, etc)

‘Data Mining’ Beyond Evaluations - Other Data Sources

• SAFE reports

• In service exam scores

• Interview, meeting and other observed behaviors

• Letters of reference

• Summative evaluations from prior training programs

– Awareness of sources of potential evaluator bias

Leveraging Resident Management System

(RMS)Tools - if Available

• RMSs – becoming more feature rich

– CCC Online Functionality – Assign Files, Upload all required data for Preliminary Review, etc

– Curriculum

• Goals and Objectives and learning outcomes by rotation

• Teaching and Assessment methodologies

– Evaluation tool development

• Sharing between programs and institutions

Leveraging Resident Management System

(RMS)Tools - if Available

• Conference attendance statistics

– Core competencies linked to specified conferences

– Attaching conference materials for later reference

• Procedures and levels; linked procedure evaluations

• Resident portfolio toolsQI participation and outcomesScholarly Activity logs

Leveraging Resident Management System

(RMS)Tools, if Available

Leveraging Your Program Coordinator

Evaluations

Milestone End of Rotation (Total)

Medical Knowledge

Patient Care

Interpersonal & Communication Skills

Professionalism

Systems-Based Practice

Practice-Based Learning & Improvement

Creating a Resident Performance ProfileStep 1 – Defining what to track (Example from Program)

In-service Assessments (MK; PC)

Routine procedure technical skills

assessment: Level 2-3

Complex procedure technical skills

assessment: Level 3-4

Medical Knowledge Assessments

Creating a Resident Performance ProfileStep 1 – Defining what to track

Case Logs / Clinical Experience (PC)

VAGINAL DELIVERY

CAESAREAN SECTION

PEDIATRICS

PEDIATRICS UNDER 3

CARDIAC

ENDOVASCULAR

Creating a Resident Performance Profile Step 1 – Defining what to track

Quality Improvement/Patient Safety

QI Project

QI Committee Participation

QI Course Work (e.g., IHI)

SAFE Report / Adverse Event Review

Patient Handover Evaluations

Creating a Resident Performance ProfileStep 1 – Defining what to track

Practice-Based Learning

Patient outcomes / Case Study Presentation

Scholarly Activity: Research study

Scholarly Activity: PublicationsPresentation at Internal and National Meeting

Creating a Resident Performance ProfileStep 1 – Defining what to track

Communication

Patient Feedback

Staff Evaluation

Medical Student Feedback

Patient Handover Evaluations

Creating a Resident Performance ProfileStep 1 – Defining what to track

Patient Communication Data – Questions and Scale

19

Resident Assessment

“How well did the resident physician…”

Very Poor Poor Fair Good Very Good N/A

1. Greet you in a way that made you feel comfortable? 1 2 3 4 5 N/A

2. Treat you with respect? 1 2 3 4 5 N/A

3. Show interest in your ideas about your health? 1 2 3 4 5 N/A

4. Understand your main health concerns? 1 2 3 4 5 N/A

5. Pay attention to you (look at you, listen carefully)? 1 2 3 4 5 N/A

6. Let you talk without interruptions? 1 2 3 4 5 N/A

7. Give you as much information as you wanted? 1 2 3 4 5 N/A

8. Talk in terms you could understand? 1 2 3 4 5 N/A

9. Check to be sure you understood everything? 1 2 3 4 5 N/A

10. Encourage you to ask questions? 1 2 3 4 5 N/A

11. Involve you in decisions as much as you wanted? 1 2 3 4 5 N/A

12. Discuss next steps, including any follow-up plans? 1 2 3 4 5 N/A

13. Show care and concern? 1 2 3 4 5 N/A

14. Spent the right amount of time with you? 1 2 3 4 5 N/A

Open-ended questions Open Response

15. What did you like about your resident’s communication?

16. How can this resident improve?

17. Do you have any other comments, questions, or concerns?

Creating a Resident Performance Profile

Step 2 - Link data sources to milestones

Evaluations Milestones

Milestone End of Rotation (Total) 24

Medical Knowledge MK A1

Patient Care PC A1-10

Communication ICS A1-2

Professionalism PROF 1-5

Systems-Based Practice SBP A1-2

Practice-Based Learning PBLI A1-4

Quality Improvement/Patient Safety

QI Project PBLI1-2

QI Committee Participation PBLI1-2

QI Course Work (e.g., IHI) PBLI1-2

SAFE Report / Adverse Event Review PROF1/2; SBP1

Patient Handover Evaluations SBP1; ICS1

Milestone Data Translation to Numerical Data

Creating a Resident Performance Profile

Step 3: CCC defines performance ranges

At or Above Expectation:

2.8 and higher

Below Expectation:

1.7 – 2.7

Remediation:

Below 1.7

Example:

For all aggregate milestone evaluation scores for a PGY 3, the CCC defines these ranges:

Hypothetical CCC Defined Ranges:

Creating a Resident Performance Profile

Step 4 – Set conditional formatting

Example: Aggregate milestone evaluation data cells

Highlight cells to apply the conditional formatting

Creating a Resident Performance Profile

Step 4 – Set conditional formatting

Click on “Conditional Formatting”“Highlight Cells Rules”

Creating a Resident Performance Profile

Step 4 – Set conditional formatting

Select, “Greater Than” “Less Than” or “Between” to Set Value RangesChoose the corresponding fill color (e.g., red, yellow, green)

Creating a Resident Performance Profile

Step 4 – Set conditional formatting

Creating a Resident Performance Profile

Compiling and centralizing data

Resident Performance Profile:

Step 5: Enter in data

Creating a Resident Performance Profile

Visual Trends and Detailed Data

When Remediation is Evident

Employing Resident Management System (RMS)Tools - if

Available to Demonstrate Performance in a Peer Context

• Aggregate reporting and graphic summaries• Peer or departmental average, individual average, minimum and

maximum scores, standard deviation or listing of all scores

What To Do with an Underperforming Resident?

1. Look at your tracker for low scores… probe…

What can you do?

1. Engage the resident at the first sign of troubling results

2. Formulate an action plan (talk with your faculty)

Assign a mentor

Are institutional resources needed (DIO?)

3. PD must commit to close monitoring and follow-up

Case Study #1 (Disclaimer-all names and Depts and

Details have been altered)

• SITUATION:

– Neurosurgery resident

– Low in-service exam scores – 1 %

– Unable to keep up with ‘paper work’ and electronic charting - esp. discharge summaries

– Great interpersonal skills

– Outstanding surgical skills

– Former concert musician

• ACTIONS:

• RESULTS

Case Study #2 - (Disclaimer-all names and Depts and

Details have been altered)

• SITUATION:

– Advanced Radiology Fellow

– Nursing complaints about his responsiveness

– Good clinical skills

– Excellent test scores

– Widely divergent evaluations (Faculty vs Nursing)

• ACTIONS:

• RESULTS

Using other resources to assist in remediation

• Educational testing

• Consultation with appropriate specialists

• Culture and Linguistic Specialists

• Well Being Resources

• Mentors

Electronic Toolbox for You!

http://med.stanford.edu/gme/GME_Community.html

• Our Video on Conditional Formatting

• The Practice Excel File that Accompanies the Video (Refer to tab PGY3 for practice): Resident Performance Profile Tool

Tools Can Be Downloaded @ www.gme.stanford.edu

GME Community Templates

Questions & Contact Information

• Ann Dohn: adohn1@stanford.edu

• Nancy Piro: npiro@stanford.edu