Milestones 2.0 – The Next Generation · • Faculty rating process and understanding of the ......

Laura Edgar, EdD, CAE

Milestones 2.0 – The Next Generation

1

© 2017 ACGME

Full time employee of ACGME

Disclosures

2

© 2017 ACGME

WHY NOW?

How and What we Learned

Milestones 2.0

Harmonized Milestones

Topics

3

© 2017 ACGME

Because we said we would…

WHY NOW?

4

© 2017 ACGME

Collaborative research

With certification boards, program director associations, institutions, researchers in medical education

Mixed methods (both “quantitative” and “qualitative” research)

Early validity studies in emergency medicine, internal medicine, pediatrics, family medicine, neurosurgery

Focus groups

Annual education conferences, society meetings, institutional visits

How We Learned

5

© 2017 ACGME

Five Sources of Validity Evidence• Content

• Review of Milestones language developed by each specialty

• Response Processes• Faculty rating process and understanding of the Milestones language

• Internal Structure• Intra-reliability of the CCC judgments

• Relations with Other Variables • Correlations with Board scores, patient outcomes, and case logs

• Consequences• Understanding the needs of the various stakeholder groups and the manner in which

Milestones data might be interpreted by these different audiences

6

© 2017 ACGME

Numerous studies published and in processThe emergency medicine milestones: a validation study.Korte RC, Beeson MS, Russ CM, Carter WA; Emergency Medicine Milestones Working Group, Reisdorff EJ. Acad Emerg Med. 2013 Jul;20(7):730-5. doi: 10.1111/acem.12166.

Initial Validity Analysis of the Emergency Medicine Milestones.Beeson MS, Holmboe ES, Korte RC, Nasca TJ, Brigham T, Russ CM, Whitley CT, Reisdorff EJ. Acad Emerg Med. 2015 Jul;22(7):838-44. doi: 10.1111/acem.12697. Epub 2015 Jun 25.

The Internal Medicine Reporting Milestones: Cross-sectional Description of Initial Implementation in U.S. Residency Programs.Hauer KE, Clauser J, Lipner RS, Holmboe ES, Caverzagie K, Hamstra SJ, Hood S, Iobst W, Warm E, McDonald FS. Ann Intern Med. 2016 Sep 6;165(5):356-62. doi: 10.7326/M15-2411. Epub 2016 May 10.

7

© 2017 ACGME

Numerous studies published and in processThe Internal Medicine Reporting Milestones: Cross-sectional Description of Initial Implementation in U.S. Residency Programs.Hauer KE, Clauser J, Lipner RS, Holmboe ES, Caverzagie K, Hamstra SJ, Hood S, Iobst W, Warm E, McDonald FS. Ann Intern Med. 2016 Sep 6;165(5):356-62. doi: 10.7326/M15-2411. Epub 2016 May 10.

Correlations Between Ratings on the Resident Annual Evaluation Summary and the Internal Medicine Milestones and Association With ABIM Certification Examination Scores Among US Internal Medicine Residents, 2013-2014.Hauer KE, Vandergrift J, Hess B, Lipner RS, Holmboe ES, Hood S, Iobst W, Hamstra SJ, McDonald FS. JAMA. 2016 Dec 6;316(21):2253-2262. doi: 10.1001/jama.2016.17357

8

© 2017 ACGME

Numerous studies published and in processThe Effect and Use of Milestones in the Assessment of Neurological Surgery Residents and Residency Programs.Conforti LN, Yaghmour NA, Hamstra SJ, Holmboe ES, Kennedy B, Liu JJ, Waldo H, Selden NR. J Surg Educ. 2017 Jun 21. pii: S1931-7204(17)30278-7. doi: 10.1016/j.jsurg.2017.06.001. [Epubahead of print]

Examining the Functioning and Reliability of the Family Medicine Milestones.Peabody MR, O'Neill TR, Peterson LE. J Grad Med Educ. 2017 Feb;9(1):46-53. doi: 10.4300/JGME-D-16-00172.1.

9

© 2017 ACGME

Numerous studies published and in processValidity Evidence From Ratings of Pediatric Interns and Subinternson a Subset of Pediatric Milestones.Turner TL, Bhavaraju VL, Luciw-Dubas UA, Hicks PJ, Multerer S, Osta A, McDonnell J, Poynter S, Schumacher DJ, Tenney-Soeiro R, Waggoner-Fountain L, Schwartz A; and the Association of Pediatric Program Directors Longitudinal Educational Assessment Research Network–National Board of Medical Examiners Pediatrics Milestones Assessment Group. Acad Med. 2017 Jun;92(6):809-819. doi: 10.1097/ACM.0000000000001622.

Competent for Unsupervised Practice: Use of Pediatric Residency Training Milestones to Assess Readiness.Li ST, Tancredi DJ, Schwartz A, Guillot AP, Burke AE, Trimm RF, Guralnick S, Mahan JD, Gifford KA; Association of Pediatric Program Directors (APPD) Longitudinal Educational Assessment Research Network (LEARN) Validity of Resident Self-Assessment Group. Acad Med. 2017 Mar;92(3):385-393. doi: 10.1097/ACM.0000000000001322.

10

© 2017 ACGME

Numerous studies published and in processThe pediatrics milestones: initial evidence for their use as learning road maps for residents.Schumacher DJ, Lewis KO, Burke AE, Smith ML, Schumacher JB, Pitman MA, Ludwig S, Hicks PJ, Guralnick S, Englander R, Benson B, Carraccio C. Acad Pediatr. 2013 Jan-Feb;13(1):40-7. doi: 10.1016/j.acap.2012.09.003. Epub 2012 Nov 17

A multi-source feedback tool for measuring a subset of Pediatrics Milestones.Schwartz A, Margolis MJ, Multerer S, Haftel HM, Schumacher DJ; APPD LEARN–NBME Pediatrics Milestones Assessment Group2,3. Med Teach. 2016 Oct;38(10):995-1002. Epub 2016 Mar 30.

11

© 2017 ACGME

Numerous studies published and in processComparison of Male vs Female Resident Milestone Evaluations by Faculty During Emergency Medicine Residency Training.Dayal A, O'Connor DM, Qadri U, Arora VM. JAMA Intern Med. 2017 May 1;177(5):651-657.

Program Director Perceptions of the General Surgery Milestones Project.Drolet BC, Marwaha JS, Wasey A, Pallant A. J Surg Educ. 2017 Mar 23. pii: S1931-7204(16)30373-7. doi: 10.1016/j.jsurg.2017.02.012. [Epub ahead of print]

Milestone assessment of minimally invasive surgery in Pediatric Urology fellowship programs.Smith PH 3rd, Carpenter M, Herbst KW, Kim C. J Pediatr Urol. 2017 Feb;13(1):110.e1-110.e6. doi: 10.1016/j.jpurol.2016.08.012. Epub 2016 Sep 15

12

© 2017 ACGME

Numerous studies published and in processMany other studies have also been published – an updated list will be on the Milestones webpage

13

© 2017 ACGME

Quantity and quality of feedback to residentsIndividualized learning plans

CCC processImproving assessment

Dealing with residents in difficulty

CurriculumFacilitates examination of curriculum

Helps with curricular improvements

Faculty developmentImproving assessment of faculty

CQI nature of system

Qualitative evaluation: Benefits

14

© 2017 ACGME

Logistics and data entry/transfer

Assessment processesNegative wording of some Milestone sets

Faculty struggling – need for faculty development

Construct misalignment (old evaluation forms do not fit with developmental model)

Assessment processes (cont.)Language in subset of Milestones (selected specialties)

Harmonize some subcompetenciesacross specialties

Synthesize multiple assessments into a Milestone judgment

Time and resources (“RVUs always win”)

Qualitative evaluation: Challenges

15

© 2017 ACGME

1. Content Validity• Milestones 2.0

2. Factor Structure• “lumping and splitting” sub-

competencies in Milestones 2.0

3. CCC Processes• best practices for CCCs • reliability of Milestones ratings

4. Rater Processes• quality of assessment tools/assessment

toolbox • faculty development - culture

change/assessment as a process

5. Relationship with other variables• correlation of MK with Board scores• correlation of PC with clinical outcomes• correlation of SBP/PBLI with

institutional variables

Need Continuing Evidence of Validity

16

© 2017 ACGME

Milestones 2.0

17

© 2017 ACGME

• Competency Crosswalk• Harmonized Milestones• Planning

2016

Proposed Timeline

• Harmonized Milestones• Kick-off of Phase 1

2017

18

© 2017 ACGME

• Completion of Phase 1• Kick-off of Phase 2

2018

Proposed Timeline

• Phase 1 begins reporting new Milestones

• Continuation of Phase 2

2019

19

© 2017 ACGME

• Completion of Phase 2• Most reporting new

Milestones2020

Proposed Timeline

All specialties reporting new Milestones

2021

20

© 2017 ACGME

We start all over again!2022

Proposed Timeline

21

© 2017 ACGME

Proposed Differences

22

© 2017 ACGME

Stays the Same

Membership for each specialty will come from the community

Review Committees, Boards, PD groups and residents/fellows will be represented

What is different

We will put out a call for volunteers –anyone involved in medical education can be nominated

We will invite public members to participate

Proposed Differences

23

© 2017 ACGME

Stays the Same

Specialties will control their content (within a framework)

What is different

We will have data to lead decisions made by the specialty

Proposed Differences

24

© 2017 ACGME

Stays the Same

Survey Program Directors about the Milestones

What is different

Surveys about the Medical Knowledge and Patient Care Milestones will be sent before we begin the process*

Put the Milestones out for Public Comment after draft completed

Proposed Differences

25

© 2017 ACGME

Stays the Same

Make presentations when invited to attend Program Director meetings

What is different

We would like the opportunity to begin talking to program directors, faculty, and residents now!

Proposed Differences

26

© 2017 ACGME

Multidisciplinary groups were formed to create a set of Milestones for ICS, PBLI, PROF, and SBP

Intent is that all specialties use these with language alterations as necessary -some specialties may add additional subcompetencies

Harmonized Milestones

27

© 2017 ACGME

Why harmonization is importantFeedback

Look-back

Public perception

28

© 2017 ACGME

Motivators - FeedbackUnsolicited Feedback

milestones@acgme.org

Calls

Q&A

Solicited Feedback

Focus Groups

Listening Tour

29

© 2017 ACGME

Motivators – Look-backAreas for Improvement

Difficult to measure

Suboptimal construction

30

© 2017 ACGME

Motivators – Public PerceptionWhy does each specialty have different views of important themes in professionalism?

Why does each specialty have a different expectation for the graduating resident?

Are there really more than 200 way to evaluate systems-based practice? Aren’t Milestones considered a biopsy?

31

© 2017 ACGME

Interpersonal and Communication Skills 5 Themes

Practice-based Learning and Improvement 7 Themes

Professionalism 8 Themes

Systems-based Practice 8 Themes

Competency Crosswalk

32

© 2017 ACGME

Interpersonal and Communication SkillsTheme Count

Communication with Patients 73Communication with Team 65Records 31Coordination and Transition of Care 23Informed Consent 6

33

© 2017 ACGME

Practice-based Learning and ImprovementTheme Count

Self‐directed Learning 88QI 38Feedback 28Evidence‐based Patient Care 20Teaching 18Scholarly Activity 8Safety 4

34

© 2017 ACGME

ProfessionalismTheme Count

Behaviors and Attitudes 75Accountability 58Ethics 27Confidentiality 23Limits 22Administrative Tasks 19Feedback 10Teams 6

35

© 2017 ACGME

Systems-based PracticeTheme Thread Count

Safety (patient) and Errors 61Economics 44Coordination of Care 33Team 18Technology 18Settings 13QI/Lab Management 9Community 7

36

© 2017 ACGME

Multidisciplinary: combination of content experts, program directors, and faculty

Selected common themes that can be used across specialties

Developing a Guidebook to better explain intent and how to assess

Process for Development

37

© 2017 ACGME

Milestones will be posted for Public Comment - COMPLETED

After Public Comment and final edits Milestones will be available for each specialty when they begin the redevelopment process

Each specialty can choose whether or not to use the Milestone themes (we hope they do)

Each specialty can alter the language to fit their needs

Each specialty can choose to add additional themes

Process for Development

38

© 2017 ACGME

INTERPERSONAL & COMMUNICATION SKILLS

Patient and Family Centered Communications

Interprofessional and Team Communication

Communication within Healthcare Systems

PRACTICE-BASED LEARNING & IMPROVEMENT

Evidence Based and Informed Practice

Reflective Practice and Commitment to Personal Growth

Overview of DRAFT Subcompetencies

39

© 2017 ACGME

PROFESSIONALISM

Professional Behavior and Ethical Principles

Accountability/ Conscientiousness

Self-Awareness and Help Seeking

SYSTEMS BASED PRACTICE

Patient Safety and Quality Improvement

System Navigation for Patient-Centered Care

Physician Role in Healthcare Systems

Overview of DRAFT Subcompetencies

40

© 2017 ACGME

Internal Medicine, Neurologic Surgery, Pediatrics, and Surgery have started the process

Transitional Year will start this month.

Survey going to Program Directors on August 2nd

Call for Volunteers - August 7th

So…when will my specialty start?

41

© 2017 ACGME

Where do I find...?

42

© 2017 ACGME

Milestone ResourcesMilestone Webpagehttp://www.acgme.org/What-We-Do/Accreditation/Milestones/Overview

Milestone FAQshttp://www.acgme.org/Portals/0/MilestonesFAQ.pdf

43

© 2017 ACGME

Milestone ResourcesClinical Competency Committee Guidebookhttp://www.acgme.org/Portals/0/ACGMEClinicalCompetencyCommitteeGuidebook.pdf

Milestones Guidebookhttp://www.acgme.org/Portals/0/MilestonesGuidebook.pdf

Milestone Guidebook for Residents and Fellows NEW!!http://www.acgme.org/Portals/0/PDFs/Milestones/MilestonesGuidebookforResidentsFellows.pdf

Milestones Annual Report 2016http://www.acgme.org/Portals/0/PDFs/Milestones/MilestonesAnnualReport2016.pdf

44

© 2017 ACGME

@ACGME WWW.ACGME.ORG

Questions? Email us at: milestones@acgme.org

Thank you!

45