Embed Size (px)
Citation preview
Laura Edgar, EdD, CAE
Milestones 2.0 – The Next Generation
1
© 2017 ACGME
Full time employee of ACGME
Disclosures
2
© 2017 ACGME
WHY NOW?
How and What we Learned
Milestones 2.0
Harmonized Milestones
Topics
3
© 2017 ACGME
Because we said we would…
WHY NOW?
4
© 2017 ACGME
Collaborative research
With certification boards, program director associations, institutions, researchers in medical education
Mixed methods (both “quantitative” and “qualitative” research)
Early validity studies in emergency medicine, internal medicine, pediatrics, family medicine, neurosurgery
Focus groups
Annual education conferences, society meetings, institutional visits
How We Learned
5
© 2017 ACGME
Five Sources of Validity Evidence• Content
• Review of Milestones language developed by each specialty
• Response Processes• Faculty rating process and understanding of the Milestones language
• Internal Structure• Intra-reliability of the CCC judgments
• Relations with Other Variables • Correlations with Board scores, patient outcomes, and case logs
• Consequences• Understanding the needs of the various stakeholder groups and the manner in which
Milestones data might be interpreted by these different audiences
6
© 2017 ACGME
Numerous studies published and in processThe emergency medicine milestones: a validation study.Korte RC, Beeson MS, Russ CM, Carter WA; Emergency Medicine Milestones Working Group, Reisdorff EJ. Acad Emerg Med. 2013 Jul;20(7):730-5. doi: 10.1111/acem.12166.
Initial Validity Analysis of the Emergency Medicine Milestones.Beeson MS, Holmboe ES, Korte RC, Nasca TJ, Brigham T, Russ CM, Whitley CT, Reisdorff EJ. Acad Emerg Med. 2015 Jul;22(7):838-44. doi: 10.1111/acem.12697. Epub 2015 Jun 25.
The Internal Medicine Reporting Milestones: Cross-sectional Description of Initial Implementation in U.S. Residency Programs.Hauer KE, Clauser J, Lipner RS, Holmboe ES, Caverzagie K, Hamstra SJ, Hood S, Iobst W, Warm E, McDonald FS. Ann Intern Med. 2016 Sep 6;165(5):356-62. doi: 10.7326/M15-2411. Epub 2016 May 10.
7
© 2017 ACGME
Numerous studies published and in processThe Internal Medicine Reporting Milestones: Cross-sectional Description of Initial Implementation in U.S. Residency Programs.Hauer KE, Clauser J, Lipner RS, Holmboe ES, Caverzagie K, Hamstra SJ, Hood S, Iobst W, Warm E, McDonald FS. Ann Intern Med. 2016 Sep 6;165(5):356-62. doi: 10.7326/M15-2411. Epub 2016 May 10.
Correlations Between Ratings on the Resident Annual Evaluation Summary and the Internal Medicine Milestones and Association With ABIM Certification Examination Scores Among US Internal Medicine Residents, 2013-2014.Hauer KE, Vandergrift J, Hess B, Lipner RS, Holmboe ES, Hood S, Iobst W, Hamstra SJ, McDonald FS. JAMA. 2016 Dec 6;316(21):2253-2262. doi: 10.1001/jama.2016.17357
8
© 2017 ACGME
Numerous studies published and in processThe Effect and Use of Milestones in the Assessment of Neurological Surgery Residents and Residency Programs.Conforti LN, Yaghmour NA, Hamstra SJ, Holmboe ES, Kennedy B, Liu JJ, Waldo H, Selden NR. J Surg Educ. 2017 Jun 21. pii: S1931-7204(17)30278-7. doi: 10.1016/j.jsurg.2017.06.001. [Epubahead of print]
Examining the Functioning and Reliability of the Family Medicine Milestones.Peabody MR, O'Neill TR, Peterson LE. J Grad Med Educ. 2017 Feb;9(1):46-53. doi: 10.4300/JGME-D-16-00172.1.
9
© 2017 ACGME
Numerous studies published and in processValidity Evidence From Ratings of Pediatric Interns and Subinternson a Subset of Pediatric Milestones.Turner TL, Bhavaraju VL, Luciw-Dubas UA, Hicks PJ, Multerer S, Osta A, McDonnell J, Poynter S, Schumacher DJ, Tenney-Soeiro R, Waggoner-Fountain L, Schwartz A; and the Association of Pediatric Program Directors Longitudinal Educational Assessment Research Network–National Board of Medical Examiners Pediatrics Milestones Assessment Group. Acad Med. 2017 Jun;92(6):809-819. doi: 10.1097/ACM.0000000000001622.
Competent for Unsupervised Practice: Use of Pediatric Residency Training Milestones to Assess Readiness.Li ST, Tancredi DJ, Schwartz A, Guillot AP, Burke AE, Trimm RF, Guralnick S, Mahan JD, Gifford KA; Association of Pediatric Program Directors (APPD) Longitudinal Educational Assessment Research Network (LEARN) Validity of Resident Self-Assessment Group. Acad Med. 2017 Mar;92(3):385-393. doi: 10.1097/ACM.0000000000001322.
10
© 2017 ACGME
Numerous studies published and in processThe pediatrics milestones: initial evidence for their use as learning road maps for residents.Schumacher DJ, Lewis KO, Burke AE, Smith ML, Schumacher JB, Pitman MA, Ludwig S, Hicks PJ, Guralnick S, Englander R, Benson B, Carraccio C. Acad Pediatr. 2013 Jan-Feb;13(1):40-7. doi: 10.1016/j.acap.2012.09.003. Epub 2012 Nov 17
A multi-source feedback tool for measuring a subset of Pediatrics Milestones.Schwartz A, Margolis MJ, Multerer S, Haftel HM, Schumacher DJ; APPD LEARN–NBME Pediatrics Milestones Assessment Group2,3. Med Teach. 2016 Oct;38(10):995-1002. Epub 2016 Mar 30.
11
© 2017 ACGME
Numerous studies published and in processComparison of Male vs Female Resident Milestone Evaluations by Faculty During Emergency Medicine Residency Training.Dayal A, O'Connor DM, Qadri U, Arora VM. JAMA Intern Med. 2017 May 1;177(5):651-657.
Program Director Perceptions of the General Surgery Milestones Project.Drolet BC, Marwaha JS, Wasey A, Pallant A. J Surg Educ. 2017 Mar 23. pii: S1931-7204(16)30373-7. doi: 10.1016/j.jsurg.2017.02.012. [Epub ahead of print]
Milestone assessment of minimally invasive surgery in Pediatric Urology fellowship programs.Smith PH 3rd, Carpenter M, Herbst KW, Kim C. J Pediatr Urol. 2017 Feb;13(1):110.e1-110.e6. doi: 10.1016/j.jpurol.2016.08.012. Epub 2016 Sep 15
12
© 2017 ACGME
Numerous studies published and in processMany other studies have also been published – an updated list will be on the Milestones webpage
13
© 2017 ACGME
Quantity and quality of feedback to residentsIndividualized learning plans
CCC processImproving assessment
Dealing with residents in difficulty
CurriculumFacilitates examination of curriculum
Helps with curricular improvements
Faculty developmentImproving assessment of faculty
CQI nature of system
Qualitative evaluation: Benefits
14
© 2017 ACGME
Logistics and data entry/transfer
Assessment processesNegative wording of some Milestone sets
Faculty struggling – need for faculty development
Construct misalignment (old evaluation forms do not fit with developmental model)
Assessment processes (cont.)Language in subset of Milestones (selected specialties)
Harmonize some subcompetenciesacross specialties
Synthesize multiple assessments into a Milestone judgment
Time and resources (“RVUs always win”)
Qualitative evaluation: Challenges
15
© 2017 ACGME
1. Content Validity• Milestones 2.0
2. Factor Structure• “lumping and splitting” sub-
competencies in Milestones 2.0
3. CCC Processes• best practices for CCCs • reliability of Milestones ratings
4. Rater Processes• quality of assessment tools/assessment
toolbox • faculty development - culture
change/assessment as a process
5. Relationship with other variables• correlation of MK with Board scores• correlation of PC with clinical outcomes• correlation of SBP/PBLI with
institutional variables
Need Continuing Evidence of Validity
16
© 2017 ACGME
Milestones 2.0
17
© 2017 ACGME
• Competency Crosswalk• Harmonized Milestones• Planning
2016
Proposed Timeline
• Harmonized Milestones• Kick-off of Phase 1
2017
18
© 2017 ACGME
• Completion of Phase 1• Kick-off of Phase 2
2018
Proposed Timeline
• Phase 1 begins reporting new Milestones
• Continuation of Phase 2
2019
19
© 2017 ACGME
• Completion of Phase 2• Most reporting new
Milestones2020
Proposed Timeline
All specialties reporting new Milestones
2021
20
© 2017 ACGME
We start all over again!2022
Proposed Timeline
21
© 2017 ACGME
Proposed Differences
22
© 2017 ACGME
Stays the Same
Membership for each specialty will come from the community
Review Committees, Boards, PD groups and residents/fellows will be represented
What is different
We will put out a call for volunteers –anyone involved in medical education can be nominated
We will invite public members to participate
Proposed Differences
23
© 2017 ACGME
Stays the Same
Specialties will control their content (within a framework)
What is different
We will have data to lead decisions made by the specialty
Proposed Differences
24
© 2017 ACGME
Stays the Same
Survey Program Directors about the Milestones
What is different
Surveys about the Medical Knowledge and Patient Care Milestones will be sent before we begin the process*
Put the Milestones out for Public Comment after draft completed
Proposed Differences
25
© 2017 ACGME
Stays the Same
Make presentations when invited to attend Program Director meetings
What is different
We would like the opportunity to begin talking to program directors, faculty, and residents now!
Proposed Differences
26
© 2017 ACGME
Multidisciplinary groups were formed to create a set of Milestones for ICS, PBLI, PROF, and SBP
Intent is that all specialties use these with language alterations as necessary -some specialties may add additional subcompetencies
Harmonized Milestones
27
© 2017 ACGME
Why harmonization is importantFeedback
Look-back
Public perception
28
© 2017 ACGME
Motivators - FeedbackUnsolicited Feedback
Calls
Q&A
Solicited Feedback
Focus Groups
Listening Tour
29
© 2017 ACGME
Motivators – Look-backAreas for Improvement
Difficult to measure
Suboptimal construction
30
© 2017 ACGME
Motivators – Public PerceptionWhy does each specialty have different views of important themes in professionalism?
Why does each specialty have a different expectation for the graduating resident?
Are there really more than 200 way to evaluate systems-based practice? Aren’t Milestones considered a biopsy?
31
© 2017 ACGME
Interpersonal and Communication Skills 5 Themes
Practice-based Learning and Improvement 7 Themes
Professionalism 8 Themes
Systems-based Practice 8 Themes
Competency Crosswalk
32
© 2017 ACGME
Interpersonal and Communication SkillsTheme Count
Communication with Patients 73Communication with Team 65Records 31Coordination and Transition of Care 23Informed Consent 6
33
© 2017 ACGME
Practice-based Learning and ImprovementTheme Count
Self‐directed Learning 88QI 38Feedback 28Evidence‐based Patient Care 20Teaching 18Scholarly Activity 8Safety 4
34
© 2017 ACGME
ProfessionalismTheme Count
Behaviors and Attitudes 75Accountability 58Ethics 27Confidentiality 23Limits 22Administrative Tasks 19Feedback 10Teams 6
35
© 2017 ACGME
Systems-based PracticeTheme Thread Count
Safety (patient) and Errors 61Economics 44Coordination of Care 33Team 18Technology 18Settings 13QI/Lab Management 9Community 7
36
© 2017 ACGME
Multidisciplinary: combination of content experts, program directors, and faculty
Selected common themes that can be used across specialties
Developing a Guidebook to better explain intent and how to assess
Process for Development
37
© 2017 ACGME
Milestones will be posted for Public Comment - COMPLETED
After Public Comment and final edits Milestones will be available for each specialty when they begin the redevelopment process
Each specialty can choose whether or not to use the Milestone themes (we hope they do)
Each specialty can alter the language to fit their needs
Each specialty can choose to add additional themes
Process for Development
38
© 2017 ACGME
INTERPERSONAL & COMMUNICATION SKILLS
Patient and Family Centered Communications
Interprofessional and Team Communication
Communication within Healthcare Systems
PRACTICE-BASED LEARNING & IMPROVEMENT
Evidence Based and Informed Practice
Reflective Practice and Commitment to Personal Growth
Overview of DRAFT Subcompetencies
39
© 2017 ACGME
PROFESSIONALISM
Professional Behavior and Ethical Principles
Accountability/ Conscientiousness
Self-Awareness and Help Seeking
SYSTEMS BASED PRACTICE
Patient Safety and Quality Improvement
System Navigation for Patient-Centered Care
Physician Role in Healthcare Systems
Overview of DRAFT Subcompetencies
40
© 2017 ACGME
Internal Medicine, Neurologic Surgery, Pediatrics, and Surgery have started the process
Transitional Year will start this month.
Survey going to Program Directors on August 2nd
Call for Volunteers - August 7th
So…when will my specialty start?
41
© 2017 ACGME
Where do I find...?
42
© 2017 ACGME
Milestone ResourcesMilestone Webpagehttp://www.acgme.org/What-We-Do/Accreditation/Milestones/Overview
Milestone FAQshttp://www.acgme.org/Portals/0/MilestonesFAQ.pdf
43
© 2017 ACGME
Milestone ResourcesClinical Competency Committee Guidebookhttp://www.acgme.org/Portals/0/ACGMEClinicalCompetencyCommitteeGuidebook.pdf
Milestones Guidebookhttp://www.acgme.org/Portals/0/MilestonesGuidebook.pdf
Milestone Guidebook for Residents and Fellows NEW!!http://www.acgme.org/Portals/0/PDFs/Milestones/MilestonesGuidebookforResidentsFellows.pdf
Milestones Annual Report 2016http://www.acgme.org/Portals/0/PDFs/Milestones/MilestonesAnnualReport2016.pdf
44
