Competency Tools to Measure Resident Progress

Carol Vandenakker-Albanese, MDResidency Program Director

Department of PM&R UC Davis Health System

November 15, 2014

ACGME Outcome Project Competency-based education focused on learner

performance in reaching specific objectives Long-term initiative by ACGME to emphasize educational

outcomes in the accreditation of residency training programs

Six core competencies for resident education• Medical Knowledge

• Patient Care

• Practice Based Learning and Improvement

• Systems Based Practice

• Professionalism

• Interpersonal and Communication Skills

The ‘Next Accreditation System’ in a Nutshell

Continuous Accreditation Model – annually updated• Based on annual data submitted, other data requested, and

program trends

RRCs will measure compliance through the evaluation of annual program data elements including:• Milestones (submitted twice a year)

• ACGME Resident Survey

• ACGME Faculty Survey

• Procedural /Case Log Data

Procedural Logs: Required(Observed and Performed)

EMG/NCSAxial epidural injectionAxial facetPeripheral joint/intraarticular injectionTendon sheath/bursa injectionTrigger pointsPeripheral nerve injectionBotulinum Toxin InjectionPhenol injectionProgramming baclofen pumpRefilling intrathecal baclofen pumpUltrasound extremityUltrasound guidance

Case Logs: Optional(admission, consult, outpatient)

Acute neuromuscular painAmputationBrain injuryChronic pain managementGeriatric rehabNeurologic disordersOncologic rehabPediatric rehabPost fracture and post-operative joint arthroplastyPulmonary and cardiacRheumatologicSoft Tissue disordersSpinal Cord InjuryStroke

Milestones are based on concept of Entrustable Professional Activity(EPA’s)

Real life patient care episodes, usually composed of elements of most if not all “competencies”

Benchmark of performance is the ability to be entrusted to perform care with “indirect supervision with direct supervision available”

Progression is then the achievement of EPA’s of increasing difficulty, risk, or sophistication

Proficiency is then the achievement of the most sophisticated EPA’s required of the resident

Uses and Implications of Milestones

ACGME• Accreditation –continuous

monitoring of programs; lengthening of site visit cycles

• Public Accountability –report at a national level on competency outcomes

• Community of practice for evaluation and research, with focus on continuous improvement

Residency Programs• Guide curriculum

development

• More explicit expectations of residents

• Support better assessment

• Enhanced opportunities for early identification of under-performers

Challenges/Opportunities Culture Change and Faculty Development

• Program Directors, Designated Institutional Officials

• Faculty Selecting Review Committee Members Improving quality of faculty observations/documentation The “Community of Educators” in each specialty has

come together and agree on:• core elements of the competencies (Milestones)

• levels of performance

• core methods of assessment

Program’s Assessment of Residents

Evaluation tools Competency Committee Semi-Annual Reporting to ACGME on

each milestone for all residents Annual reporting to ABPMR on resident

status and evaluations

Toolbox: Competancy Evaluation Tools

Begin by using existing tools and faculty observations • Exams: Written, Oral, SAE

• Clinical Observation: Simulation, ROCA

• Multi-Source Feedback: 360 evaluations, chart reviews

• Procedural Skill Evaluation• Workshops

• Clinical assessments

• Case Logs

Milestone Assessment

Determine which tools you will use to assess each milestone

Determine which rotations/ faculty are best to evaluate individual milestones

Determine which milestones may require additional tools • (Yearly updated Resident CV/ self-evaluation)

27 PM&R Milestones

Patient Care: 7 Systems Based Practice: 3 Practice-Based Learning and

Improvement: 3 Professionalism: 3 Interpersonal and Communication Skills: 2 Medical Knowledge: 9

PC 2. Physiatric Physical Exam (including medical, neurological, and gait adapted for age and impairment)

Rotations:• Inpatient: General

rehab, SCI, TBI

• Outpatient: MSK, neurorehab

• Peds

Tools:• ROCA

• PE workshop

SBP 2. Team approach to enhance patient care coordination. Rehabilitation team members may include OT, PT, SLP, etc.

Rotations:• Inpatient adult and

pediatric rehab

• Consults

• Outpatient multidisciplinary clinics

Tools:• 360 evaluations

• Faculty evaluations

PROF 3. Professional behaviors and accountability to self, patients, society and the profession

Rotations:• Any (not all)

• May want to pick most stressful

Tools• Faculty evaluations

• Program director/ coordinator evaluations

• Review of CV – involvement in service activities

MK 6. Musculoskeletal Disorders.

Rotations: • Outpatient clinics:

sports and spine

Tools:• Faculty evaluations

• ROCA

• Chart reviews

• SAE/ Written test

• Mock Oral Exam

Using Milestones as Evaluation Tool

Self Assessment on milestones annually

Schedule milestone assessments throughout year

Incorporate milestones into rotation objectives/ evaluations

Incorporate selected milestones into electronic evaluations

Faculty Education

Faculty must be familiar with milestones Reminder: Milestone levels NOT equivalent

to PGY level Distribute evaluation tools at start of rotation Identify which faculty are appropriate to

evaluate which milestones Review/ re-educate faculty based on quality

of evaluations

Key Elements of QualityEvaluation

Trained Observers- Core Faculty• Common understanding of the expectations

• Sensitive “eye” to key elements

• Consistent evaluation of a given level of performance

Establish Minimum Numbers of Quality Observations (CCC)

Interpreter/Synthesizer Experts• Clinical Competency Committee (Resident Evaluation

Committee)