Embed Size (px)
Citation preview
Difficult to Assess Milestones
. . . An Open Collaboration In Teaching and Assessment
Saadia Akhtar, MD, FACEPProgram Director
Beth Israel Medical CenterAlbert Einstein College of Medicine
Rodney Omron, MD, FACEPAssistant Program Director
John Hopkins University
Lynn Roppolo, MD, FACEPAssociate Program Director
University of Texas SouthwesternParkland Health and Hospital System
• None to report
Conflict of Interest
Objectives• Identify the challenges and obstacles to
assessing these milestones• Create a more effective educational program
to effectively teach these milestones • Develop assessment tools to appropriately
evaluate residents in these milestones• Understand how to navigate the EM
milestones wiki site
The Problem . . .
A simple but less defined assessment tool . . . . . .
. . . are now expanded into 23 sub- competencies!!!
Most faculty are convinced that the RRC-EM Milestones are a solution
for which there is no known problem.
-anonymous
Developing a roadmap . . .
• 15-question survey, validated by 7 former PDs • Sent to CORDEM listserver in the Fall of 2013 • 62% (99/160) of EM residency programs responded • IRB approved study
Difficult Milestones Survey
Objectives (revised)
Begin with the end in mind . . .
Sub-competency 20: Problem Based Learning and Improvement (PBLI)
Sub-competency 20 (PBLI): Participates in performance improvement to optimize ED function, self-learning, and patient care.
Level 1 Level 2 Level 3 Level 4Describes basic principles of evidence-based medicine
Performs patient follow-up
Performs self-assessment to identify areas for continued self-improvement and implements learning plans Continually assesses performance by evaluating feedback and assessment Demonstrates the ability to critically appraise scientific literature and apply evidence-based medicine to improve one’s individual performance
Applies performance improvement methodologies Demonstrates evidence-based clinical practice and information retrieval mastery Participates in a process improvement plan to optimize ED practice
• 98/157 CORD attendees surveyed• 71% do not use formal critical appraisal instrument• 75% do not have established EBM curriculum • 78% do not use any extramural sources (ACP, Best
Evidence, EM Abstracts, or Annals of EM)
Carpenter CR . Incorporating evidence-based medicine into resident education: a CORD survey of faculty and resident expectations. Acad Emerg Med 2010
EBM Practices and Expectations from EM Physician Educators
Fres
no t
est
Oth
er E
BM
qui
z
Jour
nal c
lub
pres
enta
tion
b...
Oth
er c
onfe
renc
e pr
esen
tat.
..
Qua
lity
impr
ovem
ent
proc
es...
Cri
tica
l app
rais
al o
f th
e li.
..
Cri
tica
lly a
ppra
ised
top
ics.
..
EBM
Cri
tica
lly A
ppra
ised
To.
..
Van
derb
ilt m
atri
x ev
alua
tion
...
Trac
king
of
resi
dent
sea
rche
...
Aud
ienc
e re
spon
se s
yste
m f
...
“Kno
wle
dge
Tran
slat
ion”
shi
...
Dir
ect
obse
rvat
ion
by s
uper
...
We
do n
ot c
urre
ntly
hav
e a
...
Oth
er
0.0%
10.0%
20.0%
30.0%
40.0%
50.0%
60.0%
70.0%
80.0%
90.0%
EBM Assessment Methods (CORD survey No-
vember 2013)
Sub-competency 20: Most Common EBM Assessments Methods
Percent
Journal Club 83.3
Direct observation by supervising faculty 47.1
Critical appraisal of the literature using EBM worksheets
26.5
Quality improvement process using EBM to create a process change
23.5
Other conference presentation by EM resident 21.6
Learning Basic EBM Principles
• Journals: ACP Journal Club, EM Abstracts, Annals• Books– Emergency Medicine Decision Making by S. Weingart, 2006– User’s Guide to the Medical Literature by G Guyatt, 2008– Evidence-Based Diagnosis (Cambridge Medicine), TB
Newman, 2009– EBM: How to Practice and Teach EBM, 4th Edition, SE
Straus, 2010– Evidence-Based Physical Diagnosis: Expert Consult, S
McGee, 2012– Evidence-Based Emergency Care: Diagnostic Testing and
Clinical Decision Rules by JM Pines and CR Carpenter, 2013
Learning Basic EBM Principles• Websites and podcasts– JAMAevidence: Using Evidence to Improve Care– http://www.dartmouth.edu/~library/biomed/guides/resear
ch/ebm-resources-materials.html– http://guides.mclibrary.duke.edu/ebm– https://www.med.emory.edu/EMAC/curriculum/diagnosis/
maindiagnostic.html– http://www.annemergmed.com/content/ebemresources– http://adc.bmj.com/content/90/8/845.full.pdf– http://www.cebm.net/?o=1023– http://bestbets.org/teaching/current.php– http://ktclearinghouse.ca/– http://emjclub.com
Critical Appraisal Worksheetshttp://www.cebm.net/index.aspx?o=1157
CAT Maker and Critically Appraised Topics
Flipped Classroom: Teaching and Learning More Efficiently
• Need to be information managers, not encyclopedias of knowledge.
• Need to be life long learners… to ask questions and go out and find answers for themselves.
• Need to search and filter vast quantities of information on the internet - “Free Open Access MedEd”(FOAMed) by way of blogs, podcasts, and videos, apps and mobile websites
Flipped Classroom Example
• One hour of ‘individualized learning’ at home• Provide a clinical scenario and 3-5 questions
for residents to answer in advance• Provide some electronic resources • Interactive discussion in classroom• Evaluate residents by written submission of
their findings or conference participation
Fresno Test
• 2 clinical scenarios with open ended questions• Must complete 4 key steps of EBM practice • 7 short answer questions, 2 mathematical
calculations, and three fill-in-the-blank questions
• The only validated, standardized, and objective measure of EBM competence currently available
• 30 minute test• 13 page rubric
Fresno Test
Knowledge Translation Shift
• Residents and an EBM faculty leader spend one shift per month in the ED answering EBM queries for all of the residents/faculty seeing patients.
Sub-competency 20: Patient Follow-Up Methods Percent
Log of patient follow up that resident does through their own review of medical record
77.5
M&M or continuous quality improvement (review of specific CASES)
65.7
Case conference presentations (not related to M&M or continuous quality improvement)
47.1
Calling back patients 23.5
Continuous quality review (review specific patient care areas)
12.7
Chart review with dedicated faculty 9.8
Sub-competency 20: Self Assessment
• 62% do a self-assessment tool or questionnaire• 61% discuss with residency leadership• 55% discuss with their advisor or mentor• 42% talk about it during clinical shifts• Most review semi-annually
Sub-competency 20: Process Improvement (PI)
• 55% have PI project in the ED• 32% do a presentation at weekly conference• 28% have resident participate in a committee• 24% do not assess this
Sub-competency 21:
Patient Safety
(System Based Based Practice 1 or SBP1)
Sub-competency 21 (SBP1): Participates in performance improvement to optimize patient SAFETY.
Level 1 Level 2 Level 3 Level 4
Adheres to standards for maintenance of a safe working environment Describes medical errors and adverse events
Routinely uses basic patient safety practices, such as time-outs and ‘calls for help’
Describes patient safety concepts Employs processes (e.g., checklists, SBAR), personnel, and technologies that optimize patient safety (SBAR= Situation – Background – Assessment – Recommendation) Appropriately uses system resources to improve both patient care and medical knowledge
Participates in an institutional process improvement plan to optimize ED practice and patient safety Leads team reflection such as code debriefings, root cause analysis, or M&M to improve ED performance Identifies situations when the breakdown in teamwork or communication may contribute to medical error
Sub-competency 21: What type of educational formats do you use to teach your residents about "Patient Safety“?
Percent
M&M or CQI conferences 97.0
Simulation 58.6
Hospital based online modules 41.4
Dedicated patient safety conferences 37.4
Mock oral boards 36.4
Small group discussions 30.3
Institute of Healthcare Improvement (IHI) Modules 4.0
Other 4.0
Sub-competency 21- Tools to assess the Patient Safety Sub-competency
185
49
9
44 4857
6 9 3
Sub-competency 21: Do you use any checklists or tools to assist with "Transitions of Care" such
as the SBAR or Team Huddle?
54%44%
2%
YesNoOther
Sub-competency 21: ToolsiPASS (EPIC)
TEAM STEPPS
Bedside Rounding
Sign-Out Tab in EPIC
Standardized HMED Transfer of Care Note
Evaluation of Handoff Form
Developed Own Transition Sheet/Template
Printed tracker board and physically rounding in ED at transition time
Check list at sign out
• After implementation of multiple medical team training programs:
•Improved observed team behaviors.
•Enhanced staff attitudes toward teamwork.
•Reduced observed clinical errors.
Morey, JC, Simon, R, Jay GD, et al. Error reduction and performance improvement in the emergency departmentthrough formal teamwork training: Evaluation results of the MedTeams project. Health Serv Res. 37:1553-1581, 2002
Sub-competency 22: Systems-based Management (SBP2)
Sub-competency 22 (SBP2): Participates in strategies to improve healthcare delivery and flow. Demonstrates an awareness of and responsiveness
to the larger context and system of health care.
Level 1 Level 2 Level 3 Level 4
Describes members of ED team (e.g., nurses, technicians, and security)
Mobilizes institutional resources to assist in patient care Participates in patient satisfaction initiatives
Practices cost-effective care Demonstrates the ability to call effectively on other resources in the system to provide optimal health care
Participates in processes and logistics to improve patient flow and decrease turnaround times (e.g., rapid triage, bedside registration, Fast Tracks, bedside testing, rapid treatment units, standard protocols, and observation units) Recommends strategies by which patients’ access to care can be improved .Coordinates system resources to optimize a patient’s care for complicated medical situations
Sub-competency 22: What type of educational formats do you use to teach your residents about "Systems-based Management" to improve healthcare delivery and flow?
Percent
Didactics 69.7
Morbidity and Mortality Conference 68.7
Small group discussions 39.4
Simulation 38.4
Multi-disciplinary teaching by other hospital staff 36.4
Mock oral boards 28.3
We do not have any formal teaching on "Systems-based Management" for our residents
7.1
Online modules 5.1
Other 1.0
End of Shift
End of Rotation
Peer Evaluation
Patient Evaluation
Direct Observation Tool by Faculty
Feedback from Others
Mock Oral Boards
Simulation
M & M/CQI
None
Other
0 10 20 30 40 50 60
Sub-competency 22: Tools for Systems-based Management
www.improvediagnosis.org
Sub-competency 23: Technology (SBP3)
Sub-competency 23 (SBP3): Uses technology to accomplish and document safe healthcare delivery.
Level 1 Level 2 Level 3 Level 4
Uses the Electronic Health Record (EHR) to order tests, medications and document notes, and respond to alerts Reviews medications for patients
Ensures that medical records are complete, with attention to preventing confusion and error Effectively and ethically uses technology for patient care, medical communication and learning
Recognizes the risk of computer shortcuts and reliance upon computer information on accurate patient care and documentation
Uses decision support systems in EHR (as applicable in institution)
Sub-competency 23: Methods used to assess the use of technology for safe health care delivery
Percent
Evaluations by supervising faculty in the ED 71.6
Chart review with the resident 28.4
Conference presentation that demonstrates ability to identify and use educational resources readily available while working in the ED (e.g. PBL)
22.5
Review of billing records 17.6
No mechanism exists to assess this milestone 16.7
Identification of clinical dashboards 8.8
EM Milestones Wiki
emmilestones.pbworks.com
OR
Google “EM Milestones Wiki”
For assistance or to add to the wiki site: [email protected]
Summary• EBM • Safety • Technology • Systems Based Management • Wiki site– How to navigate– How to access resources– How to contribute
Questions
References• Tews MC, Liu JM, Treat R. Situation-Background-Assessment-Recommendation (SBAR) and
Emergency Medicine Residents' Learning of Case Presentation Skills. J Grad Med Educ. Sep 2012; 4(3): 370–373.
• Morey, JC, Simon, R, Jay GD, et al. Error reduction and performance improvement in the emergency department through formal teamwork training: Evaluation results of the MedTeams project. Health Serv Res. 37:1553-1581, 2002
• http://teamstepps.ahrq.gov• www.improvediagnosis.org
