USING A COMPREHENSIVE INTERN OSCE FOR THE INITIAL ASSESSMENT OF NAS MILESTONES CAROLYN STALVEY, MD ERIK W. BLACK, PHD NICOLE M. PARADISE BLACK, MD MED

OVERVIEW

Interns by Department

Anesthesiology (n=12)

Emergency Med (n=8)

Family Medicine (n=10)

Internal Medicine (n=40)

Categorical (n=24)

Preliminary (n=16)

Neurosurgery (n=3)

Obstetrics and Gynecology (n=3)

Otolaryngology (n=2)

Orthopedics (n=4)

Pediatrics (n=17)

Psychiatry (n=8)

Surgery (n=13)

Urology (n=3)

Interns tested: N = 123

Stations Details Categories Post-Encounter

Chest Pain unstable angina with SP

Hx, PE, Communication Differential and Read EKG

SOB Pulmonary embolism with SP

Hx, PE, Communication SOAP note

RLQ Pain Ectopic pregnancy with SP

Hx, PE, Communication Handoff note

Mock Page Answer page, Infected PICC line with SN

Interprof Comm, Hx, Clinical Reasoning

Stations Details Categories Post-Encounter

Discharge Pneumonia Discharge instruction with SP

Education, Communication

Discharge Summary, CXR

Medical Error Disclosure medication error to SP

Education, Safety, Communication

Documentation note

Sterile Gown & Glove Nurse/grader in room Safety

Informed Consent Blood transfusion, with SP

Education, Communication

Test Blueprint - Categories

Case Name History Physical Exam Patient Safety Patient Education

InterProfession Communication

Clinical Decision Making

Communication Skills

Chest Pain X X X X X Gowning and

Gloving X Informed Consent X X

Medical Error X X X Mock Page X X X Discharge/ Pneumonia X X

RLQ Pain X X X X Shortness of

Breath X X X X

Test Blueprint – Notes, other skills

Case Name SOAP Note Discharge Note Documentation of Error Handoff Note EKG

Interpretation X-ray

Interpretation

Chest Pain X Medical Error X

Discharge/ Pneumonia X X

RLQ Pain X Shortness of

Breath X

Evaluators

Standardized patients/nurses History Physical Exam Communication items

OSCE Committee Notes EKG CXR

Milestones Addressed - IM

Case Name IM Milestones

Chest Pain 2MK: Knowledge of core content; 1PC: Diagnostic tests; 1PC Historical Data Gathering; 1PC Physical Examination; 4ICS Communicate Effectively;

Gowning and Gloving 6SBP: Recognizes systems error and advocated for system improvement

Informed Consent 1PC: Patient Management; 4ICS Communicate Effectively; 4ICS: Demonstrate Compassion and Respect

Medical Error 5P: Adhere to Basic Ethical Principles; 4ICS: Communicate Effectively

Mock Page 4ICS Interprofessional Team;1PC Clinical Decision Making; 1PC: Patient Management

Discharge/ Pneumonia

2MK: Knowledge of core content; 4ICS Transition of Care, Health Records; 1PC Clinical Reasoning

RLQ Pain 2MK: Knowledge of core content; 4ICS Transition of Care, Health Records; 1PC Historical Data Gathering; 1PC Physical Examination

Shortness of Breath 2MK: Knowledge of core content; 4ICS Transition of Care, Health Records; 1PC Historical Data Gathering; 1PC Physical Examination

Milestone Alignment

High frequency Milestones (IM) 1. Works effectively within an Interprofessional

Team (9) 2. Improves via Self Assessment (7) 3. Diagnostic Tests (6) 4. Participates in the education of all members

of health care team (6) 5. Communicates Effectively (6) 6. Demonstrates Compassion and Respect to

Patients (6)

PERFORMANCE SUMMARY

Performance by Category

Category Items MEAN SD

Patient Safety 15 87.99 22.50

Communication Skills 84 81.78 9.93

History 40 76.76 15.80

Physical Examination 30 74.43 21.00

Patient Education 29 73.44 19.40

Interprofessional Communication 2 60.97 26.80

Clinical Decision Making 7 45.12 25.00

Post Encounter

Case Post Encounter Mean SD

Chest Pain Differential Diagnosis 82.16 20.61

SOB SOAP Note 77.04 17.75

Pneumonia Interpret X-ray 75.10 43.19

RLQ Pain Handoff Note 67.56 16.21

Pneumonia Discharge Note 65.75 18.72

Error Documentation of Error 57.56 22.42

Chest Pain Interpret EKG 49.96 25.61

HCAHPS Items

Case Name Patient Education

Communication Skills

Chest Pain

X

Gowning and Gloving

Informed Consent X

Medical Error X

Mock Page

Pneumonia X

X

RLQ Pain

X

Shortness of Breath

X

1. Description symptoms to look for post discharge

2. Description of medication side effects

3. Physician listened 4. Physician

explained 5. Physician respected

HCAHPS

Item (Scale: 1 – 5) Mean Physician listened 3.67

Physician explained 3.60

Physician respected 3.84

Item (Yes/No) Frequency The resident told me about possible side effects of diarrhea or loose stool. 93%

The resident told me to seek medical attention if I have chills or fever higher than 101.5. 40%

The resident told me to contact my primary care doctor if I get worse. 37% The resident told me to seek medical attention if I have problems breathing. 33%

The resident told me to seek medical attention for chest pain. 25%

The resident told me to seek medical attention if my cough gets worse. 21%

General Areas of Weakness

Discharge Instructions symptoms to watch for smoking cessation (done 76%)

Sterile Gown and Glove Nearly all contaminated themselves at some point

Informed Consent Often don't mention it is ok to refuse transfusion

(done 80%) Allergic reactions (done 87%, 59%)

General Areas of Weakness

Handoffs What needs to be followed up What to do if..

Pulmonary exam (65%) Medical Error

Many don't ask if patient understood (done 44%) Neglecting to say it will be investigated and will get

back to you (done 73%)

UF vs. Non-UF Graduates

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Informed Consent

Medical Error

RLQ Pain Chest Pain Gown and Glove

Shortness of Breath

(PE)

Pneumonia Mock Page

Mea

n Sc

ore

UF

Non-UF

p =.02

p < .01

UF vs. Non-UF Graduates

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Mea

n Sc

ore

UF

Non-UF

p = .05

PD Reactions

Program directors described the OSCE as a positive experience

Helpful method of evaluating residents and identifying issues

A strong majority expressed interest in a retest (particularly primary care)

Next Steps

Continuous improvement and evaluation of OSCE 1 Remediation? OSCE 2 Continue to increase alignment with Milestones

Thank you