Nailed it! Doctor of Pharmacy Students’ Self-Awareness of Performance on Objective

Structured Clinical Examinations (OSCE)

Kara Sermersheim, PharmD, BS

Kimberly K. Daugherty, PharmD, BCPS

Sarah E. Raake, PharmD, MSEd, BCACP, LDE

Sullivan University College of Pharmacy & Health Sciences

Louisville, KY

Disclosures: The speaker has no actual or potential conflicts of interest in relation to this presentation

Objectives

1. Discuss Doctor of Pharmacy (PharmD) students’ self-awareness related to their performance on Objective Structured Clinical Examinations (OSCE).

2. Identify the Doctor of Pharmacy students who are not self-aware based on study data.

3. Discuss how this data could be used to develop student self-awareness skills

Background: Sullivan Itself

− Sullivan University College of Pharmacy and Health Sciences− 3-year accelerated program

− Louisville, KY

− First graduating class in 2011

− Objective Structured Clinical Examinations (OSCEs) − Experiences that link didactic learning to simulated real-world cases

Why?Self-assessment of OSCE performance may help students develop self-awareness

in the practice setting

Self-awareness

Self-awareness

− Conscious knowledge of one’s own character, feelings, motives, and desires

− CAPE Outcome 4.1: students should display self-awareness defined as the ability to examine and reflect on personal knowledge, skills, abilities, beliefs, biases, motivation, and emotions that could enhance or limit personal and professional growth

− There is currently limited data relating OSCEs to student self-awareness

Oxford Online Dictionary. Definition of self-awareness. https://www.oxfordlearnersdictionaries.com/us/definition/english/self-awareness. (accessed 25 Mar 2020).Accreditation Council for Pharmacy Education. Accreditation standards and key elements for the professional program in pharmacy leading to the doctor of pharmacy degree. https://www.acpe-accredit.org/pharmd-program-accreditation/ (accessed 2019 Sept 16).

Objective Structured Clinical Examinations (OSCEs)

OSCEs as natural place to assess self-awareness

− OSCEs in pharmacy education are experiences that link didactic learning to simulated real-world cases

− Self-assessment of OSCE performance may help students develop self-awareness in the practice setting

Past examples of students lacking self-awareness in OSCEs

− Underconfident learner

− Overconfident leaner

Sample case (Directions to student)

Self-awareness survey

Grading checklist (SP completes)Pass Fail

Checklist item 1 X

Checklist item 2 X

Checklist item 3 X

Checklist item 4 X

Checklist item 5 X

Skills Section Score 80%

Checklist item 1 X

Checklist item 2 X

Checklist item 3 X

Checklist item 4 X

Communication Section 25%

Overall Score (80% skills + 20% communication) 69%

Return to the Pharmacist (RTP) No

Overall Student Performance

Sample SP checklist and comments

Why?We do these cases to assess patient care skills as well as soft skills such as,

communication, professionalism, empathy, and potentially self-awareness.

Pharmacist patient care process

Joint Commission of Pharmacy Practitioners. Pharmacists’ patient care process. https://www.pharmacist.com/sites/default/files/files/PatientCareProcess.pdf. (accessed 2020 Mar 23).

Case “buckets”

Collect Assess Plan Implement Follow-up

Communication/Law X X X

Patient Counseling X X

Patient Medication Recommendation

X X X X

Provider Recommendation

X X X X

Skills X X

Grading Checklist (SP completes)Pass Fail

Collect Checklist item 1 X

Assess Checklist item 2 X

Assess Checklist item 3 X

Plan Checklist item 4 X

Implement Checklist item 5 X

Skills Section Score 80%

Checklist item 1 X

Checklist item 2 X

Checklist item 3 X

Checklist item 4 X

Communication Section 25%

Return to the Pharmacist (RTP) No

Overall Score (80% skills + 20% communication) 69%

Overall Student Performance

The ProjectStudy Design & Statistics

Methods

– Retrospective review

– Single institution

– 3-year program

– Single cohort– 4 OSCEs completed

– 23 discrete cases

Joint Commission of Pharmacy Practitioners. Pharmacists’ patient care process. https://www.pharmacist.com/sites/default/files/files/PatientCareProcess.pdf. (accessed 2020 Feb 22).

• Reliability• Validity

Self-awareness survey

• Reliability• Validity

OSCE cases

Obtaining validity and reliability

– Microsoft Forms– Survey

– 23 individual OSCE cases

– Utilized 15 faculty members

Validity and reliability

– Lawshe method– Average CVR for all 23 cases was 0.74 (range 0.36-0.92)

– Sixteen cases were content valid (CVR ≥ 0.75)

– Seven cases had a CVR between 0-0.74

– Cronbach’s Alpha– Average Cronbach alpha for all 23 cases was 0.646 (range 0.193-0.887)

– Four cases had a Cronbach alpha of <0.5

– Nine cases were between 0.5-0.699

– Ten cases had a Cronbach alpha >0.7

Validity and reliability

– Spearman-Brown – The overall self-awareness survey (nail it/fail it) was found to be reliable

(0.829)

– Only the return to the pharmacist (0.743) and overall student performance (0.611) subscores were found to be reliable

Eisinga R, te Grotenhuis M, Pelzer B. The reliability of a two-item scale: pearson, Cronbach or spearman-brown. Int J Public Health. 2012;58(4), 637–642.

Data reviewed using Chi-squared

OSCE Grade

Failed (≤69.4%)

Passed (≥69.5%)

Survey Nail it (1)

Fail it (0)

Eisinga R, te Grotenhuis M, Pelzer B. The reliability of a two-item scale: pearson, Cronbach or spearman-brown. Int J Public Health. 2012;58(4), 637–642.

Identifying the disconnect

Overall Performance

OSCE Score

Failed

(≤69.4%)

Passed

(≥69.5%)

Survey

Nail it (1)

Fail it (0)

Return to the Pharmacist

SP Answer

No (0) Yes (1)

Survey

Yes (1)

No (0)

The ProjectResults & Limitations

Overall-Self Awareness

Case breakdown and odds Ratio

Limitations

– Single cohort of students

– Single institution

– Low stakes

ConclusionsAction Plan(s)

Action Plan

– Data-driven report card

– Moving to modular dashboard feedback mechanism

– Face-to-face meeting

Watch list

Last Name Overall Score Skills Communication RTP Collect Assess Plan Implement Follow-up/Monitor Notes Advisor

Learner 1 Smith

Learner 2 XX Jones

Learner 3 Dennis

Learner 4 Lawrence

Learner 5 Jennings

Future directions

– Student remediation– Develop activities

– Faculty advisor meetings

– Identify areas of self-improvement

– Learner and faculty development opportunities

– Review and improvement of OSCE cases not deemed valid

Final take-away

– Nail it/Fail it surveys are a useful tool to identify students who are not self-aware

Acknowledgements

– Katie F. Leslie, PhD, MS Director of The Center for Health and Wellness and Clinical and Administrative Sciences Associate Professor

Nailed it! Doctor of Pharmacy Students’ Self-Awareness of Performance on Objective

Structured Clinical Examinations (OSCE)

Kara Sermersheim, PharmD, BS

Kimberly K. Daugherty, PharmD, BCPS

Sarah E. Raake, PharmD, MSEd, BCACP, LDE

Sullivan University College of Pharmacy & Health Sciences

Louisville, KY

Disclosures: The speaker has no actual or potential conflicts of interest in relation to this presentation