Upload
irene-collins
View
232
Download
0
Embed Size (px)
DESCRIPTION
Contact Information Abby A. Kahaleh, BPharm, MS, PhD, MPH Curriculum SIG Chair 847-330-4537 (Phone) [email protected]
Citation preview
Pharmacy Curriculum Outcomes Assessment (PCOA)
Moderated by:Abby A. Kahaleh, BPharm, MS, PhD, MPHFebruary 16, 2016
Contact Information
Abby A. Kahaleh, BPharm, MS, PhD, MPH
Curriculum SIG Chair
847-330-4537 (Phone)
Presenters
1. Elizabeth A. Coyle, Pharm.D., FCCM, BCPS Assistant Dean of AssessmentUniversity of Houston College of Pharmacy
2. Tara Jenkins, RPh, PhD Associate Dean of Academic AffairsUniversity of Houston College of Pharmacy
3. Justine Gortney, Pharm.D., BCPSClinical Assistant ProfessorWayne State University College of Pharmacy
Program Description According to NABP, the Pharmacy Curriculum Outcomes
Assessment (PCOA) is an effective assessment tool for colleges and schools of pharmacy to use for evaluating student performance in the curricula. PCOA facilitates a review of a pharmacy student’s performance from year to year in relation to the entire curriculum and compares the results to national data. The purpose of this program is to provide participants with expertise from administrators and
faculty who used PCOA in their institutions.
Objectives
Discuss the benefits of using PCOA as a non-high stakes APPE-readiness exam
Discuss the benefits of using PCOA as a high-stakes APPE-readiness exam
Compare and contrast different techniques of setting minimal competency when using PCOA
Design a remediation plan for students who do not meet minimal competency on PCOA
Presenter IJustine S. Gortney, Pharm.D., BCPSDirector of Assessment, Division of PharmacyClinical Assistant ProfessorWayne State University,Eugene Applebaum College of Pharmacy and Health Sciences
Pharmacy Institution’s Stakes Attached to Student Performance
N=38Gortney JS, Bray B, Salinitri F. Am J Pharm Ed 2015; 79(9) Article 137.
5%8%
26%
61%
High -impacts pro-gressionMedium-course gradeLow-development planNone-no negative impact
Where We House PCOA: Professionalism Curriculum
Event P1 Year P2 Year P3 Year P4 Year
Professional Community
Service
Student Curriculum Student Student
Professional Meeting
Student Curriculum Student Student
Professional Event
Student Curriculum Student Student
Summative Evaluation
---- PCOA PCOA ----
Wayne State Operational Timeline
Education about Process
PCOA Thursday &
Friday
Results Receipt and Distribution
Students- August,
November, December Mid
January Distributed to Students and Faculty in April
Remediation (Student
Development)
Evaluated by SEC March-April
Advisor meetings May; Plans due in late May/Early June
Rollout: P2s 2013, P3s 2014
With Low Stakes, How Do We Incentivize Students? Social Element- Breakfast the day of exam High Performance on PCOA - Rewards
– Recognition at graduation for top two– Letter of recognition from the Dean’s Office for
top 10%– Overall examination scores above the college
mean will be rewarded with 1% bonus on your final PPS grade
Poor performance: Student development plan
WSU Results Distribution PlansStudent Class Meetings Faculty Meeting Rollout
• Review overall class results• Review how to interpret PCOA
scorecard• Distribute individual student
scorecards• Distribute individual student
development plans• P3 only: Review performance
change from P2 and reflection
• Provide comparison of the following:
• Overall school results• Comparison with national• Comparison within class and
across classes• Review timelines of student
development plans
Publish results on dashboard: http://www.cphs.wayne.edu/pharmd/strategic_assessment_2015/pcoa_data_2015_webupdate.pdf
Example of WSU P2 to P3 scorecard
Name P2 P3
Change in Score P2 to
P3
Assess the magnitude of the change in
PCOA
Confidence Rank
Overall_SS 343 369 26 -Area_1_SS 377 418 41 -Area_2_SS 338 357 19 -Area_3_SS 348 329 -19 -Area_4_SS 330 380 50 -
Overall Percentile 58 58 0 -Area 1 Percentile 66 85 19 -Area 2 Percentile 54 52 -2 -Area 3 Percentile 56 33 -23 -
Area 4 Percentile 54 53 -1 Confidence
RankPhysiology 70 70 0 7
Biochemistry 44 38 -6 6
Past: Student Development Plans / ThresholdsYear 2013 P2a 2014 P2a 2015 P2a 2014 P3b
(2013 P2)2015 P3 b
(2014 P2)Basic Bio A
B- - E
FG
L
Pharm Sci AB
I PQ
H NO
SBAS ABCD
JKLM
RS
AB
I
Clin Sci - - TUV
-
aTwo SD below WSU meanbSame as P2 2013 except Clin Sci (current P3 2 SD below P3 mean)
WSU Student Development Plan 2013-2015 (Student Driven)
Self assessment of the content area– How have you studied this content in the past?– What are your current study habits?– What do you perceive as a limitation to studying or
understanding the specific content at hand? Plan for improvement
– How will you proceed forward to improve your knowledge and skills in these areas?
• Generate a personal action plan that is practical and implementable
• Describe the timeframe you propose to do this over– What resources will you use to do this?
Self-reassessment of the content area after plan for improvement implemented
– What specific methods will you use to assess your progress? (eg. online quizzes in Access Pharmacy)
WSU Student Development Plans Modified for 2016 Cut-points modified
– Agreed 1 SD below 3yr NABP composite based on curriculum map for P2s & overall score P3s
• P2s for Basic Biomed (282) and PharmSci (280)
• P3s Overall Score (303) Revised Student Development Plan
– Development/rollout being finalized– Faculty and content-structured – Timed during 6 weeks of first APPE block
How Have We Used This Data in WSU Curricular Assessment
Limitations We Have to Date Using as low stakes; student motivation Most of our data analysis to this point has been in P2s
PCOA with P1 GPA: r=0.50, p<0.01 (N=143) PCOA with P2 GPA: r=0.53, p<0.01 (N=142) Early curriculum, science-based classes showed low-
moderate but significant correlations with PCOA scores Lacking NAPLEX correlation data due to date of exam
Does not appear on surface to relate to risk for APPE rotation failure; APPE grading scale lacking variability
At WSU, How Do We Feel this Has Benefited our Students and Program
Provided a starting block for summative evaluations Provided positive feedback for students regarding their
knowledge Provided specific assessment data to track our students’
knowledge and benchmark against others– Used specifically during curricular renewal process
• Part of a triangulation of data driving “renewal”• Plan on using as a consistency check after “renewal”
– ACPE self-study data– Higher Learning Commission (HLC) outcomes
measurement
Presenters II and IIIElizabeth A. Coyle, Pharm.D., FCCM, BCPS Assistant Dean of Assessment University of Houston College of Pharmacy Tara Jenkins, RPh, PhD Associate Dean of Academic Affairs University of Houston College of Pharmacy
Utilizing the PCOA as a High Stakes Exam
Objectives UHCOP High-Stakes Exam History PCOA Pilot Setting Cut-off Scores Remediation Questions
PCOA Project Team Elizabeth A. Coyle, Pharm.D., FCCM, BCPS
Catherine L. Hatfield, Pharm.D.
Gerida Brown
Tara L. Jenkins, RPh, Ph.D.
UHCOP Faculty
Milemarker Homegrown cumulative didactic exam Started as high stakes for P3’s in 2000
– Included questions from all 3 years of didactic curriculum
– Written by faculty– 200 case-based questions– Questions Angoffed – Mean of Angoff score from 200
questions had to be met for minimal competency
Julie Szylagyi. AJPE. 2008; 72 (5) Article 101.
Milemarker P3’s had to meet minimum competency
on Milemarker III to advance to APPEs– Average pass rate from 2000-2012 was
99% P1 & P2’s also took Milemarkers I & II
– Formative with remediation and points towards Milemarker III
New cases & validated questions were hard to accumulate
Julie Szylagyi. AJPE. 2008; 72 (5) Article 101.
PCOA Pilot
January of 2013 P3 students
– All required to take– Formative information only
P3 students still had to take Milemarker III in April 2013
Exam Comparisons
MILEMARKER III PCOA Overall Questions: 200 Questions 220 (20 “test” questions) Biomedical Sciences: 10.5% (21 questions) 16% (32 questions) Clinical Sciences: 61% (122 questions) 32% (64 questions) Pharmaceutical Sciences: 13.5% (27 questions) 30% (60 questions) Social/Behavior/Admin: 15% (30 questions) 22% (44 questions)
Results Comparison
Overall Average (% Correct): 78.9% UH 73%/ National 67% Biomedical Sciences (% Correct): 82.9% UH 73%/National 66% Clinical Sciences (%Correct): 80.4% UH 76%/National 73% Pharmaceutical Sciences (%Correct): 68.9% UH 67%/ National 57% Social/Behavior/Admin (% Correct): 76.5% UH 77%/ National 71%
Milemarker III PCOA
PCOA Benefits External measurement and benchmark of
our students’ performance in pharmacy curricula
Psychometrically sound questions written by content experts from colleges of pharmacy around the country
Ability to trend results longitudinally
Does not require faculty time to analyze and write capstone questions
Identify Cut off Score For P3 PCOA
Which Scores to Use? Percentile rank — the percentage of scores in a
distribution that is equal to or lower than the measured score.
Raw score — Total number of raw points a test taker receives based on the number of questions answered correctly; typically, for example, 20 correct answers means a raw score of 20
Scaled score — Scores that have been mathematically transformed from one set of numbers (i.e., the raw score) to another set of numbers in order to make them comparable in some way — for example, across different editions, or “forms,” of the same test
X Tuan and R Michel. R & D Connections No 16. Sept. 2011
National Scaled Scores700
361
Nat
iona
l Sca
led
Scor
e (n
orm
ed re
fere
nce
sam
ple)
357 (scaled normed referenced score change)
0Year
1 2 73 4 5 6 8
50%ile 50%ile50%ile 50%ile
50%ile 50%ile
50%ile
50%ile
UHCOP Cutoff Score* NSS vs. 10% below NSS vs. 1 or 2 STD
* PCOA Cutoff score for P3’s is 1 STD from the NSS scale score. The STD will be calculated from the PCOA student scores each year.
3 Year Comparisons
P3 Class (N)
Class Percentile
National Scaled Score
Class Ave Scaled Score
Standard Deviation
Passing Score
Number of Remediating
after 1st PCOA
Number (%) < NSS
Number ≥ 90th percentile
Number ≤ 20th percentile
P3 Class 2013 (N=109)
75th 361 405 41 N/A (pilot)
N/A (pilot) 14 (12.8%) 28 1
P3 Class 2014 (N=113)
56th 361 388 41 325 14* 45 (39.8%) 10 11
P3 Class 2015 (N=109)
65th 357 388 47 310 2 23 (21.1%) 22 3
* If we would have used 1 STD cutoff, 11/113 would have remediated
High Stakes Exams & Academic Performance
Coyle EA, Hatfield CL, Cottreau JM, Brown G, Smesny A. Poster (July 2013) AACP Annual Meeting, Chicago, IL
Remediation What kind of remediation do you use?
– Retest PCOA?– Homegrown exam?– Tailored remediation?
How do remediated students perform on rotations?
How often do you remediate?
• Meet minimum competency go on to APPE in May
PCOA Jan/Feb ALL P3's
• Meet minimum competency go on to APPE in May.
• Do not meet minimum competency, receive I in IPPE II.
• Follow a remediation plan/take a remediation course once classes end in May.
Remediation 1 March/April PCOA
• Meet minimum competency start APPE's in August .
• Do not meet minimum competency, follow remediation plan/take remediation course.
Remediation 2July PCOA*
Remediation 3Sep/Oct PCOA
PCOA Remediation Policy for P3’s
• Meet minimum competency start APPE's in January.
• Do not meet minimum competency, follow remediation plan/take remediation course.
After 3rd remediation repeat until student meets 6 year program limit. After 5th remediation I in IPPE II becomes a U.
* Not in PCOA normal testing windows
Conclusions Adapting PCOA as a high stakes
assessment was an easy transition at UHCOP
Minimum competency should be set off of the scaled score and not percentile or raw score
A clear remediation process is imperative to close the loop for high stakes exams
Q&A
Discussion Questions
1. Compare and contrast the PCOA exam with other assessment instruments 2. What are the advantages/disadvantages of various remediation plans? 3. Describe examples of successful student/faculty development plans4. What are the steps for closing the assessment feedback loop based on the PCOA results?
Thank You!