Standard #15 Assessment of Student Learning and Curricular … · 2012. 2. 22. · Page 1 of 8 Standard No. 15: Assessment and Evaluation of Student Learning and Curricular Effectiveness:

Page 1 of 8

Standard No. 15: Assessment and Evaluation of Student Learning and Curricular Effectiveness: As a component of its evaluation plan, the college or school must develop and carry out assessment activities to collect information about the attainment of desired student learning outcomes. The assessment activities must employ a variety of valid and reliable measures systematically and sequentially throughout the professional degree program. The college or school must use the analysis of assessment measures to improve student learning and the achievement of the professional competencies. The college or school must systematically and sequentially evaluate its curricular structure, content, organization, and outcomes. The college or school must use the analysis of outcome measures for continuous improvement of the curriculum and its delivery.

1) Documentation and Data:

Use a check to indicate the information provided by the college or school and used to self-assess this standard:

Required Documentation and Data:

NA Performance of graduates (passing rates of first-time candidates on North American Pharmacist Licensure Examination™ (NAPLEX®) for the last 5 years broken down by campus/branch/pathway (only required for multi-campus and/or multi-pathway programs) [SAME DATA ARE USED FOR STANDARD 3, 9, AND 15] [Download template from: http://www.acpe-accredit.org/pdf/Excel Documents/NAPLEX_CampusPathwayFiveYearReport.xls] [Download example data and chart from: http://www.acpe-accredit.org/pdf/Excel%20Documents/NAPLEX_CampusPathwayFiveYearReport_Example.xls]

Performance of graduates (passing rate of first-time candidates) on Multistate Pharmacy Jurisprudence Examination® (MPJE®) for the last 5 years [NOTE: THIS DATA VIEW IS USED WITH STANDARDS 3, 9, 15] [Download template from: http://www.acpe-accredit.org/pdf/Excel Documents/MPJE_FiveYearReport.xls] [Download example data and chart from: http://www.acpe-accredit.org/pdf/Excel%20Documents/MPJE_FiveYearReport_Example.xls] (Appendix 3E)

Performance of graduates (passing rate of first-time candidates) on North American Pharmacist Licensure Examination™ (NAPLEX®) for the last 5 years [NOTE: THIS DATA VIEW IS USED WITH STANDARDS 3, 9, 15] [Download template from: http://www.acpe-accredit.org/pdf/Excel Documents/Naplex_FiveYearReport.xls] [Download example data and chart from: http://www.acpe-accredit.org/pdf/Excel%20Documents/Naplex_FiveYearReport_Example.xls] (Appendix 3E)

Performance of graduates (Competency Area 1 scores for first-time candidates) on North American Pharmacist Licensure Examination™ (NAPLEX®) for the last 5 years [NOTE: THIS DATA VIEW IS USED WITH STANDARDS 3, 9, 15] [Download template from: http://www.acpe-accredit.org/pdf/Excel Documents/Naplex_FiveYearReport.xls] [Download example data and chart from http://www.acpe-accredit.org/pdf/Excel%20Documents/Naplex_FiveYearReport_Example.xls] Appendix 3F)



Required Documentation for On-Site Review:

Examples of instructional tools, such as portfolios, used by students to assist them in assuming responsibility for their own learning and for measuring their achievement

Page 2 of 8

Data Views and Standardized Tables:

It is optional for the college or school to provide brief comments about each chart or table (see Directions).

AACP Standardized Survey: Faculty – Question 46 (Appendix 15F)

AACP Standardized Survey: Student – Question 28

AACP Standardized Survey: Alumni – Question 17

AACP Standardized Survey: Preceptor – Question 22, 24

Optional Documentation and Data:

Other documentation or data that provides supporting evidence of compliance with the standard Examples of assessment and documentation of student performance, nature and extent of patient and health care professional interactions, and the attainment of desired outcomes; examples of how assessment data has been used to improve student learning and curricular effectiveness

2) College or School’s Self-Assessment: Use the checklist below to self-assess the program on the requirements of

the standard and accompanying guidelines:

S N.I. U

The college or school develops and carries out assessment activities to collect information about the attainment of desired student learning outcomes. The assessment activities employ a variety of valid and reliable measures systematically and sequentially throughout the professional degree program.

The college or school’s evaluation of student learning determines student achievement at defined levels of the professional competencies, in aggregate and at the individual student level

The college or school uses the analysis of assessment measures to improve student learning and the achievement of the professional competencies.

The college or school systematically and sequentially evaluates its curricular structure, content, organization, pedagogy, and outcomes.

The college or school uses the analysis of outcome measures for continuous improvement of the curriculum and its delivery.

The college or school has developed a system to evaluate curricular effectiveness.

The college or school ensures the credibility of the degrees it awards and the integrity of student work.

The college or school has mechanisms to assess and correct underlying causes of ineffective learning experiences.

The college or school’s assessments include measurement of perceived stress in faculty, staff, and students, and evaluate the potential for a negative impact on programmatic outcomes and morale.

3) College or School’s Comments on the Standard: The college or school’s descriptive text and supporting evidence should specifically address the following. Use a check to indicate that the topic has been adequately addressed. Use the text box provided to describe: areas of the program that are noteworthy, innovative, or exceed the expectation of the standard; the college or school's self-assessment of its issues and its plans for addressing them, with relevant timelines; findings that highlight areas of concern along with actions or recommendations to address them; and additional actions or strategies to further advance the quality of the program. For plans that have already been initiated to address an issue, the college or school should provide evidence that the plan is working. Wherever possible and applicable, survey data should be broken down by demographic and/or branch/campus/pathway groupings, and comments provided on any notable findings.

A description of formative and summative assessments and measures used to evaluate teaching and learning methods and curricular effectiveness, including nontraditional pathway(s) leading to the Doctor of Pharmacy degree (if applicable)

Page 3 of 8

A description of the assessment measures and methods used to evaluate student learning and achievement at defined levels of the professional competencies and educational outcomes, both in aggregate and at the individual student level

How achievement of required competencies by all students is assessed and assured on completion of the program

Comparisons with national data and selected peer-group programs (include a description of the basis for the peer-group selection) and trends over time

How feedback from the assessments is used to improve student learning, outcomes, and curricular effectiveness

The mechamisms in place to assess and correct causes of ineffective learning experiences, including the measurement of perceived stress in faculty, staff, and students and evaluation of the potential for a negative impact on programmatic outcomes and morale

How the college or school is applying the guidelines for this standard in order to comply with the intent and expectation of the standard

Any other notable achievements, innovations or quality improvements

Interpretation of the data from the applicable AACP standardized survey questions, especially notable differences from national or peer group norms

The Curriculum Committee, on behalf of the faculty, manages the development, organization,

delivery and improvement of the Pharm.D. curriculum. The Curriculum Committee also serves as

the central body for the management and systematic evaluation of curricular structure, content,

process and outcomes based on assessment data. Three members of the Curriculum Committee

also serve on the Assessment Committee and function as liaisons to communicate results of

curricular assessment and evaluation efforts to the Assessment Committee. Curricular evaluation

data, along with recommendations for improvement, are also shared with faculty individually, at

departmental meetings, full faculty meetings and retreats.

The Assessment Committee, in conjunction with the Associate Dean for Academic Affairs and

Assessment (ADAA&A), functions to develop, maintain and administer an overall plan for

evaluation of the pharmacy program, including assessment of student learning and curricular

outcomes. Assessing student learning in the classroom is promoted to faculty through

departmental activities and learning retreats. Instructors are expected to assess student learning

using a variety of assessments, both formative and summative. Cumulative exams or capstone

projects are encouraged to summatively assess student learning and promote knowledge

retention. A recent report from the College’s curriculum mapping software reveals that a variety of

assessment types are incorporated into the classroom to evaluate student learning (Figure 15-1).

Page 4 of 8

Figure 15-1. Assessment Methods

Student performance in IPPE and APPE is also assessed using multiple measures, including self-

evaluation, peer evaluation and preceptor evaluation. Remediation is available to professional

pharmacy students failing to achieve a minimum grade of “C” in a graded course or “P” in a

pass/fail course. Remediation will be tailored to the individual student and may include, but is not

limited to, one or more of the following in conjunction with academic advising: repetition of course

or pharmacy practice experience, re-examination, additional course work, reduced course load,

independent study, and campus tutoring.

In the past, there was a limited understanding among faculty of using assessment to drive

curricular change. More emphasis has been placed on the importance of “closing the assessment

loop” to make curricular changes and improve student learning. Faculty understanding of the role

assessment plays in the curriculum has improved, as reflected by comparing responses to

question 46 of the AACP Faculty Survey in 2008 vs. 2011 (Appendix 15F).

The University requires each program on campus to provide documentation of student learning on

an annual basis (Appendix 15A: University Assessment Report 2010). Historically, the pharmacy

program has received excellent ratings for its work, but the assessments used were all based on

Page 5 of 8

individual course concepts. Faculty were educated at a faculty retreat in 2011 to design

assessments aligned to Pharm.D. ABOs, conduct the assessment, make improvements based on

the results and document the assessment. (Appendix 15B: Student Learning Outcome

Assessment Form.) Effective with the 2011-2012 Assessment Report, course assessments

included in the University Assessment Report will align with the Pharm.D. program ABOs as a

measure of professional competency and educational outcomes.

The effectiveness of the curriculum and achievement of desired competencies is evaluated using a

variety of methods, including systematic course evaluation, curriculum mapping, standardized

progress exams, student portfolios, student evaluation by preceptor, first time NAPLEX and MPJE

pass rates, and AACP annual survey data. The Curriculum Committee developed a Course

Evaluation Plan (Appendix 15C: Course Evaluation Plan & Rubric) to systematically evaluate all

courses within the professional curriculum at least once every three years as a method to ensure

that the curricular structure, content, organization and pedagogy support student achievement of

the Pharm.D. program ABOs. Specific course related elements evaluated include the syllabus,

objectives, course content, teaching and learning methods, student assessment methods and

alignment to the ABOs. Active learning techniques to enhance student learning are emphasized,

as well as ensuring the course challenges students to achieve higher levels of learning through

problem solving, decision making, and/or creative thinking.

According to the Course Evaluation Plan, courses scheduled for evaluation are reviewed as soon

as possible after the course has been taught and the evaluation is completed within a six-week

period. Course faculty first self-assess their courses using a standardized rubric and forward the

rubric electronically (along with any additional materials requested) to the ADAA&A within two

weeks. A student focus group, comprised partially of student members from the Curriculum and

Assessment committees who have recently completed the course, also evaluates the course using

a standardized evaluation rubric and serve as ad-hoc members of the faculty work group. A faculty

work group (composed of faculty representation from the Departments of Pharmaceutical Sciences

and Pharmacy Practice) begin their review after the instructor self-assessment and student focus

group evaluation has been completed. A representative from the faculty work group discusses the

evaluation findings with the course instructor and garners feedback. A copy of the completed

evaluation rubric is given to the instructor, the department chair and the ADAA&A. Recommended

actions to improve course quality are expected to be implemented within the next academic year.

The plan was piloted with four courses in the fall of 2010 and fully implemented in the spring of

2011.

Page 6 of 8

The curriculum is mapped to Appendix B of the ACPE Accreditation Standards 2007 and to the

Pharm.D. program ABOs to ensure the required knowledge, skills and abilities to practice

pharmacy are taught. The curriculum is further assessed using the Pharmacy Curriculum

Outcomes Assessment (PCOA) exam, a standardized exam used to assess student progress and

curricular effectiveness. The College has administered the PCOA as a “low stakes” exam to

students in the spring of their P3 year annually since 2009. Overall, the mean scale score for

NDSU students exceeded the national average for each year the test was administered (Table 15-

2). In addition, NDSU’s mean percentile for the program has improved each year from 50% in 2009

to 57% in 2010, and 66% in 2011, indicating the NDSU students’ knowledge level is greater than

66% of students nationally. More detail on the PCOA is provided in Appendix 15D.

Table 15-2. Pharmacy Curriculum Outcomes Assessment Results by Year Administered

Test Parameters

2009 2010 2011 NDSU (n= 85)

National*(n= 842)

NDSU (n= 66)

National* (n= 823)

NDSU (n=79)

National*(n=1440)

Total Score Scale Score (mean) 367 366 355 344 367 340

% Correct (mean) 65 65 61 58 65 59 Percentile for Program

(mean) 50 ----- 57 ----- 66 ----

* Third year professional students enrolled in traditional Pharmacy Doctorate Programs

Experiential preceptors assess student competence in both the IPPE and APPE programs using a

student assessment evaluation that has been aligned to the ABOs (Appendix 15E: Student

Evaluation by Preceptor Mapped to ABOs). Student portfolios, which contain self-reflection and

examples of their work during their experiential experiences, are also assessed by preceptors and

instructors. In addition, student portfolios are assessed by the North Dakota Board of Pharmacy for

those students who sit for the North Dakota licensure exam.

Lastly, the College monitors first time pass rates for the MPJE and NAPLEX as a measure of

student success. NDSU pass rates for the MPJE exam have exceeded that of the national pass

rate for all years 2007 – 2011 (Appendix 3E). Likewise, NDSU students have exceeded the

national first time pass rate for the NAPLEX for the last five years (Appendix 3F) and scored

comparably to the national average in Competency Areas 1-3. Scores in Competency Area 3

(Assess, Recommend and Provide Health care Information that Promotes Public Health) had been

Page 7 of 8

somewhat lower than the national scores in the past four years. However, these scores have

improved since the College increased its emphasis on public health in the curriculum.

4) College or School’s Final Self-Evaluation: Self-assess how well the program is in compliance with the standard

by putting a check in the appropriate box :

Compliant Compliant with Monitoring Partially Compliant Non Compliant

No factors exist that compromise current compliance; no factors exist that, if not addressed, may compromise future compliance.

• No factors exist that compromise current compliance; factors exist that, if not addressed, may compromise future compliance /or

• Factors exist that compromisecurrent compliance; an appropriate plan exists to address the factors that compromise compliance; the plan has been fully implemented; sufficient evidence already exists that the plan is addressing the factors and will bring the program into full compliance.

Factors exist that compromise current compliance; an appropriate plan exists to address the factors that compromise compliance and it has been initiated; the plan has not been fully implemented and/or there is not yet sufficient evidence that the plan is addressing the factors and will bring the program into compliance.

• Factors exist that compromise current compliance; an appropriate plan to address the factors that compromise compliance does not exist or has not yet been initiated /or

• Adequate information was not provided to assess compliance

Compliant Compliant with Monitoring

Partially Compliant Non Compliant

5) Recommended Monitoring: If applicable, briefly describe issues or elements of the standard that may require further monitoring.

At this time, not all instructors have designed assessments for each ABO that is sufficiently

addressed in their course. Effective 2011-2012, course assessments included in the University

Assessment Report will align with the Pharm.D. program ABOs as a measure of professional

competency and educational outcomes. We will monitor progress through the Course Evaluation

Plan process and by mapping the curriculum to ABOs.

Page 8 of 8

Appendices Appendix 3E: 5 Year MPJE Data First Time Candidates

Appendix 3F: 5 Year NAPLEX Data First Time Candidates

Appendix 15A: University Assessment Report 2010

Appendix 15B: Student Learning Outcome Assessment Form

Appendix 15C: Course Evaluation Plan & Rubric

Appendix 15D: Pharmacy Curriculum Outcomes Assessment Data

Appendix 15E: Student Evaluation by Preceptor Mapped to Ability-Based Outcomes

Appendix 15F: AACP Survey Data

Page 1 of 1

Appendix 3E:

FIVE‐YEAR MPJE DATA FOR FIRST‐TIME CANDIDATES

COLLEGE/SCHOOL OF PHARMACY: North Dakota State University

Calendar Year School's Passing Rate

Candidates National Average

Passing Rate

Candidates

2007 95.51% 156 91.32% 16,343

2008 95.95% 148 91.42% 17,188

2009 99.40% 168 92.25% 18,605

2010 95.48% 177 93.89% 18,975

2011 97.92% 48 95.92% 13,116

Key:

Data entered by college/school from NABP reports

Cells containing a formula

95.51%

95.95%

99.40%

95.48%

97.92%

91.32% 91.42%92.25%

93.89%

95.92%

86%

88%

90%

92%

94%

96%

98%

100%

102%

2007 2008 2009 2010 2011

Passing Rate

First‐Time Passing Rate on MPJE2007 to 2011

School's Passing Rate

National Average PassingRate

Page 1 of 3

Appendix 3F: FIVE‐YEAR NAPLEX DATA FOR FIRST‐TIME CANDIDATES

COLLEGE/SCHOOL OF PHARMACY:

Calendar

Year

School's

Passing

Rate

Candidates National

Average

Passing

Rate

Candidates School's

Area 1

Average

Score

School's

Area 2

Average

Score

School's

Area 3

Average

Score

National

Area 1

Average

Score

National

Area 2

Average

Score

National

Area 3

Average

Score

2007 100.00% 83 95.33% 9,848 12.0 12.1 12.3 12.6 12.5 12.7

2008 96.51% 86 96.49% 10,615 12.6 12.5 12.3 12.4 12.4 12.5

2009 98.82% 85 96.48% 10,999 12.8 12.8 12.8 12.4 12.4 12.7

2010 94.51% 91 94.41% 11,361 12.6 12.6 12.6 12.5 12.4 12.8

2011 97.59% 83 96.24% 10,904 12.6 12.9 13.3 12.7 12.6 13.0

North Dakota State University

Page 2 of 3

100.00%

96.51%

98.82%

94.51%

97.59%

95.33%

96.49% 96.48%

94.41%

96.24%

91%

92%

93%

94%

95%

96%

97%

98%

99%

100%

101%

2007 2008 2009 2010 2011

Passing Rate

First‐Time Passing Rate on NAPLEX2007 ‐ 2011

School's Passing Rate

National AveragePassing Rate

12.0

12.6

12.8

12.6

12.6

12.6

12.4 12.4 12.5

12.7

11.5

12.0

12.5

13.0

2007 2008 2009 2010 2011

Student Average

Area Score

Average Area 1 Scores ON NAPLEX

2007 ‐ 2011

School's Area 1Average Score

National Area 1Average Score

Page 3 of 3

12.1

12.5

12.812.6

12.9

12.5

12.4 12.4 12.412.6

11.5

12.0

12.5

13.0

2007 2008 2009 2010 2011

Student Average

Area Score

Average Area 2 Scores on NAPLEX

2007 ‐ 2011



12.3 12.3

12.8

12.6

13.3

12.7

12.5

12.7

12.8

13.0

11.5

12.0

12.5

13.0

13.5

2007 2008 2009 2010 2011

Student Average

Area Score

Average Area 3 Scores on NAPLEX

2007 ‐ 2011



1

Appendix 15A:

2010

ANNUAL ASSESSMENT REPORT

Pharm.D. Degree Program

College of Pharmacy, Nursing & Allied Sciences NDSU

Due February 1, 2011

Contact: Donald Miller, Pharm.D. Professor of Pharmacy Practice and

Chair, Pharmacy Assessment Committee 118 Sudro Hall

2

Table of Contents

A. Overview and Response to NDSU Assessment Committee 3 B. Course Based Assessments 5

1. PHRM 475, Pharmacy Management 5 2. PHRM 480, Drug Literature Evaluation 8 3. PHRM 532, Infectious Disease Pharmacotherapy 9 4. PHRM 551/552, Pharmaceutical Care 11 5. Concept Pharmacy Lab Sequence 12 6. PSCI 368, Pharmaceutics I 14 7. PSCI 411, Pharmacodynamics I 15 8. PSCI 416, Pharmacodynamics VI 16 9. PSCI 412 and 415, Pharmacodynamics II and V 16 10. PHRM 340, Pathophysiology I 17

C. General Assessment of Outcomes (Experiential Programs) 18 1. Introductory Experiences (IPPE) 18 2. Advanced Experiences (APPE) 21 D. Program Evaluation/Indirect Assessment of Learning 22 1. NAPLEX and MJPE 22 2. Pharmacy Curriculum Outcomes Assessment (PCOA) 24 3. Self-rating on Levels of Implementation 25 E. Summary 25 Appendices

1. Abilities-Based Outcomes for Pharmacy 2. University and College Mission 3. Curricular Evaluation Plan 4. Course Evaluation Rubric 5. Pharmacy Course Assessment Form 6. APPE Evaluation Data 7. PCOA test results for 2010 8. Self-reporting of levels of implementation

2011 PHARMACY (Pharm.D. Program) ASSESSMENT REPORT

3

A. Overview and Response to NDSU Assessment Committee

Our current abilities-based educational outcomes (ABO) for the pharmacy professional degree program are attached as Appendix 1. These curricular outcomes are based on the 2004 revision of the national Center for Advancement of Pharmaceutical Education (CAPE) recommended outcomes, the 2007 standards of the Accreditation Council for Pharmacy Education (ACPE), and the Joint Commission of Pharmacy Practitioners’ Vision for Pharmacy 2015. Our College and pharmacy program mission are aligned with the University’s, with a strong emphasis on teaching (Appendix 2).

A critical part of curricular assessment is curriculum mapping. We purchased the Atlas Curriculum Mapping program and trained all faculty in its use in August 2006. All faculty have entered their course maps since the 2006-07 academic year, which allows us to track how outcomes are actually assessed in each course. Upon obtaining a comprehensive curriculum map, reports can be generated that allow us to see unnecessary duplication or omissions in the curriculum, the types and frequency of assessment in the curriculum, and the outcomes that each course addresses. We can then realign specific courses as needed, and seek specific course-embedded assessment data from faculty.

During fall 2007, Dr. Cynthia Naughton, at that time chair of the curriculum committee, and currently Associate Dean for Academic Affairs and Assessment, began organizing meetings between faculty who share common areas of curriculum. These colleagues meet to share findings from their individual review of their course maps. A desired outcome of the discussion among colleagues is that each instructor re-visits their individual course map and makes needed revisions in learning objectives, course content, teaching methods, and student assessments for future instruction. This process continues until a consensus map is agreed upon for their particular topic of study. Once consensus maps are agreed upon and posted as such, the curriculum committee can evaluate the curriculum to identify gaps in the curriculum, eliminate unnecessary duplication, ensure repetition of important concepts/topics, improve individual teaching effectiveness and methods to support longitudinal learning, facilitate an integrated and interdisciplinary approach to outcome-based education, encourage application of theory (basic sciences) to practice (clinical skills), and inform students what is expected of them.

The consensus maps were completed in 2008 and the curriculum committee recommended several changes in response to the information. These include combining and streamlining PSCI 470 and PHRM 471 into a single course, and reorganization with reduction in credits for the pharmacodynamics sequence. These changes allowed room to add credits for over-the-counter medications (in PHRM 452) and a new course in biotechnology. Four faculty attended an American Association of Colleges of Pharmacy (AACP) Curricular Summit in September 2009, at which time we used updated curriculum mapping data to assist us with proposing a substantial reorganization of the Pharm.D. curriculum. These changes were accepted by the faculty in 2010. Prepharmacy curriculum changes begin this fall, and some minor professional curriculum changes will begin next year as well.

During 2010 we also developed a comprehensive curricular assessment plan, in which all courses will be reviewed on a three year cycle (see Appendix 3). As part of this review each Pharm.D. course will need to document the curricular ability-based outcomes it addresses and provide data on assessment of student learning for at least one of those outcomes (see Appendix 4, Course Evaluation Rubric). To prepare faculty for this process, we held a faculty retreat on January

4

6, 2011 where faculty were shown examples of course-based assessments and taught about our expectations for the future.

Another area of importance is continued implementation of new national surveys developed jointly by the AACP and ACPE. The surveys can be administered electronically to graduating students, alumni, faculty, and experiential preceptors. In each survey, colleges can easily obtain their own data, and they can also request comparisons to national data or specified peer groups. In 2007 we began using the graduating student survey, in 2008 we added the preceptor and faculty surveys, in 2009 we did the alumni survey, and in 2010 we repeated the preceptor survey in addition to the graduating student survey. We discuss the surveys just briefly here, because they are indirect measures of student learning and evaluate programmatic outcomes. However, they have been useful in providing some corroborating data to other assessments of student learning.

Results of both these surveys are generally very positive and as good as or better than our peers in most respects. Students are happy with the curriculum structure and elective opportunities, physical facilities, the advanced practice rotation placement process, available rotations and the opportunity to apply clinical skills in their rotations. Some potential areas we could improve are in early practice experiences (already underway), opportunities for students to participate in research, and informing students of postgraduate educational opportunities. Continued use of these surveys will give us a chance to look at them longitudinally to see trends and help us prepare for our next accreditation visit in 2011-12.

The NDSU Assessment Committee criticized our report last year for two items that we address here. The first is incorporation of data from the Department of Pharmaceutical Sciences. We wish to point out that this assessment report is for the Pharm.D. professional curricular program, which is a joint effort of Pharmacy Practice and Pharmaceutical Sciences. We feel it would be inappropriate to separate the program assessment into two separate reports, or for the program to have two separate assessment committees. However, Pharmaceutical Sciences does assess and report separately on their graduate program.

Secondly, we agree with the need for more focus on course-based assessment of learning. We have added more of those assessments in this report, and more importantly have developed a process to assure that all Pharm.D. courses are assessed on a rotating basis. Assessment is now incorporated into our curricular review plan, in which all courses will be reviewed every three years. Course coordinators are required to state the ability-based outcomes their courses address and to submit evidence that these outcomes are actually assessed. A new Course Assessment Rubric was developed (Appendix 5). The curricular review process will be fully implemented with the 2011-12 academic year.

Members who served on the pharmacy assessment committee in 2009-10 or 2010-11 were Donald Miller, Chair (Pharmacy Practice), David Scott (Pharmacy Practice), Amy Drummond (Pharmacy Practice), Wanda Roden (Pharmacy Practice), Cynthia Naughton (Pharmacy Practice), Rebecca Focken (Pharmacy Practice), Dan Friesner (Pharmacy Practice), Elizabeth Skoy (Pharmacy Practice), Stefan Vetter (Pharmaceutical Sciences), Jagdish Singh (Pharmaceutical Sciences), Sanku Mallick (Pharmaceutical Sciences), Susan Nelson (alumnus), Adrienne Fisher (student), Jessica McLellan (student), and Amanda Hoffbeck (student).

B. Course Based Assessments

We are encouraging, through the course mapping and course review processes, faculty to

5

conduct course-embeddded assessments of learning. These may often cross several ABO learning outcomes.

1. Pharmacy Practice 475 (Management Business Plan)

Ability-based outcomes. There are two ability-based outcomes (ABOs) that are emphasized in this plan: 1) Communications (ABO 2A: written and verbal); and 2) Systems Management (ABO 5A: manage human, informational, fiscal, and technological resources using relevant legal, ethical, social, cultural, economic, and professional principles/issues to assure efficiency and cost-effectiveness).

Each 4-member student group is requested to complete a business plan with the goal of developing a new venture related to pharmaceutical care. Course concepts learned from class are to be applied to the plan. The plan is written with the idea that the chosen pharmaceutical care practice site will actually develop/incorporate this plan. Students are asked to include each of the following business plan sections: 1) Needs assessment/marketing strategy with physicians and patients; 2) Integrate patient (pharmaceutical) care process; 3) Computer/evaluation use; and, 4) Budget/references/appendices. Each draft section is worth 50 points and is graded using a rubric. The final report is graded based on a 100-point scale. Each four-member group presented their proposal during the last two weeks of class. Assessment. The assessment examines the quality improvement of the business plan instruction over the past 4 years and the continuous quality improvement (CQI) process that the instructors have used to improve this capstone course requirement. 2007. Our plan was to incorporate the business plan (written and presentation) with the Deselle Pharmacy Management textbook and use six-member groups. Each of the two instructors graded 50% of the section drafts and compared ratings after reading the first two plans. The average group score difference between the two instructors was only 0.2 point, so inter-rater reliability appeared high. Rubrics were used to grade the section drafts and the verbal presentations. Section draft score mean was 16.8 (range: 14-19.1). Verbal presentation mean score was 58.6 (range: 52.5-60) and final draft score mean was 47.6 (range: 42-50). Generally, the students underestimated the rigor of the section drafts and felt the instructor comments were too critical. However, several students reported in the summative course evaluation that the rigor of instructors’ expectations also lead to improved overall project. 2008. Based on 2007 evaluation results, we refined the business plan, decreased group size to four, encouraged communication between advisors and groups, and included a third instructor in the course. Each of the three instructors graded 33% of the section drafts and compared ratings. The average group score difference between the instructors was only 0.47 point, so inter-rater reliability appeared high. Rubrics were used to grade the section drafts and mean was 20.6 (range: 14-25). Verbal presentation mean score was 43.5 (range: 39-48) and final draft score mean was 76.7 (range: 68-84). Average group difference for the three instructors for the presentation and final draft was 0.3 points and 0.7 points, respectively. Results indicated that verbal communication skills of the students were good to excellent. There was significant improvement in the final drafts from the initial drafts, indicating the formative feedback at midcourse was valuable to students.

6

2009. We refined the business plan and the rubrics, continued group size at four, encouraged communication between advisors and groups, and added lectures on implementing PC and entrepreneurship, covered Professor Roden’s material on human resource management, and included formative assessment. Each of the two instructors graded 50% of the section drafts and mean was 22.0 (range: 18-24.5). The average group score difference between the two instructors was 1.4 points, so inter-rater reliability appeared high. Rubrics were used to grade the section drafts and the verbal presentations. Verbal presentation mean score was 45.4 (range: 39-48.5) and final draft score mean was 89.8 (range: 83-100). Average group difference for the two instructors for the presentation and final draft was 3.0 points and 2.6 points, respectively. For total points (section draft + final draft + presentation), the mean score was 157.2 (range: 144-172) and group difference for the two instructors was 1.7 points. On a percentage basis, the difference between instructors was less than 1%. Overall, the instructors were pleased with the final written project and the verbal presentation by the group members. 2010. We added page limits to the budget justification and marketing analysis sections in the appendices, added a lecture on community management, increased points for the section plan from 25 to 50 points, and developed a rubric for the final plan. Each section draft was graded and returned to each student. Each of the two instructors graded 50% of the section drafts and mean was 44.6 (range: 38-50). The average group score difference between the two instructors was 0.4 points. Verbal presentation mean score was 45.2 (range: 40-49) and final draft score mean was 92.5 (range: 83-100). Average group difference for the two instructors for the presentation and final draft was 3.1 points and 1.3 points, respectively. Each instructor felt this capstone course requirement was a lot of work, but also felt that it led to higher order learning of course content (e.g., assessment, synthesis) and application to real world issues. Overall, written communication skills between the draft section and final sections improved significantly, as demonstrated by the improvement in scores. Since, the capstone requirement is developing a business plan, the students in a group had to investigate, evaluate, and synthesize what is relevant to integrate into their business plans. Virtually all of the system management components listed in the ability-based outcomes are integrated into this project. Overall, the instructors tied in the CQI process over the last four years and added “how to” lectures, recitations on strategic planning, and financial analysis to give students sufficient information to put together a business plan. Student formative and summative comments are also used to improve the process and the plan. As evidenced in alumni and employer surveys, the College faculty needs to improve the communications and business skills of our students. Thus, in this capstone requirement, students improved their skills in systems management and in communication skills. Table 1. Assessment results, 2007-2010 Variable 2007 2008 2009 2010 N % N % N % N % Section draft

7

Points possible 25 100.0 25 100.0 25 100.0 50 100.0 Mean 16.8 67.2 20.6 82.4 22.0 88.0 44.6 89.2 Minimum 14 56.0 14 56.0 18 72.0 38 76.0 Maximum 19.1 76.4 25 100.0 24.5 98.0 50 100.0 Ave. score difference 1

1.4 4.2 0.4 0.8

Final draft Points possible 50 100.0 85 100.0 100 100.0 100 100.0 Mean 47.6 95.2 76.7 90.2 89.8 89.8 92.5 92.5 Minimum 42 84.0 68 80.0 83 83.0 83 83.0 Maximum 50 100.0 84 98.9 100 100.0 100 100.0 Avg. score difference 1

0.7 0.82 2.6 2.6 1.3 1.3

Presentation Points possible 60 100.0 50 100.0 50 100.0 50 100.0 Mean 58.6 97.7 43.5 87.0 45.4 90.8 45.2 90.4 Minimum 52.5 87.5 39 78.0 39 78.0 40 80.0 Maximum 60 100.0 48 96.0 48.5 97.0 49 98.0 Avg. score difference 1

0.3 0.6 3.0 6.0 3.1 6.2

Total Points possible 135 100.0 160 100.0 175 100.0 200 100.0 Mean 157.2 89.8 182.3 91.2 Minimum 144 82.3 165.5 82.8 Maximum 172 98.3 193 96.5 Ave. score difference 1

1.7 0.97 4.8 2.4

1 average score difference between instructors (surrogate measure of inter-rater reliability) PCOA results. The PCOA exam results (see details in section D) are also included in this assessment. In 2009, the score in “practice management” for our P3 students was 76% which is slightly better than the national P-3 score of 74%. In 2010, our P3 students score was 58%, which is better than the national P-3 score of 56%. So, overall our students are performing at or slightly above the national level in practice management. While the management course is the most significant contributor to this score, there are other confounding variables (i.e., Concept Lab, experiential education, elective coursework, and work experience) that also contribute to this achievement test measure of the “systems management” learning outcome.

2. Pharmacy Practice 480 (Drug Literature Evaluation) Ability-based Outcome: ABO subtasks 4.E and 6.B.2 are each “retrieve, analyze, and interpret the professional, lay and scientific literature to provide drug information to patients, caregivers, and other health care providers”

8

PHRM 480, Drug Literature Evaluation uses two capstone assignments to assess these abilities. Methods are summative and directly assess learning. 1. The first is a written evaluation of a single research paper on a clinical topic. Students are graded on their ability to provide a reasonably complete, thoughtful, and accurate evaluation of the paper. Students practice by discussing articles within class and by evaluating one additional graded article assignment with a small group of 4 students. They are then given one 50 minute class period to read and critique an article on their own near the end of the course. The article for spring 2010 was “Hypokalemia following polyethylene glycol-based bowel preparation for colonoscopy”, in Annals of Pharmacotherapy 2010;44:466-70. Competence on the research paper evaluation was evaluated by grades on the assignment (using a guideline on key points), which broke down as follows for the 92 students in spring 2010: Score of 90% or better (A) 70 Score of 82% or better (B) 16 Score of 75% or better (C) 5 Score less than 75% (D) 1 The results indicate students achieved a high level of competence in the task of evaluating the paper. By the benchmark standard of B or better, 86 of 92, or 93% of students performed well. In evaluating the assignment this year, the article utilized may have been a little too straightforward and thus grades were high. The most common deficiency in the task was lack of completeness in fully evaluating strengths and weaknesses of the paper. 2. Second, students are required to write a term paper that summarizes current literature on a topic for an individual drug, and that finishes with a synthesis and sound conclusion on the topic. Overall scores on the term paper were: Score of 90% or better (A) 69 Score of 82% or better (B) 15 Score of 75% or better (C) 6 Score less than 75% (D) 2 Again, results indicate a high level of competence was achieved. For this assignment, the two students who received a D grade were required to rewrite the paper until it was satisfactory. That only two students needed to rewrite is very positive and an improvement from data reported several years ago when typically 6 to 9 students/year received a D grade. Since papers were evaluated with a standard rubric, we could determine which areas were weakest. The following numbers of students received a failing score or a score of “needs improvement” on the rubric items: Unsatisfactory Needs Improvement Content and accuracy 2 13 Organization and development 0 7

9

Spelling and grammar 0 3 Documentation and reference format 0 1 Persuasive and accurate conclusions 1 24

The data indicate students have good basic writing skills at this point in their career, but often need to be more comprehensive in their content and need improvement in writing strong conclusions that show critical thinking.

These data can be correlated to items on the graduating student and preceptor surveys. For graduating students, in item #27, 98% agreed (and 43% strongly agreed) with the statement: “The Pharm.D. Program prepared me to evaluate the health sciences literature”. In the preceptor survey, 97% of preceptors agreed (and 37% strongly agreed) with the same statement regarding preparation of our students. Future Plans: The 2 weaker areas in the term paper could be reinforced and reassessed somehow in P3 year coursework (e.g. the Infectious Disease course current events paper). The class that was assessed had not been required to take the upper level English course (English 324 or 325) being implemented for general education. In future years it will be worthwhile to see if performance improves among students who took the upper level English class.

3. Pharmacy Practice 532 (Infectious Disease Pharmacotherapy)

Curriculum ABO: Communication. Communicate in a caring and respectful manner in all situations using appropriate listening, verbal, nonverbal, and written skills Curriculum ABO: Patient-Centered Care E. Retrieve, analyze, and interpret the professional, lay, and scientific literature to provide drug information to patients, caregivers, and other involved health care providers.

A current events project was added to the course over the past few years and we wanted to assess its value. Third-year Pharm.D. students researched a current event topic related to infectious diseases and prepared a written and verbal summary that was presented to their classmates. Scores on course examinations and quizzes to assess students’ knowledge of current events material were compared to core material. A survey of students assessed students’ awareness of current events related to pharmacy practice, and their understanding of the process to prepare and present information to a group. This project was published in 2010.

Results (tables from American Journal of Pharmaceutical Education 2010; 74 (4) Article 58).

10

Thus, students learned material from the current events material as well or better than the core material. Based upon a student self-assessment survey, this active-learning activity increased students’ awareness of current events related to pharmacy practice, and their understanding of the process to prepare and present information to a group. There was a positive but not significant improvement in students’ desire to prepare and present information to a group; ability to evaluate, organize, and present information in a written report; and ability to verbally present material to a group.

Future Plans: Based upon the positive results, current events presentations will be

continued in this course, and have been implemented in a second pharmacotherapy course.

4. Pharmacy Practice 551/552 (Pharmaceutical Care V/VI) Curriculum ABO: Communication

11

A. Communicate and collaborate with patients, caregivers, health care professionals, administrative and support personnel to engender a team approach to patient-centered care.

Curriculum ABO: Patient-Centered Care E. Retrieve, analyze, and interpret the professional, lay, and scientific literature to provide

drug information to patients, caregivers, and other involved health care providers. Curriculum ABO: Public Health

A. Assure the availability of effective health promotion and disease prevention services. 1. Promote public awareness of health and disease.

B. Provide population-based care. 2. Retrieve, analyze, and interpret the professional, lay, and scientific literature to provide

drug information to other health care providers and to the public. In the last few years this course sequence has required a public health poster project in

which third-year Pharm.D. students collaborated in pairs to research a public health topic relating to pharmacy practice. Each student group prepared an informational poster, while receiving feedback from a faculty mentor at each stage of the project. The students have presented their completed posters at a statewide pharmacy conference. Faculty members evaluated the posters with a grading rubric, and students completed a survey instrument that assessed the overall experience. The results of this work have also been published. Results (table from American Journal of Pharmaceutical Education 2011; 75 (1) Article 2)

In general, faculty members rated the class highly across all domains of the grading rubric. In addition, the class generally agreed that the poster project increased their awareness of public health issues related to pharmacy practice, overall knowledge of public health, and presentation skills. The implementation of a poster project was well received by students and faculty members as an effective method for enhancing public health instruction in the Pharm.D. program.

5. Concept Pharmacy Laboratory Program Objective Course Objective Corresponding to Activity

12

Objective 4: Patient-Centered Care

Retrieve drug information to solve drug related questions

Objective 5: Systems Management

Interpret a prescription identifying abbreviations, scheduled drugs, official names, and equivalents

Evaluate prescriptions for legal requirements and appropriate dosing

Select appropriate product(s) for dispensing, including generic substitution

Prepare and dispense prescriptions, compounded products and nursing home dispensing

Counsel patients appropriately on medications

Activities One of the features of our curriculum is a sophisticated pharmacy practice laboratory that students work in every week. Students spend at least one semester in the Concept Pharmacy in the first, second, and third year of the curriculum. The lab seeks to integrate classroom knowledge with practice skills and offers an excellent opportunity to assess practice skills at multiple points in the curriculum. To assess the impact of the lab and progress of students through the curriculum, lab instructors have developed several assessments around our ability-based outcomes.

During 2010, lab faculty again chose several specific competencies for assessment for the P1, P2, and P3 students. In general, students received information required to meet course and activity objectives through lectures and handouts plus 2 to 6 hours of practice for each skill in the lab. Each student had a formative assessment on each skill during their weekly activities, and a summative assessment on each skill at the end of the semester. Standard medication orders were used for the practical assessments, and students received their medication order by random distribution on the day of the competency assessment. It is important to note that during the spring 2010 semester, a benchmark of 80% was set for pass/fail. During the fall 0f 2010, this passing benchmark was discontinued for individual competencies. Remediation appeared ineffective because students often resented the remediation and failed to take it seriously. By eliminating the remediation week, all students now can spend an additional week in lab. In addition, the pass score for lab overall was raised from 70% to 74%. Overall, we feel these changes have positively improved the climate for learning.

The assessments are so numerous that we provide only some examples below.

Results Competence Summative Assessment Average Score Failed (1st Attempt) Spring 2010 Dispensing/Consultation (P2) 92% 5 Nonsterile Preparation(P2) 87% 18 Sterile Preparation (P3) 75% 20 Fall 2010 Dispensing/Consultation (P1) 95% Nonsterile Preparation (P1) 92% Nonsterile Preparation (P3) 95%

13

Sterile Preparation (P3) 94% Sterile Preparation (P1) 94%

These data show that students generally score highly on overall competence as assessed by average score. Results of the assessments are also used to reassess faculty teaching approaches for each skill. For example, due to assessment of spring 2010 and previous semester, faculty decided to remove the 80% benchmark, and remove practical remediation. In addition, scoring rubrics and standards are updated periodically to keep up with professional standards. A detailed specific example is given below. Full details of all the assessments for each class and each competency are available on request. Nonsterile Preparation

Students received information required to meet course and activity objectives through lectures, handouts, and instructional videos. Students were provided opportunities to practice extemporaneous compounding during three lab sessions (6 hours) prior to practical examination. During this lab time, students each made a total of three lotions and three ointments. The practical examination consisted of preparation of an ointment. Standard prescriptions were used for the practical and students received their prescription by random distribution on the day of the examination. Students received a copy of the practical grading rubric during the weeks prior to the nonsterile practical. Students were given the opportunity to attend one open Concept Pharmacy session where they could ask questions to the present faculty and prepare any of the required preparations for extra practice.

The average score was 47.8 points out of a possible 50 points (95%). Among student errors

19% did not document the correct quantity of ingredients. This error could have occurred by incorrect calculations, or by not correctly choosing and levigating or solubilizing agent. 17% did not document the correct procedure, and 10% did not prepare a pharmaceutically elegant preparation. It was observed that 24.4% of students did not properly prepare the prescription label, 6% did not label their medication with the correct medication, and 6% did not choose the appropriate beyond-use date of their preparation. Assessment Results of these Concept Pharmacy Assessments constantly suggest revisions to the grading rubrics and better ways to teach and practice the various skills. Changes are made each semester on an ongoing basis. Medication Therapy Management Faculty also completed a medication therapy management project with third year professional students in the spring of 2010. Assessment data and description below is taken from published results in the American Journal of Pharmacy Education, 2010, Volume 74, Issue 10.

Medication therapy management activities were integrated into the laboratory to teach the skills necessary to provide MTM services. Students provided a real-life, non-standardized patient service to volunteer faculty and staff from across the NDSU campus. Students received lectures focused on the core elements of MTM, patient-centered communication, and techniques used for point of care screenings. Paper-based MTM cases were used during weekly laboratory activities and students practiced point of care testing through peer screening. Five pharmacist faculty

14

members and one pharmacy resident served as preceptors for the encounters and evaluated students’ performance. These individuals were practicing pharmacists with experience providing MTM and disease state management services.

A pre-post encounter survey assessed students’ perceptions of their confidence and ability to provide MTM services. The pre-encounter survey was administered after the lectures and paper-based MTM cases and prior to the participant encounters. The post-encounter survey contained an additional seven items which allowed students to rate the effectiveness of the project and was administered upon completion of the project, after the small group case presentations.

Variable Question Mean

(N=80) SD

Q1 As a result of this project, I am better prepared to work with patients during advanced pharmacy practice experiences.

4.6 0.53

Q2 As a result of this project, my clinical skills have improved. 4.5 0.55 Q3 As a result of this project, my confidence in working with

patients has improved. 4.6 0.51

Q4 As a result of this project, I am better able to integrate my clinical knowledge with communication skills.

4.5 0.52

Q5 As a result of this project, I am better prepared to provide MTM services to patients during advanced pharmacy practice experiences.

4.5 0.52

Q6 The grading of this project was fair. 3.6 0.69 Q7 The participant provided me with feedback that will allow

me to improve my MTM skills. 3.9 0.86

6. PSCI 368 (Pharmaceutics I)

Pharmaceutics I covers material on dosage forms and physical properties of drugs. For the past three years, 10 questions (multiple choice type, 2 points each) were asked of the students during the first week of classes for the fall semester. The students were assured that their performances on the assessment would not affect the grades in any way. The same set of 10 questions was posed to the students during the last week of classes. Comparison of the responses from the students indicates that the overall average score increased; the standard deviation decreased (Table 1). The post-course score was significantly greater (unpaired t-test, p<0.0001) than the score of pre-course for each year. These results support the effectiveness of the course PSCI 368 for increasing knowledge.

Category Pre-course Post-course

2008 2009 2010 2008 2009 2010

Average score 8.35 7.83 7.43 15.72 16.29 17.08

Standard deviation 3.26 3.35 3.26 2.71 2.95 2.23

15

Variance 10.65 11.22 10.66 7.37 8.72 4.99

Highest score 16 14 16 20 20 20 Lowest score 2 0 2 10 6 12

However, in 2010, an additional assessment was added from the final exam to assess student

performance on open-ended questions that applied material from the course, and addressed ABO 3.D. Based on differences in chemistry, physical properties, pharmacology, pharmacokinetics, and pharmacogenomics, recommend changes in pharmacotherapeutic regimens that minimize drug interactions, reduce side effects, increase compliance, and improve therapeutic outcomes. In the first question, which required calculation of a half-life, the average score was 4.9 out of 5 points. For the second question, which required understanding of how chemical structures change at different pH’s, the average score was 7.7 ±2.2 out of 10.

The instructor concluded that students are learning the material in the course, and generally are able to apply the concepts on the final exam. However, they are having some difficulty with acids and bases. This presents a need and opportunity to alter the teaching methods for better comprehension of this topic. 7. PSCI 411 (Dynamics I)

This course provides the basic principles and foundation for pharmacology. It deals with the principles of pharmacokinetics/pharmacodynamics, drug metabolism, receptors and toxicology. As described above, 10 questions (multiple choice types, 1 correct answer) were asked to the students during the first week of classes for the spring semester, and the same set of 10 questions was asked during the last week of the semester. Comparison of the responses from the students suggests that the overall average post-course scores significantly increased (unpaired t-test, p<0.0001) in comparison to pre-course scores, confirming students’ learning in this course.

Summary of the assessment results for 2010 for PSCI 411:

Category Pre-Course Post-Course

Points Possible 20 20

No. of students 88 90

Avg. score 10.55 16.04

Standard deviation 2.60 1.80

Highest score 12 20

Lowest score 4 12

8. PSCI 416 (Dynamics VI) Student learning of knowledge was also assessed in PSCI 416, a basic pharmacology

course covering neurological drugs. As a part of the assessment process, 10 questions (multiple choice type, one correct answer) were asked to the students during the first week of classes for the spring semester with assurance that their performances on the assessment will not affect the grades. The same set of 10 questions was asked to the students during the last week of the semester. Comparison of the responses from the students suggests that the overall average post-

16

course scores significantly increased (unpaired t-test, p<0.0001) in comparison to pre-course scores.

Before Teaching After Teaching Minimum 3 64 Maximum 94 100 Mean 43 89 Median 44 92 Std Dev. 33 12 Std Error 11 4

9. PSCI 412 (Dynamics II) and PSCI 415(Dynamics V)

We further assessed students’ learning in additional courses (PSCI 412 and PSCI 415). PSCI 412 is a basic pharmacology course. The chemical structure, medicinal and pharmacological properties of therapeutic agents used in the treatment of cancerous and infectious diseases are taught in this course. PSCI 415 deals with the pharmacological properties of therapeutic agents used in the treatment disorders of the gastrointestinal/blood systems and anti-inflammatory agents. As a part of the assessment process, 10 questions (multiple choice types, 1 correct answer) were asked to the students during the first week of classes for the fall semester in both of the above courses with assurance that their performances on the assessment will not affect the grades. The same set of 10 questions was asked to the students during the last week of the semester. Comparison of the responses from the students suggests that the overall average post-course scores significantly increased (unpaired t-test, p<0.0001) in comparison to pre-course scores in both PSCI 412 and PSCI 415 (Table 2). These results suggest the effectiveness of students’ learning in the courses PSCI412 and PSCI 415. The results obtained in 2009 assessments have been included in Table 2 for comparison. Summary of the assessment results for PSCI 412 and PSCI 415: PSCI 412 PSCI 415 2009 2010 2009 2010 Category Pre-

Course Post-

Course Pre-

Course Post-

Course Pre-

Course Post-

Course Pre-

Course Post-

Course Points possible

10 10 10 10 100 100 100 100

No. of 83 73 86 84 64 65 78 72

17

students Average score

2.18 7.37 2.66 7.30 40.00 64.77 41.92 74.17

Standard deviation

1.14 1.77 1.54 2.10 13.09 20.09 14.24 16.42

Highest score 6 10 6.25 10 70 100 80 100 Lowest score 0 3 0 3.5 10 30 10 20 10. PHRM 340 (Pathophysiology I) In the past, first year (P1) students in pathophysiology have not been exposed to the process and thinking behind the pharmaceutical care process. This year students were introduced to the process with two disease state examples. Student learning was determined through graded application of a case and its subsequent disease care plan development. Data from the graded care plan revealed a mean score =7/10, median=7, SD=1.45, high=10/10, low= 3/10. This demonstrates that the majority of P1 students have sufficient knowledge and background to begin applying critical thinking skills in the development of a patient-specific disease-based care plan. For more detailed assessment, the care plan was divided into five sections: findings, therapeutic goals, recommendations, monitoring, and patient education. Students had to identify critical elements within each section for proper disease management. The section of the care plan most difficult for students was patient education where only 42% were able to integrate the correct information from the case and lecture. Other sections were scored as follows: 43% for findings, 48% for recommendations, 90% for therapeutic goals, and 90% for monitoring. To help students develop better critical thinking skills for integrating patient information with overall knowledge of disease states, more in-class examples will be incorporated into the course. These additional examples will focus on the more difficult areas of patient education, findings, and recommendations.

C. General Assessment of Learning Outcomes

1. Introductory Experiential Rotations

Introductory Pharmacy Practice Experiences (IPPE’s) are pharmacy practice experiences occurring in the first three years of the pharmacy curriculum which introduce students to the practice of pharmacy through a variety of actual practice situations. According to accreditation standards, these experiences should integrate with the curriculum and build in a progressive

18

manner to the Advanced Pharmacy Practice Experiences (APPE’s). New web-based software (E*value) is now fully implemented for management of

introductory and advanced practice rotations. E*Value is a web-based program which helps maintain accreditation standards and remove administrative burdens related to student rotation selections, course evaluations, and performance reporting. Partnering with E*Value has allowed for the department to easily collect data and formulate reports. The software allows students to maintain portfolios of their work for assessment purposes, and it allows easier summary of preceptor ratings of student competency.

ACPE accreditation standards require students to have significant experiences prior to advanced pharmacy practice experiences specifically in the areas of institutional and community pharmacy practice. In order to satisfy this requirement, the pharmacy curriculum now requires two three-week introductory experiences in actual pharmacy practice settings. The first experience, PHRM 355 (formerly PHRM 352L) is a three week experience in an institutional (hospital) pharmacy setting following the first professional year, although the first year we offered this rotation it was only two weeks in length. The second experience, PHRM 455 (formerly PHRM 451L) is a three week experience in a community pharmacy setting following the second professional year. Rotation experiences of equal length were stipulated by a clarification published by our pharmacy program accrediting body in 2009 after the completion of our first institutional IPPE experience.

While it can be a challenge to gauge the learning that occurs throughout the introductory pharmacy practice experience itself, the introductory experience allows students to integrate and practice knowledge learned throughout the pharmacy program in an actual practice environment. One of the key uses of assessment data from the IPPE program can be seen in documentation of progression through the pharmacy program toward accomplishment of ability-based outcomes as measured by practicing pharmacist preceptors (supervisors in practice experiences). Specifically, these preceptor evaluations of the student can be used to provide direct measurement of ability-based outcomes that are difficult to measure in a classroom setting.

For this assessment report, a comparison of the baseline of two years of PHRM 355 (formerly PHRM 352L) Institutional IPPE will be provided for two program ability-based outcomes: Attitudes and Values, and Communication. Then, a comparison of the same group of student evaluations for the same criteria in both PHRM 355 (formerly PHRM 352L) and PHRM 455 (formerly PHRM 451L) will be evaluated for the same ability-based outcomes. The ability-based outcomes are being assessed using aggregate student performance data for a sample of specific questions within the student IPPE performance evaluation related to the programmatic ability-based outcomes. The evaluation of the student was completed by preceptors at the conclusion of each of the three week work experiences (two week work experience in 2009 for PHRM 352L). Student scores on this evaluation were reported as excellent, average or poor. An excellent on the evaluation was assigned a numerical value of 3, an average on the evaluation was assigned a score of 2, and poor on the evaluation was assigned a score of 1. Results are reported according to mean and standard deviation. The chart also shows which programmatic ability-based outcome to which the questions can be linked. Baseline Assessment of First Year Student Performance on Programmatic Ability-based Outcomes Following an Institutional Introductory Pharmacy Practice Experience

19

Attitudes and Values Sample of Questions from Preceptor Evaluation

Linked to ABO

Mean PHRM 352L Class of 2012 In 2009 (n= 83)

Std. Dev. PHRM 352L Class of 2012 In 2009

Mean PHRM 352L Class of 2013 In 2010 (n=87)

Std. Dev. PHRM 352L Class of 2013 In 2010

Student seeks knowledge, asks questions, searches for information and takes responsibility for their own learning

ABO 1.D 2.66 0.50 2.70 0.46

Student responds openly and positively to constructive feedback and modifies behavior if necessary

ABO 1.D 2.63 0.49 2.63 0.49

Student makes decisions and performs duties in accordance with legal, ethical, social, cultural, economic, and professional guidelines

ABO 1.C 2.56 0.50 2.55 0.50

This comparison shows there was little, and in some cases no change between years in the

evaluations for student attitudes and values. Communication Question Linked to

ABO Mean PHRM 352L Class of 2012 2009 (n=83)

Std. Dev. PHRM 352L Class of 2012 2009

Mean PHRM 352L Class of 2013 2010 (n=87)


Student is able to communicate in a caring and respectful manner in all situations using appropriate listening, verbal, nonverbal and written skills

ABO 2 2.61 0.49 2.63 0.49

Student clearly communicates thoughts, and uses appropriate vocabulary, gestures, and mannerisms to convey information

ABO 2.A 2.61 0.49 2.57 0.50

Identifies and collects all information needed to respond to an information request from another health care professional using appropriate resources and technology

ABO 2.A ABO 2.B

2.44 0.52 2.64 0.48

Responds to an information request from a patient

ABO 2.A 2.28 0.45 2.52 0.51

This comparison shows there was little change in the first two questions related to the

student’s communication skills, but an increase the ability to respond to information requests from a healthcare provider and patient in the 2010 student evaluations.

20

First, with the addition of a third week to the rotation, students may have had increased opportunities and/or confidence to respond to information requests. Second, in response to comments received on student evaluations following the 2009 course, a supplemental chart reading activity was incorporated into the coursework the spring semester prior to IPPE in order to familiarize students with medical charts and the information contained within them. This may have helped to improve confidence in identifying and gathering information needed to adequately respond to information requests. Finally, students in the course in 2010 may have had additional learning opportunities in coursework during the first professional year that better prepared them for these types of tasks.

Future assessments monitor if this trend continues. Longitudinal Assessment of Student Performance on Programmatic Ability-based Outcomes Following Introductory Pharmacy Practice Experiences in Institutional and Community Pharmacy Practice. Attitudes and Values Question Linked to

ABO Mean PHRM 352L Class of 2012 2009 (n= 83)


Mean PHRM 451L Class of 2012 2010 (n=83)


Student seeks knowledge, asks questions, searches for information and takes responsibility for their own learning

ABO 1.D 2.66 0.50 2.86 0.35

Student responds openly and positively to constructive feedback and modifies behavior if necessary

ABO 1.D 2.63 0.49 2.80 0.41

Student makes decisions and performs duties in accordance with legal, ethical, social, cultural, economic, and professional guidelines

ABO 1.C 2.56 0.50 2.75 0.44

Communication Question Linked to

ABO Mean PHRM 355 Class of 2012 2009 (n = 83)


Mean PHRM 455 Class of 2012 2010 (n = 83)


Student is able to communicate in a caring and respectful manner in all situations using appropriate listening, verbal, nonverbal and written skills

ABO 2 2.61 0.49 2.82 0.39

Student clearly communicates thoughts, and uses appropriate vocabulary, gestures, and mannerisms to convey information

ABO 2.A 2.61 0.49 2.73 0.44

21

Identifies and collects all information needed to respond to an information request from another health care professional using appropriate resources and technology

ABO 2.A ABO 2.B

2.44 0.52 2.75 0.44

Responds to an information request from a patient

ABO 2.A 2.28 0.45 2.75 0.48

Student performance in each of the questions linked to the ability-based outcomes improved

and the standard deviations in many cases decreased. This shows learning has occurred throughout the program in a progressive fashion. Future assessments will look at preceptor evaluations in APPE rotations as a whole as well as each of the required advanced rotations in community and institutional pharmacy practice.

2. Advanced Experiential Rotations Advanced Pharmacy Practice Experiences (APPE’s) are pharmacy practice experiences

which occur during the final (fourth) year of the professional pharmacy program. According to the ACPE accreditation standards, “the organization of the advanced pharmacy practice experiences should provide a balanced series of required (the majority) and elective experiences that cumulatively provide sustained experiences of adequate intensity, duration, and breadth (in terms of patients and disease states that pharmacists are likely to encounter when providing care) to enable achievement of stated competencies as demonstrated by assessment of outcome expectations.”

The Department of Pharmacy Practice requires a total of eight 5-week rotations. The five required APPE experiences expose students to primary, acute, chronic, and preventive care among patients of all ages and develop pharmacist-delivered patient care competencies in the following settings:

community pharmacy hospital or health-system pharmacy ambulatory care inpatient/acute care general medicine rural The three elective APPE experiences may occur in other settings such as research,

management, drug information, education, managed care, long-term care, hospice, and home health care. These experiences complement the required experiences and provide adequate and innovative opportunities for students to mature professionally and in accordance with their individual interests. According to accreditation standards, all experiences should integrate with the curriculum and provide outcomes to reflect competencies necessary of a generalist, entry-level pharmacist. The preceptor evaluation of students is used to assess these students during each of their eight rotations.

Competencies linked to the Ability-based Outcomes are reflected in the questions that were assessed during 2009/2010. A comparison of data is also reflected from June 2010 through December 2010 (See Appendix 6). As seen in the data, NDSU fourth year pharmacy students perform above a mean of 2.5 out of 3 on every competency. It should be noted that in comparing 2009/10 to the current academic year, students in this year have not yet been evaluated on their

22

final 3 rotations, where evaluation scores will most likely be higher. If a student is rated under 70% on an evaluation, E*Value will send an email to the

Director of Experiential Programs. This results in a conversation with the student and preceptor to better coach the student and allow the student to practice and enhance performance of that skill during the next rotation.

Within the next year, based on feedback from preceptors, the Experiential Education Committee will be working on increasing the evaluation scale from 3 to 5 ratings.

In 2012, the graduating class will be the first class to have completed the formalized IPPE sequence. At that time we will be able to analyze longitudinal data for an entire 4 years of IPPE and APPE during the professional program. The following circumstances will need to be taken into consideration when evaluating the data:

1. Preceptor variability in grading, 2. Preceptor variability in expectations, 3. Individual times that a student completes a rotation. For example, a student completing an

acute care rotation during Rotation # 1 will not demonstrate the same competencies as if that student had completed the rotation during Rotation # 8.

4. Individual student goals and objectives for specific types of rotations, and 5. Student initiative and motivation for specific types of rotations.

D. Program Evaluation/Indirect Assessment of Learning 1. NAPLEX Scores

NDSU graduates= scores on their first attempt at the spring national licensing exam (NAPLEX) for the past ten years are shown below. A major revision to NAPLEX occurred in 2005, but scores are reported on the same scale to facilitate comparison with previous years. National scores are shown in parentheses.

Date Pass Rate Average Score Area 1 Area 2 Area 3 May 2001 96.3% (96.8) 103.4 (103.4) 12.2 (12.3) 12.2 (12.2) 12.3 (12.3) May 2002 100% (96.7) 109.4 (103.0) 12.8 (12.2) 12.7 (12.1) 12.7 (12.2) May 2003 100% (96.5) 108.2 (101.4) 12.6 (12.2) 12.7 (11.9) 12.6 (12.0) May 2004 96.7% (95.1) 103.9 (102.2) 12.3 (12.2) 12.1 (12.1) 12.4 (12.1) May 2005 91.9% (92.9) 106.8 (107.7) 12.5 (12.6) 12.1 (12.3) 12.2 (12.5) May 2006 96.5% (94.5) 108.6 (110.3) 12.6 (12.7) 12.1 (12.4) 12.3 (12.6) May 2007 96.2% (97.2) 108.4 (116.0) 12.0 (12.7) 12.2 (12.7) 12.3 (12.9) May 2008 100% (97.4) 115.5 (114.1) 12.7 (12.5) 12.6 (12.5) 12.5 (12.6) May 2009 100% (97.5) 121.2 (114.7) 12.9 (12.5) 12.9 (12.5) 12.9 (12.8) May 2010 95.4% (95.3) 102.9 (103.1) 12.7 (12.6) 12.6 (12.5) 12.6 (12.8)

The areas correspond to major competencies assessed by the Board exam. Area 1 is “Assure Safe and Effective Pharmacotherapy and Optimize Therapeutic Outcomes”

23

(approximately 54% of the total test), Area 2 is “Assure Safe and Accurate Preparation and Dispensing of Medications” (approximately 35% of the total test), and Area 3 is “Provide Health Care Information and Promote Public Health” (approximately 11% of the total test). NAPLEX data show the competence of our graduates is uniform among the three sub-areas of testing, and there is no one area where our students are stronger or weaker than should be expected.

NDSU graduates have historically been average to above average versus national norms in NAPLEX scores. In 2010 we were at the average. Multistate Pharmacy Jurisprudence Exam (MJPE) The MJPE is the national exam of knowledge in pharmacy law. Results for first time candidates for the past 6 years are shown below. National means are in parentheses. NDSU pass rates and scores in this specific area are consistently superior to the national averages. This indicates our single course in pharmacy law, together with experiential rotations, is effectively preparing students for practice. Candidates* NDSU Pass Rate State Pass Rate# Average Score Jan-June 2005 85 96.7% (88.5) 100% 82.34 (80.80) July-Dec 2005 42 90.5% (87.5) 93.8% 81.10 (80.44) Jan-June 2006 93 95.7% (89.4) 93.3% 82.71 (81.09) July-Dec 2006 70 91.4% (89.3) 81.3% 81.06 (80.80) Jan-June 2007 100 95.0% (91.4) 100% 83.06 (81.72) July-Dec 2007 56 96.4% (91.2) 100% 82.4 (81.5) Jan-June 2008 90 96.7% (91.8) 92.7% 83.8 (82.0) Jan-April 2009 25 100% (88.5) 87.5% 83.5 (80.8) May-Aug 2009 118 99.2% (94.3) 97.6% 84.4 (82.6) Jan-April 2010 17 70.6% (90.6) 100% 79.7 (80.8) May-Aug 2010 127 97.6% (95.5) 98.3% 83.3 (82.7) * the number of candidates is higher than the number of NDSU graduates because some candidates graduated in earlier years or graduated from other schools. # state refers to candidates intending to practice in North Dakota regardless of school graduated from 2. Pharmacy Curriculum Outcomes Assessment (PCOA) Activities As shown above, the NAPLEX national exam only reports students’ results in three very broad areas; thus it is difficult to pinpoint specific strengths and weaknesses in the curriculum. For several years our College and others have discussed the need for a more detailed curricular assessment tool. Fortunately, the National Association of Boards of Pharmacy (NABP), which maintains the national board exam, has now developed a curricular assessment exam. The Pharmacy Curriculum Outcomes Assessment (PCOA) is a psychometrically validated, 220 item multiple-choice exam designed to assess student performance within the pharmacy curriculum and assist schools in curricular evaluation. In 2009 the exam was available to all schools of pharmacy for the first time. Content of the assessment is based upon the Accreditation Council for Pharmacy Education (ACPE) Standards and Guidelines for the professional program in pharmacy and includes the following major domains: biomedical sciences, pharmaceutical sciences, social/behavior/administrative sciences; and clinical sciences. Each major domain is broken into subareas. The exam can be given to students in any year of the curriculum and longitudinal results

24

can assist Colleges in noting how students’ knowledge base develops over time, as well as giving detailed curricular feedback. In both 2009 and 2010 we administered the PCOA to students in the spring of their third professional year. Results Full results of 2010 are given in Appendix 7. In 2009, we were at the 50% percentile in curricular knowledge for that level of the curriculum. Our strongest area was Social and Administrative Sciences; however, there was no area exceptionally strong or weak. In 2010 our students scored at the 57th percentile, with the strongest area being basic biomedical sciences. Again, no particular area of weakness was identified. Among individual students over the past two years, scores ranged from the 1st to 99th percentile. Thus, while there was great variability in student ability (and probably in effort), PCOA provides some validation that our overall curriculum is sound. Future Plans Of interest, the class that scored at the 50th percentile in 2009 was the same class that had an average score on NAPLEX in 2010. Since the class that did very well on PCOA in 2010 will write NAPLEX in 2011, we are hoping that a strong PCOA predicts an excellent performance in NAPLEX as well. The variability in student performance is also of great interest to us, so we have started an IRB-approved research project to assess student success in P4 rotations. We plan to determine if PCOA predicts student performance on Advance Practice Pharmacy Practice Experience (APPE) rotations, the North American Pharmacy Licensure Exam (NAPLEX) and the Multistate Pharmacy Jurisprudence Exam (MPJE). We anticipate preliminary data will be available for next year’s assessment report. We will again administer the PCOA to our P3 class in 2011. Future use of PCOA may depend on whether it is a useful predictive tool as determined by the research described above. 3. Self-Rating on “Levels of Implementation”

Our annual assessment report continues to be shared with all pharmacy faculty. The College of Pharmacy, Nursing, and Allied Sciences held a retreat on assessment in January 2011, and the standard University survey on levels of implementation was distributed to pharmacy faculty at the end of the retreat. Faculty ratings on each item varied a fair amount but clustered on 2 for three items and 2+ for three items (Appendix 8). This is not different from past surveys. It appears again that faculty who are relatively new to assessment tend to rate the program higher than those who understand that much more remains to be done. The highest consensus was that “assessment results are beginning to be used to make changes in some courses”.

E. SUMMARY

In the past year, we have fully utilized our ability-based outcomes (ABOs) in curriculum mapping. This has improved our curricular coordination of learning outcomes, and helped all

25

faculty be aware of the need to assess learning in their own courses. The Assessment Committee has agreed that each course in the Pharm.D. curriculum must document course-based assessment data at least once every three years. For some instructors, assessment has begun at the level of pre- and post-tests of knowledge. While this is not adequate in assessing abilities, it is a positive start toward making assessment of learning part of their mindset.

Two faculty this year have published papers in the American Journal of Pharmaceutical Education based upon their assessment projects. This indicates a maturity of effort and commitment to assessment by these faculty members.

We have several goals to improve our assessment report even further for next year. We would like to focus on outcomes described in our College and department strategic plans (e.g. immunization and medication therapy management), look for additional places to teach and assess writing skills, and include a more seamless continuum of assessment between Concept Pharmacy, introductory experiences, and advanced practice experiences. We would like to make sure that every ability-based outcome is actually assessed each year. With our new curriculum evaluation plan, we should have a minimum of 11 course-based assessments done every year. We are interested in working with the English department on evaluating English 345/325 for pharmacy students and have made plans for fall to have two faculty from pharmacy and two from English read a group of papers from ENG 325 and PHRM 480 together.

We will continue to train preceptors, so that the experiential evaluation data we collect is meaningful, and continue to encourage all faculty to participate in the University peer review of teaching process, with special attention to how learning can be assessed. Another future goal is still to combine efforts between Nursing and Pharmacy. The College’s Associate Dean for Academic Affairs and Assessment can help integrate assessment across disciplines to improve the assessment process and outcomes for both programs.

No specific assistance is requested from the University Assessment Committee at this time.

1

PHARM.D. STUDENT LEARNING OUTCOME ASSESSMENT REPORT COURSE INFORMATION

Instructor:

Course:

Semester:

Level: P1 P2 P3 P4

Date Assessed:

ABILITY BASED OUTCOME ASSESSED

ATTITUDES & VALUES – Students will integrate, apply, and reinforce professional attitudes & values across the curriculum. Specific Competencies: Demonstrate honesty and integrity in all situations. Demonstrate sensitivity and tolerance for the values, dignity, and abilities of all individuals. Make decisions and perform duties in accordance with legal, ethical, social, cultural, economic, and professional

guidelines. Self-assess learning needs and design, implement, and evaluate strategies to promote intellectual growth and continued

professional competence in the areas of patient-centered care, systems management, and public health.

COMMUNICATION – Students will communicate in a caring and respectful manner in all situations using appropriate listening, verbal, nonverbal, and written skills. Specific Competencies: Communicate and collaborate with patients, caregivers, health care professionals, administrative and support personnel

to engender a team approach to patient-centered care. Demonstrate effective communication skills in inter-disciplinary relationships to assure safe, efficient, cost-effective

utilization of human, physical, medical, informational, and technological resources.

SCIENTIFIC FOUNDATION – Students will integrate and apply scientific knowledge of pathophysiology, medicinal chemistry, pharmaceutics, pharmacology, and pharmacokinetics. Specific Competencies: Use appropriate scientific terminology to convey concepts of pathophysiology, medicinal chemistry, pharmaceutics,

pharmacology, and pharmacokinetics. Demonstrate an understanding of scientific research and discovery. Based on differences in chemistry, physical properties, pharmacology, pharmacokinetics, and pharmacogenomics,

recommend changes in pharmacotherapeutic regimens that minimize drug interactions, reduce side effects, increase compliance, and improve therapeutic outcomes.

Acquire, comprehend, synthesize, apply and evaluate information about the chemistry, physical properties, pharmacology, pharmacokinetics of therapeutic agents in order to design, implement, monitor, evaluate, and adjust care plans that are patient-specific and evidence based.

PATIENT-CENTERED CARE – Students will provide evidence-based patient-centered care in cooperation with patients, prescribers, and other members of an interprofessional health care team taking into account relevant legal, ethical, social, cultural, economic and professional issues. Specific Competencies: Obtain, interpret and evaluate patient information to determine the presence of a disease or medical condition, assess

the need for treatment and/or referral, and identify patient-specific factors that affect health, pharmacotherapy, and/or disease management.

Design, implement, monitor, evaluate, and adjust patient-centered care plans that are evidence-based. Provide information regarding the selection, use and care of medical/surgical appliances and devices, self-care

products, and durable medical equipment, as well as products and techniques for self-monitoring or health status and medical conditions.

Document patient-centered care activities to facilitate communication and collaboration among the health care team. Retrieve, analyze, and interpret the professional, lay, and scientific literature to provide drug information to patients,

caregivers, and other involved health care providers. Apply quality assessment methods to the evaluation of patient-centered care.

Appendix 15B:

2

SYSTEMS MANAGEMENT – Students will manage and use resources of the health care system in cooperation with an interdisciplinary team to: promote health; provide, assess, and coordinate safe, accurate, and time-sensitive medication distribution; and to improve therapeutic outcomes of medication use. Specific Competencies: Accurately select, prepare, and dispense medications in a manner that promotes safe and effective use. Accurately prepare and compound individual or bulk medications in a manner that promotes safe & effective use. Prepare, store, and assure quality of sterile dosage forms. Provide counseling to patients, families, and care givers. Manage human, physical, medical, informational, fiscal, and technological resources using relevant legal, ethical, social,

cultural, economic, and professional principles/issues to assure efficiency and cost-effectiveness. Apply patient- and population-specific data, quality assurance strategies, and research processes to: a) assure that

medication use systems minimize drug misadventuring and optimize patient outcomes and b) to develop drug use and health policy, design pharmacy benefits and formulary systems.

PUBLIC HEALTH – Students will promote health improvement, wellness, and disease prevention in cooperation with members of an interdisciplinary team. Specific Competencies Assure the availability of effective health promotion and disease prevention services.

a. Promote public awareness of health and disease. b. Provide emergency first aid treatment and cardiopulmonary resuscitation (CPR). c. Provide patients with access to poison control and treatment information. d. Provide immunizations and health-related screenings.

Provide population-based care. a. Develop and implement population-specific, evidence-based disease management programs and protocols based

upon analysis of epidemiologic and pharmacoeconomic data, medication use criteria and review, and risk reduction strategies.

b. Retrieve, analyze, and interpret the professional, lay, and scientific literature to provide drug information to other health care providers and to the public.

c. Apply population-specific data, quality assurance strategies, and research processes to identify and resolve public health problems

ASSESSMENT METHOD

Description:

3

Benchmark:

RESULTS

ACTION TAKEN BASED UPON RESULTS

Adapted with permission from: University of Pittsburgh School of Pharmacy 1

NDSU COLLEGE OF PHARMACY, NURSING, & ALLIED SCIENCES

PHARM.D. PROGRAM COURSE EVALUATION PLAN

Standard:

The Curriculum Committee is responsible to evaluate all courses in the professional Pharm.D. curriculum at least once

every 3 years to insure the curricular structure, content, organization, and pedagogy support student achievement of

the Pharm.D. Program Ability Based Outcomes. The Ability Based Outcomes detail the professional competencies

necessary for graduates to practice pharmacy in any setting as a generalist, entry level pharmacist.

Justification:

1. “Colleges of Pharmacy must systematically and sequentially evaluate its curricular structure, content, organization,

and outcomes. The College must use the analysis of outcome measures to improve the curriculum and its

delivery.” (ACPE Standard No. 15: Assessment and Evaluation of Student Learning and Curricular Effectiveness.)

2. “The Curriculum Committee serves as the central body for the management of orderly and systematic reviews of

curricular structure, content, process, and outcomes for the Pharm.D. program at NDSU.” (College of Pharmacy,

Nursing, & Allied Science Policy Manual.)

Process:

1. The Curriculum Committee will develop an annual schedule of courses to be reviewed.

2. Instructors will be notified in advance when their courses are scheduled for evaluation.

3. Courses scheduled for evaluation will be reviewed as soon as possible after the course has been taught and

completed within a 6 week period of time.

4. Course faculty will self‐assess their course using the Course Evaluation Rubric and forward the rubric electronically

(along with any additional materials requested) to the Associate Dean for Academic Affairs and Assessment (ADAA)

within 2 weeks of request.

5. The Curriculum Committee Chair shall convene two evaluation teams:

a. A student focus group ‐ consisting of students comprised of student members from the Curriculum &

Assessment Committees who have recently completed the course as well as other selected students so

that a minimum of three students are in the focus group. Student members will also serve as ad‐hoc

members of the faculty work group.

b. A faculty work group ‐ composed of faculty representation from Pharmaceutical Sciences and Pharmacy

Practice with at least one of the members serving as a liaison to the Assessment Committee. Outside

reviewers may also be added, when available, to the faculty working group.

6. The student focus group, facilitated by the ADAA, will begin their review as soon as possible following completion

of the course with findings recorded electronically on the Course Evaluation Rubric. The evaluation will be

completed within 2 weeks and results forwarded electronically to the Curriculum Chair. .

7. The faculty work group will begin their review after the student focus group evaluation has been completed and

upon receipt of items in #5 & #7 (above) completing the evaluation within 2 weeks.

8. A representative from the faculty work group will discuss evaluation findings with the course instructor(s) &/or

coordinator.

9. The faculty & student groups will then present their findings to the full Curriculum Committee for discussion and

recommended actions if any.

10. The Curriculum Committee Chair &/or Associate Dean for Academic Affairs & Assessment will provide a copy of the

completed evaluation rubric including any recommendations from the committee to the course

instructor(s)/coordinator & to the Department Chair.

Appendix 15C:


11. Recommended changes to the course are expected to be implemented within the next academic year.

Course Review Cycle

2010 – 2011 Jan (for Fall Term Courses) ‐ Pilot June (for Spring Term Courses)

Cycle “A” PSci 413 Dynamics III (Endo/Auton) Psci 415 Pain/Inflam/GI Phrm 340 Pathophysiology I Phrm 475 Pharmacy Management Phrm 534 Rheumatology, Endo, Repro

PSci 411 Principles of Dynamics PSCI 545 Clinical Toxicology Phrm 341 Pathophysiology II Phrm 480 Drug Literature Phrm 537 Renal, Fluids, & Electrolytes Phrm 558 Nutrition/GI

2011 – 2012 Jan (for Fall Term Courses) June (for Spring Term Courses)

Cycle “B” PSci 412 Dynamics II (Oncology/ID) PSci 470 Pharmacokinetics Phrm 462 Stress Management Phrm 532 Infectious Disease Phrm 535 Neoplastic Diseases

PSci 410 Pharm Biotechnology Psci 416 Dynamics VI (Neuro/Psych) Phrm 485/685 Econ Outcomes /Assessmt Phrm 520 Peds/ Gerontology Phrm 536 Neurology/Psychiatry

2012 – 2013 Jan (for Fall Term Courses) June (for Spring Term Courses)

Cycle “C” PSci 368 Pharmaceutics I Phrm 350 Intro to Pharmacy Prax Phrm 351/L, 551/L Concept Lab Phrm 463 Issues in Hosp/Inst Pharmacy Phrm 538 Cardiovascular

PSci 369 Pharmaceutics II PSci 414 Dynamics IV (Cardiovascular) Phrm 352 Intro to Healthcare Systems Phrm 450 Self Care Phrm 452/L Concept Lab Phrm 572 Pharmacy Law

Summer ‐ Phrm 355,455,555, 552L IPPE I‐III ; Phrm 581, 582, 583 APPE


COURSE EVALUATION RUBRIC Course: Instructor(s): Reviewers: Date Self-Assessment Completed: Date Review Completed:

Meets Expectations Needs Improvement Does Not Meet Expectations

1. Course Syllabus The course syllabus follows the guidelines established by the University & College including: a complete listing of instructors & contact information, bulletin description, course objectives, applicable program ABO’s, Instructional Continuity Plan, class schedule, evaluation procedures & criteria, attendance, disability and academic integrity statements. Appropriate grammar & spelling.

In general, the syllabus follows the guidelines established by the University & College but some minor revisions are needed. List revisions needed below:

The course syllabus does not follow the guidelines established by the University &/or College. The syllabus format needs to be revised significantly in order to meet the guidelines. Several grammatical or spelling errors noted.

Coordinator assessment Meets Needs Improvement Does Not Meet Expectations

Reviewers’ assessment Meets Needs Improvement Does Not Meet Expectations

2. Clarity of Course Objectives

Course objectives are stated with both knowledge acquisition and inclusion of higher order processes (application, synthesis, and evaluation). Course objectives are clear, action orientated, achievable & measureable.

Course objectives are unclear or could be more appropriately aimed at a higher (or lower) level of knowledge. List improvements to course objectives below:

Course objectives are unclear, un-achievable, or focus solely on knowledge acquisition. Course objectives need to be re-stated.


Reviewers’ assessment

Meets Needs Improvement Does Not Meet Expectations

2/8/2012 4

COURSE EVALUATION RUBRIC – Page 2


3. Alignment of Course Objectives and Applicable* AACP CAPE Supplements* *(Pharmaceutics, Medicinal Chemistry, Pharmaco-kinetics, Drug Information, Pharmacy Practice, Social & Administrative Sciences)

Relevant AACP CAPE objectives are substantially addressed in course objectives.

Some relevant AACP CAPE objectives are missing and should be included. List AACP CAPE objectives to be included:

Many relevant AACP CAPE objectives are either not substantially address or are missing.



4. Course Content Content is relevant to current scientific and clinical practice. Course content maps to course objectives and ACPE Appendix B. No gaps in content noted. The scope and depth are appropriate for a professional pharmacy student. Content is appropriately sequenced or arranged. Pharmaceutical and clinical sciences are well integrated in the course.

Some content is not relevant or current for the practice of pharmacy today. Some course content could be deleted without affecting the course or course content needs to be added based upon ACPE Appendix B. Course sequencing could be improved. Integration between clinical and pharmaceutical sciences could be improved. Suggestions for Improvement:

Content is not appropriate for a professional pharmacy student (too elementary or too difficult). There is excessive content that does not contribute to the education of a generalist practitioner. Course missing many elements from ACPE Appendix B. Course content appears fragmented. No evidence of integration between clinical and pharmaceutical sciences.



2/8/2012 5



5. Teaching Methods & Learning Activities

A variety of active learning techniques appropriate for the nature of the course are used to enhance student learning. The course challenges students to achieve higher levels of learning through problem solving, decision making, and/or creative thinking.

Some active learning techniques are employed but infrequent or not optimal for the nature of the course. Suggestions:

Teaching methods are primarily lecture based with little or no opportunities for active learning provided. Little additional material is added to lectures beyond that what is included on the power point slide.



6. Assessment Methods

All student learning objectives are assessed. Course assessment methods are appropriate for determining how well students master the course objectives. A variety of assessments, including both formative & summative, are used. The course requires a cumulative exam or capstone project to summatively assess student learning and promote knowledge retention. Assessment method included in the University Assessment Report.

Some learning objectives are not assessed. Assessments methods not optimal to measure student’s mastery of course objectives. Formative or summative assessments missing. Cumulative exam or capstone project not required. Suggestions for Improvement:

Several learning objectives are not measured adequately by course assessments. Assessment methods are not matched to course objectives. Exams are too elementary or unnecessarily difficult.



2/8/2012 6



7. Relation of Course Objectives, Content, Learning Activities, & Assessments to Ability Based Outcomes

The course objectives, content, & learning activities are clearly related to the Ability Based Outcomes. Assessments measure student competency towards attainment of Ability Based Outcomes.

In general, course elements relate to Ability Based Outcomes. However, some clarification or re-wording for relevance is needed.. Assessments provide weak evidence of competency towards ABO’s. Suggestions:

The course is not related to the Ability Based Outcomes. It is difficult to see how the course contributes to mastery of the Ability Based Outcomes.



8. Student Learning Resources

Student learning is supported by a variety relevant and current books, journal articles, DVD’s, printed notes, or websites. Resources are easily accessible by students. Lecture notes are clear and well organized, & free of spelling &/or grammar errors.

Student learning is supported by some additional references but are outdated, not accessible, or not optimal. Student lecture notes somewhat unclear &/or in need of minor modification. Suggestions:

Students are not given any additional resources to support learning other than lecture notes. Lecture notes unclear, contain misspellings, grammatical errors, and/or are unorganized. Resources do not support student learning.



9. Allocation of Course Credit

The number of credits allocated to the course is appropriate for the amount of class time, student effort, and desired curricular goals.

The number of credits allocated to the course is excessive for the amount of class time, student effort, and desired curricular goals.

The number of credits allocated to the course is insufficient for the amount of class time, student effort, and desired curricular goals.

Coordinator assessment Meets Needs Improvement Needs Improvement

Reviewers’ assessment Meets Needs Improvement Needs Improvement

SROI Question #4 – “The Quality of this Course” score for last 3 years

Year: This class Department College



Grade Distribution Last 3 years

Year: # A’s # D’s # P # B’s # F’s #F # C’s



Appendix 15D: NDSU PCOA DATA 09- 10 - 11

Table 1. Pharmacy Curriculum Outcomes Assessment Results by Year Administered

Year 2009 2010 2011

Test Parameters NDSU (n= 85)

National*(n= 842)

NDSU (n= 66)

National* (n= 823)

NDSU (n=79)

National§

(n=1440) Total Score

Scale Score (mean) 367 366 355 344 367 340 % Correct (mean) 65 65 61 58 65 59

Percentile for Program (mean) 50 ----- 57 ----- 66 ---- Basic Biomedical Sciences (21%)


Percentile for Program (mean) 51 ----- 64 ----- 68 ---- Pharmaceutical Sciences (29%)


Percentile for Program (mean) 47 ----- 53 ----- 60 ----- Social/Behavioral/Admin Sci. (15%)


Percentile for Program (mean) 56 ----- 55 ----- 62 ---- Clinical Sciences (35%)


Percentile for Program (mean) 50 ----- 54 ----- 63 ---- * Third year professional students enrolled in traditional Pharmacy Doctorate Programs § Third year professional students with additional data forthcoming Scale Score (mean): A measure of difficulty of the items ranging from 0 (lowest ability) to 700 (highest ability). It is used as a basis of comparison across exam administration years and between major content areas. For example, a student scoring 200 on Basic Sciences and 250 on Pharmaceutical Sciences is more knowledgeable in Pharmaceutical Sciences than Basic Sciences. % Correct (mean): Identifies the mean percentage of items answered correctly by program year. Percentile for Program (mean): Gives performance information in relation to other students within the same program year. A Percentile for Program Year score of 60 implies that students performed equally or better than 60^ of all other students within that same program year.

Table 2. School Subtopic Means – mean % of Items Answered Correctly 09-11

Subtopics 2009 2010 2011

NDSU National NDSU National NDSU National

Basic Biomedical Sciences Anatomy & Physiology 71 65 71 65 64 60

Pathology/Pathophysiology 60 63 52 50 64 59

Microbiology 59 59 68 62 67 56

Immunology 74 65 77 66 54 46

Biochemistry/Biotechnology 58 63 69 56 73 64

Biostatistics 63 62 58 56 59 52

Pharmaceutical Sciences

Medicinal Chemistry 52 55 35 40 48 48

Pharmacology 64 62 51 51 57 52

Pharmacognosy and Alternative and Complimentary TX 65 67 52 48 51 40

Toxicology 61 66 47 49 50 41

Bioanalysis/Clinical Chemistry 40 70 41 36 81 71

Pharmaceutics/Biopharmaceutics 48 49 59 54 50 44

Pharmacokinetics/Clinical Pharmacokinetics 65 64 55 53 48 46

Pharmacogenomics/Genetics 88 91 83 73 63 60

Extemporaneous Compounding/Parenteral/Enteral 54 55 47 40 18 27

Social/Behavioral/Administrative Pharmacy Sciences

Healthcare Delivery Systems 60 57 52 52 78 69

Economics/Pharmacoeconomics 67 67 37 50 57 68

Practice Management 76 74 58 56 68 59

Pharmacoepidemiology 69 55 95 81 96 79

Pharmacy Law and Regulatory Affairs 71 72 75 59 76 67

History of Pharmacy 89 78 68 61 68 69

Ethics 85 83 81 78 82 75

Professional Communications 82 80 86 88 87 81

Social and Behavioral Aspects of Practice 82 85 95 86 79 78

Clinical Sciences

Pharmacy Practice and Pharmacist-Provided Care 71 70 50 53 66 62

Medication Dispensing and Distribution Systems 67 71 77 66 82 68Pharmacotherapy - Practice Guidelines and Clinical

Trials 54 56 50 51 71 67

Pharmacotherapy - Health Promotion/Disease Prevention 76 75 86 78 81 71

Pharmacotherapy - Pharmaceutical Care 57 60 62 64 59 58

Pharmacist-provided care for Special Populations 49 50 48 50 69 57

Drug Information 84 77 72 69 89 82

Medication Safety 93 88 95 93 71 62

Literature Evaluation and Research Design 55 56 78 63 88 75

Patient Assessment Laboratory 89 83 61 64 76 74

Section A: Ability Based Outcomes Ability Based Outcome IPPE/APPE1. Attitudes and Values/ProfessionalismStudent arrives at practice site and meetings on time, meets deadlines for completion of tasks and 1 D Allresponsibilities.Student seeks knowledge, asks questions, searches for information, and takes responsibility for his/her own 1 D Alllearning.Student responds openly and positively to constructive feedback and modifies behavior if necessary. AllStudent demonstrates regard for patients, superiors, colleagues, other personnel, and property. 1 B AllStudent demonstrates empathy for patients.Student makes decisions and performs duties in accordance with legal, ethical, social, cultural, economic, and 1 C Allprofessional guidelines.Student adheres to dress code and maintains personal health and good grooming habits as put forth by the Allpractice setting.

2. Communication SkillsStudent is able to communicate in a caring and respectful manner in all situations using appropriate listening, Allverbal, nonverbal, and written skills.Student actively and appropriately engages in dialogue or discussion. Not afraid to express his/her viewpoint. 2 A AllStudent clearly communicates thoughts, and uses appropriate vocabulary, gestures and mannerisms to Allconvey information.Student acts and communicates in a self-assured manner, yet with modesty and humility. AllIdentifies and collects all information needed to respond to an information request from another 2 B Allhealth care professional using appropriate resources and technology.Responds to an information request from a patient. 2 A All

3. Scientific FoundationMakes decisions regarding complex problems that require the integration of scientific, social, Allcultural, and ethical issues with one's ideas and values.Recommends medication doses and dosage schedules for a specific patient based upon APPErelevant patient factors and drug pharmacodynamic and pharmacokinetic properties. 3 C

4. Patient Centered CareStudent establishes relationships with patients, care givers, and other health care professionals as Allnecessary to provide pharmaceutical care. 2 AEvaluates information obtained from the patient's history and physical assessment. AllStudent effectively counsels, communicates, refers and selects the appropriate medications or care plans APPEfor patients.Recommends appropriate drug therapy for a specific patient. 4 A1 APPEDocuments information related to the identification, resolution, or prevention of drug-related Allproblems in individual patients.Student effectively retrieves and evaluates medical literature and is able to analyze and apply information in 4 A5 Alldecision making.

Appendix 15E

North Dakota State University Department of Pharmacy Student Evaluation by Preceptor Mapped to ABO

2/8/201210:06 AMAppendix 15E Student Experiential Evaluation Mapped to ABO 1

Patient Centered Care (cont.) Excellent IPPE/APPERecommends appropriate drug therapy for a specific patient. APPEDocuments information related to the identification, resolution, or prevention of drug-relatedproblems in individual patients. AllStudent effectively retrieves and evaluates medical literature and is able to analyze and apply information in Alldecision making.

5. Systems ManagementAccurately select, prepare, and dispense medications in a manner that promotes safe and effective use. AllAccurately prepare/compound individual or bulk medication in a manner that promotes safe and effective use. AllPrepare, store, and assure quality of sterile dosage forms. Not P1sProvide counseling to patients, families, and caregivers. 5 B5 Not P1sApply patient and population-specific data, quality assurance strategies, and research processes to optimizepatient outcomes. 5 B1 APPESection B: Special ProjectsActivity 1: Public Health Project Completed activity was appropriate for the intended audience.Activity and supporting materials were of a professional quality.Demonstrated the ability to follow through with the activity from conception to conclusion.Activity 2:Completed activity was appropriate for the intended audience.Activity and supporting materials were of a professional quality.Demonstrate the ability to follow through with the activity from conception to conclusion.Activity 3: Completed activity was appropriate for the intended audience.Activity and supporting materials were of a professional quality.Demonstrate the ability to follow through with the activity from conception to conclusion.Section C: CommentsComments:

Identify one area in which the student excelled:

Identify one area in which the student could further develop his/her skills:

Signature certifies student completed 200 rotation hours _____________________________________________________Preceptor Signature

2/8/201210:06 AMAppendix 15E Student Experiential Evaluation Mapped to ABO 2

Page 1 of 5

Graduating Student Survey

Question: 28. The Pharm.D. Program prepared me to reflect critically on personal skills and actions and make plans to improve when necessary.

Appendix 15F: AACP Surveys

Page 2 of 5

Faculty Survey

Question: 46. The college/school uses programmatic assessment data to improve the curriculum.


Page 3 of 5

Preceptor Survey Name

Question: 22. I am aware of the mechanism to provide feedback to the college/school regarding the Pharm.D. curriculum.


Page 4 of 5

Preceptor Survey

Question: 24. The assessment tools provided to me for my site are suitable for measuring student performance.


Page 5 of 5

Alumni Survey

Question: 17. Since graduation, the college/school has solicited my input/feedback for programmatic improvement.
