Quantifying and Measuring Professionalism: Demonstrating the

Professionalism Outcomes of Pharmacy Curricula

Presenters

• Dana Hammer,RPh, PhD– University of Washington, Pharmacy

• Kristin, K. Janke, PhD– University of Minnesota, Pharmacy

• Katherine A. Kelley, PhD– Ohio State University, Pharmacy

• Steven A. Scott, PharmD– Purdue University, Pharmacy

Session Goals• Gain an understanding of the unique

challenges of measuring professionalism• Explore the unique aspects of

professionalism as it relates to interdisciplinary health care teams

• Gain and enhanced understanding of how to use professionalism assessment data to inform curricular change

Creating a Tool to Measure Professionalism Behaviors: A Multi-

Institutional Pilot

Steven A. Scott, PharmD

Kristin K. Janke, PhD

Katherine A. Kelley, PhD

Best Practices for Professionalism Assessment

• Multiple assessors

• Multiple settings

• Multiple methods

• Formative

• Beginning early in the curriculum

• Improve existing tools rather than create new

Lynch, et al, 2004

ActivityWhen working with colleagues, I behave in a ways that communicate my respect for them.

1.Strongly agree

2.Agree

3.Neutral

4.Disagree

5.Strongly disagree

Barriers to Measuring Professionalism

• Definitions tend to be abstract

• Context specific

• Reluctance to report or address unprofessional behaviors

• Ceiling effects

Ginsburg et al, 2000

Case Example• A student you have on rotation dresses

professionally and appropriately everyday. You have noticed that the student tends to talk down to and act rudely toward the office staff members at your practice site.– Identify/name the professional behaviors of

this student?– What professional attitudes would you attribute

to this student?

Professionalism: Internally held vs Externally Exhibited

• Attitude or behavior?– I am respectful of others– I dress professionally on rotation

• Attitude or behavior?– I am trustworthy– I protect patient confidentiality

Challenges Associated with Measuring Professionalism

• Often more difficult to measure professional behavior during the didactic curriculum than during the experiential curriculum– Fewer opportunities– Situations are often simulated vs. real-life– Expectations may not be consistent– Faculty deal with many more students than in the

practice setting

Challenges Associated with Measuring Professionalism

• The major focus of assessments and assessors is to evaluate the student’s knowledge and skills, not necessarily behaviors.

• Professionalism is more difficult to assess than knowledge or the demonstration of a skill and usually plays a minor role in any course grade.

• Most assessors have not been trained to evaluate professional behaviors. (Bryden, et al, 2010)

Challenges Associated with Measuring Professionalism

• Assessments of professionalism are often sporadic and isolated and thus the professional development of students is difficult to monitor. (Lucey and Souba, 2010)

• Assessment of professionalism is often carried out by a variety of individuals who have not observed the student previously.

• Isolated incidences of unprofessional behavior may not be communicated and thus undesirable behaviors must be rediscovered as the student matriculates through the curriculum.

Challenges Associated with Measuring Professionalism

• Volunteer faculty, other healthcare professionals, and patients are often uncomfortable making summative judgments about student professionalism after limited interactions. (Swick, H., 2007, Hickson, et al, 2007)

• Other healthcare professionals may focus more on roles than on behaviors.

• Peers are often hesitant to provide honest evaluations. (Reed, et al, 2008)

Challenges Associated with Measuring Professionalism

• Behavioral assessments conducted prior to graduation are prone to being influenced by student motivation to achieve a desired grade, rather than being reflective of the student’s true value system. (Brown and Ferrill, 2009)

• Although valid assessment tools have been developed, no standard approach to measuring professionalism has emerged. (Stern DT, et al, 2006)

Multi-Institutional Development

• 7 institutions* collaborated to create an instrument to measure behavioral aspects of professionalism

• GOAL: use previous literature as a foundation to create an instrument that minimizes ceiling effects

• Pilot: Spring 2009 in three schools

*Big 10: Purdue University, University of Illinois Chicago, University of Iowa, University of Michigan, University of Minnesota, Ohio State University, University of Wisconsin

Multi-Institutional Work

• 7 institutions* collaborated to pilot the instrument

• GOAL: cross-validation of instrument• Spring 2010 with first and third year

students

*Purdue University, University of Illinois Chicago, University of Iowa, University of Minnesota, Ohio State University, University of Wisconsin, AND University of Kansas

The Instrument • Professionalism Assessment Tool (PAT)• Foundational instruments:

– Physicianship Evaluation Form (Papadakis, 2001)

– American Pharmacists Association white paper on student professionalism (2000)

– Hammer et al 2000– Chisholm et al 2006

The Instrument

• Domains– Reliability, responsibility and accountability– Lifelong learning and adaptability– Relationships with others– Upholding principles of integrity and

respect– Citizenship and professional engagement

Self-assessment Ratings

• Foundations – Pangaro, 1999

• RIME: Reporter, interpreter, manager, educator

– Wanted a basic level• Novice

– Miller, 1990• Knows, knows how, shows, shows how, does

The Final Ratings“Know”: I understand these responsibilities, but may perform one or

more inconsistently, at times.“Know how”: I understand these responsibilities and perform them in

a reliable, consistent and accountable manner. “Show”: Without prompting or support from instructors, preceptors or

managers, I determine when and how to engage in these responsibilities.

“Show how and Does”:  I am confident in assisting others with these responsibilities or proposing or creating options to fulfill these responsibilities. 

“Teach how”: I have mastered these responsibilities and desire to learn more and share my learning with others.  I demonstrate maturity, confidence and an ability to educate others in these areas through the use of evidence and strong interpersonal skills.

Factor NamesTotal Items

Maximum Value

Possible M (SD)Upholding Principles of Integrity and Respect

8 40 31.3 (7.2)

Lifelong Learning and Adaptability 7 35 22.3 (6.4)

Relationships with Others 9 45 31.7 (7.2)

Reliability, Responsibility, and Accountability

5 25 17.8 (5.0)

Citizenship and Professional Engagement

4 20 12.6 (4.1)

Kelley KA., et al. A Multi-Institutional Pilot for Cross-Validation of a Tool to Measure Professionalism Behaviors. Amer J Pharm Educ, 2011 article 179.

Cross-validation

• Internal consistency was good (all 5 factors coefficient alpha >0.9)

• Both exploratory and confirmatory factor analysis techniques were used

• Factor analysis demonstrated that all items were correctly loaded under their assigned domain

Limitations

• 155 surveys removed due to incomplete data

• Homogeneity of the survey sample (all 7 pilot schools are public research intensive)

Future Directions

• Longitudinal data collection to determine growth over time

• Use of instrument by external evaluators (preceptors and faculty)

• Examine possibilities in curricular revision

Lessons Learned

• Measuring professionalism has value• Assessing professionalism is possible• Building on previous literature is helpful• Multi-institutional work can be productive• There is more work to be done!

– Multiple assessors & multiple settings– Longitudinal: beginning early, using multiple

measures– Formative (for student and curriculum!)

Challenges, Opportunities, and Best Practices for Interprofessional

Professionalism

Dana Hammer, RPh, PhD

Closing the Loop: Using Assessment Data to Improve

Professionalism CurriculaGroup Discussion

Discussion Questions

• How do you assess professionalism in your curriculum? Or how would you like to assess professionalism?

• How does professionalism assessment data feedback to improve the curriculum? Or how would you like to use professionalism data to improve your curricula?