1. Meaningful rubric could facilitate honest and useful evaluations• Consistent standard- Mirroring ACGME

Milestones approach.• Emphasize constructive, specific comments.

2. Training on evaluations process may confer reliability among evaluators

• Remove concerns re: anonymity and retaliation.• Focus on purpose of evaluation.• Reinforce need for constructive criticism.

3. Promotion and Tenure Process Transparency

• Consider track and qualification of learners.• Consider the competing priorities faculty face

depending on their academic track and departmental mission.

Theory & Conclusion

How do clinical teaching faculty interpret feedback from rotators?Tara F Carr MD, Guadalupe F Martinez, PhD

AMES-OMSE Teaching ScholarsDepartment of Medicine, University of Arizona College of Medicine, Tucson AZ

Introduction RecommendationsSaturated Themes (cont)

References1. Epstein RM, Siegel DJ, Silberman J. Self-monitoring in clinical practice: a challenge for medical educators. J Contin Educ Health Prof. 2008 28(1):5-13.2. Arah OA, Hoekstra JAB, Bos AP, Lombarts KM. New tools for systematic evaluation of teaching qualities of medical faculty: results of an ongoing multi-center survey. PLoS ONE 2011, ;6(10):e25983.3. Overeem K, Driessen EW, Arah OA, Lombarts KM, Wollersheim HC, Grol RP. Peer mentoring in doctor performance assessment: strategies, obstacles and benefits. Medical Education 2010, Feb;44(2):140-7.4. Corbin J, Strauss A. Basics of Qualitative Research: Techniques and Procedures for Developing Grounded Theory. SAGE Publications, Inc, 2007. 5. Creswell JW, Miller DL. Determining Validity in Qualitative Inquiry, TheoryInto Practice, 2000; 39:3, 124-130.

• Self-monitoring is a challenge for medical educators.1

• Providing concrete plans and structured assessment is more helpful for faculty improvement than independent change.2,3

• While giving and receiving feedback is emphasized through Graduate Medical Education, there are barriers to the use of that feedback on the part of the faculty.

Purpose of the study:• Describe and form a theory about how Medicine

faculty interpret written evaluations from clinical rotators.

• Describe and form a theory about whether and how Medicine faculty use feedback to inform their own educational practice and philosophy.

.  

Overall Research Questions

1. Method of Feedback is Problematic• Current grading scale is poor- 7/12

• Desire for more substantive feedback- 6/12

2. Concerns with Evaluators• Poor feedback from evaluators- 8/12

Not enough time spent with the evaluatorEvaluators may be scared to give honest feedbackEvaluators don’t care about evaluations process Evaluators need education on the process and identifying teaching moments

3. Perceptions Regarding Evaluation use in Promotion & Tenure Process

• Reviews Conducted within the Context of Clinical Responsibilities and Academic Contract is Important- 5 (Assist Prof)

• Filtering of Evaluations Through Relevant Superiors or Averaged Over Time is Highly Valued- 6 (1 Prof; 3 Assoc Prof; 2 Assist Prof)

MethodsQualitative Methodology - Grounded theory4

Purposeful selection Triangulation for Qualitative reliability and validity5 • 1. 12, 15-60 minute one-on-one semi-structured

open ended Interviews• 2. Observation reflection notes Analysis with Nvivo10© Coding/Data Reduction →

Research Site and Sample• 12 Subspecialty faculty in a division within the

Department of Medicine at a southwestern academic center.

• Faculty appear to interpret evaluations based on perceptions they have about feedback methods and the evaluator’s ability to assess and recognize teaching skills. Moderate use in informing practice.

• The perceived current role of evaluations in P&T process further shapes the educational philosophy of these faculty members.

• Limitations - Small sample size; interviews not transcribed verbatim; limited to Medicine faculty perspective; overrepresentation of junior faculty.

Saturated Themes

Review (when available to them) Do Not Review

Professor 1 0

Associate Professors 3 0

Assistant Professors 6 1

Research Assistant 0 1

0.5

1.5

2.5

3.5

4.5

5.5

6.5

Faculty who Review their Evaluations

# w

ho re

view

None Unsure "Fair Amount" Highest Extent

Professor 0 0 0 1

Associate Professor 0 0 2 1

Assistant Professor 1 2 3 1

Research Assistant 1 0 0 0

0.25

0.75

1.25

1.75

2.25

2.75

3.25

Extent to Which Faculty Internalize Feedback

# of

resp

onse

s

3-Selective →Inform practice

2-Axial→Issues/Reaction

1-Open→Broader perceptions

Years experience as faculty member at site:Range: 0.5-24 Average 6.58

• Do clinical teaching faculty use learner evaluations to inform their teaching practices?

• What are their interpretations of the value of these evaluations, and what are implications on their own development?

“The verbal feedback from my colleague/boss made me more cognizant of my behavior and I modified it appropriately.”

“I think if you are an academic physician with a teaching role, the evaluation should … [be used, but] taken in context. People who are mostly clinical should be held to a higher standard…”

“Inexperienced rotators might not know when the teaching is happening.”

“I think the comments a the bottom in free text, under the numbers are helpful.”

The numbers are worthless. If someone gives me a 7 or 8 with a baseline to get all 9s; if I get a 7, am I really bad at something or is 7 still technically superior?”