Learner Feedback on the Pediatrics Milestones … Feedback on the Pediatrics Milestones Assessment Project Sue Poynter, MD1, Teri Turner, MD, MPH, MSEd2, Christa Chaffinch, MA3, Patricia

Learner Feedback on the Pediatrics Milestones Assessment Project

Sue Poynter, MD1, Teri Turner, MD, MPH, MSEd2, Christa Chaffinch, MA3, Patricia J. Hicks, MD, MHPE4

For the APPD LEARN – NBME Pediatrics Milestones Assessment Group

1Cincinnati Children’s Hospital, 2Baylor College of Medicine, 3National Board of Medical Examiners, 4Children’s Hospital of Philadelphia

• Purpose: • Test the feasibility of assessing fourth year medical students’ and pediatric

interns’ inpatient performance using a subset of nine Pediatrics Milestones

• Objectives:

• Direct observation assessment utilizing multisource feedback (MSF) and structured clinical observations (SCOs) for rounds and patient history-taking.

• Evaluate reactions of all participants to a subset of Pediatrics Milestones based on participation in assessments, feedback conversations, and surveys

• A Learner Feedback Survey was used to determine learner satisfaction with their pilot experience regarding frequency and value of feedback received.

Pediatrics Milestones Assessment Pilot

• Methods • Data collection by handheld app

• During direct observations: MSF and SCO (rounds/history taking)

• Key features of Feedback • Immediately following SCOs

• Aggregate feedback session with learners at completion of a rotation

• Faculty development

• Measures of success • Surveys, interviews, and focus groups

Pediatrics Milestones Assessment Pilot

Learner Survey Methodology

Question Types

2 frequency

80 learners surveyed; 63 responded (79%)

3 yes/no/not

sure

2 agree/disagree

5 free-text comment

items

Learner Survey - Analysis

• 63/80 (79%) respondents after 6 months of data collection

• Mixed methods approach • Quantitative: word counts, responses to non-comment

items

• Qualitative: text analysis of learner comments • Grounded theory / Inductive analysis approach

• 4 coders

• Trustworthiness check

• Initial coding and constant comparison

• Development of themes, concept maps, exemplars

Responses by Site

10

3

14

12

16

8

University of Virginia Health System

University of Michigan CS MottChildren's Hospital

Phoenix Children's Hospital

Cincinnati Children's Hospital

Children's Hospital of Philadelphia

Baylor College of Medicine

Orientation to Pilot

7 4 52 (83%)

0% 20% 40% 60% 80% 100%

Did you receive orientation to or explanation about your participation

in the pilot?

No Not Sure Yes

Pilot Expectations

3 19 41 (65%)

0% 20% 40% 60% 80% 100%

Did the study meet your expectations?

No Not Sure Yes

Frequency of SCO feedback




End of Rotation Feedback

1

1

1 6

2

30

28

21

28

0% 20% 40% 60% 80% 100%

It helped me understand how those with whom I work perceive my performance. It was useful for constructing future goals or identifying a developmental path.

Thinking about the feedback you received at the end of your rotation, please indicate your level of agreement with the following statements.

Strongly disagree Disagree Neutral Agree Strongly agree

Learner Survey Methodology

Question Types

2 frequency

80 learners surveyed; 63 responded (79%)

3 yes/no/not

sure

2 agree/disagree

5 free-text comment

items

All survey respondents responded to this item.

Please briefly explain the purpose of the study and what, if any, advantages you expected.

Receiving Feedback/multiple sources and timing:

“I expected to benefit from receiving feedback from multiple individuals, including faculty, residents, and ancillary staff.”

“To improve 4th year med students' clinical learning and skills by having multiple points of feedback from attendings, residents, and staff before, during, and after the rotation.”


New Novel System:

“To study the process of assessing residents and medical students based on a set of milestones to be achieved rather than the traditional system.”


Performance improvement:

“…to examine better ways to evaluate and provide feedback to residents that is constructive and consistent. Specifically, by providing concrete ideas of current performance as well as what constitutes the "next level" it helps house staff recognize and realize concrete goals for improvement.”


• 44 of 63 learners left free-text responses

• 41 learners responded “yes”, the study did meet their expectations (65%)

• 3 responded “no”, it did not

• 19 responded “not sure” if the study met their expectations

Please briefly explain why the study met or did not meet your expectations.

• Specific, Descriptive Feedback : “I received constructive feedback that was much more extensive and specific then what we normally receive...”

• Feedback as a Guide: “…feedback provided was far more informative with regards to how I was performing as a sub-intern as well as providing concrete goals for improvement…”

• Quality of Feedback: “…was given good, constructive feedback at the end including where I fell on the [Pediatrics] Milestones ratings and what the expectations were for somebody at the sub-I level.”


“Behind the scenes”

“The observers were non-obtrusive and I received some specific and valuable feedback.”

“I did not feel that the method of evaluating hindered by ability to do my daily work and yet I was provided with a wealth of information on my performance.”


47 of 63 learners left free-text responses Of those who did not respond,

14 reported having “never” been given feedback; 2 others reported no change or “n/a”.

Based on the feedback you received describe what you have done or plan to do differently on rounds.

Preparation & Planning: Try to be more prepared with how I want

to explain things to families in language they can understand.”

Self-Efficacy or Confidence: “Become more confident in stating

what my thoughts are regarding the diagnosis and plan.”

Aspects of Self: “I plan to be more discerning in which topics I choose to explore with patients as I have a tendency to over-explain the reasoning… I also will work to speak slower as I have always had a tendency to speak too quickly.”

Rationale: “Discuss with the group why I am doing what I plan to do with my patients.”

“…To incorporate reasoning.”

Based on the feedback you received describe what you have done or plan to do differently on rounds.

34 of 63 learners left free-text responses

Of those who did not respond to this item,

27 reported having “never” been given feedback;

the other 2 reported “n/a”

Based on the feedback you received describe what you have done or plan to do differently when you elicit a patient history.

• Verbal Strategies: “Talking to the patient in more layman terms.”

• Non-verbal Strategies: “Be more attentive and sensitive about my patients' and their families' reactions.”


• Prioritization: “…address the most concerning aspects of a patient's history first.”

• Focused/Direct: “…use more open ended questions.”

• Depth/Specificity: “…more probing questions regarding complications of past medical history.”

• Organization/Structure: “Elicit history in chronological order to increase clarity of history.”


All learners responded Of those, 6 contained content not coded, such as:

“None” “None I can think of” “Nothing at present” “Unsure of recommendations”

What suggestions do you have for improving the feedback you receive to help you in your

professional development as a physician?

Specificity and coaching “I felt that this [study] allowed me to receive more specific feedback and gave me tangible things to work on.” “I'd rather receive constructive criticism and take a hit to my ego, than continue to perform at a sub-optimal level.”



Constraints “I think that if this were to be implemented as the status quo throughout residency programs, there would have to be protected time for observation.” “[The only suggestion I would have would be] trying to have more input for many members of the team I worked with over the course of the month.”





Emerging Themes on Learners’ Plans to Incorporate Feedback

• Balance open-ended and direct questions (20%),

• Improve communication with patients and families, particularly clarify and confirm shared understanding (36%),

• Thoroughness with questioning and building care plans (40%).

Emerging Themes for Pilot Improvements

• Feedback should be specific, with examples of both “positive” and “negative” behaviors (30%),

• Should come from a variety of team members (17%)

• Feedback sessions should be scheduled regularly (23%).

Study Limitations

• Potential Bias

• Possibly primed respondents with questions/terms before they completed comment items: “feedback”, “advantages”, “constructing future goals”

• Generalizability

• Time of year, academic year varied for sub-interns

• Response rate

• Free-text responses not required

• Inability to correlate response to learner performance

• Anonymity of respondents

Results Summary

• 63 (80%) responses have been received.

• More than 65% of learners indicated their expectations were met.

• The frequency of real-time feedback varied based on the nature of the observation (88% on rounds; 57% taking a patient history).

• Nearly 88% of survey respondents agreed/strongly agreed that end-of-month feedback was useful for creating future improvement goals.

• Survey narrative comments contained emerging themes of how learners were targeting areas for improvement based on the feedback they received during the pilot.

Conclusions

• Overall, learners indicated a high level of satisfaction that pilot participation allowed more frequent, focused feedback that proved useful in improving their current performance and constructing future improvement goals.

• Despite study limitations, the Learner Survey results support the utility of the Pediatrics Milestones as an assessment method to provide learners with formative feedback.

Collaborators

APPD LEARN Pediatrics Milestones Assessment Project Group (Study data collection sites)

(I = Investigator, C = Coordinator): Baylor College of Medicine: Teri Turner, MD, MPH, MSEd (I); Melodie

Allison, C-TAGME (I). Children's Hospital of Philadelphia: Patricia J. Hicks, MD, MHPE (I); Jeanine Ronan, MD (I);

Rebecca Tenney-Soeiro, MD (I); Dawn Young, MSEd (C). Children's National Medical Center: Aisha Barber Davis, MD

(I). Cincinnati Children's Hospital: Javier Gonzalez del Rey, MD (I); Sue Poynter, MD (I). Phoenix Children's

Hospital/Maricopa Medical Center: Grace L. Caputo, MD, MPH (I); Vasudha Bhavaraju, MD (I); Ryan Bode, MD (I);

Erin Kuroiwa, MHI (C). University of Michigan: Hilary Haftel, MD, MHPE (I), Kristen Wright, MD (I), Rosalind Moffitt

(C). University of Virginia: Linda Waggoner-Fountain, MD (I), Lisa Morris (C). Emory University: Susie Buchter, MD (I).

Stony Brook Long Island Children’s Hospital: Robyn Blair, MD (I); Rachel Boykan, MD (I); Maribeth Chitkara, MD (I).

University of Florida: Cynthia Powell, M.Ed. (I), Nicole Paradise Black, MD (I); Lilly Chang, MD (I). University of

Illinois at Chicago: Amanda Osta, MD (I); Michelle Barnes, MD (I); Emri Tas, MD (I); Jen McDonnell, MD (I); Jordan

Hupert, MD (I), Karen Hayani, MD (I), Jonathan Miller, MD (I). University of Louisville: Sara Multerer, MD (I);

Kimberly Boland, MD (I); Olivia Mittel, MD (I). University of Texas Health Sciences Center: Sandra Arnold, MD (I);

Mark C. Bugnitz, MD (I). Inova Fairfax Hospital for Children: Meredith Carter, MD (I). Boston University: Daniel J.

Schumacher, MD, MEd (I). Vermont Children's Hospital: Ann Guillot, MD (I); Karen Leonard, MD (I). Winthrop-

University Hospital: Jill Leavens-Maurer, MD (I); Ulka Kothari, MBBS (I); Robert Lee, DO (I). Wright State University:

Ann Burke, MD (I).

National Board of Medical Examiners: Stephen G. Clyman, MD; Thomas Rebbecchi, MD; Colleen

Canavan, MS; Christa Chaffinch, MA; Melissa J. Margolis, PhD; Margaret Richmond, MS; Kathleen M. Rose; Leta Rose;

Yelena Spector, MPH.

Association of Pediatric Program Directors Longitudinal Educational Assessment Research

Network: Alan Schwartz, PhD; Patricia J. Hicks, MD, MHPE.

Documents

Learner Feedback on the Pediatrics Milestones … Feedback on the Pediatrics Milestones Assessment Project Sue Poynter, MD1, Teri Turner, MD, MPH, MSEd2, Christa Chaffinch, MA3, Patricia