Embed Size (px)
Citation preview
Preparing Final Clerkship Performance Evaluations
A Guide for Clerkship Directors and Evaluation Teams
2015-16
Table of Contents
• Evaluation Criteria • Data collection• Interpreting evaluation data:
– Patient Care (RIME)– Professionalism and Interpersonal
Communication• Final grades• Narrative evaluations• Appeals• Resources and Contacts
Evaluation Criteria
Grades and MSPE reporting
• Students may earn a grade of Pass or Pass with Distinction in each of three domains:– Patient Care– Professionalism and Interpersonal
Communication– Final Exam
• Grades for each domain are reported separately in the MSPE
Criteria for Pass
• Patient Care:– Direct observations of clinical skills complete (2 per
clerkship)– RIME Interpreter
• Professionalism/Interpersonal Communication:– No significant or consistent concerns– Patient logs, other assignments complete
• Exam score above clerkship passing threshold
Criteria for Pass with Distinction
• All Passing criteria must be met AND
• Patient Care:– RIME Manager
• Professionalism and Interpersonal Communication (IPC):– Multisource feedback requested: 1 non-MD staff member, 1
patient, 1 peer– Consistent evidence of both Exceptional Professionalism
and IPC with patients AND members of the medical team
• Exam score above clerkship-specific threshold for Pass with Distinction
Data Collection
Data Collection
• Ideally– 100% of potential evaluators – will complete electronic evaluation forms in
E*Value.– within a week of an evaluation being
assigned.
Data collection
• In reality– The minimum acceptable return rate is
50%.– Final evaluations must be submitted within
4 weeks of the end of the clerkship. – Some evaluators will not want to use
E*Value.
Recommendations
• Keep an eye on return rates – start reminding and re-requesting early.
• Clerkships may need to tailor approaches to data-gathering to suit the needs of different groups, sites, or individual evaluators
Recommendations
• Avoid going into an evaluation team review or submitting final evaluations with a return rate under 50%. Evaluations based on input from fewer than 50% of potential evaluators are unlikely to stand up in an appeal.
• All evaluators who submit information (residents, faculty, fellows, non-MD staff, peers) should be listed as Contributing Evaluators in the final evaluation form.
Data Collection
• For ideas, resources, and practical support in boosting evaluation return rates, contact – Jen Deitz ([email protected])– Gretchen Shawver ([email protected]) or– Elizabeth Stuart ([email protected])
Interpreting Evaluation Data:Patient Care (RIME)
Two paths to distinction
Students may earn Pass with Distinction by meeting criteria throughout the clerkship OR improving to meet criteria by the
end of the rotation
Performance
Time
Threshold for Distinction
Expected Transitions
MANAGERMANAGER
POM Core clerkship
Sub-I Residency +
INTERPRETERINTERPRETER
REPORTERREPORTER
Core clerkship students are expected to be in the Interpreter stage.
Interpreting Data from Patient Care (RIME) forms
• For a student in the Interpreter stage:
– Nearly all evaluators will select Consistently True for items in the Reporter section.
– Data will include a mix of Sometimes and Consistently True for items in the Interpreter section.
Reporter Items
Interpreter Items
Alert
• If significant number of evaluators - throughout the rotation - have selected Sometimes or Rarely True on Reporting items – the student has not met expectations for performance in a core clerkship and should receive a non-passing grade.
• Clerkship directors should speak directly with individual evaluators to confirm that a student is not consistently demonstrating the skills required in the Reporter stage.
Pass with Distinction
MANAGERMANAGER
POM Core clerkship
Sub-I Residency +
INTERPRETERINTERPRETER
REPORTERREPORTER
Pass with Distinction requires consistently strong Reporting and Interpreting (solid bars) with evidence of being in the transition to the
Manager stage.
Interpreting Data from Patient Care (RIME) forms
• For a student in the Manager stage:
– Nearly all evaluators will select Consistently True for items in the Reporter and Interpreter sections.
– Data will include a mix of Sometimes and Consistently True for items in the Manager section.
Manager/Educator Items
Broadly
• Students in the Manager stage are functioning above the expected level – at the level of a sub-intern or beyond.
Q & A
Q: Should core clerkship students really be expected to manage patient care?
Q & A: “Manager”
A: The term Manager is not meant to imply independent implementation of patient care plans. Students who are Managers demonstrate a sense of personal responsibility for knowing as much as possible about their patients and ensuring that they receive optimal care. Managers see themselves – and are seen by others – as patients’ primary providers and advocates.
Q & A: M without I?
Q: The evaluators for one of my students marked Consistently True for all of the Manager items, but only Sometimes True in the Interpreter section. Does the student meet criteria for being a RIME Manager?
Q &A: M without I?
A: No. Pass with Distinction requires consistently strong Reporting and Interpreting skills in addition to the skills and attitudes represented by the Manager items on the evaluation form.
Q & A: Outliers
Q: What if one evaluator checked Rarely or Sometimes True on an item – when all other evaluators marked Consistently True?
Q & A: Outliers
A: When reviewing data for both Patient Care and Professionalism/Interpersonal Communication, Evaluation Teams should look for trends and themes - over time and across evaluators.
Outliers
In sorting through outliers, consider:• Role/identity of the evaluator• Setting• Time the evaluator spent with the
student• Timing during the rotation
Q & A: Ratings vs. Comments
Q: What if all of the ratings in the Interpreter section are “consistently true”, but the narrative comments suggest difficulty with synthesizing, prioritizing, and presenting clinical information?
Q & A: Ratings vs. Comments
A: Evaluation teams should use both ratings and comments to determine whether a student is performing solidly at the Interpreter or Manager level.
Q & A: Mean scores
Q: Can we use a mean score cutoff to determine whether a student’s performance meets criteria for PWD?Updated
for 2015-16
Q & A: Mean scores
A: Mean scores can be helpful as an initial screen. Mean scores do not take into account:
– Contact between student and evaluator– Training level of the evaluator– Improvement over time
Do not rely exclusively on mean scores.
Updated for 2015-
16
Interpreting Evaluation Data:Professionalism and Interpersonal
Communication (IPC)
Interpreting data from the Professionalism/IPC form
• For a final grade of Pass, there should be a consistent trend of meeting expectations, with no significant or consistent concerns.
Fundamentals of Professionalism and IPC
Interpreting data from the Professionalism/IPC form
• For Pass with Distinction, data from multiple evaluators should show a consistent trend of exceptional performance in the following subdomains:
– Interactions with patients– Interactions with teams– Self-management
Updated for 2015-
16
Exceptional Professionalism/IPC
Updated for 2015-
16
Exceptional Professionalism/IPC
• In the following example, multiple evaluators have reported seeing evidence of exceptional performance across all three subdomains.
• Assuming that the student requested multisource feedback – and that there were no concerns about professionalism - his or her performance would meet criteria for PWD for professionalism/IPC.
Updated for 2015-
16
Professionalism/IPC - example
Count Option
16Extends him/herself beyond usual duties to ensure patients' comfort or well-being.
17 Advocates respectfully and diplomatically on behalf of patients.10 Serves as patients' preferred source of information and/or support.7 Makes an extra effort to support or help fellow students and others excel.8 Without prompting, takes on extra work to help the team/preceptor.
7Supports the team by paying attention to the needs and care plans of patients other than those assigned.
5 Maintains composure and manages conflict in difficult situations.
10Makes an extra effort to participate in learning opportunities beyond those required.
11 Seeks and responds openly and proactively to feedback.
11Demonstrates an advanced degree of personal responsibility and accountability– beyond being punctual and reliable.
0 I have not observed any of the above behaviors.1 I have not spent enough time with the student to make an assessment.
Multisource Feedback (MSF)
• To meet criteria for Pass with Distinction, students must request multisource feedback from– 1 non-MD staff member– 1 patient– 1 peer
• A response by peers, non-MD staff, patients is not required.
• Clerkships must put systems in place to record students’ MSF requests.
Q & A: Multisource Feedback
Q: What if a non-MD staff member reports concerns about professionalism? Does that disqualify the student for PWD?
Q: What if a patient or non-MD staff member submits a glowing description of a student’s Interpersonal Communication? Shouldn’t that information count toward Pass with Distinction?
Q & A: Multisource Feedback
A: The content of multisource feedback is not considered in determining whether a student has met criteria for Pass with Distinction
Positive comments from MSF content may be used as examples in the final summative evaluation.
Multisource Feedback
Anonymized multisource feedback comments should be included in the formative narrative section of the final evaluation – and explicitly labeled as MSF.
Updated for 2015-
16
Final Grades
Final grade options
• Options
– N - Continuing– Fail – Marginal Pass– Pass
• Pass with Distinction will be recorded separately for each performance domain
N grade
• N is for failed exams.
• N should not be used for:
– Marginal/non-passing performance in the domains of Patient Care or Professionalism/Interpersonal Communication
– Missed time from the clerkship
– Incomplete assignments
• Use Marginal Pass or suspend the evaluation if a student has not completed all clerkship requirements
More on N
Use when… Consequences
N • All aspects of performance meet criteria for passing, but student does not pass the final exam.
• Should not be used for marginal or failing performance in any other domain.
• Except in case of emergency, failure to attend the final exam session without prior permission from the clerkship director will result in referral to the Committee on Performance, Professionalism and Promotion (CP3) for unprofessional behavior.
Must retake exam
After second failed attempt, student takes oral exam or suitable alternative, to be determined by the clerkship director.
N grades must be corrected within 12 months of the end of the clerkship.
See MD Program Handbook for additional details.
Marginal Pass vs. Fail
Use when… Consequences
MP • Performance fails to meet criteria in one domain (other than exam).
• Mild concern about Patient Care or Professionalism/Interpersonal Communication
• Reviewed by CP3.• Remediation plan to be
determined by the clerkship director.
• 3 Marginal Passes in clerkships may be considered grounds for dismissal.
F • Performance fails to meet criteria in more than one domain.
• Significant concern about Patient care or Professionalism/Interpersonal Communication
• Reviewed by CP3.• Remediation plan to be
determined by the clerkship director.
• 2 Failing grades in clerkships may be considered grounds for dismissal
See MD Program Handbook for additional details.
Q & A: Evaluation Review Shortcuts?
Q: Does the full evaluation team need to review files for all students, or just those who seem likely to meet criteria for Pass with Distinction in one or more domains?
Q & A: Shortcuts?
A: CBEI introduced not only a new grade to recognize exceptional performance, but also a new process to ensure a fair and balanced review for each student.
All students should benefit from the process of a full review.
Q & A: Evaluator Lists
Q: Does the final evaluation in E*Value need to list every individual evaluator by name?
Q & A: Evaluator Lists
A: All contributors should be listed individually, by name.
If there is concern about protecting the identity of an individual evaluator, the clerkship may opt to list the names of all evaluator who were asked to contribute to the final evaluation.
Narratives
Guidelines for Narratives
Summative vs. Formative
Guidelines for Summative Narratives
Summative narratives should be:
– 100-200 words long
– Framed as cohesive paragraphs (not lists of quotes) reflecting student performance in each of the key domains:
• Patient Care• Medical Knowledge• Professionalism and Interpersonal
Communication
Patient Care: RIME
Narrative comments for patient care should address students’ skills in
•Reporting
•Interpreting
•Managing patient care
Patient care – essential elements
• Interviews
• Exams
• Presentations
• Organization, synthesis and summarization of information
• Selective attention to pertinent details
• Differential diagnoses
• Prioritization of problems and tasks
• Ability to manage day-to-day tasks in patient care
Knowledge
• Narrative comments on knowledge might address exam performance, acquisition and application of new knowledge, and/or ability to link basic science to clinical care
Professionalism and Interpersonal Communication
Narratives should address•Interactions with patients•Interactions with teams•Self-management
– Motivation– Initiative– Response to feedback
Updated for 2015-
16
MSPE format
Formative evaluation: Two sections
• The final evaluation form includes two sections for formative comments:
– Formative comments from clerkship director
– Individual evaluators’ comments and Multisource feedback (anonymous/de-identified)
Updated for 2015-
16
• Summative evaluation involves a final judgment and description of a student's performance during a clerkship.
• Formative evaluation provides ideas
and recommendations for further learning and improvement.
Formative vs. Summative
Formative vs. Summative: An Analogy
A cook asks for an evaluation of his soup
Adapted from Stanford School of Medicine Clerkship Evaluation Tutorial, 2006
Summative & Formative: Analogy
Summative evaluation would answer the question:
Formative evaluation would answer the question:
How was the soup? What might be done to improve the soup?
Yuck!Pretty good. Very tasty.A culinary masterpiece!
Needs a little more salt. Heat it up a bit more.
Some Tabasco would round out the flavor.
Formative vs. Summative
• Summative narratives submitted in E*Value are intended to be cut and pasted verbatim into the MSPE.
• Formative narratives are not included in the MSPE. Formative comments in E*Value are seen by:– Students– Advising Deans– Members of the Committee on Performance,
Professionalism, and Promotion (CP3).– Members of the Clerkship Evaluations Committee
Formative vs. Summative
• Summative comments should address performance - weak or strong - in every domain.
• Avoid the temptation to put strengths in the summative section and weaknesses in the formative.
Q & A: Formative vs. Summative
Q: I had a student who did the most amazing physical exams but who had a very hard time organizing presentations.
Should I put positive comments about his exams in the summative section and leave his difficulty with presentations for the formative section?
from Stanford School of Medicine Clerkship Evaluation Tutorial, 2006
Q & A: Formative vs. Summative
A: No. The summative narrative is intended to provide an honest, accurate, and complete description of a student’s performance during a given clerkship. Both "positive" and "negative” aspects of performance should be included.
Should I include…?
• “Negative” or less than glowing comments should be included in the summative section if they reflect:
– Significant or consistent trends in performance over time
– Features of performance that did not change despite mid-rotation feedback
– Characteristics of performance that should be known to potential residency programs
Q & A: Concerns
Q: I have concerns about a student. He responded to mid-clerkship feedback and deserves to pass the rotation – but I worry that he will have difficulty in future rotations and beyond. Should I mention the concerns I had in the formative section?
Q & A: Concerns
A: Yes. The Committee on Performance, Professionalism, and Promotion (CP3) relies on information in both the formative and summative narratives in monitoring student progress. Although formative comments are not included in the MSPE, they are helpful to CP3 as a formal documentation of concerns.
Concerns
• The final evaluation form includes a confidential question to allow clerkship directors to report concerns about student performance.
• These flags are viewed by Advising Deans, Educators for CARE faculty, and the Assistant Dean for Clerkship Education. They serve as tool for identifying trends in student performance..
• Confidential concerns are not seen by students and not reflected any where in the MSPE.
Narrative Evaluations: 2014-15 Trends
• 10 evaluations reviewed for each clerkship - periods 1-5, 2014-15
• Word count– Range: 22-426
• Improved:– Coverage of all relevant domains– Inclusion of formative comments – Synthesis vs. lists of quotes
Updated for 2015-
16
Raising the Bar: Summative Comments
• Comparability across sites– Use keywords (e.g. excellent,
outstanding, very good) consistently or not all
– Consistent length, quality across sites
Updated for 2015-
16
• Match comments to final grade
• The following terms imply PWD-level performance:– Exceptional– Sub-intern/intern level– Above expectations
Raising the Bar: Summative Comments
Updated for 2015-
16
Raising the Bar: Formative Comments
• Use RIME scores to guide feedback from the clerkship director
• Label multisource feedback explicitly
Updated for 2015-
16
Resources
• For more information on narrative evaluations, including guidance as to whether information should be formative or summative and suggestions for framing “negative” ideas, contact:
– Jen Deitz ([email protected])– Elizabeth Stuart ([email protected])
Appeals
Appeals
• Some students will appeal their final clerkship grades and/or the content of their narrative evaluations.
Appeals
• According to the SOM policy on evaluation appeals, the process begins with a discussion between the student and the clerkship director or Advising Dean.
• Unresolved disagreements are referred to the Clerkship Evaluations Committee.
See MD Program Handbook for additional details.
Appeals process
• The CEC will initially contact the appealing student and the clerkship director to discuss the evaluation in question.
• The CEC will then assemble a committee to review the student’s evaluation file to determine whether:– the evaluation process was conducted fairly– the final evaluation was based on sufficient
information– the final grade and narrative warrant revision.
Appeals process
• For an appeal to be considered, a written request must be made to the Division of Evaluation within eight weeks of the date that the final evaluation was submitted in E*Value.
Appeal ≠ Failure
• A request for an appeal does not constitute failure on the part of the clerkship evaluation team.
• Appeals provide an opportunity for learning and calibration. They serve an important function in achieving the broader goal of optimizing consistency in the evaluation across clerkships and sites.
