AECOM COGME Seminar Implementing the Competencies 360 Degree Evaluations Catherine C. Skae, MD Director, Pediatric Residency Program Children’s Hospital

AECOM COGME SeminarImplementing the Competencies

360 Degree Evaluations

Catherine C. Skae, MD

Director, Pediatric Residency Program

Children’s Hospital at Montefiore

Outline

Definition of 360 degree evaluation ACGME Suggested Best Methods Organization/ Timing Results/ Scholarship Future

Outline

Definition of 360 degree evaluation ACGME Suggested Best Methods Organization/ Timing Results/ Scholarship Future

Definition

Consist of measurement tools completed by multiple people in a person’s sphere of influence

Evaluators are superiors, peers, subordinates, patients, and families

Two practical challenges: constructing the surveys and orchestrating data collection

Outline

Definition of 360 degree evaluations ACGME Suggested Best Methods Organization/ Timing Results/ Scholarship Future

ACGME Toolbox Suggested Best Methods of Evaluation

1= Most Desirable:Patient Care- Work within a team

Practice-Based Learning & Improvement- Use of information technologyFacilitate learning of others


1= Most Desirable:Professionalism-

Ethically sound practiceSensitive to culture, age, gender, disability issues

Systems-based Practice- Understand interaction of

their practices w/ the larger system Advocate for patients within the healthcare system


2= Next Best Method:Patient Care- Caring & respectful behaviors

Counsel & educate patients and familiesPerformance of medical procedures

Interpersonal & Communication Skills- Creation of therapeutic relationshipsListening skills

Professionalism- Respectful, altruisticSyst-Based Practice- Practice cost-effective care


3= A Potentially Applicable Method:Patient Care- Interviewing;

Developing and Carrying out Management PlansMedical Knowledge-

Investigatory & Analytic thinkingKnowledge and Application of Basic Sciences

Practice-Based Learning & Improvement- Analyzing own practice for needed improvements

Outline


Faculty-Continuity Clinic

Intern/Resident

Faculty-Inpatient/ Elective


Junior Residents

Intern/Resident Senior

Residents


Intern 360 Degree EvaluationCHILDREN’S HOSPITAL AT MONTEFIORE

EDIATRIC ROTATION EVALUATION• JUNIOR/SENIOR RESIDENT: _________________________• • INTERN: ________________________________ (1, 2= below expectations; 3 = meeting expectations; 4, 5 = exceeding expectations)• ALL 1,2, AND 3 RATINGS REQUIRE COMMENTS AT END OF FORM

• I. PATIENT CARE• Fails to gather essential and accurate information via history and P.E. 1 2 3 4 5 N/A Gathers essential and accurate information via

history /P.E.• Experiences difficulty in developing and carrying out plans 1 2 3 4 5 N/A Develops and carries out patient management plans

appropriately• Has difficulty taking responsibility and lacks accountability for decisions 1 2 3 4 5 N/A Demonstrates responsibility and accountability for decisions• Has difficulty performing technical procedures 1 2 3 4 5 N/A Performs technical procedures skillfully

• II. MEDICAL KNOWLEDGE• Does not demonstrate independent investigatory and analytic thinking 1 2 3 4 5 N/A Demonstrates independent investigator and analytic

thinking• Does not have adequate knowledge base of basic and clinical sciences 1 2 3 4 5 N/A Knows and applies basic and clinical sciences• • III. INTERPERSONAL SKILLS AND COMMUNICATION• Does not use effective listening skills to elicit information 1 2 3 4 5 N/A Uses effective listening skills to elicit information• Does not work effectively as member of health care team 1 2 3 4 5 N/A Works effectively as a member of a health care team• Has difficulty in creating & sustaining therapeutic relationships 1 2 3 4 5 N/A Creates and sustains therapeutic relationships


Junior Residents


Residents

PhysicianAssistants/

Nurses



Junior Residents

SocialWorkers


Residents


Nurses


Social Work 360 Degree Evaluation

• ALBERT EINSTEIN COLLEGE OF MEDICINE- CHILDREN’S HOSPITAL AT MONTEFIORE• PEDIATRIC ROTATION EVALUATION

• • SOCIAL WORKER: _________________________• • RESIDENT: _______________________ (1,2 = below expectations; 3 = meeting expectations; 4,5 = exceeding expectations)•

• DATE OF EVALUATION: ___________________________•

• I. MEDICAL KNOWLEDGE• Knows how to make appropriate Social Services referrals 1 2 3 4 5 N/A

• • • II. INTERPERSONAL SKILLS AND COMMUNICATION• Communicates well with Social Workers 1 2 3 4 5 N/A• • • III. PROFESSIONALISM• Exhibits a positive attitude toward work 1 2 3 4 5 N/A

• Attends and contributes to Interdisciplinary Team Rounds 1 2 3 4 5 N/A ••• PLEASE ADD COMMENTS:•


Interns/ Junior Residents

SocialWorkers

Patients/Families


Residents


Nurses



Interns/ Junior Residents

SocialWorkers

Patients/Families

Intern/Resident

Standardized

Patients

Senior

Residents


Nurses


Outline


Results/ Scholarship

Assessing Competencies of Pediatric Residents in an Outpatient Setting.

Sandra Braganza, Iman Sharif, Philip Ozuah.

Objectives: To determine whether the assessment of pediatric residents’

ACGME competencies by peers, support staff, and parents differs from the assessment by preceptors alone.

To identify particular ACGME competencies for which differences between preceptor and peer/staff/parent assessments are most pronounced.


Methods: 14 statements assessing:

Interpersonal & Communication Skills- Communication and Empathy

Professionalism- Respect & Integrity


Results: 271 surveys completedCompared to preceptors, resident peers reported more

problems with Communication and Empathy. Preceptors noted more problems with Communication than

did parents. Parents were more likely to report problems with Empathy

and Respect than were preceptors.

Conclusion: 360 degree assessment was useful in eliciting different evaluations of competencies by different observers that interact with pediatric residents.


Parental Assessment of Quality of Ambulatory Care: How Do Attendings and Residents Compare?

Andrew D. Racine, Sheila Liewehr, Laura Weingart, Scott D. Krugman. Eastern Society for Pediatric Research. March

2005.

Objective: To compare the quality of pediatric ambulatory care reported by parents of children with resident versus attending primary care providers.


Results: 473 surveys were returnedNo differences were found between residents and attendings in aggregate scores for any of the 5 quality domainsBut on 2 individual items: does the doctor explain things to your satisfaction, and can the doctor arrange for referrals- attending scores were higher.

Conclusions: Parental assessment of the overall quality of care delivered by pediatric residents compares favorably to that delivered in similar settings by attending pediatricians.


Determining the Essential Components of Professionalism and Interpersonal/ Communication Skills.

Laura Dattner, Philip Ozuah. Pediatric Academic Societies- May 2005.

Objective: To determine what pediatric patients and their parents consider to be essential components of professionalism and interpersonal/ communication skills.


Methods: Anonymous web-based survey for hospitalized patients and their parents. Respondents were asked eight questions about the residents who cared for them.


Results: 253 resident evaluation surveys were completed.Overall, residents received very positive evaluations.

Questions which predicted a global score of 5 were: “Does Dr. X. speak to you in ways that are clear and easy to understand?” “Does Dr. X. include you in decisions and planning of care?” “Does Dr. X. listen to your concerns?”


Conclusions: Essential components of professionalism and interpersonal/ communication skills which independently predicted an excellent global score were:

listening to concerns

including families in decisions and planning of care

clear communication


Can We Rely on Adolescent Standardized Patients to Assess Communication Skills?

Reliability of Adolescent Standardized Patients in Assessing Professionalism.

The Utility of Unannounced Standardized Patients in Medical Education.

Philip Ozuah, Marina Reznik, and Sandra Braganza. Pediatric Academic Societies- May 2005.


Adolescent standardized patients (SP) are a reliable method for use in assessing residents’ competency in communication skills.

There was a strong correlation between faculty and adolescent SP’s ratings of professionalism.

Residents displayed lower levels of professionalism in unannounced standardized patient encounters.

Outline


The Future

• Peers

• Fellows

• Inpatient Nurses

• Unit Clerks

• myevaluations.com

Thank You!

Documents

AECOM COGME Seminar Implementing the Competencies 360 Degree Evaluations Catherine C. Skae, MD Director, Pediatric Residency Program Children’s Hospital