ACGME General Competencies: Evaluation Methods

ACGME General Competencies:

Evaluation MethodsRichard E. Hawkins, MD

Deputy Vice President, Assessment Programs

National Board of Medical Examiners

Overview of Presentation

Introduction to ACGME Competencies Three important assessment

methods Assessment of the individual

competencies Comprehensive assessment

approaches

ACGME General Competencies

Medical knowledge Patient care Interpersonal and communication skills Professionalism Practice-based learning and

improvement Systems-based practice

Views on the General Competencies

Not well constructed / not portable or coherent Not evidence-based Theoretical basis questionable

? Influence of content and context

Introduction - important stimulus for action Two “competencies” essential for practice Emphasis on outcomes (vs structure &

process) More comprehensive view regarding

evaluation

ACGME General Competencies = KSA + 2

Medical knowledge Patient care Interpersonal and communication skills Professionalism

Practice-based learning and improvement

Systems-based practice

Selecting Assessment Methods: Factors

Validity Reliability Feasibility Credibility / acceptability Quality of feedback provided

Behavioral impact

Assessing the Competencies

Thoughts on implementation: Start with an inventory Refine and/or document existing

approaches Apply new assessment methods to holes Develop methods collaboratively

Apply locally based upon resources and needs Critical role of professional organizations –

PD Associations Understand and take advantage of overlap

between competencies and related assessment methods.

Three Assessment Tools

Portfolios Multi-Source Feedback / 360o

Evaluations Chart-stimulated Recall (CSR)

Portfolios

“…a collection of products prepared by the resident that provides evidence of achievement related to a learning plan.” (ACGME)

“A collection of material that records and reflects on learning events and processes. A sample of work products that demonstrates accomplishments.” (ABMS)

Portfolios

Stimulus for reflection and self-directed learning + repository of evidence

Contents (evidence) depend upon learning and assessment objectives - may include just about anything

Useful for assessing competencies difficult to measure: PBLI, SBP

Reliability, validity, and feasibility issues

Portfolios: Critical Components

Learning objectives “Evidence”

Written work Video-tapes (mini-CEX, SPs) / Audio recordings

(patient communication) Structured faculty observation / rating

“Reflection on action” – essential Reflection on past experiences to guide further

learning Summarizes the learning that has occurred

with discussion of evidence supporting the attainment of the learning objectives

Portfolios: Hybrid Example

Learning objective Analyze the quality of a recent article

using evidenced-based medicine principles

Activities Learner chooses review article on topic

of personal interest Prepares presentation using EBM criteria

(JAMA)

Portfolios: Hybrid Example

Activities “Reflects” on what was learned about

topic and EBM criteria for reviews after presentation

May be self-assessment only or with mentor

Contents in portfolio Copy of presentation Brief written summary of self reflection

Portfolios: Challenges

Well-suited for formative assessment Summative: need clear definitions of

learner goals, content, and explicit evaluation criteria

Must have clear learning objectives Evidence must be of sufficient quality Should demonstrate actual learning Initial development of portfolios

challenging

Portfolios: Conclusions

Learner centered and learner driven Emphasizes reflection, self-assessment Need clear learning objectives Good evidence for use as formative tool

? Future as summative “evaluation tool” Requires fairly significant “up-front”

investment Embraces continuum of education and

practice ? Future role in CPD, MOC, MOL, credentialing and privileging…

360° Evaluations

Definition Evaluation completed by multiple

individuals, usually from different perspectives and based upon observations in different contexts

Raters: self, faculty, peers, nurses, students, patients, other health care providers (medical assistants, social workers, technicians, discharge planners, etc.)

360° Evaluations

PATIENTS

ATTENDINGS

NURSING

PEERSRESIDENT

SELF

360° Evaluations: Strengths

Captures different perspectives: Patients and nurses - evaluate humanism,

professionalism, communication Peers – work ethic, team approach,

professionalism Others – unique observations on key attributes

Supplemental approach for other competencies: Communication / IPS, Patient Care, SBP

360° Evaluations: Limitations

Usual limitations of global rating forms: Reliability: Patients - need 20 – 80 ratings Validity:

Nurse ratings – depends upon rating site and rater background

Individual categories highly related

Logistics of data collection, entry, and analysis

Learner resistance and denial issues

360° Evaluations: Conclusions

Uses - professionalism, humanism, team approach and patient focused skills / attitudes

Raters should provide ratings based upon the context of observation and qualifications

Implementation (Norcini, Med Educ 2003;37:539) Clear purpose, communication of criteria, rater

training, monitoring and feedback

Communication of objectives through assessment importance of team approach and patient-centeredness

Chart Stimulated Recall (CSR)

Extension of medical record review Uses actual patient records as the stimulus

for discussion and template for assessment Expand and elaborate on medical record

entries: Capture information not revealed in routine

audits Explore contextual factors underlying clinical

decisions Further assess FOK, problem solving, clinical

reasoning, diagnostic and therapeutic management


Value points to deficiencies in medical record audit in making judgments regarding physician competence Filling in missing data or context (that result in

underestimation of health care quality) Rationale for diagnostic and therapeutic

decisions

ABEM CSR Research Project

Option for recertification 3 cases / case specific criteria:

Reliability 0.54 Correlation with real cases 0.70 Consistent with SP encounters

Eliminated in 1993 Labor intensive and expensive for ABEM Time consuming for candidates

CPSO Peer Review Program: CSR

8 records + typed summary + interview Generic

Knowledge, data acquisition, problem solving, patient management, comprehension of pathophysiology, resource utilization

Inter-rater reliability .75 - .90 Correlations: MCQ .56; SP .70; oral

exam .53; OSCE .31

Calgary: Chart Audit + CSR

Chart Audit Presenting complaint Differential diagnosis Secondary diagnoses

CSR Hx / PE details Tests considered Other treatments Follow-up details Contextual info:

Practice factors Patient factors System factors Payment method Trainee present

Jennett, JCEHP 1995;15:31


Advantages: Relevant - based on examinee’s patients Addresses higher cognitive skills Reinforces importance of documentation Targets difficult to assess competencies:

Patient care, PBLI and SBP Supplemental information on other

competencies: Medical knowledge – application Communication and IPS – written

communication

The Competencies

ACGME competencies defined as they apply to Internal

Medicine

Medical Knowledge

Defined as demonstrating a command of established and evolving biomedical, clinical and social sciences and the application of that knowledge to patient care and the education of others.

Included in this context are:

Medical Knowledge

An open minded and analytical approach to acquiring new knowledge

The ability to access and critically evaluate current medical information and scientific evidence

Medical Knowledge

Acquisition of applicable knowledge of the basic and clinical sciences that underlie the practice of internal medicine

The application of this knowledge to clinical problem solving, clinical decision-making and critical thinking.

Medical Knowledge

Fund of knowledge: In-training examination / Other written

examinations

Application of knowledge / higher cognitive skills Medical record review – CSR Mini-CEX Conference participation

Analytical approach / critical evaluation skills Overlap with Patient Care and PBLI competencies

Interpersonal and Communication Skills

These skills enable physicians to establish and maintain professional relationships with patients, families, and other members of health care teams.

Included are the abilities to:


Provide effective and professional consultation to other professionals and sustain therapeutic and ethically sound professional relationships with patients, their families, and colleagues;

Use effective listening, nonverbal questioning, and narrative skills to communicate with patients and families;


Interact with consultants in a respectful, appropriate manner; and

Maintain comprehensive, timely, and legible medical records.


Direct Observation: Mini-CEX Standardized patients Structured clinical observations

360o Evaluations: Incorporates the direct observations of: Peers, patients, nurses, students…

Written Communication Medical record reviews Consultant evaluations Portfolio task

Professionalism

Residents are expected to demonstrate behaviors that reflect a commitment to continuous professional development, ethical practice, and understanding and sensitivity to diversity, as well as a responsible attitude towards patients, the profession and society.

Include are the abilities to:

Professionalism

Demonstrate respect, compassion, integrity, and altruism in relationships with patients, families and colleagues;

Demonstrate sensitivity and responsiveness to the gender, age, culture, religion, sexual preference, socioeconomic status, beliefs, behaviors, and disabilities of patients and professional colleagues;

Professionalism

Adhere to principles of confidentiality, scientific / academic integrity, and informed consent; and

Recognize and identify deficiencies in peer performance.

ProfessionalismProfessionalism

Performance evaluations (rating scales) Longitudinal / episodic observations

360o Evaluations Patients, peers, nurses, students…

Direct observation Real or SPs (gender or cultural issues)

Professionalism

Recognize and identify deficiencies in peer performance.

Participation in peer assessment and quality of care review (PBLI)

Patient Care

Defined as compassionate, appropriate, and effective care which encompasses the promotion of health, prevention of illness, treatment of disease, and end of life.

At the cornerstone of competent patient care are the abilities to:

Patient Care Gather accurate, essential information

from all sources, including medical interviews, physical examinations, medical records, and diagnostic / therapeutic procedures;

Make informed recommendations about preventive, diagnostic, and therapeutic options and interventions that are based on clinical judgement, scientific evidence, and patient preference;

Patient Care

Develop, negotiate, and implement effective patient management plans and integration of patient care; and

Competently perform the diagnostic and therapeutic procedures inherent to the practice of internal medicine.

Patient Care Data-gathering (Hx, PE, communication) /

patient education and counseling / informed decision making: Direct observation (Mini-CEX, SP) Patient satisfaction (part of 360o) – preferences

Overlap with Communication, Medical Knowledge

Patient management / use of scientific evidence: Medical record review +/- CSR Journal article review / Literature search Portfolio task Conference participation

Overlap with PBLI, SBP, Medical Knowledge

Patient Care (continued)

Procedural / technical skills: Direct observation Performance evaluations Computer / mannequin simulators Medical record review (+/- CSR)

Documentation Indications, Interpretation of results

Practice-based Learning and Improvement

The ability to use scientific evidence and methods to investigate, evaluate, and improve patient care practices.

This effort encompasses the abilities to:


Identify areas for improvement and implement strategies to enhance knowledge, skills, attitudes, values, and processes of care;

Analyze and evaluate practice experiences and implement strategies to continually improve the quality of patient care;


Develop and maintain a willingness to learn from errors and use errors to improve systems or processes of care; and,

Use information technology and/or other available methodologies to access and manage information, support patient care decisions, and enhance both patient and physician education.

PBLI: Important Message

Need for life-long learning Current practice can be improved Traditional CME not effective

Value of experiential learning / relevance Emphasizes the continuum of training and

practice Endorses self-assessment and life-long learning Facilitates and structures self-assessment Consistent with models of practitioner learning

and behavioral change

Validation

Contemplation

Assessment

Definition of Educational Need

Educational Action

Reward

Desired Practice Actual Practice

Handfield-Jones, 2002

PBLI: Elements

Derived from CQI principles: Determine improvement needs Identify and apply an intervention Measure impact of the intervention

PBLI Steps

1. Monitor practice

2. Reflect on or analyze practice to identify learning or improvement needs

3. Engage in learning or plan improvement

4. Apply new learning or improvement

5. Monitor impact of learning or improvement

*Lynch et al. Teach Learn Med, 2004;16:85-92

PBLI Steps and Assessment Approaches

1. Monitor practice

2. Reflect on or analyze practice to identify learning or improvement needs

3. Engage in learning or plan improvement

4. Apply new learning or improvement

5. Monitor impact of learning or improvement

Performance ratings

Medical record review

Project

Portfolio

*Lynch et al. Teach Learn Med, 2004;16:85-92

PBLI: Portfolio Contents Evidence:

Medical record review (+/- CSR) Critical incident log Self assessment information Conference participation: AM report,

journal club Literature search / EBM based Participation in QA / PI activities

Project

Systems-based PracticeSystems-based Practice

This encompasses both an understanding of the contexts and systems in which health care is provided and the application of this knowledge to improve and optimize health care.

Included are the abilities to:

Systems-based Practice

Understand, access, and utilize the resources, providers, and systems necessary to provide optimal patient care;

Understand the limitations and opportunities inherent in various practice types and delivery systems and develop strategies to optimize care for the individual patient;


Apply evidence-based, cost-conscious strategies to prevention, diagnosis, and disease management; and

Collaborate with other members of the health care team to assist patients in dealing effectively with complex systems and improve systematic process of care.

‘Translation’ of SBP Elements

Use resources, providers and systems wisely to provide optimal care

Understand pros and cons of various delivery systems

Practice evidence-based, cost-effective care

Function as a team-member to advocate for patients and improve systems of care

SBP: Basic Concept

Awareness of and responsiveness to the larger context and system of health care

Ability to effectively call on system resources to provide care that is of optimal value

L. Headrick, Sept 24, 2004


…ethical principles underlying rationing, distributive justice, fidelity (to individual patients, and stewardship (for a population of patients).”

Gordon Moore

SBP: Relevance and Importance

Physician action alone not enough to ensure optimal health outcomes

System view is critical to understanding patient outcomes, safety, value, and quality

PBLI focuses on change at the provider level; SBP focuses on the interdependencies of a system

M. Splaine, Sept 24, 2004

SBP: Assessment Methods Primary:

Portfolio task / project 360o Evaluations (nurses, case managers,

social workers, discharge planners) Structured Oral Discussion MR review + CSR

Secondary: Performance ratings

Inpatient and ambulatory; Discharge rounds Conference participation Case-based discussion:

Paper vignettes / SP cases

SBP: Portfolio Tasks Do a detailed process flow chart of a

patient visit and identify all systems involved

Understand a patient adverse event using a root cause analysis approach

Reflect on a critical incident where individual patient interests conflict with system priorities or resource constraints

Describe methods used to advocate for patient home health care needs

PBLI and SBP

PBLI: EBM, statistics, audit methods, IT, self-assessment, population health, attitude of CI…

SBP: microsystems, root cause analysis, process flow charts, distributive justice, cost-effectiveness

-Complex, multi-component competencies

-Different methods for teaching (didactics, small group, supervised clinical experience) and assessment

-Resource intensive: cost-efficient strategies

Used with permission of the ACGME, 10/2001.

Documents

ACGME General Competencies: Evaluation Methods