Competence By Design (CBD)

Assessment and Evaluation

Raja Rajamohan

Assessment

• Focuses on learning, teaching and outcomes

• Interactive process - informs faculty how well their students are learning what they are teaching

• Used to make changes in the learning environment and improve students learning and study habits

• This information is learner-centered, course based, and not graded

Evaluation

• Evaluation is a systematic and objective review of an ongoing or completed project or programme

• The aim is to determine the relevance, effectiveness and fulfilment of objectives

Angelo, T and Cross, K.P. 1993. Classroom assessment techniques a handbook for college teachers. Jossey-Bass A Wiley Imprint, San Francisco, CA. Pp 427.

Assessment Evaluation

Formative: ongoing, to improve learning

Summative: final, to gauge quality

Process-oriented: how learning is going

Product-oriented: what’s been learned

Diagnostic: identify areas for improvement

Judgmental: arrive at an overall grade/score

How are we doing this now?

• Assessment- Daily - staff

• Evaluations- Monthly/ End of rotation - site coordinator- Bi-annual – Evaluation committee/Program Director- Exams / OSCE / Simulation

RCPSC – End of 5th yearABA – PGY 2 – 5 every yearAKT - 3rd, 4th & beginning of 5th

Basic sciences exam - 2nd yrOral exams - 2x / year starting in 2nd year

Daily assessment forms

• Knowledge• History taking and physical examination skills• Technical skills• Communication skills • Clinical reasoning skills• Health promotion• Teaching• Collaboration • Comments section – Strengths / Areas for

improvement

A very good day

Problems with existing system

• Lacks validity - Assess a restricted range of competencies

• Have limited inter-rater and intra-rater reliability - Do not discriminate between residents or performances (i.e., they lack reproducibility)

• Rater errors - distribution such as the leniency/severity error (doves/hawks) - central tendency error (failure to use the entire rating scale)- correlation (halo effect) - influenced by prior impression

• Even when practice behaviors are observed on a daily basis, they frequently are not documented or not done in a timely fashion

Improving In-Training Evaluation Programs.J Gen Intern Med. 1998 May; 13(5): 317–323.Jeffrey Turnbull, Jean Gray, JohnMacFadyen

The Need for CBD

• Evidence that our existing work-place based assessment tools, such as ITERS, are inefficient and ineffective

• The existing tools focus on documenting whether or not a resident was present, rather than assessing skill level or guiding future learning

The Need for CBD

• Assessment tools promote inefficient, checklist-style feedback

• There is concern about the quality and quantity of direct observation/assessment of a resident’s abilities

• Supervisors are often unsure of how/what to assess and as a result provide ambiguous feedback such as “very good” or “needs more work”

• Lack of clear learning paths allows some supervisors to teach based on their own interests – rather than focus on the curriculum and the needs of the trainee

The Need for CBD

• There is increased culture of “failure to fail”

• The onus rests on the teacher/assessor to demonstrate that a trainee is struggling or falling behind

• It can be difficult to gather this evidence and residents (who are not ready) can be passed from one rotation to the next

The Need for CBD

• Residents are often unclear of the learning objectives of their program

• They’re not sure what they need to learn, when they need to learn or how they will be assessed and what they will be assessed on

• Residents spend much of their final year preparing for exams, instead of refining valuable clinical skills and gaining the confidence to practice independently

Competence By Design (CBD)

• Outcomes-based approach to the design, implementation, assessment and evaluation using an organizing framework of competencies

• Curriculum is organized around the outcomes expected of a resident and advancement is dependent on having achieved those expected outcomes

Competence By Design (CBD)

CBD organizes residency training into four developmental stages and clearly lays out markers for teaching and learning at each stage

Each learning experience, focuses on the identified outcomes for that stage

EPA

An Entrustable Professional Activity is a key task of a discipline(i.e. specialty or subspecialty) that an individual can be trustedto perform in a given health care context, once sufficient competence has been demonstrated.

EPA

• EPAs integrate multiple milestones from various CanMEDS Roles

• Some people think of an EPA as a basket or a bundle that holds numerous milestones

• RCEPAs allow for authentic, work-based assessment that is targeted at the daily tasks of a physician

Milestone

A milestone is an observable marker of an individual’s ability

EPA and Milestones

• EPA - basket or a funnel that aggregates or integrates a bunch of milestones.

Vision of Assessment:EPAs and Milestones

22

Entrustable Professional Activities (EPAs): for assessment• EPAs are the key tasks of

a discipline that a practitioner needs to be able to perform.

Milestones: for teaching• Milestones are the

abilitiesexpected of a health professional at a stage of development, based on the CanMEDS Roles.

CBD : Vision of assessment

EPAs and milestones provide staff and residents clear expectations of the skills and abilities they need at each stage of training

Help plan teaching and learning opportunities

Based on observations of resident’s ability to perform the EPA staff will assess if they can be entrusted to perform this EPA in the future

As a teacher, how will CBD affect me?

Each rotation will include a series of competency milestones that must be taught and learned

Residents will start to see us as someone who helps them develop their skills

Based on direct and indirect observation, short focused feedback will be given to learners and brief records of the observed performance will be completed

As a teacher, how will CBD affect me?

As a supervisor, you will observe residents as they perform an EPA and you will coach the residents to improve their performance

Refer to the milestones associated with an EPA to give feedback on specific elements of the task

As you become familiar you may ‘unpack’ an EPA (i.e. look at all of the milestones) to plan teaching or help a resident who is struggling to progress

Milestones and EPAs within Each Stage of Training

27

• Provide clear learning and teaching paths

• Remove need to teach or evaluate everything at once – focused on discrete clinical tasks

• Allow for regular, meaningful feedback, tracking and coaching

• Generate practical and demonstrable evaluation data

Assessment for learning

EPA : Providing neuraxial anesthesia for ASA 1 and 2 patients undergoing scheduled or emergency non-obstetrical surgery

Key Features

This EPA includes the technical skill of administering a spinal or epidural anesthetic, as well as the decision-making regarding proper patient selection, ensuring procedure appropriateness, identifying contraindications, full discussion with patient of risks/benefits/complications, and postoperative disposition including addressing postoperative VTE prophylaxis and resumption of other anticoagulant/antiplatelet medications.

EPA : Providing neuraxial anesthesia for ASA 1 and 2 patients undergoing scheduled or emergency non-obstetrical surgery

Assessment plan:

Part A: Patient management

Supervisor does assessment based on direct observation and/or case review and debrief

Use Form 1. Form collects information on:Type of surgical procedure: general surgery; gynecology; urology; orthopedic surgery; plastic surgery

Collect 5 observations of achievement At least three assessors

Part B: Procedure

Supervisor does assessment based on direct observation

Use Form 2. Form collects information on:Procedure: spinal/epidural

Collect 10 observations of achievement No more than 2 observations by the same assessor

EPA : Providing neuraxial anesthesia for ASA 1 and 2 patients undergoing scheduled or emergency non-obstetrical surgery

Relevant milestones (Part A)

1 F ME 1.4.1 Perform focused clinical assessments with recommendations that are well-documented (key points ; history, physical examination of spine, review investigations including coagulation profile; awareness of latest guidelines from professional bodies like ASRA)

2 TP ME 2.4.3 Establish and perform patient-centred perioperative anesthetic management plan for neuraxial anesthesia

3 F ME 2.4.4 Ensure that the patient and family are informed about the risks and benefits of each treatment option in the context of best evidence and guidelines

4 F ME 3.1.10 Determine the most appropriate anesthetic management in relation to patient’s condition and surgical procedure

5 F ME 3.3.12 Describe the indications, absolute and relative contraindications, risks and alternatives for neuraxial blocks

6 F COM 1.6.3 Assess a patient’s capacity to understand and appreciate the issues and risks, participate in decision-making and their capacity to give informed consent

7 F COM 5.1.2 Document clinical encounters to adequately convey clinical reasoning and the rationale for decisions to offer neuraxial blocks (over GA)

8 F COL 2.2.1 Identify communication barriers between anesthesiologists and surgeons

9 F COL 2.2.2 Communicate clearly and directly to promote understanding, manage differences and resolve conflicts

10 F S 2.3.1 Identify unsafe clinical situations involving learners and manage them appropriately

11 F P 2.2.2 Monitor institutional and clinical environments and respond to issues that can harm patients or the delivery of health care

EPA : Providing neuraxial anesthesia for ASA 1 and 2 patients undergoing scheduled or emergency non-obstetrical surgery

Relevant milestones (Part B)

1 F ME 1.3.7 Apply knowledge of the anatomy of the spine and spinal cord, including surface anatomy and deeper structures

2 TTD ME 3.1.10 Apply appropriate monitors correctly for the planned procedure

3 TTD ME 3.4.6 Demonstrate effective procedure preparation, including gathering required equipment and optimal positioning

4 F ME 3.4.2.1 Seek assistance as needed when unanticipated findings, difficulties, or changing clinical circumstances are encountered 5 F ME 3.4.7 Perform neuraxial anesthesia in a skillful, fluid, and safe manner

6 F COM 1.2.1 Optimize the physical environment for patient comfort, privacy, engagement, and safety

7 F COM 5.2.7 Appropriately document anesthetic care and technique in an accurate, complete, timely, and accessible manner

8 F S 1.1.2 Use technology to develop, record, monitor, revise, and report on learning

9 F S 1.1.3 Demonstrate a structured approach to monitoring progress of learning in the clinical setting

10 S F 2.3.1 Identify unsafe clinical situations involving learners and manage them appropriately

11 F P 2.2.6 Demonstrate a commitment to patient safety and quality improvement through adherence to institutional policies

12 F P 2.2.5 Monitor institutional and clinical environments and respond to issues that can harm patients or the delivery of health care

How will it work ?

• Royal College’s MAINPORT ePortfolio (an enhanced version of MAINPORT developed for CBD) to track and record learning and assessment via milestones and Entrustable Professional Activities (EPAs)

• However, many university partners already have systems in place that pose challenges to how data is shared, accessed and integrated

• The Royal College is in the process of finalizing a clickable prototype of MAINPORT ePortfolio, which will clarify the data entry processes and assessment workflows

How Will it Work? Learner

33

Learner

Supported by technology

Plan learning• Views Learning Plan EPAs &

milestones• Selects training activities • Requests observation

from Observer

Participates in learning• Works with Observer &

peers• Reflects on learning• Contributes evidence

of learning

Reflects on observations & learnings

Participates in observation encounter

How Will it Work? Observer

34

Observer

Supported by technology

Supports learning• Works with learners• Contributes narrative• Reviews evidence

Prepares for observation• Reviews and accepts

requests for observation• Reviews EPAs and

milestones

Participates in observation encounter

Evaluates EPA/milestones & records data via

technology

What Happens to the Observation Data?

35

Program Director

Supported by technology

Guides learning• Meets with learners regularly• Assigns EPAs to Observer• Reviews learner progress using technologyReviews program status• Reviews program data and investigates issues • Prepares for Competence Committee meetings

Competence Committee

Assess overall achievement of EPAs & milestones• Review learner EPAs/milestones• Approve changes to learner status

• Promote learner to next stage of training• Request Royal College certification• Modify Learner program plan• Monitory overall progress of learner

• Provides group decision process rather than single summative assessment

Supported by technology

Competence Committee

• Critical component of CBD

• Review and make decisions related to the progress of residents and ensure they achieve the national standards established by each Royal College discipline

Competence Committee -Composition

• Reports to the Residency Training Committee

• Chaired by a faculty member who is not the program director

• Minimum 3 members

• Composed of training committee members and clinical supervisors

Competence Committee -Responsibility

• Monitor the progress of each resident in demonstrating achievement of the EPAs or independent milestones within each stage

• Synthesizing the results from multiple assessments and observations to make decisions related to:

I. promotion of residents to the next stage of trainingII. review and approve individual learning plans to address areas

for improvementIII. determine readiness to challenge the Royal College

examinationsIV. determine readiness to enter independent practice V. determining that a trainee is failing to progress within the

program

Competence Committee -Responsibility

• Monitoring the outcome of learning or improvement plan established for a resident

• Maintaining confidentiality and promoting trust

Challenges

• Determining the EPAs and Milestones for each stage

• Number of observations for each EPA• Daily assessment ?• Electives / rotations in other programs• Data - Mainport or local• Training and support for faculty/training

committee

• Assessment is a key component of CBD, with a focus on methodic work-based assessment

• Program Directors, Observers and Learners share responsibility for learning in residency

• Opportunity to improve the way we teach and assess trainees