Simulation / High Stakes Exams - Rocky Mountain · PDF fileEducation and High Stakes Exams Neil Gibson, MD, MSc, ... Remains useful for flight training. ... Israeli Board Examination

Use of Simulation in Medical Education and High Stakes Exams Neil Gibson, MD, MSc, FRCPC,FACP

Division of General Internal Medicine

University of Alberta

Irene Ma, MD, MSc, FRCPC, FACP

Division of General Internal Medicine

University of Calgary

Conflict of Interest

None to declare

Objectives

By the end of the session, the participants will be able to:

1) Understand the uses of simulation in teaching and assessment

2) Understand the assessment requirements for high-stakes examination

What is Simulation?

An instructional medium used for education, assessment, and research

Reproduction of certain characteristics of clinical reality

Several modalities – Ogden PE et al. Am J Med 2007

Partial-task trainers (commercial, homemade, animal models)

Full body task trainers

High fidelity mannequins

Standardized patients

Virtual reality

Computer software

Chiniara G et al. Med Teach 2012

A Day in Ottawa at the College….

…..

Outline

1. Simulation is useful and is here to stay

2. For simulation to remain useful, research agenda needs to adapt

3. The use of simulation in assessment opens up a realm of assessment opportunities

Simulation is useful and is here to stay

1. What is the evidence?

Copyright © 2012 American Medical

Association. All rights reserved.

Technology-Enhanced Simulation for Health Professions Education: A Systematic Review and Meta-analysis

JAMA. 2011;306(9):978-988 609 studies (137 RCTs, 67 two-group design, 405 pretest-postest)









12 Ma IWY et al. Acad Med Sep 2011; 86(9):1137-47.

In the case of Central Venous Catheterization, 20 studies (9 pretest posttest; 9 two-group; 2 RCTS)






1. The evidence is increasingly robust for its educational utility


2. Simulation makes sense. Analogy from flight simulation

http://en.wikipedia.org/wiki/File:Antoinette_sim_1909.jpg


2. Simulation makes sense. Remains useful for flight training.

Photo Credit: NASA. http://gimp-savvy.com/cgi-bin/img.cgi?ailsorHdFtW9c861266


3. Simulation has already been here for some time

Asmund Laerdal with Resusci-Anne, in about 1970.

Cooper J B , Taqueti V R Qual Saf Health Care 2004;13:i11-i18


2. Simulation has already been here for some time

Michael Gordon with Harvey in the early 1970s



2. Simulation has already been here for some time, even “high-fidelity” simulators


Dr Stephen Abrahamson (seated) and Dr Judson Denson with Sim One in the late 1960s

Nat Geo Nov 1970




How to Kill Off a Good CRM Program

1. Not integrating CRM into LOFT, PT, and other operational training

2. Failing to recognize the unique needs of your own airlines culture

3. Allowing the CRM Zealots to run the show

4. Bypassing research and data gathering steps

5. Ignoring the checking and standards pilots

6. Having lots of diagrams, boxes, and acronyms

7. Making the program a one shot affair

8. Using pop psychology and psycho babble

9. Turning CRM into a therapy session

10. Redefining the “C” to mean charismatic

Taggart WR. THE CRM Advocate 1993; 93(1):11-2

simulation [ti] AND medical education AND year [dp] Search: PubMed, November 9, 2012

Much research has been done on medical simulation

For simulation to remain useful, research agenda needs to adapt We need to stop asking:

“Does it work?”

“If you teach them, do they learn?”

Instead, we need to ask questions that help advance educational theory, curriculum development and implementation

1. Simulation is just a tool

2. Education is complex


Simulation is just one tool


Education is complex

For simulation to remain useful, research agenda needs to adapt Instead, ask educationally useful questions:

How best to teach using simulation?

How do learners learn?

When do they learn?

Why do some not learn?

When to use to simulation?

For simulation to remain useful, research agenda needs to adapt Instead, ask educationally useful questions:

How best to teach using simulation?

How do learners learn?

When do they learn?

Why do some not learn?

When to use to simulation?


Canadian Network for Simulation in Healthcare (CNSH)


Organophosphate Poisoning



Septic shock



Septic shock

Eosinophilic Fasciitis



Septic shock

UTI, pneumonia




Septic shock

UTI, pneumonia



For simulation to remain useful, research agenda needs to adapt Simulation is NOT the answer for every educational activity we have




























Use of simulation in assessment

1. Assesses performance at a higher level

Triangle of knowledge / clinical skills competence / performance Miller 1980

Use of simulation in assessment

1. Assesses performance at a higher level

Triangle of knowledge / clinical skills competence / performance Miller 1980

Does

Shows how

Knows how

Knows

SPs as a form of simulation has been widely implemented Objective structured clinical examination

Medical school

LMCC Part II

Royal College of Physicians and Surgeons of Canada

USMLE Step 2 Clinical Skills examination (since 2004)

National Board of Osteopathic Medical Examiners (NBOME) Comprehensive Osteopathic Medical Licensing Examination Level 2-Performance Evaluation (COMLEX-USA Level 2-PE).

Use of High-Fidelity Simulation in High Stakes Examinations Royal College of Physicians and Surgeons of Canada – Internal Medicine

Combined auscultation videos with SP, 2003 – Hatala R et al. Acad Med 2005

Harvey® cardiopulmonary simulator

Israeli Board Examination in Anesthesiology

ER Trauma management

Resuscitation – ACLS

OR - hypertension post induction

Adjustment of ventilation in response to ABG results

Framework for Simulation-based Assessment

1. Valid

2. Reliable

3. Educational Impact

4. Acceptability

5. Cost

Van der Vleuten CPM. Advances in Health Sciences Education. 1996

Validity

“Validity refers to the degree to which evidence and theory support the interpretations of test scores entailed by the proposed uses of the tests.” American Psychological Association. Standards for Educational and Psychological Testing. 1999

Traditional Trinitarian view of validity: Guion RM 1980

1. Criterion validity

2. Content validity

3. Construct validity

Replaced by umbrella term: Construct validity

Encompasses all hypothesis testing in the pursuit of validity

In the absence of a gold standard, for the “new” exam in question to be valid… Exam should be relevant and representative of the tasks in question

Those who are more competent should score higher than those who are less competent

Those who score higher on the exam should in practice make more accurate diagnoses/provide better treatment plans than those who score lower

Those who score higher on the exam should be those who did well during residency

Evidence of Construct Validity

Forrest FC et al. Br J Anaesth 2002;88:338-44

Devitt JH et al. Anesth Analg 1998;86:1160-4


1. Valid

2. Reliable

internal reliability, inter-rater reliability, intra-rater reliability


4. Acceptability

5. Cost


Optimizing Reliability

Standardize the examination

Examine on multiple occasions (how frequent?)

Sample multiple domains (knowledge, skills, attitude)

Use multiple methods (simulation, MCQ, track clinical outcomes, peer assessment,

3600 evaluation)

Trained raters, blinded

☐ Some or all of the above?

Evidence of Inter-rater reliability

Review of 28 performances by 3-5 raters

Inter-rater reliability ICC 0.79-0.85

96% agreement in pass/fail decisions

Weller JM et al. Br J Anaesth 2003;90:43-7


1. Valid

2. Reliable

internal reliability, inter-rater reliability, intra-rater reliability


4. Acceptability

5. Cost





4. But for simulation in assessment to be useful, research agenda needs to adapt

Conclusion

Simulation for education and assessment seems new and exciting, although it actually has been here for some time.

We argue that that

1. Simulation is useful (and there is robust evidence to suggest that it is so) and is here to stay.

2. For simulation to remain useful, research agenda needs to adapt in order to advance learning theories


4. Research agenda needs to establish its validity and reliability

Documents

Simulation / High Stakes Exams - Rocky Mountain · PDF fileEducation and High Stakes Exams Neil Gibson, MD, MSc, ... Remains useful for flight training. ... Israeli Board Examination