9/30/19

1

Assessing Simulation-Based Learning Across the Curriculum: A Call to Action

Education Leadership ConferenceOctober 19, 2019

Kathy Lee Bishop, PT, DPT, CCSSharon L. Gorman, PT, DPTSc, GCSEllen Wruble, PT, DScPT, MS, CWS, FACCWS Molly Hickey, PT, DPTNicki Silberman, PT, DPT, PhDEric Stewart, PT, DPTErin Thomas, PT, DPT

Disclosures

Sharon Gorman and Nicki Silberman are members of the ACAPT Simulation in Physical Therapy Education Task Force

We have no conflicts of interest

Objectives

● Integrate sound pedagogy for sequencing and scaffolding simulation-based learning experiences (SBLEs) within curricula to maximize impact and value on student clinical reasoning skills and professional behaviors.

● Examine opportunities within curricula to integrate SBLEs across content areas and patient populations.

● Identify data points that will enable assessment of SBLEs outcomes both within curricula and post-graduation.

● Evaluate the utility of existing assessment tools in measuring identified outcomes.

● Explore collaborations that can yield large outcome data sets wherein optimal SBLE design, frequency, and outcomes can be analyzed relative to resource expenditures.

9/30/19

2

Interactive, implementation-based session

What is the appropriate timing, placement, and frequency of Simulation-Based Learning

Experiences within a curriculum?

Assumptions

1. We recognize the potential and realized benefit of simulation-based learning.2. We acknowledge that simulation is beneficial in learning roles and

responsibilities.3. We agree that both low and high fidelity simulation have value.

9/30/19

3

INACSL Standards of Best Practice: Simulation

Definitions Simulation-Based Learning Experience (SBLE) - Array of structured activities that represent actual or potential situations in education and practice

High Fidelity - In healthcare simulation, refers to simulation experiences that are extremely realistic and provide a high level of interactivity and realism for the learner (human, manikin, task trainer, or virtual reality)

Low Fidelity - Not needing to be controlled or programmed externally for the learner to participate (case studies, role playing, task trainers)

Environmental/Engineering Fidelity - Degree to which simulated environment replicates reality and appearance of real environment

Psychological fidelity- Level of realism associated with a particular simulation activity

www.ssih.org

Definitions Fictional Contract - Concept which implies that an engagement in simulation contract between instructor and learner: each has a part to make the simulation worthwhile

Task Training - Model that represents a part or region of the human body vs. Integration/Clinical Reasonings

Standardized Patients - Person who has been carefully coached to simulate an actual patient so accurately that the simulation cannot be detected by a skilled clinician

Simulated Patients - Individual trained to portray a real patient in order to simulate a set of symptoms or problems used for healthcare education, evaluation, and research

Debrief - Formal, collaborative, reflective process within the simulation learning activity

www.ssih.org

9/30/19

4

Current Simulation-Based Learning Experiences

Variations in Practice

Focus on Acute Care

PT OnlyInterprofessional Only

Self-Efficacy

Study Satisfaction

CALL TO ACTION

Dosing (timing, placement, frequency)

Assessment and OutcomesIntentional Scaffolding/Curricular Thread

Top of License Practice

Demonstrate:

* Adaptive Learning

* Competence* Accountability

* Evolved Clinical Reasoning

EXCELLENCE

Dosage within the Curriculum

Review of Literature - Dosing

What is the frequency and duration of SBLEs within your curriculum?

9/30/19

5

Mori B, Carnahan H, Herold J. Use of simulation learning experiences in physical therapy entry-to-practice curricula. A systematic review. Physiotherapy Canada 2015; 67(2);194-202.

Purpose - Can SBLEs be meaningfully incorporated in PT entry-to-practice curricula to positively affect learner’s knowledge, skills, and attitudes? Systematic review

● MEDLINE, CINAHL, Embase Classic+Emase, Web of Science (~1946-2013)○ Simulation activities for specific skills○ Interactive computer games or programmed simulation learning activities○ Simulation for managing a case presentation○ Using simulation to represent clinical education

Challenges● Variable MERSQI scores across studies ● Outcome tools (Assessment of Physiotherapy Performance, confidence, anxiety)● Short lived carry over of simulation (1 week), length of simulation dosage● Cost $35,000 - $50,000

○ Simulation space/equipment/faculty○ Faculty time to develop cases, manage SBLE, debriefing

● Basing SBLE on educational theory○ Kolb stages (concrete, reflective, abstract conceptualization, active experimental)○ Blocked practice versus random practice

Melling M. et al. Simulation Experiences in Canadian Physiotherapy Programmes: A Description of Current Practices. Physiotherapy Canada 2018; 70(3) 262-271.

Purpose - Explored the definitions of simulation, its current use, and perceived

benefits and barriers in Canadian entry-to-practice physiotherapy programs.

● Qualitative, descriptive study that examined 8 PT programs in Canada using

a semi-structured interview approach.

Themes:

1. Variability in the definition of fidelity - should include a component of

complexity

2. Variability in simulation use

3. Benefits and barriers to the use of simulation - increased authenticity and

stimulated students to develop clinical reasoning and integrate lecture

material into clinical practice

Dosage - Panel Survey

Survey Questions Hours or Sessions N

How often do your students participate in high fidelity simulation in the DPT curriculum? Consider each occurrence as ‘one’.

Year 1 (1 - 8 times)Year 2 (2** - > 10 times)Year 3 (2** - 10 times)

665

How many hours do your students participate in high fidelity simulation in the DPT curriculum? Consider total time in simulation – Prebrief/scenario/debrief/reflection.

Year 1 (3 - > 10 hours)Year 2 (4 - > 10** hours)Year 3 (4 - > 10 hours)

665

How often do your students participate in high stakes-graded simulation in the DPT curriculum? Consider each occurrence as ‘one’

Year 1 (0** - 6 times)Year 2 (0** - 8 times)Year 3 (0 - 4 times)

665

How often do your students participate in skill building (non-graded) simulated learning experiences in the DPT curriculum? Consider each occurrence as ‘one’

Year 1 (1 - > 10 times)Year 2 (2** - > 10 times)Year 3 (0 - >10 times)

665

How often do your students participate in skill assessment (non-graded) simulated learning experiences in the DPT curriculum?

Year 1 (0 - 8 times)Year 2 (0 - 8 times)Year 3 (0 - 10 times)

445

** = Majority

9/30/19

6

Simulation-Based Learning: Dosage

Adapted from INACSL Standards of Best Practice Clinical Simulation in Nursing (2016) 12, S5-S12

Assessment & Outcomes

Standard Process for AssessmentCriterion ● Determine Method of Participation Evaluation

● Directed by objectives/outcomes/intent● Guided by type: formative, summative, high-stakes

Formative ● Progress toward achieving outcomes; Provide ongoing feedback● Support clinical competencies● Identify/close gaps: knowledge/skills● Assess readiness for real-world

Summative ● Discrete point in time; Safe learning environment (includes orientation to environment/equipment)

● Level of fidelity necessary to achieve outcomes; Standardized format and scoring methods● Theoretically based method passing/cut scores ● Video recording (review - multiple trained evaluators)● Valid/Reliable instrument***; Provide rater training; interrater reliability● Inform participants of evaluation process; Summative feedback

High-Stakes ● At end of learning process; may be used to assess gaps in knowledge/safety issues● Based on specific objectives● Explain consequences; predetermined parameters for terminating scenario● Pilot test scenario; trained/non-biased objective raters/evaluators● Use comprehensive tool (checklist/rubric clearly outlines desirable/undesirable behaviors)● AFTER participant has been exposed to multiple SBLE including evaluations

9/30/19

7

Simulation-Based Learning: Assessment ToolsInterprofessional Tools

TeamSTEPPS Team Assessment

43-Item questionnaire; individual team members' perceptions of team foundation, functioning, performance, skills, leadership, climate/atmosphere, and identity

ICCAS 20 item Likert-type self-assessment IP competencies (pre/post)

PACT contains 5 instruments: two are self-report, pre-post assessments; 3 observational rating tools developed for raters with different levels of experience. Extensive training for raters to use

W(e)Learn IP Program Assessment Scale

30 item Likert-type; evaluation of learning activity including faculty facilitators, content, organization (post)

SPICE 2 10 Item Likert-type; Measures attitudes IP healthcare teams (medical students) (pre/post)

IPEC Competency Self-Assessment

42 core competency statements; self-efficacy related to collaborative practice at the healthcare degree program level through individual student self-assessment.

• Extremely limited number published or presented• Unvalidated versions exist, are being used• Low hanging fruit for research?

Valid?• Extremely limited data• Terminal clinical experience assessment difficulties• Lack of reproducibility of existing toolsReliable?• Many attitude-specific• Skill and behavior-level assessments lacking• Integration into clinical practice, lack performance-based measures

Applicable?• IPE versions may not translate in whole or in part• SP ratings capture some aspects only• Limited PT-specific, setting specific (ACCS) or population specific (TASPE)Specific?

Challenges in assessing SBLEs in PT

Intentional Scaffolding/Curricular

Threading

9/30/19

8

Intentional Scaffolding and Curricular Threading

Pedagogy

Evidence of scaffolded learning in education, nursing, and medicine

SBLE

Evidence for efficacy of individual SBLE

Best Education Practices

Goal is to understand evidence for purposeful curricular threading of simulation in PT education

Learning theories that support SBLE

Repetitive Practice and Curricular Integration are Essential to Effective Simulation-Based Learning

Repetitive Practice:

● Rehearsal/repetition: Most effective for nondeclarative memory formation● Elaboration: More effective for declarative memory formation

○ Includes debriefing

Scaffolding:

● Build knowledge upon a foundation to achieve a higher level● Repair, deconstruct, or break down knowledge● Clean knowledge by clarifying concepts

Includes sequential support and feedback

9/30/19

9

Practice-Based Education

Learning for practice through practice

Grounded in situated learning theory, practice-based education provides opportunity for learners to have authentic experiences

Learning for practice must be done early to help provide the scaffolding for continued contextual learning

Simulated-based learning is one of the identified practice-based education pedagogies in our pursuit of excellence in PT education.

Creating Adaptive Learners

● Educators should have an intentional approach to fostering development of adaptive learners with a focus on excellence and high expectations

● Adaptive learning occurs in situations that mirror the complexity and uncertainty of practice where students must learn problem solving and clinical reasoning

● Adaptive learners will:○ Engage in continuous learning○ Include situation and self-awareness and assessment○ Embrace feedback○ Adapt and function in uncertain environments○ Reflect upon and learn from new experiences○ Incorporate new learning into practice

● Simulations can be designed for uncertainty, demanding the learner to use problem solving and clinical reasoning skills

How do we apply this evidence?Promoting Excellence in a PT curriculum

9/30/19

10

Interprofessional Focus

Physical Therapy

Focus

Simple Complex

Range of Best Practices

LateMid

dle

Early

Simulation Activities

Num

ber/

spec

ificit

y of B

ody S

yste

ms I

nvol

ved

Objectives

● Integrate sound pedagogy for sequencing and scaffolding simulation-based learning experiences (SBLEs) within curricula to maximize impact and value on student clinical reasoning skills and professional behaviors.

● Examine opportunities within curricula to integrate SBLEs across content areas and patient populations.

● Identify data points that will enable assessment of SBLEs outcomes both within curricula and post-graduation.

● Evaluate the utility of existing assessment tools in measuring identified outcomes.

● Explore collaborations that can yield large outcome data sets wherein optimal SBLE design, frequency, and outcomes can be analyzed relative to resource expenditures.

Call to Action: Is the juice worth the squeeze?

High or low fidelity???

I can’t do it!Can I debrief effectively?

We need to take risks and innovate in order

to excel!

9/30/19

11

Creating a Body of Evidence

Multi-Center Data Collection

Simulation Based Learning Experiences

Collaborative Sharing

Excellence

Assessment

Dosa

ge

Scaffolding and Curricular Integration

Questions

Kathy Lee Bishop - klbisho@emory.edu

Sharon Gorman - sgorman@samuelmerritt.edu

Ellen Hakim-Wruble - ewruble@udel.edu

Molly Hickey - mahickey@utica.edu

Nicki Silberman- ns592@hunter.cuny.edu

Eric Stewart - esstewar@udel.edu

Erin Thomas - erin.thomas@osumc.edu

ReferencesBlackstock F, Watson K,Morris N, Jones A, Wright A, McMeeken J, Rivett D, et al. Simulation can contribute a part of cardiorespiratory physiotherapy clinical education: Two randomized trials. Sim in Healthcare. 2013;8(1):32-42.

Greenfield BH, Jensen GM, Delany CM, Mostrom E, Knab M, Jampel A. Power and promise of narrative for advancing physical therapist education and practice. Phys Ther. 2015;95(6)924–933.https://doi.org/10.2522/ptj.20140085.

INACSL Standards Committee (2016, December). INACSL standards of best practice: SimulationSM Simulation design. Clinical Simulation in Nursing, 12(S), S5-S12. http://dx.doi.org/10.1016/jecns.2016.09.005.

Higgs J., Jensen, G., Loftus S., Christensen N. Clinical reasoning in the health professions. Edinburgh: Elsevier; 2019: pg. 448-449.

Jensen, G., Mostrom, E. Hack, L., Nordstrom, T. and Gwyer, J.: Educating Physical Therapists, Slack Incorporated, 2018

Levac D, Glegg SMN, Camden C, Rivard LM, Missiuna C. Best practice recommendations for the development, implementation, and evaluation of online knowledge translation resources in rehabilitation. Phys Ther. 2015:95(4) 648–662. https://doi.org/10.2522/ptj.20130500.

Levett-Jones T, McCoy M, Lapkin S, Noble D, Hoffman K, Dempsey J, Arthur C, Roche J. The development and psychometric testing of the Satisfaction with Simulation Experience Scale. Nurse Educ Today. 2011, doi:10.1016/j.nedt.2011.01.004.

Mai JA, Pilcher RL, Frommelt-Kuhle M. Fostering interprofessional collaboration and critical thinking between nursing and physical therapy students using high-fidelity simulation. J Interprof Edu Prac.2018:10(Mar);37-40. https://doi.org/10.1016/j.xjep.2017.11.002.

9/30/19

12

References (continued)Melling M. Duranai M. Pellow B. et al. Simulation Experiences in Canadian Physiotherapy Programmes: A Description of Current Practices. Physiotherapy Canada 2018; 70(3) 262-271.

Mori B, Carnahan H, Herold J. Use of simulation learning experiences in physical therapy entry-to-practice curricula. A systematic review. Physiotherapy Canada 2015; 67(2);194-202.

Nithman RW, Spiegel JJ, Lorello D. Effect of high-fidelity ICU simulation on a physical therapy student's perceived readiness for clinical education. J Acute Care Phys Ther. 2016;7(1):16–24. doi:10.1097/JAT.0000000000000022.

Ohtake PJ, Lazarus M, Schillo R, Rosen M. Simulation experience enhances physical therapist student confidence in managing a patient in the critical care environment. Phys Ther. 2013:93(2);216–228. https://doi.org/10.2522/ptj.20110463.

Pawlik AJ, Kress JP. Issues affecting the delivery of physical therapy services for individuals with critical illness. Phys Ther. 2013:93(2); 256–265. https://doi.org/10.2522/ptj.20110445.

Plack, M., Driscoll, M.Teaching and Learning in Physical Therapy, From Classroom to Clinic. 2nd Ed. Slack Incorporated, 2017

Pritchard SA, Blackstock FC, Nestel D, Keating JL. Simulated patients in physical therapy education: systematic review and meta-analysis. Phys Ther. 2016:96(9);1342–1353.https://doi.org/10.2522/ptj.20150500.

Silberman NJ, Litwin B, Panzarella KJ, Fernandez-Fernandez A. High fidelity human simulation improves physical therapist student self-efficacy for acute care clinical practice. J Phys Ther Edu.2016:30(1);14–24.

Society for Simuation in Healthcare. SSH Resources. Dictionary. www.SSIH.org. 2019. Accessed September 30, 2019.