Games to Improve Clinical Practice and Healthcare Administration

Gaming for Organizational, Clinical and Patient Risk Management

Comprehensive Unit-based Safety Program (CUSP) for Enhanced Recovery Protocol for Surgery

“Simu-Leader” (a gaming application) YouTube link

What were we trying to do?Johns Hopkins Medicine implemented the Comprehensive Unit-based Safety Program (CUSP) and has demonstrated significant outcomes. Armstrong Institute conducted a brief table top game in 2012 to facilitate learning and reinforce best practices amongst hospital leadership. Based on those results a grant was requested to build an automated application to achieve the same results on a larger scale.

Two developed are:

1. Patient Risk: Application geared to improve the safety and quality of acute patient care , strongly impacted by all levels of organizational leadership. Game provides evidence-based methods for developing the knowledge and skills necessary for effective safety leadership

2. ERAS/CUSP: Gamification and gaming interfaces measurably improve patient outcomes by implementing enhanced recovery after surgery (ERAS) practices in hospitals through applying the Comprehensive Unit-based Safety Program (CUSP).

Overall Study Aims§ Specific Aim #1: Evaluate the comparative effectiveness of simulation and traditional methods of management instruction on multi-level learning outcomes.

Ø H1a-c: Compared to traditional instruction, simulation will produce significantly higher individual level a) learner reactions, b) learning, and c) behavior change.

§ Specific Aim #2: Compare the impact of ad hoc and intact management team configurations on multi-level learning outcomes and the quality of strategy generation in organizational leadership simulations.

Ø H2a-c: There will be a significant difference due to the team configuration manipulation for individual level a) learner reactions, b) learning, and c) behavior change.

Ø H3a-c: There will be a significant difference due to the team configuration manipulation for team level creativity of ideas generated in terms of a) novelty, b) usefulness, and c) feasibility of ideas.

§ Specific Aim #3: Evaluate the impact of a mindfulness pre-training intervention on multi-level learning outcomes and management team performance. Pre-training and pre-performance interventions are effective, efficient ways to enhance learning and performance outcomes.

Ø H4a-c: Compared to no mindfulness training, mindfulness training will increase the efficacy of the organizational leadership simulation by producing significantly higher individual level a) learner reactions, b) learning, and c) behavior change.

Ø H5a-c: Compared to no mindfulness training, mindfulness training will produce significantly higher team level creativity of ideas generated in terms of a) novelty, b) usefulness, and c) feasibility of ideas.

ERAS/CUSP Overall Aims

Task 1: Develop a public domain version of a CUSP for ERAS educational kit for national distribution.

Ø Johns Hopkins APL to leverage software engineering expertise as well as existing applications to address the requirements. APL will design, develop, test and deliver the software application

Task 5: Assess the Adoption of CUSP for ERAS and Evaluate the Effectiveness of the Intervention in the Participating Hospitals.

Ø Johns Hopkins APL will develop software and provide a credible and systematic data collection schema allowing for data collection. The software application will provide a robust back-end database structure for in-stride assessments, metadata visualization and distributed access.

Ø Subtask 5.1: Design and develop data collection protocols and analysis plan. Support the data collection protocol provided by AI collaborators. Make software application available to participating hospitals to voluntarily input and share data.

Key Metrics, Measures & Performance

How$Represented?$

How$Collected?$

Expected$Format$

Weigh:ng$

Transparency*

Reliability*

Leadership*

Strategic*Plans*

Impact*

8*8*8*other*metrics*

Performance in the application was measured in a variety of ways:• Individual progress + translation to practice• Team effectiveness within scenario(s)• Between team comparisons conditioned on mindfulness training

Armstrong Institute will define each metric for JHU/APL to include

Teams Team v TeamMeasures

Cond v CondMeasures

Candidate metrics for response/resolution:• Rate of re-admission• Patient falls• Acquired infections• Preventable VTE

Detailed design connecting data needs to GUI

Software Design & Computer Science

Technical Collaboration

As elements of the application are developed in parallel by Johns Hopkins Medicine, Armstrong Institute and JHU/APL, we anticipated evolving requirements. Therefore, the Agile software development methodology best suited the development team.

“Agile software development is a group of software development methods in which requirements and solutions evolve through collaboration between self-organizing, cross-functional teams. It promotes adaptive planning, evolutionary development, early delivery, continuous improvement, and encourages rapid and flexible response to change.”

The web application is updated frequently during the development process in order to facilitate frequent feedback from collaboration discussions.

Participant Rhythm1

Player Feedback

Software Architecture

Client Layer

Amazon Cloud Server

Windows Server 2012

MS IIS Webserver

Production Web App

MS SQL Server Database

Web App

Game ManagerGame Session

Game Session…

Game Session

Security Services

Game Engine

Text Chat

Player Interface

Principal Inv. I/F

Performance Measurement Reporting

Web Page Access

Spreadsheets

PDFs Database

Designed Reports

Game Navigation ConceptConcept discussions with Armstrong Institute produced a hub-spoke structure which migrated to a more functional physical representation of users’ expected normal work environment.

Shown here is early concept art used to scope the application and conduct early usability tests.

User Interface via Web Application (prototype)

Web Portal on Amazon Web Services

Software Demonstration

Human Factors & Usability

Human Factors Tasking

In Progress: CompletedRound 1, N=9 Round 2, N=9

Completed N=4

Human Factors TaskingConcept Vetting - Preparation

Human Factors TaskingAlpha Testing & Usability Testing

Background Questionnaire

Proctored Activity Scenarios

Usability Questionnaire

Retrospective Interview
















What did the participants find important during the session?What problems did they encounter? (Flanagan, 1954).

Their perception of the session and their self-reflection may offer insight into how the simulation was used and how it could be improved (Rosson & Carroll, 2002).

Questions included:• Are the instructions clear?• Do you feel you know what to do/where to navigate?• What would make this easier for you?• Can you recall any activities being particularly successful? Unsuccessful?• What did you like the most? Dislike the most?

Human Factors TaskingAlpha Testing & Usability Testing – Results (3/3)Post-Session Usability Questionnaire

• Round 1 – July (purple), n=9. Round 2 – November (blue), n=9. • Four participants completed both rounds.

Metrics and Verification

Scoring Algorithm

§ Outcome = Pos. tactics + Neg. tacticsØ Pos. Behaviors = (6RCpos + 3TRpos + BCpos) * ((TL +1) + BS)Ø Neg. Behaviors = (-2.2RCneg + -3TRneg + -1.6BCneg) * (TL +1)

§ Term definitions/MOPS/Tactic DomainsØ RC = Responsibility, role clarity, and feedback tacticsØ TR = Time and resources tacticsØ BC = Building capacity tacticsØ TL = Transformational leadership tacticsØ BS = Boundary Spanning tactics

§ Term values will be based on proportion of (+) and (-) actions selected during game (TL & BS will be a count of actions)

§ Initial weights determined through adapted ProMES process

How much does each indicator contribute to effective organizational performance?

Three primary types of contingency shapes:

-100-80-60-40-20020406080100

80 85 90 95 100PERCENT PASSING INSP.

EFFECTIVENESS

Diminishing Returns

-100-80-60-40-20020406080100

80 90 100 110 120 130TRAINING MET

EFFECTIVENESS

Linear Critical Mass

Images: (Pritchard, 2009)

Selecting Different Tactic Sets

5 7 9 11 13 15 17 19 21 23 25 27

050

150

250

# of Tactics Selected

SSI

5 7 9 11 13 15 17 19 21 23 25 27

0100

200

300


RA

5 7 9 11 13 15 17 19 21 23 25 27

0100

200

300


LOS

5 7 9 11 13 15 17 19 21 23 25 27

050

150

250

350


Sat

Surgical Site Infections Patient Readmissions

Length of Stay Patient Satisfaction

Impact of “positive” and “negative” tactics

positive negative overall

-50

050

100

150

200

250

300

Surgical Infections


0100

200

300

Readmission


0100

200

300

Length of Stay


050

100

150

200

250

300

350

Patient Satisfaction

Risks for our small budget game§ On-demand help is not available during game play increasing the level of software reliability and testing

§ Technology barriers that exist at the player location, equipment or access

§ Depth and/or breadth of scenario data required to represent a realistic decision environment

§ Voice communication implementation for system users

§ Adjudication formulation represents ‘real’ hospital outcomes

§ No timing mechanism included to compel player action

Summary

Major Take-AwaysØ Everyone wants gaming and it is ‘accepted’Ø Professional game success is measured differentlyØ Represent “real life” with plausible actions and effectsØ Technical skills and experimentation over graphicsØ Experimental testing focuses on impact over entertainment

The game and initial results perform well against the study hypotheses. Participants from around the world have registered and data collection is on-going. We anticipate

significant engagement and persistent effects for individuals and organizations participating.

Scott Simpkins;; [email protected];; 240-228-3718Principal Staff, Principal InvestigatorSupervisor, Health Concepts & Systems Analysis

Health Systems Engineering Group

Patient Risk Management Summary

The safety and quality of acute patient care is strongly impacted by all levels of organizational leadership, yet few evidence-based methods for developing the knowledge and skills necessary for effective safety leadership currently exist. Simulation clearly improves technical and non-technical competencies of healthcare workers, as well as organizational learning and continuous improvement. However, simulation has not yet been broadly applied to the development of leadership for patient safety.

This work assembled a multi-disciplinary team with unique and diverse expertise (simulation, training, gaming, engineering, patient safety leadership, business and management, social science of creativity, human factors and organizational psychology), and capitalizes on existing projects and strong relationships with state, national and international healthcare organizations to evaluate a game for patient safety leader development.

ERAS/CUSP SummaryTo measurably improve patient outcomes by increasing the implementation of enhanced recovery after surgery (ERAS) practices in hospitals, through the use of an adaptation of the Comprehensive Unit-based Safety Program (CUSP). CUSP for ERAS shall be developed as an integrated combination of clinical and cultural interventions. § Develop a consensus description of the essentials of ERAS, develop consensus recommendations for implementation of ERAS bundles for various surgeries and determine accurate measures of effectiveness of CUSP for ERAS.

§ Develop an adaptation of the CUSP protocol and materials to be applied to ERAS.

§ Recruit hospitals to implement ERAS in a phased approach, beginning with 100 hospitals in the base period, and expanding to include approximately 200-250 hospitals during each option period, in order to cumulatively include 750 or more hospitals. Participating hospitals should be from all States, Puerto Rico and the District of Columbia.

Education

Games to Improve Clinical Practice and Healthcare Administration