Intelligent Dashboard for Augmented Reality basedIncident Command Response Co-ordination

Mark Vassell, Olivia Apperson, Prasad Calyam, John Gillis, Salman AhmadUniversity of Missouri-Columbia, USA

{mdvy96, oamr6}@mail.missouri.edu; {calyamp, gillisj}@missouri.edu; {ahmadsa}@health.missouri.edu

Abstract—Communication in a mass casualty disaster sceneis limited and difficult for medical personnel in the absenceof necessary communication infrastructure and collaborationtechnologies. It leads to misdirected and delayed triage of scene-wide critically injured patients, especially when there is a largevolume of patients needing diverse care levels. In this paper, wedescribe a novel Intelligent Dashboard that provides augmentedreality benefits with minimal human communication throughintegration of a standardized Incident Command System (ICS)with Internet of Things (IoT) such as heads-up displays, virtualbeacons, QR-code cards, and wireless mesh network elements. Weconduct a usability evaluation with a two-incident ‘Task Force 1Rescue’ simulation to show the ease-of-use and effectiveness ofour Intelligent Dashboard. Our work lays the foundation for next-generation ICS for an Incident Commander to deploy resourcesat the right locations more efficiently, and reduce triage time,mitigate over/under triage, and thus increase triage care levels.

I. INTRODUCTION

Communication in a mass casualty disaster scene is limitedand difficult in the absence of necessary communication infras-tructure and collaboration technologies. It leads to misdirectedover-triage/under-triage and even delayed triage of scene-widecritically injured patients, especially when there is a largevolume of patients needing diverse care levels. Such a situationcan be expensive for hospitals who have limited medical staffand supplies, as well as can potentially lead to loss of lives.

Today, there is lack of integrated technology platformsthat allow a Medical Director or Incident Commander(s) tocollaboratively sense disaster scene status events. Suitabletechnologies for efficient co-ordination can help them to strate-gically delegate triage responsibility to first responders anddirect them remotely using audiovisual communication. Theycan also help with emergency response in disaster scenariosthat requires all first responder groups such as Fire, Police,EMS, and health services to co-ordinate closely rather thanact as isolated units [1]. Further, Incident Commanders couldbe equipped to use mobile cloud technologies and wired/wire-less network diversity to connect to external databases (e.g.,blood bank, patient data, medical resource pools that updateavailability of beds, ambulances, expert opinion, and so on)to have adequate situational awareness for effective disasterrelief response. Wearable technologies can provide hands-free communication capability to doctors/nurses and otherresponders and can improve collaboration.

This work was supported by the Wallace H. Coulter Foundation, and theNational Science Foundation under Award No. CNS-1359125. Any opinions,findings, and conclusions or recommendations expressed in this publicationare those of the author(s) and do not necessarily reflect the views of theWallace H. Coulter Foundation or the National Science Foundation.

The state-of-the-art practice in hospitals to handle masscasualty disaster triage is to use a standing protocol such as“Code Silver” [2]. This protocol is based on a hierarchal Inci-dent Command System (ICS), whose technology is providedby companies such as Intermedix [3]. Existing ICS deploy-ments rely on an operational infrastructure as the communi-cation backbone to coordinate efforts between ICS divisions.They prominently feature text-based web-applications for inci-dent response status notifications, and offers basic synchronoustext messaging as a way for synchronous communicationsbetween the Incident Commander and First Responders. Inaddition, they assume outdated communication modes in masscasualty disaster triage that involves the use of traditionalradio handsets between responder personnel in absence ofcommunication infrastructure.

Moreover, they do not take advantage of wearable tech-nologies such as Google Glass [4] and Recon Jet [5] heads-updisplays that allow for hands-free communication and forcemultiplication, enabling a single individual to perform moretasks that earlier needed more individuals to perform. Thisis essential when caring to needs of critical patients andco-ordinating with other support medical staff. Coupling theICS web-applications with the use of wearable technologiesalong with other Internet-of-Things (IoT) devices (e.g., virtualbeacons, wireless mesh network elements) and a mobile cloudplatform as shown in Figure 1 can provide the benefitsof augmented reality. This in turn can transform the ICSworkflows by notably improving the situational awareness foreffective co-ordination between involved staff in a disasterrelief situation.

In this paper, we describe a novel Intelligent Dashboard thatprovides augmented reality benefits with minimal human com-munication through integration of a standardized ICS with IoTdevices such as heads-up displays [4] [5], virtual beacons [6],QR-code cards [7], and wireless mesh network elements. Thiswork builds upon our earlier work on developing a “Panacea’sGlass” framework that features an architecture for a basicheads-up display integration featuring Google Glasses within awireless mesh network environment [8]. More specifically, inthis paper, we detail our development of a ‘Responder TheaterDashboard’ (also referred to as the Intelligent Dashboard inthe remainder of this paper) for our “Panacea mobile cloud”1 that demonstrates how to integrate IoT devices with the

1Panacea in English means a solution or remedy for all difficulties ordiseases, and references the Greek goddess of universal remedy viz., Panacea.Our effort could be equated to a mobile cloud service with wearable tech-nologies and other IoT devices that provide tools for Panacea when treatingmany patients simultaneously in austere environments.

Fig. 1: Illustration of the Panacea mobile cloud setup in a field deployment

hierarchal ICS. The ultimate goal of the integration is toallow supply management as well as patient/responder trackingin order to provide improved situational awareness, whichin turn helps in prioritization of triage co-ordination of theIncident Commander and Responders. To show the ease-of-use and effectiveness of our Intelligent Dashboard approach,we present results from a usability evaluation featuring atwo-incident simulation following a ‘Task Force 1 Rescue’template (complies with a common search and rescue protocolapproved by US Federal Emergency Management Agency),and an expert ICS user providing opinions.

The remainder of this paper is organized as follows: Sec-tion II describes our Panacea mobile cloud hardware and soft-ware components integrated within our Intelligent Dashboardfor communication in a mass casualty disaster triage scenario.Section III describes our experimental evaluation results of ourIntelligent Dashboard. Section IV concludes the paper.

II. PANACEA MOBILE CLOUD SOLUTION

In this section, we first describe the technology requirementsfor effective co-ordination of the Incident Commander andResponders at a disaster scene. Following this, we describe theIoT devices integration and the Intelligent Dashboard to meetthe disaster medical triage co-ordination needs effectively.

A. Disaster Scene Response Requirements

The hierarchical ICS structure in [2] is mainly comprised ofthe head Incident Commander who instructs three subgroups:the Logistics Section Chief, the Planning Section Chief, andthe Medical Branch Director. These staff also have authorityover various other support personnel that are allocated toparticular incidents initiated within the ICS. When an incidentoccurs, a prior template is used to register the incident, andappropriate staff personnel associated with the template areautomatically notified to respond. As responders arrive at thescenes and conduct relief activities, information is passed fromthese scenes to the heads of the subgroups, and then up to theIncident Commander to interpret and provide more instructionand deploy additional resources (e.g., ambulances, medicalsupplies, domain experts). Such a hierarchy ensures that theIncident Commander receives all necessary information and

has situational awareness to appropriately respond to thevarious events at the incident scenes at all times.

At a given scene, particularly if there is a mass casualtyincident, the co-ordination intensity and demands are muchmore rigorous. If local communication systems are affecteddue to the incident, then medical theater environments haveto be setup closer to the patients needing critical care, andthe ICS structure has to be initiated at the scene location.The technologies to facilitate the medical responder theaterenvironment need to operate offline (i.e., without Internetconnection) and have to provide for their own battery powerednetworking and computation, at least on a temporary basis inthe worst case scenario that is typical soon after any majordisaster incident.

As described in Section I, pertinent technologies suchas wearable heads-up displays, virtual beacons, wirelessmesh network elements along with a quick-deploy and self-contained mobile cloud setup with an Intelligent Dashboardare critical for effective co-ordination and communication.To have a large scale of operations, low cost factors alsoneed to be considered in the design of the ICS augmentingtechnologies. However, if parallel infrastructures exist (e.g.,communication networks, ambulance routing), the ICS shouldbe able to leverage these capabilities (e.g., for exchanginginformation with other larger databases of Disaster Responsegroups, higher order computational resource usage with inci-dent related geographical data sets, seeking inputs from remoteexpert sources, requesting patient data and other medicalresources) to provide more accurate and cost-effective medicaltriage that avoids delays and exhaustion of resources.

B. IoT Devices Integration

To meet the technology requirements identified in previoussubsection, we have developed the Panacea mobile cloudsolution, whose deployment in the field is illustrated in Fig-ure 1. We can see the various triage scenes being networkedwith wireless coverage in the disaster affected area, and thesituational awareness and co-ordination being orchestratedfrom the Incident Command Center with a “Responder TheaterApplication” customized for a particular scene incident. Thenetworking is established at the disaster affected area through

an off-the-shelf, yet reliable/fault-tolerant configuration ofmeshed access points (Custom RouterBOARD routers withcombined 2.4 GHz and 5 GHz access points, Ubiquiti Net-works Unifi AP Outdoor 5 GHz) that can be quickly setupup in a ‘plug-and-play’ manner and powered by rechargeablebatteries (up to 24 hours of operation with extra battery swap-ping). The mobile cloud router with server in our setup is alow-power consuming Raspberry Pi appliance that can controlthe network at each access point and ensure communicationbetween all points, even if one of the points experiencesfailures.

For the heads-up display devices, we are using two wearabletechnologies available in the market, one is the Google GlassExplorer Edition, and the other is the Recon Jet CommercialEdition which was released to consumers early in Summer of2015. These devices connect to the Panacea wireless networkor any other available wireless network, and can communicatewith the mobile cloud server and Responder Theater Applica-tion. The audio and video streams from the heads-up displaydevices allow for synchronous communication between theIncident Commander and the Responders through a WebRTC-based plug-in. The audio and video chat feature allows theIncident Commander to visually view and track a scene statusfrom the Responder’s point of vision, and can help with bettersituational awareness to provide patient care instructions.

Additionally, data streams of patient status, responder sta-tus (e.g., inactive/active, care-in-progress/care-pending/care-resolved or other status based on custom settings in ICStemplates) or supply status (e.g., depletion rate, maximumamount of supply to be stocked, current supply stock levels)are relayed through the heads-up display to the ResponderTheater Application via the use of QR codes and virtualbeacons. For the virtual beacons capability in Panacea mobilecloud platform, we use Estimote Beacons that are commonlyused for micro-location status tracking and for collecting con-textually relevant information of interest in a mobile setting.The virtual beacon information when used on a map overlaycan be valuable to quickly assess the status of the variousscene contexts and visualize important places/objects, andinitiate appropriate co-ordination measures. To have positionalaccuracy and energy efficiency in an urban environment, wefound that Bluetooth Low Energy (BLE) variety of EstimoteBeacons are the most suitable technology due to their lowconsumption of power both on the beacon hardware, and onthe hands-free devices that receive the signals.

In our current implementation for virtual beacon integra-tion capability, we use the Google Glass capability to haveResponders scan QR codes as part of a Glassware applicationcalled QR Lens, and convey virtual beacon status to ourResponder Theater Application. By simply saying “Recognizethis” verbally, a Responder can make his/her Google Glassdevice open up a QR code scanner that reads any programmedQR code associated with virtual beacon. The codes can bemapped to the various status codes of patient status, responderstatus and supply status. Thus, again we reduce the amountof manual work involved on the part of the Responders increating situational awareness at the Incident Commanderlevel via the notifications of the status in the ResponderTheater Application. Moreover, Incident Commander can also

send notifications to Responders’ heads-up display devicesto update them on any status information from his point ofview. Thus, our scheme helps increase the scale of patientsbeing cared and tracked, in contrast to the case of usingonerous traditional triage tags and verbal communication forco-ordination of status at a disaster incident scene.

Fig. 2: QR codes along with virtual beacons used as the replacementfor the traditional triage paper tags

Fig. 3: Map section of Panacea’s Cloud

C. Intelligent DashboardThe Intelligent Dashboard whose screenshot is shown in

Figure 3 has several software features that allow for managingthe Staff directory, Incident create/update/archive/delete ac-tions, Facilities tracking actions and organizing text messagesand auto-generated notifications of various status of patients,responders and supply items. We can see that the ‘Dashboard’page captures the entire situational awareness of multipleincidents, and the latest response status. Some of the details ofthe features we have implemented in the Intelligent Dashboardare described in the following.

1) Incidents: An ‘Add Incident’ tab will transfer the Inci-dent Administrator (could be the Hospital Incident Comman-der or one of the subgroup leaders mentioned in Section II-A)to the ‘Add Incident’ template. This template can be chosenfrom many variants and is designed to provide an easy andquick way to alert staff about an emergency incident that hasoccurred. Both ‘Drill/Exercise’ or ‘Real Incident’ are availablevia a dropdown menu to be chosen when creating an Incidentform. Next, the Incident Commander can write a message to besent out to the incident-associated staff along with a missionstatement. To add additional staff to an incident, there is a‘Contacts’ option to select additional staff who will be addedto the incident response and will receive this notification. Thefinal part of the ‘Create Incident’ form includes the contactinformation of the facility of the supplies and location of theemergency scene. The ‘Incidents’ tab has a ‘View Incidents’

feature that lists all the currently active incidents in the ICS.Resolved incidents can be archived, while the unresolvedincidents will be at the top and ordered by their status andtime. The incident status options include: “Active”, “Activeand Escalating”, “Active and Deescalating”, and “Resolved”,and these options text can be customized.

2) Status Notifications: As shown in Figure 3, notificationsare messages sent to a large group of responders, such as whenan incident is created or a new development in an emergencyhas occurred. This feature is important in that it provides aquick and easy way to broadcast essential information to theentire staff in an emergency situation. In order to utilize thisfeature, the Incident Commander can create these notificationson the Dashboard, where they are then sent throughout thedatabase to all of the selected staff. These notifications willbe received by the staff to guide them to take action whennecessary. Additionally, the notifications can be configured tobe sent to users of e.g., Google Glass to view them across thetop of their screen for taking necessary actions.

3) Facilities Tracking: When the Incident Commanderclicks on the ‘Facilities’ tab, s/he will be directed to a pagethat lists the facilities associated with the ICS. If they wishto see contact information and other details about the facility,they can do so by clicking on the name of the facility. Thiswill take them to another page containing the information, andthey will have the ability to edit this information by clickingthe ‘Edit’ button located to the right of the name of the facility.On the main ‘Facilities’ page, the Incident Commander willbe able to see a variety of information on supplies, such asblood bags, beds, and medicine that will be located next tothe name of the facility.

4) Video and Audio Feeds: Using the ‘Video Feeds’ tab,the Incident Commander will be able to view the video feedsof staff in the field using heads-up display devices. A list of allthe available feeds are displayed along with associated sceneinformation for the Incident Commander to select the sceneand staff to communicate for provide instructions or guidance.The video feeds are streamed over the wireless network theyare connected to, and then linked to this page on the IntelligentDashboard. The Incident Commander can also add new userswho have heads-up display devices, if the need arises.

III. USABILITY EVALUATION OF INTELLIGENTDASHBOARD

In this section, we describe a simulation that was conductedfrom an Incident Commander standpoint to demonstrate theease-of-use, and effectiveness in co-ordination of multiple in-cidents through our Intelligent Dashboard with IoT integration.

An expert ICS user who administers and regularly usesthe Intermedix ICS was presented with a two-incident sit-uation that needed response. The two incidents related toa severe fire incident and a multiple car crash at differentlocations, respectively. The expert was asked to create anincident using a ‘Task Force 1 Rescue’ template (complieswith a common search and rescue protocol approved by USFederal Emergency Management Agency) that has been pre-populated with the associated staff expected to respond, andother protocol information regarding facility and prioritizationpolicies for routing ambulances, medical supplies and addi-tional responders. As responders (test subjects) arrived at the

two scenes, heads-up displays were used for audio and videocommunication and various notifications from the scenes to theIncident Commander, and vice versa occurred as the responseactivities progressed via the various IoT devices integratedwith our Intelligent Dashboard. At the end of the simulation,the expert was asked to rate the ease of use and effectivenessof the various features of our Intelligent Dashboard.

Table I shows the expert ratings that range from 1 to 5,where a rating of 1 is low and 5 is high. We can see that theexpert found the Intelligent Dashboard was easy to setup andcustomize for incidents, evident through a 5 rating. Expert alsowas satisfied and gave a 5 rating in tasks of adding extra staffand managing the facilities information based on the notifica-tions being exchanged via the Intelligent Dashboard. However,the expert was not fully satisfied with the audio and videocommunications with the wearable technologies, and gave a3 rating. Although the audio and video communication washelpful in the co-ordination, there was a need for better visualsituation awareness at the scenes to see around the scenesrather than just from the point-of-view of the responders.

TABLE I: Usability Experiment

Pancea’s Cloud Intelligent Dashboard Usability (Range 1 - 5)Setup and Customization 5Staff Templates 5Incident Template 5Facilities Template 5Video Feeds 3

IV. CONCLUSION AND FUTURE WORK

In this paper, we described our Panacea mobile cloud solu-tion that features a novel Intelligent Dashboard that providesaugmented reality benefits regardless of any physical/networkinfrastructure available in the surrounding environment. Wedescribed how our solution can operate with minimal humancommunication through integration of a standardized IncidentCommand System (ICS) with Internet of Things (IoT) suchas heads-up displays, virtual beacons, QR-code cards, andwireless mesh network elements. Through a usability eval-uation and platform experiments, we showed the ease-of-useand effectiveness of our Intelligent Dashboard in helping anIncident Commander.

REFERENCES

[1] D. White, “Advanced Health and Disaster Aid Network”,Final Report JHU/APL NSTD-07-896 to NIH, 2007.

[2] “Patient Surge: Code Silver”, Technical Publication ofUniversity of Missouri-Columbia Hospital, 2014.

[3] Intermedix Solutions for Health Care -http://www.intermedix.com

[4] Google Glass - https://www.google.com/glass[5] Recon Jet - http://www.reconinstruments.com/products/jet[6] Estimote Virtual Beacons - http://www.estimote.com[7] QR Code Generator - http://www.qrstuff.com[8] J. Gillis, P. Calyam, A. Bartels, M. Popescu, S. Barnes,

J. Doty, D. Higbee, S. Ahmad, “Panaceas Glass: MobileCloud Framework for Communication in Mass CasualtyDisaster Triage”, IEEE Mobile Cloud, 2015.

[9] Trickle ICE - https://tools.ietf.org/html/draft-ietf-mmusic-trickle-ice-02