Embed Size (px)
Citation preview
University of South FloridaScholar Commons
Graduate Theses and Dissertations Graduate School
2006
RSVP: An investigation of the effects of RemoteShared Visual Presence on team process and teamperformance in urban search and rescue teamsJennifer L. BurkeUniversity of South Florida
Follow this and additional works at: http://scholarcommons.usf.edu/etd
Part of the American Studies Commons
This Dissertation is brought to you for free and open access by the Graduate School at Scholar Commons. It has been accepted for inclusion inGraduate Theses and Dissertations by an authorized administrator of Scholar Commons. For more information, please [email protected].
Scholar Commons CitationBurke, Jennifer L., "RSVP: An investigation of the effects of Remote Shared Visual Presence on team process and team performance inurban search and rescue teams" (2006). Graduate Theses and Dissertations.http://scholarcommons.usf.edu/etd/2467
RSVP: An Investigation of the Effects of Remote Shared Visual Presence on
Team Process and Performance in Urban Search & Rescue Teams
by
Jennifer L. Burke
A dissertation submitted in partial fulfillment of the requirements for the degree of
Doctor of Philosophy Department of Psychology
College of Arts and Sciences University of South Florida
Major Professor: Michael D. Coovert, Ph.D. Walter C. Borman, Ph.D.
Michael T. Brannick, Ph.D. Mark S. Goldman, Ph.D. Robin R. Murphy, Ph.D.
Date of Approval: April 10, 2006
Keywords: human-robot interaction, rescue technology, shared mental models, communication analysis, field research methods
© Copyright 2006, Jennifer L. Burke
Dedication
For my family: my mother Jane, who always found something good to say, and told me
she was proud of me; my sisters Jill and Krista, who patiently listened to my tales of woe
when things were not going well; my father Garfield, who inspired me to go for it in the
first place; sweet daughter Sara, who dealt gracefully with a busy student-mom (“I can’t
cook dinner, I have homework!”); and for my husband Michael, fellow traveler through
life, whose love and support kept me going through the adventures of the past five
years—as it has throughout our lives together.
Acknowledgements
There are many people who played a part in this study coming to fruition, from
funding agents that supported the research to fellow students who helped with the nitty
gritty details. Whatever the role, I extend my heartfelt gratitude for your help. Thanks go
to DARPA and NSF for their support; SSRRC staff members Sam Stover, Tim Slusser,
and Colleen Cleveland; CSE staff members Valerie Mirra and John Giannoni; USF
students Rahul Agarwal, Laura Barnes, Jeff Craighead, Thomas Fincannon, Matt Long,
David Richards, Ashley Gray, Jamie Griffiths, Brian Day, and Rachel Sessions; CMU
students Jason Geist, Mike Scherwin, Elie Shammas, and Anthony Kolb; and many
others, including Bob Dolci at NASA-Ames, Jim Bastan at NJTF-1, Howie Choset from
CMU, Jean Scholtz from NIST, Mark Micire, and Erika Rogers (who bought me the little
red book in which to begin the writing process).
Special thanks go to my committee members: Wally Borman, who gave me good
counsel on rater training, and lots of positive reinforcement; Mike Brannick, who always
made time to answer my (many) questions on methodological issues; Mark Goldman,
who gave me my first research job, and whose own research sets the standard by which I
measure; and finally, to my mentors and advisors, Mike Coovert and Robin Murphy.
Mike’s guidance, support, and encouragement inspired me, and his quiet questions
challenged me to think and stretch to come up with new ideas. I can only hope to “be like
Mike” in my career. Lastly, my thanks go to Robin Murphy, who made it possible for me
to go places and do things I never dreamed possible. The day I visited her lab was a
turning point in my career, opening the door to a future in a brand new field—human-
robot interaction. I look forward to our future collaboration!
i
Table of Contents
List of Tables ..................................................................................................................... iii List of Figures .................................................................................................................... iv Abstract ................................................................................................................................v Chapter 1: Introduction .......................................................................................................1 Chapter 2: Related Work ....................................................................................................8 Teams.......................................................................................................................8 Distributed Teams......................................................................................10 Distributed Teams in US&R......................................................................11 Description of Technical Search Team Operations .......................11 Communication and Coordination Challenges in Distributed
Teams ....................................................................................................14 Shared Mental Models ...................................................................16 Team Communication and Shared Mental Models .......................20 Shared Visual Presence..................................................................22 Robots ........................................................................................................25 What is a Robot?............................................................................25 Human-Robot Interaction ..............................................................27 Human-Robot Team Performance .................................................28 Situation Awareness and Team Processes .....................................30 Chapter 3: Mobile Robots as Shared Visual Presence: Approach....................................32 Hypotheses.............................................................................................................36 Chapter 4: Method ............................................................................................................38 Setting, Participants, and Apparatus ......................................................................38 Setting ........................................................................................................38 Participants.................................................................................................39 Apparatus ...................................................................................................40 Design ....................................................................................................................41 Location (Collocated vs. Distributed Teams) ............................................41 Remote Shared Visual Presence (RSVP or no-RSVP) ..............................41 Measures ................................................................................................................42 Team Performance Measure ......................................................................43 Team Process Measures.............................................................................44 Shared Mental Model Measures ................................................................45
ii
RASAR-CS................................................................................................46 Procedure ...............................................................................................................51 Data Collection ..........................................................................................51 Data Analysis .............................................................................................54 Data Editing and Preparation .........................................................54 Data Coding and Rating.................................................................54 Statistical Analyses: Multilevel Regression...................................56 Chapter 5: Results ..............................................................................................................64 Descriptive Analyses .............................................................................................65 Demographics ............................................................................................65 RASAR-CS Analyses ................................................................................66 Descriptive Analyses for Study Variables .................................................72 Multilevel Regression Analyses ............................................................................76 Team Performance Measure ......................................................................76 Supplemental Regression Analyses ...............................................78 Shared Mental Model Measure..................................................................80 Supplemental Regression Analyses ...............................................81 Team Process Measure ..............................................................................83 Supplemental Regression Analyses ...............................................84 Summary of Hypotheses and Findings ..................................................................85 Chapter 6: Discussion ........................................................................................................94 RSVP and the Model .............................................................................................96 Location and Site Effects .......................................................................................98 Theoretical and Practical Implications.................................................................104 Theoretical Implications ......................................................................................104 Practical Implications...........................................................................................107 Limitations ...........................................................................................................110 Parting Thoughts and Future Directions ..............................................................111 References........................................................................................................................114 Appendices.......................................................................................................................122 Appendix A: Script for Search Task Scenario....................................................123 Appendix B: Demographic Survey Questionnaire .............................................125 Appendix C: Complete Correlation Table for Study Variables..........................127 About the Author ................................................................................................... End Page
iii
List of Tables Table 1 Distribution of teams across experimental conditions .....................................42 Table 2 Team process dimensions ................................................................................45 Table 3 Robot-Assisted Search and Rescue Coding Scheme (RASAR-CS) ................48 Table 4 Raw agreement, Kn, and K for the four RASAR-CS categories ......................56 Table 5 Means/proportions, standard deviations, and percentages for
demographic survey data .................................................................................66 Table 6 RASAR-CS statement frequencies and percentages........................................68 Table 7 RASAR-CS form x content comparisons of statement frequencies and
proportions for speaker-recipient dyads...........................................................70 Table 8 RASAR-CS statement proportions by location and RSVP conditions............71 Table 9 Means, standard deviations, and correlations for study variables of
interest (N = 50) ..............................................................................................73 Table 10 Multilevel regression analysis of location and RSVP effects on team
performance .....................................................................................................77 Table 11 Supplemental multilevel regression analyses for team performance...............78 Table 12 Multilevel regression analysis of location and RSVP effects on shared
mental model....................................................................................................81 Table 13 Supplemental regression analyses for shared mental model............................82 Table 14 Multilevel regression analysis of location and RSVP effects on team
process..............................................................................................................84 Table 15 Supplemental regression analyses for team process ........................................85 Table 16 Study hypotheses and evidence of support ......................................................86
iv
List of Figures Figure 1. Organizational Structure of US&R Task Force (FEMA,1992) .......................13 Figure 2. Klimoski & Mohammed’s Framework for Explaining the Role of
Team Mental Models in Team Performance ...................................................19 Figure 3. Model of Team Performance in Robot-Assisted Technical Team
Search...............................................................................................................34 Figure 4. Inuktun Micro-VGTV Robot System ..............................................................40 Figure 5. A Mannequin Hand Hidden in the Search Space Serves as a Visual
Cue ...................................................................................................................44 Figure 6. SA Indicators in the RASAR-CS.....................................................................49 Figure 7. A 3-Person Team of Researchers Manned Each Task Scenario Site...............53 Figure 8. Baseline and Ordercode Model Estimates of Performance .............................63 Figure 9. Mean Performance Scores Plotted by Location and Use of RSVP .................99 Figure 10. Bar Graphs Showing Mean Performance Scores at NASA-Ames and
NJTF-1 Sites According to Location (Distributed or Collocated) and Use of RSVP..................................................................................................102
v
RSVP: An Investigation of the Effects of Remote Shared Visual Presence on Team
Process and Team Performance in Urban Search & Rescue Operations
Jennifer L. Burke
ABSTRACT
This field study presents mobile rescue robots as a way of augmenting
communication in distributed teams through a remote shared visual presence (RSVP)
consisting of the robot’s view. It examines the effects of RSVP on team mental models,
team processes, and team performance in collocated and distributed Urban Search &
Rescue (US&R) technical search teams, and tests two models of team performance.
Participants (n=50) were US&R task force personnel drawn from high-fidelity training
exercises held in California (2004) and New Jersey (2005). Data were collected from the
25 dyadic teams as they performed a 2 x 2 repeated measures search task entailing robot-
assisted search in a confined space rubble pile. Team communication was analyzed using
the Robot-Assisted Search and Rescue coding scheme (RASAR-CS). Team mental
models were measured through a team-constructed map of the search process. Ratings of
team processes (communication, support, leadership, and situation awareness) were made
by onsite observers, and team performance was measured by number of victims
(mannequins) found. Multilevel regression analyses were used to predict team mental
models, team process, and team performance based upon use of RSVP (RSVP or no-
RSVP) and location of team members (distributed or collocated). Results indicated that
the use of RSVP technology predicted team performance (β = -1.322, p = 0.05), but not
vi
team mental models or team process. Location predicted team mental models (β = -0.425,
p = 0.05), but not as expected. Distributed teams had richer team mental models as
measured by map ratings. No significant differences emerged between collocated and
distributed teams in team process or team performance. Findings suggest RSVP may
enhance team performance in US&R search tasks. However, results are complicated by
differences detected between sites. Support was found for both models of team
performance, but neither model was found sufficient to describe the data. Further
research is suggested in the use of RSVP technology, the exploration of team mental
models, and refinement of a modified model of team performance in extreme
environments.
1
Chapter 1
Introduction
Distributed team performance is becoming one of the most popular and critical
research areas in industrial-organizational psychology. The “perfect storm” combination
of workforce globalization, the proliferation of teams as an organizational structure, and
the onslaught of increasingly complex technology has made the concept of distributed
teams a necessary reality without a clear understanding of what it takes to make them
work. The communication and coordination challenges posed by distributed teaming are
readily evident as a hindrance to effective team performance (Olson & Olson, 2003).
These challenges are sometimes exacerbated rather than alleviated by the insertion of
new technology designed to address them. This study, which explores the effects of one
type of technology (robots) on distributed team performance, is motivated largely in the
interests of making robots a help, not a hindrance, in distributed team tasks of the future.
This study presents mobile rescue robots as a way of augmenting communication
in distributed teams through a shared remote visual presence consisting of the robot’s
view. The research has a theoretical focus, testing portions of an existing model of team
performance (Klimoski & Mohammed, 1994), and probing further to examine questions
as to how certain constructs within the model are related. It moves from theoretical
constructs to a quasi-experimental investigation that will tease apart some of the issues
pertaining to the relationships between team mental models, team processes and team
performance. The question is addressed within the purview of the Federal Emergency
2
Management System, which became part of the U.S. Department of Homeland Security
in March, 2003. FEMA's continuing mission within the new department is to lead the
effort to prepare the nation for all hazards and effectively manage federal response and
recovery efforts following any national incident. Emergency/disaster management is a
system composed of many (distributed) ad hoc teams; one of these is Urban Search and
Rescue (US&R).
Urban search and rescue has been posed by the DARPA/NSF study on human-
robot interaction (Burke, Murphy, Rogers, Lumelsky, & Scholtz, 2004) as an exemplar
domain for human-robot interaction (HRI). US&R involves the rescue of victims from
the collapse of a man-made structure. The environment can be characterized as a pile of
steel, concrete, dust, and other rubble and debris. The areas are perceptually disorienting;
they no longer look like recognizable structures due to the collapse, it is dark, and
everything is covered in gray dust from concrete or sheet rock. Robot assisted search and
rescue in this field domain, requires that small shoe-box sized physically situated robots
operate under these unstructured, outdoor environmental conditions in real-time to
visually search areas that are either too narrow for safe human or canine entry or
generally unsafe for human exploration The robots are short, providing a viewpoint from
less than one foot off the ground. This exacerbates any “keyhole effects” (Woods, Tittle,
Feil, & Roesler, 2004). These domain and agent characteristics present many challenges
that distinguish US&R from other human-robot interaction settings, e.g. manufacturing,
entertainment and office-oriented applications.
The relationship between humans and robots in US&R is different than
manufacturing, office, or even security applications of robots. Robots must physically
3
team with people to perform any activity. Because of their small size and the mobility
challenges imposed by the US&R environment, robots must be carried in backpacks to
the voids targeted to be searched. Humans must interpret the video, audio, and thermal
imaging data provided from the robots and fuse it with other data sources (e.g., building
plans) and knowledge (e.g., time of day) in order to identify victims and structural
anomalies as well as conduct and coordinate large-scale rescue efforts. The information
extracted from the robot’s search must be abstracted and propagated up a hierarchy of
decision makers as well as distributed laterally among search specialists. Therefore, the
human-robot team must cooperatively transform data into information and levels of
knowledge. This means human-robot interaction in US&R must consider distributed
information transfer and cooperation.
Though the use of robots as remote shared visual presence is presented
specifically within the domain of US&R, the potential applications have far-reaching
implications for propagation throughout the emergency management system structure,
encompassing all of the federal response and recovery efforts following any national
incident. It is believed that using the remote presence provided by the robot as the basis
for establishing mutual knowledge among distributed teams can increase communication
efficiency in distributed teams by building shared awareness, or common ground.
Communication efficiency has been shown to reduce workload and can result in better
performance (MacMillan, Entin, & Serfaty, 2004). Moreover, the use of mobile robots as
a visual resource allows for assessment of the situation while avoiding information
overload, and can assist in resource allocation by incident command. It can also provide
reliability/redundancy in communication support to existing communication channels,
4
and help with accountability of resources and personnel—all critical issues in firefighting
and emergency response (Jiang, Hong, Takayama, & Landay, 2004).
This study hypothesizes that the shared visual presence provided by the robot’s
eye view can serve as common ground for the distributed team onsite, and for others
removed from the site. Robot-assisted technical search presently requires a 2:1 human-to-
robot ratio (Burke & Murphy, 2004a). The robot operator bears the brunt of the cognitive
load in teleoperating the robot and searching the remote environment; the tether-handler
can provide some physical assistance through manuevering the tether, but mostly relies
on communication with the operator to participate in the cognitive aspects of the task.
The tether-handler can share the cognitive load by assisting with the search, but lacks the
same point-of-view as the operator, since she is typically several meters away and cannot
observe the visual image provided by the robot. The robot’s view offers a medium for
building a shared mental model of the remote search space, allowing for feedthrough of
awareness information by positions, orientation and movement of both the robot and
other artifacts in the visual environment (Gutwin & Greenberg, 2004).
Shared visual presence in the remote environment can enhance team
communication and coordination, serving as a source for conversational grounding
(common ground), and facilitating the use of gaze awareness, deictic references and
targeted communication, as well as providing a feedback source (visual evidence of
understanding). Operators and tether-handlers currently utilize verbal communication to
create team mental models of the search environment, and to support mutual knowledge
of both the task and their respective roles and actions in performing the task. By
providing the tether-handler with the same remote visual presence experienced by the
5
robot operator via an auxiliary monitor, more contextual information can be conveyed
using targeted communication and deictic references toward artifacts in the environment.
The robot operator can convey gaze awareness by camera manipulation or changes in the
robot’s configuration. Feedback between the two team members may be enriched by
providing the visual channel as a conduit for confirmatory communication on an implicit
level.
For example, the tether-handler may suggest that the robot operator take a closer
look at a particular artifact in the remote environment. When the robot operator focuses
on the intended object, the tether-handler has visual confirmation of the operator’s
understanding. In addition, using the visual image provided by the robot to build mutual
knowledge may allow team members to anticipate each other’s informational needs, and
provide needed information without being asked. This simple switch from explicit to
implicit coordination in teams can increase communication efficiency, and has been
linked with effective team performance in high stress situations (Morris, Rouse, & Zee,
1987). Therefore it is expected that the use of mobile robots as a shared visual presence
in remote environments may lead to more effective distributed team performance in
robot-assisted technical search tasks. Moreover, as wireless communication technology
improves, this shared visual presence does not have to be limited to the distributed team
onsite. A team member “looking over the shoulder” of the robot operator from a site
well-removed from the Hot Zone can assist in the search task, offering a fresh pair of
eyes that are not subject to the physical and cognitive stressors present onsite.
As mentioned earlier, this study is a theoretical piece, presenting a context- and
task-specific model of team performance that tests relationships between shared (team)
6
mental models, team process, and team performance as outlined in Klimoski and
Mohammed’s (1994) framework explaining the role of team mental models in team
performance. This model draws on Kraiger and Wenzel’s (1997) proposed framework for
mental models as well, which situates shared mental models in a nomological net of
antecedents and consequent effects (including team process and performance.) Unlike
Klimoski and Mohammed’s model, however, the Kraiger and Wenzel framework makes
no attempt to explain how shared mental models influence team process and performance.
This proposal tests a portion of Klimoski and Mohammed’s theoretical model, that shared
mental models do contribute to team performance directly and indirectly through greater
team capacity and more effective team processes. A further theoretical contribution is
made by exploring more specifically how these team mental models are created, and
whether particular team processes (communication, backup/support, leadership, situation
awareness) are affected by their quality. In this study, the remote shared visual presence
provided by the robot is posited as a team resource that increases the team’s capacity
(readiness) and serves as the common ground from which team mental models are
constructed. Through communication analysis and other measures, I intend to trace the
formation of the team mental model, examine its influence on team processes and
investigate the effects, if any, on team performance.
To understand the research presented in this study, there are some terms and
background information you need to know. The following chapter will give you some
background on teams and distributed teams, and on the communication and coordination
challenges that confront distributed teams. The concepts of shared awareness and shared
mental models are discussed, as is research on shared visual space. Next, a short review
7
of the research on robots and human-robot interaction is recounted, focusing on the
studies that relate to the search and rescue domain. Armed with this information, you can
then wade into the approach outlined in Chapter 3, which explains the proposed model of
performance in robot-assisted technical search teams, and enumerates specific
hypotheses. This is followed by the method section (Chapter 4), which details the
specifics of the field study, and the analyses used to examine the research questions.
Results are presented in Chapter 5, and are followed by discussion and conclusions in
Chapter 6.
8
Chapter 2
Related Work
This chapter reviews important characteristics of teams and distributed teams,
explains some of the communication and coordination challenges that confront
distributed teams, and presents relevant research on human-robot interaction. It begins
with a brief discussion of the characteristics that define teams and how we look at team
processes and performance. Next, the communication and coordination challenges that
confront distributed teams are enumerated. The concepts of shared awareness and shared
mental models are introduced here, as is research on shared visual space. Finally, a short
review of the research on robots and human-robot interaction is recounted, focusing on
the studies that relate to the search and rescue domain.
Teams
Teams are an important organizing structure in the workplace, a fact reflected in
the burgeoning literature and research on team performance, processes, and training. The
increasing size, complexity and globalization of organizations have fostered the use of
teams to speed development and delivery of products and services in a timely and cost-
effective manner. Moreover, the technology-driven changes in work settings often
necessitate coordinated efforts between team members (Coovert & Foster Thompson,
2001). Teams exist as they perform cyclically over context and time, interacting among
themselves and others (Ilgen, Hollenbeck, Johnson, & Jundt, 2005), and have an identity
as a work group within a defined organizational function (e.g., a technical search team
9
within a USAR Task Force unit) (West, Borrill, & Unsworth, 1998). There are many
definitions of a work team, most of which share the notions of interdependence and
shared goals. For the purposes of this study, a work team is defined as “…two or more
people with different tasks who work together adaptively to achieve specified and shared
goals” (Brannick & Prince, 1997). More than thirty years of research on the factors which
contribute to team performance has yielded a plethora of models and theories of team
functioning. Theoretical models of team work typically focus on the inputs, processes and
outputs that characterize team effectiveness (Hackman, 1987; McGrath, 1984; Salas,
Dickinson, Converse, & Tannenbaum, 1992), though recent models have included more
complex representations of teams to include temporal influences (Marks, Mathieu, &
Zaccaro, 2001), mediational effects (Ilgen et al., 2005), and multilevel constructs
(DeShon, Kozlowski, Schmidt, Milner, & Wiechmann, 2004). Key to all of these models
and theories is the assumption that various team variables and processes contribute to
successful team functioning. Variables on the environmental, organizational, team and
individual levels can influence the processes seen as integral to effective team
performance: e.g., communication, coordination, situation awareness, leadership,
adaptability, and support/backup behavior. In this study, the input variable of interest is
the team level psychological construct of shared mental models. This construct is
discussed in detail later.
Distributed Teams
Distributed or virtual teams are those whose members are mediated by time,
distance or technology (Driskell, Radtke, & Salas, 2003). Distributed teams are usually
project or task-focused groups. The team membership may be stable (e.g., an established
10
sales team) or change on a regular basis (e.g., in project teams). Members may come
from the same organization, or from many different organizations. They can be co-
located and work in the same physical space, but usually are thought of as working
interdependently in remote, geographically separated workspaces. Moreover, they may
work at different times, i.e. asynchronous work-cycles. Other factors can influence team
processes in distributed teams, such as whether the teams are assembled for a single,
time-limited project or on a long term basis, whether team members know each other and
have worked with each other before, and whether they expect to have any
interaction/shared work in the future.
Research on distributed teams has consistently reported special challenges in
communication and coordination which negatively affect team performance (Hinds &
Bailey, 2003; Thompson & Coovert, 2003). The difficulties in establishing shared context
across distributed teams can lead to increased conflict and confusion between team
members, and less satisfaction with team processes and outcomes. These difficulties are
discussed in greater detail later in this chapter.
All of the characteristics of distributed teams described above are present in the
US&R organizational structure, which consists of teams of teams.
Distributed Teams in US&R
Distributed teams in US&R are both project-and task-focused, in that they are
created specifically to perform certain tasks in response to disaster incidents, and are
mobilized as part of a larger emergency management effort, or project. Responders must
go through a rigorous training and certification process to be eligible to serve on a
regional or national US&R Task Force, and are often members of local fire rescue
11
departments. Members of the same Task Force may have worked together before;
however, in most responses that require activation of US&R functions, teams are drawn
from all over the country, so it is very likely that rescue workers will be working
alongside others from different teams for the single response incident. Teams work in 12-
hour cycles, and though there is some overlap, they must coordinate their efforts with
others as they enter or leave the Hot Zone (immediate disaster area). In the Hot Zone,
team members operate in deconstructed, unfamiliar environments and rely heavily on
radio communication as they work. The physical environment is dangerous and workers
are required to wear heavy personal protective equipment, which exacerbates cognitive
fatigue by making even the simplest tasks (breathing, walking) effortful. The work itself
is highly stressful and time pressured, with serious (life-threatening) consequences for
error. Technical search (Figure 1) is one of the four US&R functions: search, technical
support, medical, and rescue or extrication. These four operations represent sub-
specialties within the task force. Technical search teams are the particular type of
distributed team examined in this study.
Description of technical search team operations. While no two disasters are
managed precisely the same way, US&R technical search operations often begin with a
manual reconnaissance of the area of damage, called the hot zone. Victims on the surface
or easily removed from light rubble are extracted immediately as encountered. After
reconnaissance, the command staff determines what the safest strategy is to effectively
search the hot zone for survivors within the rubble. In areas that are deemed safe for
humans to investigate, canine teams may be sent forward. In most cases, technical search
teams wait until called for. When a dog has indicated signs of a survivor in an area,
12
technical search specialists are summoned onto the pile. The command staff attempts to
minimize the number of people in the hot zone, so technical search teams wait at the
“forward station” of the hot zone perimeter until called over the radio or assigned an area
to search. Technical search specialists may carry a fiber-optic boroscope, thermal imager,
or a video camera mounted on a wand for a visual inspection of the rubble, depending on
the verbal description of the void or the specific request of a particular device by the
leader. If a survivor is found, the search team and command staff brings in the medical
and rescue teams, who call on members of the technical support team as needed. Before
leaving the void, the technical search team marks the exterior of the void with symbols
indicating that it has been searched, the structural condition, and presence of
survivors/remains.
Figure 1. Organizational structure of USAR Task Force (FEMA, 1992).
The visual inspection of a void is most often done with a boroscope or a camera
on a wand. These technologies generally cannot penetrate more than 12 feet into a void,
whereas robots are well-suited for voids longer than 20 feet. Regardless of tool, the
search activity takes on the order of 3-30 minutes, and a technical search team may spend
most of a 12-hour shift waiting, and then work furiously for a few minutes. The
command staff may periodically evacuate the hot zone and cease all operations so that
13
14
dds
ical search task
ith
, as
rdination of activities pose significant challenges to
can affect team performance. Communication in distributed
s c
al
e
m
technical search specialists can apply sensitive acoustic listening devices. This also a
to the cognitive stress.
The field data collected in this study used the robots for a visual technical search
task, where robots served as “cameras on wheels.” The visual techn
consists of four activities in order of importance: search for signs of victims, report of
findings to the team or task force leader, note any relevant structural information that
might impact the further investigation of the void, and estimate the volume that has been
searched and map it relative to the rubble pile. In this case, the technical search teams
operated a robot instead of a boroscope or thermal imager. Technical search, along w
the other primary tasks of victim rescue and extraction, medical care and patient transfer,
requires close coordination of efforts with both co-located and remote team members
well as prompt and accurate information transfer to incident command, local medical
authorities, and others involved in the response. The communication and coordination
challenges faced by distributed US&R teams are by no means unique, but are certainly
exacerbated by the extreme environment and other stressors.
Communication and Coordination Challenges in Distributed Teams
Communication and coo
distributed teams which
team an suffer from the loss of information gleaned through “back channels” such as
physical gestures, body language, and interaction with artifacts in the environment. AI or
computer-mediated technologies that do not support transmission of contextu
information are impoverished and provide less visibility and feedback, both of which ar
needed for establishing and maintaining mutual knowledge, i.e. knowledge that tea
15
high
d,
re
derstood as intended (Clark & Marshall, 1981). Three
rimary pposes
s
members share and know they share (Krauss & Fussell, 1990). Computer-mediated
communication’s impact on mutual knowledge is likely to be greater for tasks where
individual team members possess a large quantity of unique information, and where
contextual information between remote sites differs. This problem is attenuated by
requirements for complexity, workload and interdependence, and can lead to confusion
among team members and errors in performance (Cramton, 2001).
Coordination of activities, which requires shared awareness, or common groun
is difficult when team members are distributed, and often requires more confirmatory
communication, since many of the back channel types of awareness mentioned above a
unavailable. Successful collaboration among distributed team members requires situation
awareness – ongoing awareness of what each person is doing, status of task, and the
environment (Endsley, 1995) and conversational grounding – working with each other to
ensure messages are being un
p sources for common ground are common group membership (which presu
a set of common knowledge), linguistic co-presence (hearing the same verbalizations),
and physical co-presence (inhabiting the same physical setting). Physical co-presence
provides multiple resources for building common ground, most prominently visual co-
presence. Shared awareness and conversational grounding are integral components in
creating shared mental models.
Shared Mental Models
The concept of shared mental models has been invoked to explain team dynamic
and performance for many years (Mathieu, Heffner, Goodwin, Salas, & Cannon-Bowers,
2000). Shared, or team mental models represent efforts to simplify events or
16
rdinate
l
oup
. Wellens
e
ls
ntal
and
responsibilities to make them more tractable, and are an emergent characteristic of the
team. Shared mental models serve an orienting and coordinating function in team
processes. They contain organized knowledge, and may hold a variety of content, e.g.,
representations of tasks, situations, response patterns, or working relationships (Klimoski
& Mohammed, 1994). Shared mental models are thought to improve team performance in
several ways. First, they enable team members to form accurate explanations and
expectations for a task. Second, shared mental models allow team members to coo
actions and adapt behavior to task demands. Lastly, they can facilitate information
processing (Kraiger & Wenzel, 1997). Examples of applications of the shared menta
model construct include Orasanu’s shared situation models (1990) and Wellens’ gr
situation awareness model (1993). Orasanu’s shared situation models among air crew
members included shared understanding of a problem, and team member roles
described group situation awareness among distributed decision making teams as th
sharing of a common perspective regarding current environmental events, their meaning
and projected future status.
Shared mental models can be described in terms of their focus or content.
Cannon-Bowers, Salas and Converse (1993) listed four types of team mental models:
equipment/technology, task, team interaction, and team. Equipment/technology mode
include knowledge of operating procedures, equipment functions, limitations, and likely
failures. Task models’ content includes task procedures and strategies, environme
constraints, likely contingencies and scenarios. Team interaction models reflect role
responsibilities, information sources, interaction patterns and communication channels.
Lastly, team models contain knowledge of team members’ knowledges, skills, abilities
17
ut
ent of
measures and their interrelationships to adequately assess a) how team members perceive,
process or react to external stimuli, b) how they organize or structure task-related
knowledge, c) common attitudes or effect for task-relevant behavior, and d) shared
expectations of behavior. In the proposed framework, antecedents of shared mental
models are classified as environmental, organizational, team or individual. These
determinants affect the development of shared mental models in terms of knowledge,
behaviors, and attitudes. Shared mental models, in turn, may affect both team
effectiveness and team performance. The authors note the reciprocal nature of the
relationships, showing that these outcomes can have an influence on all of the preceding
factors in the framework.
In their seminal article, Klimoski and Mohammed (1994) use the term “team
c
003), representing efforts to simplify events or
responsibilities to make them more tractable. Team mental models are thought to affect
decisio
and other characteristics. Shared mental models can further be characterized in terms of
knowledge type (e.g., declarative, procedural, structural), level of specificity (abstract,
concrete), or function. They can provide information about what an element is, how it
works, or why it is needed. Moreover, they may contain not only types of knowledge, b
also behaviors and attitudes (Kraiger & Wenzel, 1997). This makes the measurem
shared mental models a complicated undertaking. Indeed, Kraiger and Wenzel argue for a
construct-oriented approach, i.e., identifying a nomological net of related concepts,
mental model” to define the shared mental model construct as an emergent characteristi
of the team, similar to Cooke et al. (2
n-making, team dynamics and performance, and to enhance the quality of
teamwork skills and team effectiveness. Training, team composition and life cycle,
18
tal
ong with leadership)
influen
2).
communication patterns and cohesion are listed as determinants of team mental models.
They acknowledge that content and form of team mental models can vary, and depend
largely on the function (mission/task/subtask) of the team and its members, as well as the
situational context. Further, team mental models reflect internalized beliefs, assumptions
and perceptions, and exist to the extent they are apprehended by the team members at
some level of awareness. Klimoski & Mohammed placed the existence of team men
models in a framework of team performance as a factor (al
cing performance directly and indirectly through its effect on team capacity, i.e. a
team’s latent potential for demonstrating effective process and performance (Figure
Figure 2. Klimoski & Mohammed’s framework for explaining the role of team mental models in team performance (1994).
19
bers’ potential for performance, 2)
le to the team. A team’s capacity (4),
or read
ing
g
nd effectiveness.
Suppor s
iving
to
Figure 2 walks you through the framework, beginning with the factors thought to
determine team capacity: 1) the individual team mem
team composition and size, and 3) resources availab
iness, has an impact on team process (7) and performance (8) contingent on the
availability of some orienting factor that enables the team to harness that capacity, to put
it to work; team mental models and leadership (5 and 6) are 2 such factors. The orienting
and coordinating aspects of team mental models can create smooth functioning teams
through shared cognition. Interestingly, the authors imply leadership can serve as a
guiding factor when that shared cognition is not possible:
“…absent the availability of TMMs a leader can serve to guide the team, serv
such executive functions as assigning information gathering activities among team
members, doing information integration and interpretation, adjudicating disagreements
and/or directing individual team members into action” (p.431.)
It may be that RSVP technology can assume a leadership role (or help team
members do so) by performing some of these executive functions (Coovert & Burke,
2005). RSVP technology may benefit other team processes (communication,
support/backup behaviors, situation awareness) in similar fashion. For example, givin
team members access to RSVP may increase communication clarity a
t/backup behaviors may occur more frequently due to quicker detection of error
and better monitoring capabilities. Team situation awareness may be enhanced by g
all team members access to previously restricted sources of information, enabling them
make projections as to each other’s information needs before being asked. The model
utilized in this study posits RSVP as a team resource that is used to help create a team
20
s and
ce,
f
n in
2)
est work
am
“comm
work
ld
specifically with human police SWAT teams).
y in this article recorded real robot-user interaction as it occurred
context and situation, work process and domain-specific information were needed to
situation model “on-the-fly”. Therefore it serves as a test of the theoretical construct
relationships outlined in Klimoski and Mohammed’s (1994) model of team performan
specifically looking at the influence of technology as a resource enabling the creation o
richer shared mental models among team members.
Team Communication and Shared Mental Models
Research on team communication and shared mental models in police SWAT
teams and other domains offers similar findings regarding the role of communicatio
building shared awareness. Jones and Hinds’ qualitative analysis (Jones & Hinds, 200
of police SWAT teams (an “extreme team” domain similar to US&R) is the clos
conceptually to the goals of this study. Jones and Hinds explored the importance of te
communication in the development of a shared mental model (which they termed
on ground”), and noted the implications for SWAT team performance. They
observed police SWAT teams in training exercises, and identified leader roles in
establishing common ground and coordinating distributed team member actions as factors
transferable to system design for coordinating distributed robots. Jones and Hinds’
studied distributed SWAT teams (people) to model a team of distributed robots that cou
work together in similar fashion (but not
In contrast, the field stud
between team members and a single robot to inform the development of coordinated
human-robot systems within the organizational structure of US&R.
In a study of military command and control exercises, (Sonnenwald & Pierce,
2000) found that frequent communications between team members about the work
21
of
ember communication, (Orasanu, 1990) found
.
tion)
e effects of fatigue.
6
t
maintain shared situation awareness in dynamic, constraint-bound contexts. In a study
the cognitive functions of cockpit crew m
that captains of high performing crews explicitly stated more plans, provided more
explanations, and made more predictions, which were articulated for the whole crew
This enabled crew members to contribute relevant information or strategies from their
specialized perspectives, and to interpret requests and commands unambiguously.
(Foushee, Lauber, Baetge, & Acomb, 1986) studied the effects of fatigue on crew
coordination and performance, and suggested that team processes (e.g., communica
contributed to the development of shared mental models in crews. They found that
superior performance was associated with more task-related communications among
crew members, specifically more commands, suggestions, statements of intent,
exchanges of information, and acknowledgements. They also found that crews with
mental models based on shared experiences were able to overcome th
Mathieu et al. (2000) noted that team communication mediated the relationship
between mental model convergence and team effectiveness in a laboratory study using 5
undergraduate dyads who "flew" a series of missions on a personal-computer-based fligh
combat simulation. This study uses the findings regarding the criticality of shared
awareness in team-based, dynamic work domains as a justification for exploring team
communications in the US&R domain.
Shared Visual Presence
Prior research investigating the effect of shared visual presence in collaborative
physical tasks shows that the availability of a shared visual workspace can positively
impact performance and team process in two ways: through creating awareness of the
22
ons,
subtasks within the collaborative process of
-
ed
l
f
und to
task state (i.e., situation awareness) and through providing an efficient resource for
conversational grounding (Gergle, Kraut, & Fussell, 2004b; Kraut, Fussell, & Siegel,
2003; Kraut, Gergle, & Fussell, 2002).
Research investigating the role of visual information in collaborative physical
tasks decomposed the visual information available when people share physical co-
presence into 4 categories: participants’ heads and faces, participants’ bodies and acti
task objects, and work environment/context (Kraut et al., 2003). Each type of visual
information offers certain benefits in various
maintaining situation awareness and grounding conversation. Video conferencing
systems usually focus on the participants’ heads and upper bodies, which affords limited
benefit in attaining situation awareness and common ground. Research on workspace
oriented video systems that provide input on task objects and work environment/context
suggests it is likely to be more useful in supporting SA and conversational grounding
((Fussell, Setlock, & Kraut, 2003; Fussell, Setlock, & Parker, 2003).
Investigations comparing different shared fields-of-view revealed that a scene-
oriented view of the workspace was more conducive to performance than headmount
video. In a study examining distributed participants performing a collaborative physica
task (Kraut et al., 2003) the use of head-mounted video did not seem to aid performance
in terms of speed or accuracy. It did change the nature of communications between the
participants, allowing for use of deictic references to task objects. However, the
limitations of the field-of-vision provided by head-mounted video led to more queries
designed to establish a shared field of view. In a follow-up study comparing the effects o
head-mounted video to workspace-oriented video, the scene oriented video was fo
23
such
-
on
h
rs
cond
that
a)
be superior in terms of task completion, communication efficiency and user ratings of
work quality, suggesting that providing remote helpers with a wide-angle, static view of
the workspace was most valuable (Fussell, Setlock, & Kraut, 2003). The researchers
noted the difficulties in gaining the advantages of static cameras in mobile settings
as emergency telemedicine or remote repair. Finally, in a related “gaze” study using eye
tracking technology during a collaborative physical task (Fussell, Setlock, & Parker,
2003), results indicated that the remote helper’s gaze was most directed at task (task
objects, pieces/tools, and worker’s hands), i.e. targets relevant to gathering information
about steps to be completed and task status.
Additional studies conducted by Gergle, Kraut, & Fussell suggest that a comm
visual referent facilitates the development of shared mental models of what each other
knows or assumes of the situation, which is critical for team members to coordinate
activities necessary to accomplish team goals. They reported that pairs with shared visual
space in a collaborative puzzle task performed more quickly and accurately, and were
more likely to use deictic references, and less likely to explicitly verify their actions wit
speech; that is, they relied on observed actions to provide the necessary communicative
and coordinative cues (Gergle, Kraut, & Fussell, 2004a; Gergle et al., 2004b). They
examined a collaborative puzzle task manipulating the extent to which team membe
viewed the same work area – a remote assistant had access to the same view, a 3-se
delayed view or no view. They also manipulated various features of the visual space,
such as proportion of shared field of view, spatial perspective, and color drift. Dyads
had a simultaneous shared view performed about 1/3 faster than did the dyads in the
delayed view or no view condition. Results showed that having a shared visual space
24
hen
the information is visually complex and n vocabulary for describing the world is
pants. Sequential analysis of conversational discourse between team
membe
rlier,
they had a shared visual space (though these results
vary de of
d the
remote
om
ies
ffect
helps team members understand the current state of the task, and enables them to
coordinate activities and communicate efficiently, and b) is even more important w
o simple
available to partici
rs revealed that they used the visual information in two ways: 1) as a more
efficient, less ambiguous source of confirmation (thus team members are less likely to
verify w/speech) and 2) for coordination cues, e.g. team members can detect errors ea
and remedy them before their actions become nested and more difficult to untwine.
Overall, these studies show that dyads working together on a collaborative task
were 30% faster on average when
pending on the features of the shared visual space), and that the availability
shared visual space supports communication by creating a shared awareness of the task
state and by serving as an efficient resource for conversational grounding. In each of
these studies, one of the participants is operating in the task environment and the other is
assisting remotely in the process. In robot-assisted search, both the robot operator an
tether-handler are removed from the task environment—the robot serves as the
presence in the environment for both of them. The robot operator teleoperates the robot
using the Operator Control Unit (OCU), which provides audio-visual information fr
the robot’s camera as the robot moves through the remote environment. The tether-
handler can manipulate the robot grossly through manuevering the tether, and can
sometimes physically see the robot when it is first inserted into the void, but mostly rel
on communication with the operator to participate in the task. My interest is in how
providing both team members with a common view of the remote work space may a
25
team pr
it
al
es not
environ s
e their
e
ocesses and performance in robot-assisted team tasks. This work extends the
exploration of shared visual presence into a real world domain application. Moreover,
provides a rare realtime observation of human-robot interaction in a work setting.
Robots
What is a Robot?
The term robot came from Karl Capek’s 1921 play R.U. R. (Rossum’s Univers
Robots). It was used to describe a race of menial workers, “artificial humans” created
from a vat of biological parts to serve as slave labor for real humans. Science fiction
books and movies transformed robots into mechanical creatures, and propitiated their
menial stance by portraying them as factual-minded automatons that mimicked human
qualities without understanding.
In reality, an intelligent robot is a mechanical creature which can function
autonomously and interact with its world (Murphy, 2000). Intelligence implies it do
perform in a mindless fashion, while autonomy means it can adapt to changes in the
ment (or itself) and continue to reach its goal. It is important to note that a robot’
goals are generated by a human, not by the robot itself. This is a critical distinction
pointed out by Clancey (2004) that restricts one from referring to a robot as a
collaborative team member. Brooks (2002) defines two principles that distinguish robots
from computers: situatedness and embodiment. Robots are situated in that they are
embedded in the world, and interact with the world through sensors which influenc
behavior. They are embodied in that sense of having a physical body that experiences th
world in part through the influence of the world on that body. Like computers, robots
26
re
n the worlds of entertainment, work and everyday life.
nally been used for the three D’s: dull, dangerous or dirty
g,
e
ut
arch Projects Agency (DARPA). Mobile robots have
manitarian concerns, and are the primary focus in
search
n
g
ator has
supervision). Others have built upon the notion of shared control, where the robot does
have evolved from research laboratories and military/industrial applications, and a
rapidly gaining a presence i
Robots have traditio
work. Industrial robots have been developed for economic reasons in manufacturin
agriculture and service industries, to increase productivity and reduce inefficient human
resource allocation, particularly in hard-to-staff menial labor positions. Because th
original goal was precision and repeatability for use in mass production, little effort was
put into machine intelligence or human factors considerations. As the space program
evolved, the need for artificial intelligence, i.e. robots capable of learning, planning,
reasoning and problem-solving, spurred research sponsored not only through NASA, b
also by the Defense Advanced Rese
developed more from safety and hu
nuclear, space exploration, military and rescue applications. This study is directed toward
human-robot interaction with mobile robots. While the pervading notion in past re
has been the substitution of robots for people, the current trend is toward robots as
assistive technology, i.e. designed to complement humans rather than replace them.
The current state of the art in mobile robots is situated autonomy (the robot acts
on its own using information from its sensors), though teleoperation is more common i
practice. Teleoperation is when a human operator controls a robot from a distance usin
sensors and a display. (This differs from remote-control operation, where the oper
visual contact with the robot). Some applications have moved to semi-autonomous
control, where the robot is given an instruction or task to do on its own (but under
27
ore
obotic
by cognitive functionality but rather according to the
domain
into
, and
with a
the
tric of
Yanco,
the dirty work and the human does that which requires finesse. Certainly there are m
autonomous applications in the commercial sector (Honda’s Asimo, Sony’s Aibo r
dog), but systemic problems have slowed the rate of development in military and
governmental application.
Human-Robot Interaction
Human-robot interaction is a relatively new field. Studies in human-robot
interaction are often categorized not
functionality of the robots in question: industrial, professional service, or
personal service robots (Thrun, 2004). Urban search and rescue (US&R) robots fall
the professional service category, along with those in the medical field, the military
space applications (e.g., Ambrose et al., 2000; Endo, MacKenzie, & Arkin, 2004; Pineau,
Montemerlo, Pollack, Roy, & Thrun, 2003), where robots are intended to work
human to meet the human’s goals.
Human-Robot Team Performance
Studies concentrating on improving task performance in human-robot teams
appear to be universally robot-centric, testing either the robot’s abilities or some
component of the robot as a factor influencing performance, and ignoring the role of
human. This provides no insight into the larger human-robot team. Professional service
robots assist people in attaining their professional goals; therefore, a logical me
human-robot team performance is whether the team’s goal is achieved. However, the
question of how to measure human-robot interaction in terms of team performance is
largely ignored, with the notable exceptions of Bruemmer et al. (2005); Marble,
Bruemmer, & Few (2003); Nourbakhsh et al., (2005); Scholtz, Young, Drury, &
28
or
-focused, this study relies on methodologies taken from the psychology
commu
ke,
d
environ
s
and
the
(2004); Yanco, Drury, & Scholtz, (2004). Since these methods are generally usability-
evaluation
nity and encapsulated in the RASAR-CS scheme described in Chapter 3.
To date, no one has attempted to examine human-robot team performance from
the “human” side, i.e., through investigation of team processes. Situation awareness and
team processes have been identified as important elements needed for effective
communication and coordination of activities in robot-assisted US&R operations (Bur
Murphy, Coovert, & Riddle, 2004).
The annual Robocup Rescue Competition has been used to compare human-robot
teams’ performance in a rescue-oriented domain (Scholtz et al., 2004; Yanco et al.,
2004). However, the physical setting and conditions are quite different from those
experienced at a disaster site, and the robots used are not fieldable, i.e., they are designe
for short competition rounds in the NIST testbed rather than for a true disaster
ment (Murphy, Blitch, & Casper, 2002). Moreover, the people on these teams are
robot developers rather than rescue professionals, and have neither the training nor the
skills of the intended end-users. Nourbakhsh et al. (2005) have created a simulated
US&R environment in which humans, agents and robots operate in high-fidelity game-
generated simulations of the NIST US&R test arenas, allowing them to test various
configurations of heterogeneous agents and robots in actual Robocup competitions as
well. Marble, Bruemmer and associates at the Idaho National Engineering and
Environmental Laboratory have measured human-robot team performance in search task
conducted in laboratory settings and in field locations, varying the levels of control
autonomy (Bruemmer et al., 2005; Marble et al., 2003). These experiments, by far
29
l
end-users in ecologically valid disaster response
t
red
phy,
ies,
a
most advanced in terms of experimental design and analysis, are gauged toward
evaluation of robot systems characteristics, and participants are typically high schoo
students.
Field studies conducted with true
settings offer a more realistic look at human-robot team performance in terms of curren
capabilities. Four field studies conducted by the Center for Robot-Assisted Search and
Rescue (CRASAR) prior to this study recorded real robot-user interaction as it occur
between team members and a single robot to inform the development of coordinated
human-robot systems within the organizational structure of US&R (Burke & Murphy,
2004b; Burke, Murphy, Coovert et al., 2004; Casper & Murphy, 2002; Casper & Mur
2003). In each of these, situation awareness (Endsley, 1988) is a key construct for
understanding (and improving) human-robot interaction. In the two most recent stud
team processes, particularly communication between team members, have emerged as
critical to the development of situation awareness in robot-assisted team tasks.
The first CRASAR field study was an ethnographic study of Florida Task Force 3
members using robots to search for a victim (Casper & Murphy, 2002), while the second
was an analysis of data collected during the use of rescue robots at the World Trade
Center disaster (Casper & Murphy, 2003). The Florida Task Force 3 study suggested that
two operators are needed to interpret multiple sensor data while navigating due to the
simultaneous nature of activities described as part of the technical search task (searching
for victims and structural inspection). Casper and Murphy’s (2003) analysis of video dat
collected during the World Trade Center disaster response found that operators’ lack of
awareness regarding the state and situatedness of the robot in the rubble impacted
30
h process. In Burke et al. (2004), team members created
shared mental models (mutual knowledge) of the search space through talking about the
environment, the robot’s situatedness in that environment, and search strategy. Operators
who reported to team members about the environment being searched by the robot and
search strategy were rated as having better situation awareness (Burke & Murphy, 2004b).
If the tether-handler or another distributed team member had access to the same robot’s
eye view, the effort required to establish mutual knowledge, or common ground, would
be far less. This is important because analyses comparing the performance of operators
performance of human-robot teams. Operators also had difficulty linking current
information obtained from the robot to existing knowledge or experience. Both the
Florida Task Force and World Trade Center human-robot interaction studies reveal
difficulties in operator teleproprioception and telekinesthesis, consistent with the
problems described in Sheridan (1992).
Situation Awareness and Team Processes. In the two most recent field studies
(Burke & Murphy, 2004b; Burke, Murphy, Coovert et al., 2004) conducted with 33 teams
(robot operator-tether handler), communication analyses revealed that 50 – 60 % of
operator communication during a technical search task was related to building and
maintaining situation awareness. One of the challenges presented was the fact that the
robot operator was cognitively overloaded; he couldn’t drive and look at same time.
Because the tether-handler did not share the same viewpoint, the operator alone had to
interpret the robot’s eye view, and used talking with the tether handler (who might
sometimes have an external view of the robot) to build common ground so that the tether-
handler could assist in the searc
31
rated as having good or poor situation awareness in a victim search scenario revealed that
those with high SA were 9 times as likely to locate the victim as those with low SA.
This chapter has covered a great und. Teams, distributed teams, and
distributed teams tion and
e)
rse
a team’s capacity, enabling the creation of richer shared mental models
ng tive
deal of gro
in US&R have been described, as have the communica
coordination challenges in distributed teams. The concepts of situation awareness, shared
awareness, common ground, and shared mental models (all variants on a similar them
are introduced, and the research on shared visual presence as a conduit for these
constructs is reported in detail. Finally, robots, human-robot interaction, and relevant
studies on human-robot interaction in US&R are presented and discussed. These dive
topics all play a part in the current study, which is described more fully in the coming
chapters. The current study is a theoretical piece, testing portions of Klimoski and
Mohammed’s model of team performance (1994). RSVP is presented as a team resource
that increases
amo team members. These richer mental models can, in turn, foster more effec
team processes and enhance team performance. The approach for testing the model
(which includes the creation of a more context- and task-specific model—a submodel, if
you please) is now explained.
32
ses and
el
present
ting
a
Chapter 3
Mobile Robots as Shared Visual Presence: Approach
This chapter outlines the proposed model of team performance for human-robot
teams in technical search, and presents specific hypotheses. The research questions
addressed in this study are as follows: Does using the robot as a remote shared visual
presence affect robot-assisted team performance? Team process? Can this remote shared
visual presence facilitate the performance of human-robot teams where human team
members are distributed? The model proposed in this study posits that use of a remote
shared visual presence (RSVP) by team members will lead to creation of richer team
mental models. These richer team mental models will in turn enhance team proces
performance.
As described in Chapter 2, there are well-established theoretical models
circumscribing the role of shared mental models in team performance. Kraiger & Wenz
(1997) place shared mental models within a nomological net of determinants and
outcomes, along with measurement implications. Klimoski & Mohammed (1994)
a framework for explaining the role of team mental models in team performance (Figure
2). These theoretical frameworks are used to situate this study within the body of exis
research. Indeed, the proposed model and study serve as a test of some of the theoretical
constructs and relationships posited in these frameworks. The model presented in this
study is more specific in two ways. First, it is limited to the US&R environment, and
particular task within that environment. This is important because the dynamic, high
33
a
situation-
l.,
ight on some
of the cloudier issues related to team mental models: how do these shared mental models
r,
adership, situation awareness) and performance?
Figure 3 shows the conceptual model of team mental model formation for robot-
assisted technical search. It is a synthesis of the results of previous field studies and is
consistent with Kraiger & Wenzel’s (1997) framework for mental models, but is
contextually task-specific, and focuses on team communications as central to the
development of the team situation model. The model is explained following the diagram,
bottom upward. The human-robot team consists of two operators and one robot. The
operators are the robot operator and the tether-handler. The robot operator is the person
who directly operates the robot. The tether-handler handles the robot tether or safety line.
The input to the robot operator is data (video) from the robot processed through the
Operator Control Unit (OCU). The OCU serves as the user interface. Both the robot
stress, high consequence nature of this task in this environment adds cognitive load to
seemingly low-complexity task. The data collected is deliberately context- and
specific to inform analysis of this technology-altered task. Second, this model posits that
team communication can be used as a measure of the shared mental model (Cooke et a
2003; Kiekel, Cooke, Foltz, & Shope, 2001; Orasanu, 1990; Orasanu, 1995)—in that the
quality of the shared model is related to the presence of various indicators in team
communication: that they talk more about goal-related aspects of the task (environment,
robot situatedness, search strategy, information synthesis), engage in more planning and
reporting, and anticipate each other’s information needs and provide information without
being asked (communication efficiency). Thus, this model attempts to shed l
actually contribute to team processes (communication, backup/support behavio
le
34
k
f the void space (and at
some p
s
this study), the tether-handler is given a separate monitor that
display
operator and the tether-handler can view the OCU; the robot operator generally looks
constantly at the OCU, and the tether-handler typically only looks at the OCU
intermittently as responsibilities permit (often he is located several meters away from the
robot operator.) This means that the tether-handler has a different perspective on tas
progress and the overall situation since he can see the exterior o
oints what the robot is doing), and can feel the robot’s movements through the
tether (e.g., whether it is moving forward and drawing more line). To explore the effect
of creating a remote shared visual presence between the two team members (one of the
two dependent variables in
s the same robot’s view which appears in the robot operator’s OCU.
Figure 3. Model of team performance in robot-assisted technical team search.
35
ften the case with US&R teams)
these sh of the
problem
goes fu at fusion takes place via communication between the human
ls
(Fiore &
1990; Sonnenwald & Pierce, 2000). The formation of a team situation model is an
or awareness of a given process
explicit shared
mental
situatio ith
technic here 50-60 % of operator communications was
Each team member has a situation mental model representing his or her current
understanding of the task, system, and team member roles. Situation models extend
beyond the more static mental models of the system, task and team to represent the
dynamic, present state of the system. The model posits a team (shared) situation model as
a fusion of the two individual situation models into a common ground (Jones & Hinds,
2002). The team situation model concept follows Orasanu (1990), who suggested that
teams faced with novel situations or emergencies (as is o
must also develop shared situation models for the specific problem. Important parts of
ared situation models according to Orasanu include shared understanding
, goals, information cues, strategies and member roles. The model in this study
rther and assumes th
team members, which is consistent with research on team processes and mental mode
Schooler, 2004; Foushee et al., 1986; Kanki, Lozito, & Foushee, 1989; Orasanu,
emergent process aptly described in (Fiore & Schooler, 2004)
“…only as one articulates one’s understanding
does it truly become known to oneself and others. Thus, the act of making knowledge
facilitates the development of not only one’s own mental model but also a
model.” (p.143).
In other words, communication plays a consistent role in both the operators’
n models and the team situation model. This is supported by previous work w
al search operators and teams, w
36
2004b;
backup
perform
While the quality of the user interface will have some impact on individual SA,
this study is restricted to the formation of the team situation model through team
communication, and its influence on team process and performance. This restriction
permits the study scope to be tractable.
Hypotheses
Hypotheses are presented related to each of the two experimental conditions—the
presence of RSVP technology, and whether the teams are collocated or distributed. If the
use of RSVP has the hypothesized effect, then differences will emerge between teams
that have access to the technology and teams that do not. Based upon what is known
about distributed team performance compared to that of collocated teams, the collocated
teams will outperform the distributed teams. If using RSVP makes a difference, however,
it may close the gap between collocated and distributed teams in terms of team process
and performance. Specific hypotheses are listed below:
H1: Teams having access to RSVP technology will generate richer, more accurate
team situation models than teams not having access to the technology, as
measured by search maps generated by the team, and frequencies of SA indicators
in the RASAR-CS.
related to building and maintaining SA (Burke & Murphy, 2004a; Burke & Murphy,
Burke, Murphy, Coovert et al., 2004).
In the model, the team situation model affects team processes (communication,
behaviors, leadership/initiative, and team SA), which in turn affect team
ance.
37
H2: Teams having access to RSVP will exhibit more effective team processes, as
measured by observer ratings and by frequencies of team process indicators in the
RASAR-CS.
H3: These (RSVP) teams will accordingly perform better in the search task scenarios,
uted teams in the
search task sce utcome scores.
VP technology, then the differences between
tilize
as measured by performance outcome scores.
H4: Collocated teams will generate richer, more accurate team situation models than
distributed teams, as measured by search maps generated by the team, and
frequencies of SA indicators in the RASAR-CS.
H5: Collocated teams will exhibit more effective team processes than distributed
teams, as measured by observer ratings, and by frequencies of team process
indicators in the RASAR-CS.
H6: Collocated teams will accordingly perform better than distrib
narios, as measured by performance o
H7: If there is a main effect for use of RS
collocated and distributed teams will be significantly less in the teams that u
RSVP than in those that do not.
38
R
t it
nd
d-- multilevel regression.
Setting, Participants, and Apparatus
Setting
h
Chapter 4
Method
This chapter describes the details of the study. Due to the cost and rarity of US&
field exercises, the data used is archival. The problem with archival data analysis is tha
is constrained, in that you can’t always ask more questions. However, this data was
collected with a goal in mind, and therefore is presented as a quasi-experimental field
study with great attention and effort directed toward addressing threats to the validity a
reliability of the results. The following sections discuss the participants, apparatus, and
setting, the quasi-experimental design, measures used, and procedures followed. The final
section describes the type of analysis use
Data collection took place in two different locations during US&R training
exercises. The first exercise was conducted in May, 2004 at NASA-Ames Research
Center in Menlo Park, CA. NASA-Ames is home to one of the most advanced Urban
Search and Rescue teams in the country. NASA's Disaster Assistance and Rescue Team
(DART) was organized almost twenty years ago in the agency's attempts to comply wit
disaster preparedness regulations for federal facilities. DART comprises nearly 140
personnel from the Research Center who have been trained and certified in a variety of
emergency response and recovery systems and techniques. Most its members are also a
part of California US&R Task Force 3. The training exercise that took place was a
39
rt
vited to
r Station at
urst in Toms River, New Jersey and has leased property from the Federal
Govern s of
s) chosen through
convenience sampling of the intended end-user population (US&R task force personnel)
via their participation in scheduled training exercises conducted at NASA- Ames, Moffett
Field, CA, and Lakehurst NAVAIR, Lakehurst, NJ. Study participants at the NASA-
Ames site included 13 responders from other parts of the country who came to participate
in the training exercise with DART members. All study participants at the New Jersey
site were members of NJTF-1. Participant demographics were collected, including age,
gender, and relevant experience. The majority were males (90%) between the ages of 35-
54 (76%). Participants received no payment for their participation; it was viewed as a
training accustomed to putting in long
“Technology Meets Responder” event in which various new technologies were
introduced and used by the responders during the course of the training exercise. As pa
of the event, approximately 30 responders from all over the United States were in
attend and participate with DART members in the exercise. The second exercise was held
in February, 2005 at the Lakehurst Naval Air Warfare Center in Tom’s River, NJ. New
Jersey Task Force-1 (NJTF-1), the state’s own Urban Search & Rescue Task Force, is
based there. The team consists of career and volunteer fire, police, and EMS personnel
from all 21 counties in New Jersey. The task force is quartered at the Naval Ai
Navy/Lakeh
ment at Navy Lakehurst to support the logistical and training operational need
the team. This training exercise was a similar “Technology Meets Responder” event, with
participants drawn solely from NJTF-1.
Participants
Participants were 62 men and women (31 2-person team
experience, and US&R task force members are
40
hours of unpaid training just to attain/ma ir certification as US&R teams. The
particip
rticipants not being able to complete both runs. Power analysis for this study
showed
equipped with a color CCD camera on a tilt unit and two-way audio through a set of
ers on the robot and Operator Control Unit. The operator is given
basic c
intain the
ants knew that we were collecting data for research purposes, but they did not
have specific information about what we were studying other than human-robot
interaction. The final sample used in this analysis was reduced to n=50 (25 teams), due to
some pa
that to attain power = .80 in a single-group repeated measures design at alpha =
.05, given an estimate of r = .80 as the average correlation for the repeated measures, a
total of 15 teams are required to detect a medium size effect (Stevens, 2002).
Apparatus
The robot systems used in the study were Inuktun Micro Variable Geometry
Tracked Vehicle (VGTV) robots. Each robot system consists of a small, tracked platform
microphones and speak
ontrol capability: traversal, power, camera tilt, focus, illumination, and height
change for the polymorphic robot (Figure 4.)
Figure 4. Inuktun Micro-VGTV robot system.
41
le
sence (RSVP), described
elow:
Location (collocated vs. distributed teams). The robot operator and tether-handler
y-si di se visually in the other. Due to the
constrain arating walls or corners on the pile), for the distributed
artic e told to face away from each other (see Appendix A - Task
o Instruc peci ). T ers in the distributed condition
were allowed to talk, but not use gestures, eye contact, or other behaviors associated with
working side-by-side.
era connected to the OCU; in the other task, only the robot operator has
access
nd remote
as poss
s
Design
This is a 2 x 2 repeated measures design, partially crossed, counterbalanced for
location and remote shared visual presence. Each team participated in 2 of the 4 possib
conditions. The 2 IVs are location and remote shared visual pre
b
work side-b de in one con tion, and are parated
setting ts (no sep
condition p ipants wer
Scenari tions for s fic wording eam memb
Remote Shared Visual Presence (RSVP or no RSVP). In one condition, the robot
operator and tether-handler have access to the same visual image (robot’s view) via a
second DV cam
to the visual image.
In order to determine the effects of the independent variables (location a
shared visual presence) on the various dependent variables (shared mental models, team
process, performance), efforts were made to limit or control any extraneous or nuisance
variables that might influence the results. For example, testing conditions were as nearly
ible the same for each team. A standardized protocol for running each team
through the experimental task was developed and followed. The instructions were given
the same way each time; runtimes and observations were consistent and accurate in term
42
Table 1
Distribution of teams across experimental conditions
Collocated Totals
of detailed documentation. Experimenters were friendly, but minimized conversation
with participants during the tasks, and avoided discussing the experiment, extraneous
information about the robots or the research, etc. The number of teams nested in each
combination of experimental conditions is shown in Table 1.
Condition Distributed
RSVP 15 teams 14 teams 29 teams
No RSVP 10 teams 11 teams 21 teams
Totals 25 teams 25 teams 50 teams
Measures
Measures used in this study include an initial demographic survey questionnaire
administered to individual participants, and dependent variable instruments assessed at
the team level.
Survey questionnaires were designed to collect demographic data from
participants regarding their background prior to the task. Participants provided
information about age, gender, years of experience in firefighting, US&R, mil
self reported skill with various technologies (Appendix B). Age and years of experien
itary, and
ce
were codified by range, and technology skill was rated on a 1 (low)-5 (high) Likert scale.
e,
am processes, and shared mental models. Team performance is measured as an outcome
ch task scenario. Team processes are measured through
The dependent variables in this study fall into three categories: team performanc
te
score on a confined space sear
43
onsite o earch
ert et
Team Performance Measure
Four visual cues were placed in each of the search spaces. These target objects
were mannequin pieces (upper/lower arm, lower leg, hand, foot, nose/mouth portion of a
face) and indicators of possible human presence (clothing pieces, webbing strips used by
firefighters.) These objects were placed in varying degrees of visibility—some were lying
in plain view in the rubble, others were partially buried under the rubble (Figure 5).
Teams scored 3 points for each target object located during a run (0-12 possible points).
Teams were not penalized for errors in identification (e.g., if they identified a mannequin
piece as part of an arm when it was actually a lower leg), and received an extra .5 point
for location of significant target cues that were not hidden in the searchspace as part of
the task. Examples: finding 2 pennies, a pen. These are important because teams were
instructed to note anything that might indicate human presence in the rubble.
Identification of non-human related debris (big rocks, wood, rebar) was not scored.
bserver ratings and coding frequencies obtained from the Robot-Assisted S
and Rescue communication coding scheme (RASAR-CS) (Burke, Murphy, Coov
al., 2004). Shared mental models are assessed using a spatial map construction measure
and coding frequencies from the RASAR-CS. The measures and RASAR-CS
communication coding scheme used to assess each of these variables are described
below.
Figure 5. A mannequin hand hidden in the search space serves as a visual cue (as seen through the robot operator’s OCU).
Team Process Measures
Team process variables are communication effectiveness, support/backup,
adaptation of the team performance dimensions used in the TADMUS (Tactical Decision
in the early 90’s (Smith-Jentsch, Johnston, & Payne,
ngs on the Team Process dimensions as
shown in Table 2.
leadership, and team situation awareness. These variables are measured using an
Making Under Stress) program
1998). These are operationally defined as rati
44
45
Table 2
Team process dimensions
Team Process Dimension Facet
Communication Clarity of communications Degree of non-task related communications
Monitor each other for overload and needed
Overall support effectiveness
Leadership Clear agreement on priorities
Overall communication effectiveness
Support Prompt correction of team errors
assistance
Guidance and feedback for decision making/problem solvingOverall team leadership
areness Used all available sources of information
Passed information to right person without being asked Overall level of team SA
Situation Aw
Provided situation updates
Team process ratings were made by an onsite observer during each run, utilizing a
1-4 point Likert scale (low-high). In addition to the separate dimension ratings, a global
team process rating is formed from the mean of the four dimension ratings to be used as a
dependent variable in part of the analysis process. Other indicators of team process are
drawn from the RASAR-CS through analysis of communication dyad, form, content and
function.
Shared Mental Model Measures
The shared (team) situation model is measured in two ways: through pertinent
statement frequencies/proportions (SA indicators) rendered through RASAR-CS coding,
and through a spatial map created by the team onsite during the exercise showing the
46
Likert
n
ss:
fficient labeling of details, and notations regarding start point and path
aveled.
RASAR-CS
Team interactions are classified using the Robot-Assisted Search and Rescue
Coding Scheme (RASAR-CS). The RASAR-CS (Table 3) draws on the FAA’s
Controller-to-Controller Communication and Coordination Taxonomy (C4T) (Peterson,
Bailey, & Willems, 2001), which uses verbal information to assess team member
interaction from communication exchanges in an air traffic control environment. Like the
C4T, the RASAR-CS is domain-specific, capturing not only the “how” and “what” of
US&R human-robot team
member situation awareness.
RASAR-CS addresses the goals of capturing team process and situation
awareness by coding each statement on four categories: 1) conversational dyad: speaker-
recipient, 2) form: grammatical structure of the communication, 3) content: topic of the
communication, and 4) function: intent of the communication.
Team processes are examined using the dyad, form, function and content categories to
results of their search. Each team produced a hand-drawn map of the search as part of the
task scenario. These maps are scored on similarity/accuracy/coverage using a 1-4
scale (1=Low, 4=High.) Maps were scored by a subject matter expert familiar with the
task scenario and with US&R search protocols. The SME was provided with a schematic
of the search space containing “ground truth” as to the placement of the mannequi
pieces in the voidspace. The following points were considered in the rating proce
indications of time, scale or compass direction, use of 2D and 3D representation, level of
detail and su
tr
s, but also the “who,” as well as observable indicators of team
47
e which team members are interacting and what they are communicating about.
nt and function
dicators of the f n aw
other elements as indicators of the highest level of situation awareness, planning,
rojection and pro unction
ere ge Q-sort tech
l. (2004).
determin
Team situation awareness is explored using elements of the conte
categories. For example, when statements are coded for content, certain elements serve as
in irst two levels of situatio areness, perception and comprehension;
p blem-solving (Figure 6). Elements in the content and f
categories w nerated using a nique (Sachs, 2000), as reported in Burke et
a
Figure 6. SA rs in the RASAR-CSindicato .
48
Table 3
Robot-Assisted Search and Rescue Coding Scheme (RASAR-CS) Category Elements Definitions
Sender/Recipient 1. Operator-Tether Handler Operator: individual teleoperating the robot 2. Tether Handler-Operator Tether Handler: individual manipulating the tet
and assisting operator with robot
specialist
5. Tether Handler-Researcher 6. Researcher-Tether Handler 7. Operator-Other Other -individual interacting with the operator w
is not a tether handler or researcher
9. Tether Handler-Other 10. Other-Tether Handler Statement Form 1. Question Request for information
3. Answer Response to a question or an instruction
not a question, instruction or answer Content 1. Environment Characteristics, conditions or events in the search
environment
3. Robot situatedness Robot’s location and spatial orientation in the environment; position
4. Information synthesis Connections between current observation and priobservations or knowledge
6. Navigation Direction of movement or route
her
3. Operator- Researcher Researcher: individual acting as scientist or robot
4. Researcher-Operator
ho
8. Other-Operator
2. Instruction Direction for task or activity
4. Comment General statement, initiated or responsive, that is
2. Robot state Robot functions, parts, errors, capabilities, etc.
or
5. Victim Pertaining to a victim or possible victim
7. Search Strategy Search task plans, procedures or decisions
,
2. Plan Projecting future goals or steps to goals
cise rming a previous statement or observation
6. Convey uncertainty Expressing doubt, disorientation, or loss of
7. Provide information Sharing information other than that described in ing
members
8. Off task Unrelated or extraneous subject
Function 1. Report Sharing observations about the robot, environmentor victim
3. Seek information Asking for information from someone 4. Clarify Making a previous statement or observation more
pre 5. Confirm Affi
confidence in a state or observation
report, either in response to a question, or offerunsolicited information
8. Correct error Correcting errors made by self or team
49
nvolve
ns between individuals. Three dyad codes classify statements made by the
operato
er -
d in the
nd
be a
g
scribes the topic of communication, and consists of eight
elements: 1) statements related to robot arts, errors, or capabilities (robot
tate), 2
Speaker-recipient dyad codes were developed based upon the anticipated
roles/individuals present in a US&R environment (Table 3). The primary dyads i
the operator and tether-handler (the person manipulating the robot’s tether during
teleoperation), operator/tether-handler and researcher, or operator/tether-handler and
another person not involved in the scenario. Ten dyads were constructed to describe
conversatio
r to another person: operator-tether handler, operator-researcher, or operator-
other. Similarly, three codes classify statements made by the tether-handler to others:
tether-handler - operator, tether-handler -researcher, tether-handler -other. The
remaining four classify statements received by the operator or tether-handler from
another person: researcher-operator, other-operator, researcher- tether-handler, oth
tether-handler. In this study there were only 55 statements (approximately 0.5%) that
included the category element ‘other’; therefore these statements were not include
analysis. Verbalizations between individuals which did not include the team members
were not coded (e.g., communications between members of the research team).
The form category describes the grammatical structure of the communication, a
contains the elements: question, instruction, comment or answer. (A statement can
whole sentence, or a meaningful phrase or sentence fragment.) Statements not matchin
these categories are classified as undetermined.
The content category de
functions, p
s ) statements surrounding the robot’s location, spatial orientation in the
environment, or position (robot situatedness), 3) statements describing characteristics,
50
tions and prior observations or knowledge
hesis), 5) statements concerning the victim (victim), 6) indicators of
directio ans,
ask
, and is
om
recise
or offering unsolicited information (provide information), and 8) correcting
errors m
the data collection process is explained. This is followed by a detailed description of the
conditions or events in the search environment (environment), 4) statements reflecting
associations between current observa
(information synt
n of movement or route (navigation), 7) statements reflecting search task pl
procedures or decisions (search strategy), and finally 8) statements unrelated to the t
(off task).
The function category is used to classify the purpose of the communication
comprised of eight elements: 1) sharing observations about the robot or environment
(report), 2) projecting future goals or steps to goals (plan), 3) asking for information fr
someone (seek information), 4) making a previous statement or observation more p
(clarify), 5) affirming a previous statement or observation (confirm), 6) expressing
doubt, disorientation, or loss of confidence in a state or observation (voice uncertainty),
7) sharing information other than that described in report, either in response to a
question,
ade by self or other team members (correct error). The function elements of
report and provide information merit explanation, as they appear very similar. Report
involves perception and comprehension of the robot state, robot situatedness, the
environment, information synthesis, or the victim. Any other information shared by a
team member, in answer to a question or on his or her own, is classified as provide
information (e.g., navigation).
Procedure
This section describes the procedures used in data collection and analysis. First,
51
ating, and
Data C
e.
report to one of the two search
exercis
ed a
ot
f
e
e
data analysis process, including data editing and preparation, data coding and r
application of statistical procedures.
ollection
Prior to performing the search task exercises, participants completed a pre-
training survey and received 1.5 hours basic robot awareness training. This training
included an opportunity to observe and operate the robots in open, visible spac
Participants were organized into 2-man teams for the search task exercises. Some
participants chose their own teammate, but most were assigned by the researcher based
on where they were sitting during the training. Each team participated in 2 20-minute
confined space search task scenarios over a 2- day period. Teams were assigned a
treatment condition by the experimenter and a time to
e locations on the pile. (Other training activities were going on during the data
collection process, so participants were involved in other events throughout the 2-day
period.)
Human-robot teams (2 people: 1 robot) were videotaped as they perform
technical search task to capture how the robot operator and tether-handler used the rob
to search for signs of victims. The method of data collection was a modified version o
the procedure used in (Burke, Murphy, Coovert et al., 2004). A team of 3 researchers
(Experimenter, Videographer, Team Process Rater) used 2 Sony digital videocameras to
record the teams as they performed each of the 2 search task scenarios (Figure 7). On
camera was attached to the robot’s Operator Control Unit to record the view through the
robot’s camera. This camera was also used by the tether-handler as the RSVP in 2 of th
52
experimental conditions. A second camera was held by the Videographer to
simultaneously capture a view of the operator and the tether-handler.
The Experimenter explained the task scenario to each team, answered any
questions about robot capabilities/operation (but NOT about search strategy, navigation,
victim location or other mission related information) and made sure that conditions were
administered correctly according to the data condition assignment sheet. The
Experimenter also recorded critical events (such as finding a mannequin piece) during
each run. The Videographer filmed each team as they ran the scenario. The Onsite Rater
observed and rated each team using the team process rating measure described above, and
collected the maps generated by each team. All three experimenters monitored each
others’ tasks to make sure controls were not violated.
In the first run, the participants self-organized to use the robot, i.e. they decided
r
ndix A for the specific instructions). During
participants switched roles, and again were given standardized
instruct
ng
id not complete the feedback surveys onsite. These participants
receive
ed a
4
who would fill each role, and were given standardized instructions to search the void fo
victims or signs of human presence (see Appe
the second run,
ions regarding the search task. Upon completion of the second task scenario,
participants returned to the classroom and completed a post-training feedback survey
containing individual ratings of effectiveness, usefulness, ease of use, satisfaction, alo
with self ratings on the team process variables described in the Measures section. In a few
cases, participants d
d a followup letter and survey to complete and return by mail. Complete survey
data was obtained for each team member included in this study. The 50 runs yield
total of 16 hours, 40 minutes of videotape for analysis.
53
Figure 7. A 3-person team of researchers manned each task scenario site. (From left to right: Videographer g
explain
prepares to film a team; tether-handler looks downward into voidspace; Onsite Rater readies her ratinsheet; robot operator is obscured behind Onsite Rater; Experimenter oversees preparations; onlookersobserve prior to the next run.)
Data Analysis
Data analysis provides a way of organizing the data for the study. This section
s the three steps of the data analysis process: data editing and preparation; data
coding and rating; and application of statistical procedures. The section on statistical
analysis includes a detailed description of the multilevel regression modeling technique
applied to the data.
Data editing and preparation. The data analysis process began with data editing
and preparation, which time synchronized the robot’s camera videotape with the
matching operator videotape to produce a side-by-side video recording of the robot and
54
gs
l
ed descriptions of the disaster
rill and data collection procedures, and then reviewed definitions for all the codes.
al examples selected from other data sets (video recordings) were reviewed, and
enhance reliability. The
m en tat in
A fixed number of statements was established (9,954 statements across the 25
teams) to be coded. Raters coded each statement across four categories: dyad (speaker-
recipient pair), form (grammatical structure of the communication), content (topic) and
function (intent of the communication). Ten of the 50 team observations (20%) were
coded by all four raters for reliability (n = 1,486 statements). Both raw agreement indices
and int 4.
t, it
the operator manipulating the robot. The discourse between team members during the
search task scenario was transcribed to produce a fixed number of meaningful statements
for coding. Editing and transcription took approximately 6 man-months. These recordin
and transcriptions were then used to code statements made by team members with the
Observer Video-Pro (Noldus, Trienes, Hendriksen, Jansen, & Jansen, 2000) behaviora
analysis software.
Data coding. Data coding is the next step in the process: raters were trained in
using the coding system, a fixed number of statements to be coded was decided upon, and
the actual coding required approximately 280 man-hours. Coders were three graduate
students (drawn from the psychology, business, and computer science departments) and
one undergraduate psychology student who were trained by the author to code the
videotapes. During 10 hours of coder training, raters review
d
Behavior
coding guidelines were developed to reduce ambiguity and to
ajority of the training c tered on coding s ements together and reach g consensus.
errater reliability estimates are reported for the four coding categories in Table
While Cohen’s Kappa (1960) is the most frequently used coefficient of rater agreemen
55
hich
ter
ices and/or Brennan and Prediger’s Kappan
(1981) ng the
tent
ate to
is not so much a measure of raw agreement as it is of improvement in agreement beyond
chance. Cohen’s Kappa is based upon a main effects model of rater independence, w
only takes into account the differences in frequencies with which raters used different
rating categories. Brennan and Prediger’s Kappan is a variant of K based upon the null
model, which solves the problem created when raters use categories at different rates
(maximum agreement is reduced when this occurs). The most current research on ra
analysis suggests reporting raw agreement ind
in addition to the more traditional Cohen’s K in the interests of best capturi
magnitude of agreement (Von Eye & Mun, 2005). The lower estimates for the con
and function categories of the RASAR-CS reflect the difficulty in orthogonal coding of
dimensions with multiple elements. However, ratings in all categories exhibit moder
excellent agreement (Fleiss, 1981; Landis & Koch, 1977).
Table 4
Raw agreement, Kn, and K for the four RASAR-CS categories
RASAR-CS Category Raw Agreement Brennan & Prediger’s Kn Cohen’s K
Dyad .84 .80 .71
Form .71 .61 .56
Content .52 .46 .43
Function .44 .42 .36
Statistical Analysis: Multilevel Regression. Finally, statistical procedures are
applied to the data. In addition to standard descriptive and correlational analyses, this
56
teams across levels of experimental conditions, and the nature of the data, which
were co s
le
here
in schools; in cross-cultural studies, where individuals are
nested
e
study uses multilevel regression analyses to examine the effects of location and RSVP on
team mental models, team processes and team performance. Multilevel analysis is
appropriate for this data for two reasons: the complex nesting structure of repeated trials
within
llected at both individual and team levels. To clarify how multilevel regression i
applied here, a brief overview of multilevel modeling is presented, followed by an
example using variables in the current study.
Multilevel modeling is a type of statistical analysis designed to work with
hierarchical or cross-classified data. Psychological research often asks questions
involving the relationship between individuals and groups, based upon the assumption
that individual persons are influenced by the properties of the groups to which they
belong. Similarly, groups are thought to be influenced or shaped by the individuals that
comprise the group. This creates a hierarchical system with individuals and groups at
different levels, and variables describing each at these levels. Observations from
individuals examined within a group context often violate the assumption of
independence made in traditional statistical models. Multilevel analysis draws samp
data from each level of a hierarchical population, and uses variables at higher levels to
adjust the regression of lower level dependent variables on lower level explanatory
variables (Hox, 2002). Examples of multilevel research can be found in education, w
students might be nested with
within units by nationality or ethnicity; and in business, where individuals are
nested within departments in organizations. Multilevel analysis can be applied to less
obvious data structures, e.g. repeated measures studies, where events/observations ar
57
nt
nts, or
perimental conditions, and
sites.
ysis
ple
vel also
-
al levels, to examine the direct
effects
me
nested within individuals; or meta-analyses, where individuals are nested within differe
studies altogether. In this study, observations are nested within pairs of participa
teams. These observations/teams are in turn nested within ex
Multilevel analysis offers statistical and conceptual advantages over traditional
statistical analysis. Traditional statistical tests tend to aggregate lower level variables’
data into a smaller number of hierarchical units or conversely, to disaggregate the higher
level unit into data on a lower level. In the first case, information is lost and the anal
loses power. In the latter, the smaller number of grouping units is expanded into multi
values for a larger quantity of individual units, which are then treated as independent
information from that population of units. This can result in misleadingly significant
outcomes. Conceptually, analyzing hierarchical or nested data all at one level makes it
easy to commit the ecological fallacy of assuming inferences made at the group le
hold at the individual level (Robinson, 1950); or to incorrectly form inferences about
higher level constructs based upon lower level data, known as the atomistic fallacy (Diez
Roux, 2000). The ultimate goal of multilevel analysis is to answer questions about
relationships between variables at different hierarchic
of individual and group level explanatory variables on an outcome of interest, and
to see if group level variables moderate individual level relationships by testing for
interactions between levels.
A multilevel regression model (aka random coefficient model, hierarchical linear
model, variance component model) assumes a hierarchical data set, with a single outco
variable measured at the lowest level, and explanatory variables at all levels. In this
58
ata
oint
also entered at this level: Location (1=Distributed, 2=Collocated)
rly, the four team process dimensions
rcept,
slope and error (residual) term. In Equation 1, the intercept β0j is the overall average
performance by a team on a run (occasion). The slope β1xij is the regression coefficient
study, multilevel regression is used to generate separate models for performance, map
score, and a team process composite measure. The following example uses the
performance outcome to illustrate how the models are built.
This study used data from 25 teams with 2 observations (occasions) per 2-person
team. So that we can look at both individual and team level data across observations, d
were arranged so that occasions are nested within participants, giving a sample of 100
cases for analysis. Thus outcome variables are measured at the lowest hierarchical level.
Outcome variables were performance score (0-12 points), map score (1-4 pts.), and
global team process rating (1-4). Explanatory variables at level 1 include firefighting
experience, US&R experience, and technology experience, each measured on a 5-p
Likert scale (low/high). Because each team completed runs in two different conditions,
experimental IVs are
and RSVP (1=RSVP, 2=no RSVP). Simila
(Communication Effectiveness, Support, Leadership, and SA) are entered at level 1, each
measured on a 4-point Likert scale. Explanatory variables entered at level 2 include site
(1=NASA-Ames, 2 = NJTF-1), and order of conditions (1-11 listing various orders in
which teams went through experimental conditions).
To analyze the data, separate regression equations are created for each occasion to
predict the outcome variable y by the explanatory variable x.
yij = β0j + β1xij + eij (Equation 1)
The equation’s form is that of a traditional regression equation, consisting of an inte
59
r
l 2
s
n
m quantities with means = 0 and a normal distribution, so that
varianc
el is
r, it
is consi
(slope) for the explanatory variable x. The intercept and slope make up the fixed part of
the equation. The error eij is the part of performance not predicted by the fixed regression
part of the relationship, where i is the occasion and j is the team. The error term has a
mean of 0 and a variance to be estimated. In a multilevel analysis, the level 2 groups, in
this case teams, are regarded as a random sample from a population of teams, thereby
leading to the expression of the intercept β0j as β0 + u0j. With multiple teams, the erro
term u0j is the deviation of the jth team’s intercept from the overall value, and is a leve
residual which is considered to be the same for all occasions in group j. The residual i
now partitioned into a level 1 component eij and a level 2 component, u0j corresponding
to each level in the hierarchy. The variance between level 2 groups is σ2u, and the
variance between occasions within a given level 1 group is σ2e. The regression model ca
thus be expressed as
yij = β0j + β1xij + u0j + eij (Equation 2)
where uj and eij are rando
es σ2u and σ2
e can be estimated. These variances are assumed to be uncorrelated
since they are at different levels, and are known as the random parameters of the model.
The intercept and slope (fixed parameters) can also be estimated. This multilevel mod
sometimes called a variance components model. The model can be expanded to include
multiple explanatory variables at any level (both categorical and continuous); howeve
dered prudent to begin with an intercept-only model and build the model by
adding theoretically important variables. The model is estimated using the iterative
60
ntil
n
del fit is evaluated using the deviance statistic, defined as -
2*log (
ributed
gure
s the ‘ordercode’ variable. This is a dummy code to assess whether the order in
which teams participated in experimental conditions had any effects on the outcome.
Looking at the upper model in Figure 8, the intercept β0j (4.350/0.312 = 13.94)
and level 1 error variance eij (8.615/1.723 = 5) are both significant, the level 2 error
variance u0j is not (0.546/1.298 = 0.42), and the baseline model deviance is 505.121. In
generalized least-squares method (IGLS)1, where the model is computed repeatedly u
a specified level of convergence is reached (that is, the model is no longer changing from
iteration to iteration.) As seen in the example that follows, the model provides standard
errors for the intercepts, regression coefficients and variance estimates. These are used i
assessing the significance of coefficients using the Wald statistic, which is the ratio of a
coefficient to its standard error (Wald, 1943). It is tested as a Z-value compared to a
standard normal distribution at a given p-level (p=.05 is the level of significance used
throughout this study). Mo
Likelihood) where Likelihood is the value of the function at convergence and log
is the natural logarithm. Many models can fit a data set, but a smaller deviance typically
corresponds to a better fit of the data. When models are nested, the change in deviance
can be tested for significance using a chi-square test, where the difference is dist
as a chi-square with degrees of freedom corresponding to the change in number of
parameters of the model.
In this model, performance is assumed to be normally distributed, and is notated
as y ~ N (XB, Ω) where XB is the fixed part of the model and Ω is the random part. Fi
8 shows the baseline, intercept-only model, followed by the more general model that
include
1 IGLS is the default procedure used in MLwiN 2.0 (Rasbash, Steele, Browne, & Prosser, 2005), the statistical software used to estimate model parameters in this study.
61
the lower model depicted in Figure 8, the explanatory variable ‘ordercode’ has been
added to the equation. It is not significant (-0.012/0.101= 0.12), signaling that the order
of experimental condition does not predict performance. Note that though the coefficients
for the intercept and level 2 error variance u0j vary slightly, their significance level has
not changed. The model deviance does not appear to have significantly changed (χ2 =
0.13, ns.) Therefore ‘ordercode’ does not appear to contribute to the overall prediction of
performance in this study. However, it is included in all analyses to control for possible
effects on other variables.
Multilevel regression allows the testing of variables at both individual and team
vels within the same analysis. Is it more complicated than traditional regression?
n be. The results in
terms of the fixed regression estimates are very close to those obtained through
traditional multiple regression, and offer a more conservative estimate of the standard
errors of these parameters. Though the effect size estimates may be similar, multilevel
modeling can reveal differences in variance across units of analysis at different levels,
making it a useful tool in situations such as this study, where there are not only data
nested within teams, but explanatory variables present at both the individual and team
level.
le
Depending on the number of parameters and interactions tested, it ca
Figure 8. Baseline and ordercode model estimates of performance.
62
63
nd major
it
dings from this study are clearer when presented according to the
question (performance, shared mental model, team process), so the
finding
a lot of significant results from supplemental analyses
to
be
Chapter 5
Results
This chapter presents descriptive analyses for participant demographics a
study variables, and findings from the communication analysis conducted using the
RASAR-CS, followed by the results for the multilevel regression analyses of team
performance, shared mental models, and team process. Typically results are presented
according to hypotheses, beginning with Hypothesis 1, etc. However, upon reflection
seems that fin
dependent variable in
s are reported in that fashion. However, all findings are clearly linked to the
hypotheses in question, so you will have no trouble connecting them. In addition, a
summary chart displaying all hypotheses and evidence of support can be found at the
close of this chapter.
As you read, you will see
looking at variables other than location and RSVP, many of which are not directly tied
the hypotheses. These are not “fishing expeditions”, but are designed to explore the
relationships between various constructs in the model described in Figure 3. All will
made clear in the end. Findings are reported as statistically significant at p < 0.05 unless
noted otherwise.
64
Descriptive Analyses
arizes the results from demographic survey questionnaire given
ants. The majority were male 90%) between the ages of 35-54 (76%).
ting, US&R, and chnology ex ience were included in the mu vel
nalyses as po tial moderators of perform e in human-robot teams.
very few particip nts reported having any mi perience (l than 20%),
a pos nat variable for the purposes of this analysis.
self-ratings w re averaged across questions 8-11, which can be found in
Analyses
section repor findings fro he communication analysis conducted using
S. The hyp theses stated the end of Chapter 3 include results from the
R-CS as a seconda metric of support. In the interests of clarity, the findings
n ar the es in conjunction with t ultilevel
analyses that f low.
encies an percentages m the RASAR-CS are reported in Table 6.
am member exchanges, and tether handlers initiating 46%. Over half of all coded
stateme
Demographics
Table 5 summ the
to particip s (
Firefigh te per ltile
regression a ten anc
Because a litary ex ess
this was dropped as sible expla ory
Technology e
Appendix B.
RASAR-CS
This ts m t
the RASAR-C o at
RASA ry
reported in this sectio e linked to hypothes he m
regression ol
Statement frequ d fro
Communications were balanced between team members, with operators initiating 47% of
te
nts were comments (53%), with the remainder divided equally among the other
categories (answer, question, instruction).
65
Means/
Table 5
proportions, standard deviations, and percentages for demographic survey data
Survey Question M (SD) Total % NASA-Ames % NJTF-1 %
Age (5 groups) 3.44 (0.95) 1. under 25 4 4 5
3. 35-44 40 39 40 4. 45-54 36 32 41 5. 55 and up 12 14 9 Gender -- 1. Male 90 93 86 2. Female 10 7 14 Firefighting Experience 3.42 (1.81) 1. 0-3 years 30 40 18 2. 4-7 years 4 0 9 3. 8-11 years 4 7 0
5. over 15 years 52 46 59
1. 0-3 years 36 40 32
3. 8-11 years 16 21
2. 25-34 8 11 5
4. 12-15 years 10 7 14
US&R Experience 2.16 (1.21)
2. 4-7 years 34 14 60 8
4. 12-15 years 6 11 0 . over 1
Military Experience -- 5 5 years 8 14 0
1. Yes 18 25 14 2. No 82 75 86 Technology Experience 2.9a (0.88) 1. Very Little 8 10 4 2. Some 18 18 18 3. Moderate 52 43 64 4. Above Average 20 25 14 5. Expert 2 4 0 Note. Total N = 50. NASA-Ames n = 28, NJTF-1 n = 22. a A technology experience composite rating was created based upon respondents’ self ratings of experience
In terms of content, teams communicated most often about the robot’s state (30%)
and navigation (22%), with the environment, search strategy, and robot situatedness
categories each claiming another 10% of the total. In the function category, team
communications were focused on reporting, planning, and confirming (29%, 23%, a
with remote-controlled vehicles, video games, video cameras, technical search technology, and robots.
nd
66
pectively).
munication between team members,
s x content co ere generat operator-
tether and tether-operator dyads (Table 7). The upper portion of the table categorizes
operator-tether utterances by form and content: for example, operators made
8 fied as answers to teth dlers. This represents 17% of all
operator-tether statements in terms of form ose 807 answers, re classified as
being about the environment. Looking acro table, you can se statements about
the environment made up 15% of all opera er statements in of content
(n=704 statements). Proportions within the re easily calcula ividing the
s requency by the appropriate cat half of the table
c rances made by tether-handle erators.
and tether-handlers made r kinds of statem verall in terms of
content. The one content category that stands out is navigation. Operator statements
regarding navigation totaled 18% of all sta ts (n = 852), and t -handler
s about navigation totaled 29% (N = 1,333). This means that tether-handlers
talked to the operators 60% more often abo igation than the o rs talked to them
a ting when you consi t the operators are the ones doing the
navigating. Turning to the form category, t s another striking difference between
t tether-handlers gave over 4 s as many instructions to the operators (N
lear that the
majority of those (54%) were instructions about navigation (N = 651). Referring back to
Table 6, instructions made up 15% of all statements coded in the study (N = 1,506).
18%, res
To gain some insight into the patterns of com
peaker-recipient dyad x form mparisons w ed for the
07statements classi er-han
. Of th 100 we
ss the e that
tor-teth terms
table a ted by d
tatement f egory total. The lower
lassifies utte rs to op
Operators simila ents o
temen ether
tatements
ut nav perato
bout it. This is interes der tha
here i
eam members: time
= 1,203) as operators gave to them (N = 253). Looking within the table, it is c
67
gory Frequency Percent
Table 6
RASAR-CS statement frequencies and percentages
Coding Dimension/Cate
Dyad Operator-Tether 4629 47%
er-Operator cher-Operat
esearch 9 earcher 8 1%
er-Tether 2 54
Teth 4543 253
46% Resear or 3% Operator-R
ser 20
172%
Tether-ReResearch 14 1% 99 100%Form Comment 5251
4 r 83
6
53%Question 161 16% Answe 15 16%Instruction 150 15% 9954 100% Content Robot state Navi
3001 30% gation 2203 22%
ent 1237 strategy
tedness 2 7 7%
6 n synthe
54
Environm 12% Search 1013 11% Robot situa 98
7310%
Victim Off task 60 6%Informatio sis 175 2% 99 100%Function Report 2893
5
tio
4% 333 3%
Correct error 132 1% 9954
29% Plan 225 23%Confirm 1818 18% Provide informa n 1053 11% Seek information 1040 11% Clarify 430 Convey uncertainty
100% Total number of statements = 9,954 for 50 occasions. Categories are ordered by frequency. M = 199, SD = 69.
68
speaker-recipient dyads
Table 7
RASAR-CS form x content comparisons of statement frequencies and proportions for
Operator-Tether
Content tegory tal
Proportion of total
Environment 100 447 147 704 0.15
Dyad Form Content
Answer
Comment
Instruction
Question
cato
10 Information Synthesis 19 65 1 0 5
tion 148 527 83 94 852 0.18 20 83 1 20 124 0.03
ss 66 2 18 93 449 0.10 276 8 100 203 1412 0.31 106 3 32 59 512 0.11 72 3 8 74 481 0.10
807 2 253 700 4629 1.00
1 9 0.02 NavigaOff task Robot Situatedne 72 Robot State 33 Search 15 Victim 27 Form category total 869 Proportion of total 0.17 0 0.05 0.15 1.00
perator
Answer
C ent
Instr
Questio
Contencategortotal
Proportion of total
nvironment 95 302 13 118 528 0.12
.62 Tether-ODyad Form Content
omm uction n t y
EInformation
10 54 5 6 75 0.02 139 4 651 139 1333 0.29 9 7 1 23 103 0.02
uatedness 88 3 24 59 524 0.12 191 4 390 225 1290 0.28 62 1 111 92 438 0.10
1 8 69 252 0.06 tegory total 624 1 1203 731 4543 1.00
Synthesis Navigation 04 Off task 0 Robot Sit 53 Robot State 84 Search 73 Victim 30 45 Form ca 985 Proportion of total 0.14 0 0.26 0.16 1.00 .44 Total operator statements N = 4,629. Total tether-handl atements N = 4,543.
tatements, 80% were made b er-handlers to operato he tether-
insert the ro nto the accessible voidspace and manipulate the tether
teleoperates the t through the void. While it has been shown that the
er st
Of those s y teth rs. T
handler’s role is to bot i
as the operator robo
69
“backse
e
at driver” phenomenon is common in human-robot teams (Burke, Murphy,
Coovert et al., 2004), this is an unusually skewed ratio. Could it be due to one of th
experimental conditions in the study?
Table 8
RASAR-CS statement proportions by location and RSVP conditions Location RSVP
Coding Dimension/Category Distributed Collocated RSVP No RSVP
Dyad Operator-Tether 0.47 0.47 0.44 0.50 Tether-Operator 0.44 0.45 0.48 0.40 Operator-Researcher 0.03 0.03 0.03 0.03 Researcher-Operator 0.03 0.02 0.02 0.03 Tether-Researcher 0.02 0.02 0.02 0.02 Researcher-Tether 0.02 0.01 0.01 0.02 Form Answer 0.15 0.16 0.15 0.17
0.53 0.51 0.56 Instruction 0.14 0.16 0.18 0.10 Question
Environment 0.13 0.11 0.12 0.12
Navigation 0.21 0.23 0.23 0.21
Robot situatedness 0.12 0.09 0.09 0.12
Search 0.09 0.10 0.10 0.09
Comment 0.53
0.18 0.16 0.17 0.17
Content
Information synthesis 0.01 0.01 0.02 0.01
Off task 0.08 0.06 0.06 0.07
Robot state 0.30 0.31 0.31 0.29
Victim 0.06 0.09 0.07 0.08
Function Clarify 0.04 0.04 0.04 0.05 Confirm 0.18 0.18 0.18 0.19 Convey uncertainty 0.03 0.03 0.03 0.04 Correct Error 0.01 0.01 0.01 0.01 Plan 0.21 0.22 0.24 0.18 Provide information 0.10 0.10 0.10 0.10 Report 0.30 0.31 0.29 0.32 Seek information 0.12 0.11 0.11 0.11
70
ntal
ted teams. Collocated teams gave slightly more instructions, and talked less about
robot situatedness. Looking at the RS
perform nce (r = 0.36, p = 0.01). Situation awareness was related to map score (r = 0.33,
Taking a quick look at the RASAR-CS proportions according to experime
conditions in Table 8, there were few notable differences between distributed and
colloca
VP teams and no-RSVP teams, there are several
differences that stand out. Tether-handlers talked to operators more often when they
shared the robot view (48% vs. 40%), and gave more instructions (18% vs. 10%).
Like the collocated teams, RSVP teams talked less about robot situatedness (9%
vs. 12%). They did slightly less reporting (29% vs. 32%), and engaged in more planning
compared to no-RSVP teams (24% vs. 18%). These observations cannot be taken at face-
value since the two conditions were crossed and nested within teams. However, they
serve as interesting indicators of relationships to look for in later analyses.
Descriptive Analyses for Study Variables
Means, standard deviations, and correlations between team performance scores,
map scores, team process dimension ratings and selected RASAR-CS categories are
reported in Table 9. The incorporation of all 26 category element variables used in the
RASAR-CS makes for an ungainly correlation table, so only those elements that played a
role in the reported results are included. The complete correlation matrix is available in
Appendix C. The overall mean performance score was rather low (M = 4.35, SD = 3.06),
with scores ranging from 0-12.5. Mean map score was 2.54 (SD = 0.96), with scores
ranging from 1-4. Team process means hovered around 3 on a 4-point Likert scale. There
was no correlation between map score (shared mental model measure) and performance
score. Of the four team process dimensions, only Communication was related to
a
71
p = 0.02), underscoring the importance of SA in creating shared mental models.
Correlations between team process dimensions were all significant, ranging fr 1
0.64.
llocated teams gave slightly more instructions, and talked less about robot
situatedness. Looking at the RSVP teams and no-RSVP teams, there are several
differences that stand out. Tether-handlers talked to operators more often when they
shared the robot view (48% vs. 40%), and gave mo nstr s v
the collocated teams, RSVP teams talked less about robot situatedness (9% vs. 12%).
They did slightly less reporting (29% vs. 32 ore planning compared
to no-RSVP teams (24% vs. 18%). Th o t n t
since the two conditions were crossed n w e r a
interesting indicators of relationships to look for in later analyses. In the RASAR-CS
cate t t l e
dim ons (r = 0.29-0.4 e i w r
significant (r = 0.27, p = 0.06). The tether-operator dyad w neg ly associated wi
SA v i t
app hed sig - r i
teth r e er tings on these team process
dim on A h ro ti f in c s w lso ted oor a
evidenced by the negative relationship (r = -0.29). Statements about navigation were not
asso ed th , t w a ic neg e re nsh i p score (
) c ra e e wer sitiv rel t s
om .03 to
Co
re i uction (18% s. 10%). Like
%), and engaged in m
bseese rva ions ca not be taken a face-value
and ested ithin t ams. Howeve , they serve s
gor
ensi
(r =
roac
er-h
ensi
ciat
8, p
ies, the operator-tether dyad was significan ly rela ed to a l four t am process
7), and the r lationship w th map score as ma ginally
as
he
ative
aders
th
-0.42, p < 0.01), and a negati e relationsh p with Le hip dimension
nificance as well (r = 0.26, p = 0.07). Mo e communicat ons from the
andle to th op ator are associated with poor ra
s. hig er p por on o stru tion as a rela to p SA, s
wi
.01
SA
. In
but
ont
here
st, s
as
arch
signif
statem
ant
nts
ativ
e po
latio
ely
ip w
ated
th ma
o map
r = -
0.3 < 0 core
72
Tabl
e 9
for s
tudy
var
iabl
es o
f int
eres
t (N
=50
)SD
2
3 6
10
12
13
14
15
Mea
ns, s
tand
ard
devi
atio
ns, a
nd c
orre
latio
ns
Varia
ble
M
1
4 5
7 8
9 11
1.
3.
06
Pe
rform
ance
4.
35
1
2.
Map
Scor
e 0.
96
0.13
8 1
0.
339
3.
Com
mun
icati
0.77
0.
359*
0.
101
0.
010
0.48
5 4.
3.
02
0.71
-0
.041
0.77
9 0.
444
5.
3.08
0.
64
0.04
3 0.
152
0.
769
0.29
1 6.
Situ
atio
n 3.
08
0.63
0.
054
0.
707
0.02
1 th
er
0.47
0.
13
0.04
9 0.
268
0.
059
8. T
ethe
r-Ope
rato
r 0
.45
0.13
0.
025
0.
217
9. In
stru
ctio
n 0.
15
0.11
0.
230
-0
0.13
1
10. N
avig
atio
n
0.22
0.
10
-0.0
45
-*
0.
007
11. R
obot
Si
tuat
edne
ss
0.11
0.
05
-0.2
57
0.11
3
0.43
5 0.
30
0.09
-0
.110
0.
043
0.
769
13.
0.0
2
1
31
0*
1
0.
642*
1
0
0.39
4*0.
465*
0.47
9**
0.00
1 0
0.
351*
0.
287*
0.
382*
* 0.
467*
1
0.
013
0.04
3 0.
001
-0.1
89
-0.2
57
- 0.41
7**
- 0.90
0**
0.19
5 0.
188
0.00
3 0
-0.0
98
-0.1
86
- 0.67
1**
0.73
3*1
0.68
1 0.
497
0.04
2 0
-0.0
23
0.08
9 -0
.044
-0
.229
- 0.
471*
* 0.
528*
0.54
4**
0.87
2 0.
538
0.76
4 0.
11
0
0 0.
028
0.11
2 0.
165
-0.3
55*
- 0.44
9**
-0.2
96*
1
0.84
7 0.
439
0.01
1 0.
001
0.03
7 0.
061
-0.0
04
0 -0
.303
* -0
.052
0.
75
0.67
6 0.
361
0.83
9 0.
976
1 0.
718
024
0.10
3 0.
022
-0.1
91
-
0.22
9 0.
477
0.87
8 0.
2.54
on
3.
18
Supp
ort
0.11
1 0.
0.
029
Lead
ersh
ip
0.35
3*
*
0.01
2
Awar
enes
s 0.
327*
*
* 1
0.
005
7.
Ope
rato
r-Te
*
0.
734
0.00
6
-0.1
78
-0.1
86
1
0.86
1 0.
071
.217
0.
06
-0.2
89*
*
0.
108
0.19
5 0
0.37
9**
1
0.75
4 0.
001
0
-0
.052
0.07
2 0.
719
1 0.
252
12
. Rob
ot S
tate
-0
.002
-0
.046
-0
.132
-0
.03
1
0.44
7 0.
99
0.03
3
Sear
ch
0.10
0.
05
0.20
9 0.
429*
* -0
.052
0.
093
0.0.
173
0.06
4 -0
.24
0.34
8*
1
0.14
5 0
0.72
0.
52
0.87
66
1 0.
093
0.18
5 0.
013
14. R
epor
t 0.
30
0.11
-0
.018
-0
.108
0.
281*
0.
001
0.10
7 0.
102
0.22
1 -0
.27
- 0.41
4**
-0.1
51
0.25
4 0.
239
- 0.55
0**
1
0.
900
0.45
4 0.
048
0.99
2 0.
459
0.47
9 0.
124
0.05
8 0.
003
0.29
7 0.
075
0.09
4 0
15. P
lan
0.21
0.
11
0.30
8*
0.00
4 0.
141
-0.0
28
-0.1
16
-0.0
72
-0.3
46*
0.49
4**
0.81
0**
0.37
5**
- 0.50
0**
-0.1
98
0.36
6**
- 0.64
4**
1
0.03
0 0.
981
0.32
7 0.
847
0.42
3 0.
621
0.01
4 0
0 0.
007
0 0.
168
0.00
9 0
*p
= 0
.05,
**p
= 0
.01.
73
arch strategy than
navigation were able to creat t least, as measured by the
robot state and robot situatedness were unrelated to the
outcom
being
less
nts in
rior
r is
(r = 0.43, p < 0.01), suggesting that teams that talked more about se
e a richer shared mental model (a
map score). The categories of
e variables, though the negative relationship between robot situatedness and
performance was nearly significant (r = -0.26, p = 0.07). However, they were each
negatively associated with navigation (r = -0.30 for both). This is partly due to their
in the same coding dimension, but statements in one content category do not preclude
statements in another. It may be that teams that talked more about navigation talked
about the robot in general, focusing on direction instead. The proportion of stateme
the report category was positively related to Communication ratings (r = 0.28). Teams
that had a higher proportion of statements reporting what they were seeing in the search
space were rated has having more effective communication. Finally, the function
category plan was associated with performance (r = 0.31), signaling that teams that
engaged in more planning were more successful in locating victims.
Some interesting findings have emerged from these analyses. Some confirm p
research findings: 1) planning and effective communications are related to performance;
and 2) search strategy and situation awareness are important in creating a shared mental
model of the search. Other findings are quite novel: something is going on with the
tether-handler giving all those instructions about navigation! Having communications
from the tether-handler detract from team SA is unexpected, since prior research has
shown that more frequent communication between the operator and tether-handle
associated with better SA and more effective performance (Burke & Murphy, 2004a).
However, before speculating as to why this occurred, there are further analyses to report
74
ough
ubsequent equations to control for order effects in other conditions. Thus, Model B
e baseline model for Models C-G. There were no effects for individual
cient ht nc p ignificance. No e
location (Model D) on performance. This model contradicts Hypothesis 6, which stated
that colloca s would perform distributed teams. More positive results
were found in the next mo odel te hypothesized diffe in perfo e
between teams with and with ut RSV . T ative regression co t for R β
= -1.322/SE 0.611) signifies a positive rel n RSVP rforman
(rec ling that the RSVP variable is coded
model is significantly reduced as well (χ2 = 4.52). The next step in the analysis was to test
for site effects, i.e. to look for differences s accord here t
we ollec SA-Am , NJ -1 Unfortunately, I fo : Model F’s
drop in Deviance and the significant Beta or the site variab 2.103/S
0.5 left u kable e ce of fe performance by site. (This is
unfo tunate in the sense that the variance rformance was effec ficantl
other than the independent variables in question.) However, after re-entering
which may shed some light on the findings from the RASAR-CS.
Multilevel Regression Analyses
Team Performance Measure
Table 10 displays model parameters for the regression of location and RSVP on
team performance. No effects were found for the order in which teams ran thr
experimental conditions, as seen in Model B. I chose to keep the Order variable in
s
serves as th
experience of any type (firefighting, US&R, or technology), though the regression
coeffi for firefig ing experie e ap roached s ffects were found for
ted team better than
del. M E sts for rences rmanc
o P he neg efficien SVP (
ationship betwee and pe ce
al 1=RSVP, 2 = no RSVP). The deviance in the
between the team ing to w he data
re c ted (NA es = 1 TF = 2). und them
weight f le (β = E =
73) nmista viden dif rences in
r in pe ted signi y by
something
75
the RSV e
he
the results
Regression Coefficients
P variable into the equation along with order and site and using Model F as th
baseline for comparison, the regression coefficient for RSVP remained significant and t
model deviance was significantly reduced (χ2= 5.67). Despite the fact that there were site
effects, the use of RSVP does have a positive effect on performance. Therefore
reported in Table 10 provide support for Hypothesis 3 (RSVP teams→better
performance), but not for Hypothesis 6 (collocated teams→better performance).
Table 10
Multilevel regression analysis of location and RSVP effects on team performance
M Deviance ∆ Deviance df Variables β SE
odel
A. 505.12 -- -- Null
5
-0.012 0.101
C 500.98 4.14 3
rder hting Experience Experience
echnology Experience
0.003 0.322 -0.245 0.159
0.100 0.170 0.256 0.345
D 5 ion 3
8 0
E. 500.59 4.52* 1 51
2*
5 1
F. 492.62 12.49* 1 3
07* 3 3
G 490.15
2.47 1
rder
ion
-0.081 6*
0.096 7 1
H 4
8*
4 8 6
B. 05.11 0.01 1 Order
.
OFirefigUS&RT
. 02.69 2.42 1
Order Locat
-0.03 0.984
0.100.61
Order RSVP
-0.0-1.32
0.100.61
Order Site
-0.03-2.1
0.090.57
.
OSite Locat
2.12 0.935
0.560.59
. 86.95 5.67* 1
Order Site RSVP
0.034 2.16-1.416*
0.090.550.58
*p < 0.05.
76
d
process ratings to predict team performance. Though not
ere entered
d
l
Supplemental regression analyses. In addition to testing the main hypotheses
relating to location and RSVP, I examined the relationships between team process an
performance by using the team
specifically stated as a hypothesis, this relationship is both part of Klimoski &
Mohammed’s team performance framework (Figure 2) and part of the model of robot-
assisted team performance (Figure 3) proposed in this study. Since no particular team
process dimension was theorized to be more important than another, all four w
simultaneously into the regression equation. Table 11 lists the regression coefficients an
standard errors for the four team process dimensions, along with comparisons of mode
fit.
Table 11
Supplemental multilevel regression analyses for team performance Regression Coefficients
Model Deviance ∆ Deviance df Variables β SE
A 505.12 --
Null
Order -0.033
0.093
Site Communication Support Leadership
1.860* 1.605* -0.827 0.849
0.546 0.340 0.5330.559
Site Plan Robot situatedness Robot state
2.231* 7.123* -20.802* -5.198
0.489 2.848 5.698 2.798
B. 492.62 12.50* 2 Site -2.107* 0.573
C. 478.43 14.19* 4
Order
Situation Awareness
0.077
-0.826
0.103
0.530
D. 457.78 34.85* 4
Order
Tether-operator
-0.154
-5.065*
0.084
2.174 *p < 0.05.
77
sured
tgun
ASAR-
y,
ficant
bles
bot
e, and tether-operator. The first three are theoretically important
accordi
tion
tion
e
s
f performance. Talking more
bout the robot (state or situatedness in the environment), however, seems to detract from
Communication appeared to have significant influence on performance. After
controlling for order and site, deviance in the model was significantly reduced (χ2 =
14.19).
This naturally leads to the question of whether team communication as mea
by the RASAR-CS would show any predictive power for team performance. With 26 of
the 30 possible coding categories used in this study, it made no sense to take the sho
approach of trying all category elements as predictors. To narrow the field of
possibilities, I first looked to the elements classified as indicators of SA in the R
CS: environment, information synthesis, robot state and situatedness, search strateg
plan and report. I then reviewed zero correlation tables to see if there were any signi
relationships between these RASAR elements (or any others) and primary study varia
(both dependent and independent). I chose to look at the following variables: plan, ro
situatedness, robot stat
ng to prior research, and were significantly related to one or more major study
variables. The tether-operator proportion expresses that part of the team’s communica
that is initiated by the tether handler speaking to the robot operator. This proportion
displayed some interesting relationships with several study variables (e.g., team situa
awareness), and was chosen for that reason. When entered simultaneously into th
baseline model controlling for order and site, three of the four RASAR proportion
variables were significant predictors of performance (Table 11). Model deviance wa
significantly reduced from 492.62 to 457.78 (χ2 = 34.84). As noted in earlier studies,
planning in human-robot teams is a positive predictor o
a
78
rmance, as does more frequent communication from the tether handler to the
ier.
S Mental ea
Results from the multilevel regression analysis investigating fects of
location and RSVP on shared m tal m l presented in Table 12. There were no
significant effects for previous experience a order effects in th l mode
compu tions. The regression coefficient fo e significa n site
entered into the regression (Model D). The order in which team
experimental conditions did influence their ores, which sugg team
constructed better maps in some conditions than in others. It may be that there was
simply practic t-- te ew be r n the second run atter w
combination of location and RSVP con io ere in. Site wa actor
redicting map score.
No support was found for Hypothesis 1 (RSVP teams → better shared mental
models). Teams with RSVP were no better at constructing maps of their search than
teams without it, as evidenced by the lack of change in model deviance when RSVP was
entered into the regression equation (using Model D for comparison). However, the
location of team members played a role in predicting teams’ mental map scores:
distributed teams drew better maps of their search than collocated teams (β = 0.425/SE =
0.174). This is contradictory to the outcome predicted in Hypothesis 4 (Collocated
teams→ better shared mental models). This model was a significantly better fit (χ2 =
5.33) and interestingly, the order effects disappeared.
team perfo
robot operator. These findings are similar to those reported from the RASAR-CS earl
hared Model M sure
the ef
en ode s are
nd no e initia l
ta r order becam nt whe was
s went through
map sc ests that s
a e effec ams dr tte maps o , no m hat
dit ns they w s not a f in
p
79
ental model
nts
Table 12
Multilevel regression analysis of location and RSVP effects on shared mRegression Coefficie
Model ce ce s β
Devian ∆ Devian df VariableSE
3 --
A 270.8 -- Null . 14 1
r 64
3
C. 267.09 0.04 3
r perienc
Experience y Experienc
63 10 03
008
3 7 6 6
D. 266.81 0.33 1 Order -0.065*
16 0.033
2
E. 266.46 0.67 1 71*
152 4 4
261.80 5.33* 1 Location
-0.425* 5
0.174
B 267. 3.69 Orde -0.0 0.03
OrdeFirefighting Ex e 0.0US&R Technolog e 0.
-0.0
0.0
0.030.050.080.11
Site 0.1
0.20 Order RSVP
-0.0 0.
0.030.18
F.
Order -0.042
0.03
Note. * p < 0.05.
e
Supplemental regression analyses. Following the procedure for analyzing team
performance, I tested for effects of team processes on shared mental models. Again, this
was not a hypothesis specified for this study, but an exploratory effort to look for possibl
relationships between the team process ratings and map scores. The four team process
ratings were entered simultaneously into the regression equation. Table 13 shows the
regression coefficients and standard errors for the four team process dimensions along
with model fit comparisons. After controlling for order effects, ratings of team situation
awareness were predictive of the team map scores (β = 0.53/SE = 0.173), and model
deviance was significantly reduced (χ2 = 9.57), suggesting that teams with better SA were
more likely to share a mental model of their search process.
80
egression analyses for shared mental model
Regression Coefficients
Table 13
Supplemental multilevel r
Model Deviance ∆ Deviance df Variables β SE
A 270.83 -- --
Null
B. 267.14 3.69 1
Order -0.064
0.033
C. 257.56 9.57* 4
Order
Support Leadership Situation Awareness
-0.059*
-0.100 0.051 0.530*
0.030
0.168 0.187 0.173
D. 238.86 28.28* 5
Search
Robot situatedness Instruction Tether-operator
7.059*
1.441 -1.274 0.732
1.633
1.830 1.172 0.91
Communication -0.072 0.112
Order
Navigation
-0.042
-2.129*
0.509
0.990
9 *p < 0.05.
ch,
regression coefficients for robot situatedness, instruction, and tether-operator were not
Five proportion categories from the RASAR-CS were chosen to examine the
influence of communication elements on the creation of a shared mental model. Sear
navigation, and robot situatedness were chosen based on theory and findings from prior
research. Instruction and tether-operator proportions were chosen because they were
inversely related to team SA ratings in this study. The proportion of communication
devoted to search strategy was predictive of the team’s shared mental model (β =
7.059/SE = 1.633), in that more talk about search strategy is associated with higher map
scores. Navigation emerged as a negative predictor (β = -2.129/SE = 0.990), in that less
talk proportionately about navigation is associated with higher map scores. The
81
t. The change in model deviance (χ2 = 28.28) is significant, and the order effects
e
The four team process dimension ratings were averaged to create a mean team
process rating for the purposes of this analysis. Table 14 presents the model parameters
a om ns of f the l location and RSV s on te ocess. No
effects were found for experience,
o rfo and m core n s we d for e f the
experimental conditions as well. R ion coefficients for n (β = -0.053/SE =
0.109) and RSVP (β = 0.150/SE = 0.105) were non-significant, and none of the models
te d p a bet of t d n the null model. Therefore, no support was
found for Hypothesis 2 (RSVP tea more effective team sses) n
ypothesis 5 (Collocated teams → more effective team processes).
Supplemental regression analyses. For the final set of analyses, I chose the
RASAR-CS categories of plan, report, and tether-operator to see if these patterns of
communication predicted mean team process ratings. Plan and report are RASAR team
process indicators that have proven important in prior research, and the tether-operator
proportion has drawn attention in earlier analyses. All three variables significantly
predicted mean team process ratings (Table 15). Plan (β = 2.058/SE = 0.587) and report
(β = 1.661/SE = 0.533) were both positive predictors of team process. Teams that focused
more of their communications on these two purposes functioned more effectively overall.
Higher proportions of tether-operator communication were negatively associated with
significan
once again disappeared.
Team Process M asure
nd c pariso it for ana ysis of P effect am pr
order of condition or site, as were in previous analyses
f pe rmance ap s . U fortunately, no effect re foun ither o
egress locatio
s et rovided ter fit he ata tha
ms → proce or for
H
82
effects on team process
R
Table 14
Multilevel regression analysis of location and RSVP
egression Coefficients Model D De Va le
β SEeviance ∆ viance df riab s
A. 152.8 --
Nu 9 ll
B 150.5 39
Order -0
0.
C 150.1 0.04
Order Firefight nce US R E e Technolo
-0 0.-0-0
0.010.0.0.
0.24 1
Order Location
-0 24 -0.053
0.019 0.109
E. 148.47 2.03 1
Order RSVP
-0.032 0.150
0.018 0.105
F. 146.87 3.64
0.017 0.102
. 0 2. 1 .027 018
. 2 3
ing Experie& xperienc
gy Experience
.028 011 .022 .005
8 029 044 059
D. 150.26 .0
1 Order Site
-0.029 -0.196
Note. *p < 0.05.
team process ((β = -2.042/SE = 0.397), confirming the negative relationships observed in
Table 9 with the team process dimensions of Leadership and SA. The model deviance
was greatly reduced (χ2 = 29.42), providing a much better fit of the data.
83
process
Regre nts
Table 15
Supplemental regression analyses for team
ssion CoefficieModel Deviance ∆ Deviance df Variables
β SE
A. 152.89 --
Null
150.50 2.39 1
Order
Order Plan
B. -0.027
0.018
C. 121.08
29.42* 3
Report Tether-operator
-0.022 2.058* 1.661* -2.042*
0.015 0.587 0.533 0.397
Note. *p < 0.05.
inding
RASAR-CS communication analyses, and
multile ing to the outcome measure in
questio elation to the hypotheses listed
in Chapter 3. Table 16 provides a quick review of the hypotheses and evidence of
suppor o not directly address the study
hypoth eam performance pictured in Figure 3.
r reviewing the hypotheses.
Summary of Hypotheses and F
Results from descriptive analyses,
s
vel regression analyses have been reported accord
n. Now it is time to pull these results together in r
t. Many of the significant findings in the analyses d
eses, but rather the proposed model of t
These findings are summarized in relation to the model afte
84
Table 16
Study hypotheses and evidence of support
Supported?
Hypothesis Regression RASAR-CS
Analyses Analyses
H1: Teams having access to RSVP technology will generate richer, more accurate team situation models than teams not having access to the technology, as measured by search maps generated by the team, and frequencies of SA indicators in the RASAR-CS.
No No
H2: Teams having access to RSVP will exhibit more
ratings and by frequencies of team process
No Partial
search task scenarios, as measured by performance
team situation models than distributed teams, as
and frequencies of SA indicators in the RASAR-
processes than distributed teams, as measured by observer ratings, and by frequencies of team process indicators in the R
effective team processes, as measured by observer
indicators in the RASAR-CS.
support
H3: RSVP teams will accordingly perform better in the
outcome scores.
Yes ---
H4: Collocated teams will generate richer, more accurate
measured by search maps generated by the team,
CS.
No No
H5: Collocated teams will exhibit more effective team
ASAR-CS.
No No
H6: Co llocated teams will accordingly perform better than distributed teams in the search task scenarios, as measured by performance outcome scores.
No ---
85
ble
s
with
s by
l feedback he was receiving than thinking about the task
and his
15),
H1: The first hypothesis stated that teams with RSVP would generate richer, more
accurate shared mental models, as evidenced by map scores and frequencies of RASAR-
CS indicators of SA. In multilevel regression analyses summarized in Table 12, RSVP
was not a significant predictor of map scores. Supplemental regression analyses (Ta
13) revealed that the RASAR-CS categories search and navigation did predict map
scores, as did ratings on the team process dimension SA. More statements about search
strategy were positively associated with map scores. This is consistent with prior research
linking more frequent communications about search with better SA (Burke & Murphy,
2004b). However, RSVP teams did not talk appreciably more about search than team
with it. Navigation statements were negatively related to the shared mental model
measure, i.e., better map scores came from teams that talked less about navigation.
Comparisons of RASAR-CS proportions between RSVP and No RSVP conditions
revealed that tether-handlers in the RSVP condition initiated more communications
the operator, many of which were instructions about navigation. It may be that having
RSVP detracted from the creation of a shared mental model between team member
creating an attentional tunneling effect: when the tether-handler had RSVP, he spent
more time reacting to the visua
role in accomplishing it.
H2: The second hypothesis predicted teams with RSVP would exhibit more
effective team processes, as measured by team process ratings and indicators in the
RASAR-CS. There were no effects on team process mean scores from RSVP in the
regression analyses summarized in Table 14. Supplemental analyses revealed that the
RASAR categories plan and report were positive predictors of team process (Table
86
o
that
ing,
m to
summa s
a
ors were
and that tether-handler communications were negatively associated. In the RASAR-CS
comparisons between RSVP and No RSVP conditions, there were notable differences in
these categories. First, there was more involvement of the tether-handler in RSVP teams
because he could see what was occurring in the search space. RSVP teams did less
reporting than teams without RSVP, likely for the same reason—it was unnecessary t
report some observations about the search environment since both members could see
what was happening. As mentioned earlier, more instructions were given, suggesting
the problem-holder role was shared between team members. Though this seemed to play
a negative role in creating the team mental model, it is not necessarily a bad thing—
statement function comparisons for operators and tether-handlers revealed that 18% of
operator statements and 31% of tether-handler statements were classified as planning.
RSVP teams had 25% more planning statements than did No RSVP teams, and plann
as noted above, predicts more effective team processes. Taken together, these four
observations (increased tether-handler involvement, less reporting, more instructions,
more planning) suggest that team processes in RSVP teams are at least different—
whether they are more effective is arguable, but the increase in planning would see
support their being beneficial.
H3: The hypothesis that RSVP teams would perform better in the search task as
measured by performance scores was supported in the multilevel regression analysis
rized in Table 10. Even after accounting for site effects that emerged in previou
models, RSVP positively predicted performance (β = -1.416/SE=0.586) and produced
significantly better fit to the data (χ2 = 5.67, df = 1). Though RASAR-CS indicat
not included in the hypothesis, supplemental regression analyses revealed that the
87
would
ms. Other
and the
h environment has been associated with better SA in past
studies why it
;
into play
r—but this may be due to the level of the measure
used. S
f
e 12 were actually in the opposite direction:
proportion of search statements was a positive predictor of performance (Table 11). In
light of the larger proportion of planning statements attributed to RSVP teams, this
seem to strengthen the case for the effectiveness of team processes in those tea
significant predictors of performance in the RASAR-CS were robot situatedness
proportion of statements initiated by the tether-handler. Both of these were negative
predictors: that is, fewer statements about robot situatedness and less communication
from the tether-handler were associated with better performance. Talking about the
robot’s situatedness in the searc
, and better SA is typically linked with better performance—it is not clear
seemed to detract from performance in this study. The mixed benefits from increased
involvement of the tether-handler have already been discussed. It is easy to blame poor
team performance on the tether-handler’s hijacking of the operator’s navigator role
however, this is not a new phenomenon, and it may be that other factors come
here (perhaps those dreaded site effects). The team process dimension Communication
also predicted performance. This is not surprising, as the importance of effective
communication is a point well understood in team performance research. What is
surprising is that SA was not a predicto
A is defined somewhat differently as a team measure, relying more on the sharing
of important information about the environment rather than just being aware of it.
H4: The fourth hypothesis stated that collocated teams would generate richer,
more accurate shared mental models, as evidenced by map scores and frequencies o
RASAR-CS indicators of SA. This hypothesis was not supported. Results from
regression analyses summarized in Tabl
88
distribu effects
ent,
ltilevel regression analyses.
was
to
iab e
differen m of
ask
,
ted teams had higher map scores (β = -0.425/SE=0.174). There were order
present in earlier models, but these became insignificant when the location IV was
entered into the model. RASAR-CS analyses did not support this hypothesis:
comparisons between collocated and distributed teams revealed no appreciable
differences in statement proportions categorized as SA indicators (search, environm
information synthesis, plan, and report). However, collocated teams did talk less about
robot situatedness, a RASAR category that was negatively related to performance in
mu
H5: Collocated teams were hypothesized to exhibit more effective team
processes, as measured by team process ratings and RASAR-CS indicators. Location
not a significant predictor of team process ratings. This is surprising since it is generally
accepted both in research and the workplace that collocated teams function more
effectively than distributed teams. It is possible that the distributed condition
manipulation was weak, or that the mean team process ratings lacked enough variance
show differences. In the RASAR-CS analyses by location, there were no apprec l
ces in the pattern of communication between team members, the for
statements, in content, or in function.
H6: The hypothesis that collocated teams would perform better in the search t
as measured by performance scores was not supported in the multilevel regression
analysis summarized in Table 10. Location did not predict performance. There were no
differences between collocated and distributed teams in the RASAR-CS categories
associated with performance. Notably, there were no differences in planning or reporting
as observed in the RSVP comparisons.
89
dy
iables
d mental models, team process, and team performance. The
e
er
ng to the
rch
atings of
al
ies bode well
Six of the 7 hypotheses relating to RSVP and location from Chapter 3 are
presented in Table 16, along with indicators of whether support was found in the stu
analyses. These hypotheses were tests for main effects of the two independent var
(RSVP and location) on share
seventh hypothesis, which deals with the potential interactive effects of RSVP on
performance by location, is not directly testable in light of the complexities posed by th
crossed and nested data, and the observed site effects. It is discussed, however, in Chapt
6.
Looking at Table 16, it seems that the investigation of the effects of RSVP and
location on shared mental models, team process, and team performance yielded very
little: RSVP predicted performance, and nothing else; location predicted shared mental
models, but not in the way expected. Yet the results are full of significant findings that
tell much more than what appears in the table, and a review of these findings is merited.
Results of the supplemental analyses are presented in relation to the three dependent
variables of map score, mean team process ratings, and performance outcome.
RSVP did not predict better shared mental models, but the RASAR
communication categories search and navigation, and the team process ratings for
situation awareness did. So we do know something about what’s contributi
development of the shared mental model between team members. More talk about sea
contributes to good SA. This is a point made in earlier studies comparing SA r
operators (Burke & Murphy, 2004a), and it is reified here. Getting caught up in the
details of navigation, on the other hand, is not conducive to forming rich shared ment
models of the search process. Moreover, the RASAR-CS’s predictive abilit
90
ssisted
s:
better SA (Burke & Murphy, 2004a). In this
study it s clea
communication patterns between team members are critical in team functioning, and
since the model of team performance in robot-assisted search in Chapter 3 theorizes that
the shared mental model is formed through team communication, we can make a link
between the creation of shared mental models and effective team process.
Finally, these analyses showed that team process ratings for Communication
predicted performance scores, thus establishing a link between effective team
communication and better team performance. The RASAR-CS regression analyses
pinpointed some of the specific categories and patterns of communication that were
associated with performance: plan, robot situatedness, and tether-operator
for its abilities to capture the process of creating shared mental models of robot-a
technical search. It is also evident that the relationship between team processes and
shared mental models is a reciprocal one, in that the team process ratings for SA
predicted better map scores.
More effective team processes were predicted from three RASAR-CS categorie
plan, report, and tether-operator communications. Again, past research has shown that
operators who focused on goal-directed communication (planning the search, reporting
on what is seen in the environment) had
i r that these communication functions influence team processes overall. The
role of the tether-handler in the search process was revealed to have much import on
effective team functioning, as more communications from the tether-handler to the
operator seemed to have a negative effect. A large part of these communications were
about navigation, which were deleterious to the development of a shared mental model
between the two team members. In all, the significance of these findings show that
91
communications. The import of the plan and tether-operator categories has been
discussed at length, but the impact of robot situatedness has not. In past research,
operators who talked about the robot’s location and spatial orientation in the search
environment (position) performed bette esults were puzzling. Robot
s a
yses
e
del of
.
r. Here, the r
situatedness was not a significant predictor of the shared mental model, yet there a
strong negative relationship with performance (β = -20.802/SE = 5.698). Too, the RSVP
teams talked less about the robot’s position, as did collocated teams. Clearly, this is a
topic to be explored in future research.
In this chapter, results from descriptive analyses, communication anal
conducted using the RASAR-CS, and multilevel regression analyses of team
performance, shared mental models, and team process have been presented. Results hav
been linked with 6 of the 7 study hypotheses, and with the team performance mo
robot-assisted technical search (Figure 3). What does it all mean? And yes, what about
that 7th hypothesis? That, dear reader, is the topic for discussion in the next chapter
92
gs from the study are discussed, focusing on the
mmunication in distributed US&R teams through a shared remote visual
presenc
Chapter 6
Discussion
In this chapter, the findin
hypothesized interaction between the two main study variables as a way of interpreting
the results. The main effect found for RSVP on performance (but not shared mental
models or team process) is discussed in relation to the model of robot-assisted technical
search team performance that was introduced in chapter 3. The effects found for the
influence of location on the development of a shared (team) situation model (and
subsequent lack of effect on team process and performance) are considered, and linked
with the site effects noted in the multilevel regression analyses of performance.
Theoretical and practical applications of these results are broached, as are the limiting
factors that bound the findings. The chapter closes with some parting thoughts and
conclusions about the future of RSVP in human-robot teams.
This research began by proposing the use of mobile rescue robots as a way of
augmenting co
e consisting of the robot’s view. I hypothesized that by helping team members
build a shared mental model, the use of mobile robots as a shared visual presence in
remote environments might lead to more effective distributed team performance in robot-
assisted technical search teams. This led to two main research questions: 1) does using
the robot as remote shared visual presence affect team process and performance; and 2)
can RSVP facilitate performance in distributed human-robot teams? These two research
93
effect for use of RSVP technology, then the differences between
collocated teams and distributed team ess in the teams that utilize
RSVP t
nce
do)
SVP
no
d the
st,
eam
questions culminated in 7 hypotheses, 6 of which have been discussed in the previous
chapter. The last hypothesis states:
H7: If there is a main
s will be significantly l
han in those that do not.
Results from the first 6 hypotheses show that there was an effect for using the
robot as RSVP, i.e., it did seem to help performance. However, the process did not unfold
as predicted in the model and hypotheses delineated in Chapter 3. There was no evide
that RSVP contributed to the shared mental model held by team members, and conflicting
support for its influence on team processes. As far as the location of team members goes,
I anticipated that collocated teams would perform better all around (as they typically
and hypothesized in H7 that if RSVP had an effect, then the distributed teams with R
might do a little better than the distributed teams without it (not as well as collocated
teams, but better). What happened instead was this: the collocated teams didn’t do better,
after all. In fact, the distributed teams had better shared mental models, and there were
differences at all between the collocated and distributed teams in terms of team process or
performance.
So, RSVP worked, but not as predicted, and it is unclear whether it helpe
distributed teams catch up to the collocated ones in terms of performance, because the
collocated teams didn’t perform better in the first place. Two main questions arise. Fir
if RSVP didn’t help the teams form better shared mental models, or have better t
processes, then how did it influence performance? Second, what’s up with the collocated
teams not performing better than the distributed teams? To answer the first question, look
94
e
een
,
to the theoretical model in Chapter 3. As for the second question, I believe this is where
the site effects come into play. Let’s look at each of these in turn, and perhaps some light
will be shed on H7.
RSVP and the Model
To review the model of team performance in robot-assisted technical search
(Figure 3), RSVP was posited to augment communication between team members by
giving the tether-handler the same robot data from the remote search environment as the
robot operator. By having the same visual referent, the team members would form a
richer shared mental model of the search environment and process as they performed th
search. This shared situation model, formed through enhanced communications betw
the team members, would in turn positively affect team processes (Communication
Effectiveness, Support/Backup Behavior, Leadership/Initiative, and Team Situation
Awareness), thus leading to better team performance. Let us look at the relationships
(signified by lines/arrows) among the constructs in the model and examine where the
findings apply. First, the central dotted-line box that holds the shared (team) situation
model is crossed by the bi-directional line (communications) between the robot operator
and tether-handler, representing that the shared mental model is formed through
communications between team members. This is paralleled in the results by the
predictive relationship between the RASAR-CS categories search and navigation and the
map score which measured the team mental model. The arrow connecting the shared
(team) situation model box to the team process box appears in the results of the
supplemental analyses of team process, where the RASAR-CS categories of plan, report
and tether-operator predicted the mean team process scores. (An arrow going back from
95
re
the
we
(via
and
d
s. Is the
es.
VP and
s to
ituation model through the robot operator and tether-handler’s own
mental models of the search process er, the unshared data
receive
the
of
team process to the shared situation model can be added to illustrate the reciprocal natu
of the team process/shared mental model relationship observed in the supplemental
analyses for the shared mental model. Recall that the team process ratings for SA
predicted the map scores.) Next, the arrow from team process to team performance in
model is mirrored in the results by the fact that team process ratings for Communication
predicted performance scores. Through the findings of the supplemental analyses,
have traced the process of shared mental model formation through communication
the RASAR-CS) and established a reciprocal link between the shared mental model
team process; lastly, we have shown that the team process of communication is linke
with team performance (thereby validating a portion of Klomoski & Mohammed’s
model). What has not been identified is how RSVP contributed to that proces
model described in Figure 3 deficient? I think not, but it does not completely match what
was measured in the study. I assumed that the effects of RSVP would be captured in the
communications between team members, and to some degree, they were; differences
between RSVP teams and no-RSVP teams were observed in the RASAR-CS analys
However, the hypothesis stated in H1 does not account for the path between RS
the shared situation model (“….teams having access to RSVP technology will generate
richer, more accurate team situation models…”. In the actual model, RSVP contribute
the shared (team) s
and environment. Moreov
d by each of the team members (e.g., the OCU interface for the robot operator,
and the part of the search space visible to the tether-handler from where he inserted
robot into the void) is not accounted for. It may be that combining these different kinds
96
re are
ental model created
together might clarify exactly what it is in RSVP that helps the team perform more
effectively. Too, the model is somewhat deficient in that it does not take into account the
other factors which may contribute: the individual, team, environmental and
organizational antecedents listed in Kraiger and Wenzel’s framework for shared mental
models (1997), or the resources available and other factors feeding into team capacity in
Klimoski and Mohammed’s model of team performance (1994). What this model did do
is illuminate the processes that go on in robot-assisted technical search teams, and
demonstrate the value of RSVP as a team resource. To understand its (RSVP)
contribution toward team performance, however, the model must be expanded to include
ithin the model.
Location and Site Effects
Turning to the location question: why did the collocated teams not outperform the
distributed teams, and what do the site effects have to do with it? To answer these
questions, comparisons of performance by location and RSVP condition must be made
input with the data from the robot acting as RSVP contributes to each team member’s
individual situation model in a unique way. While it is possible to make some inferences
about each individual’s mental model by looking at what he or she talked about, the
obviously some internal cognitions that are not voiced by team members. So, while
analyzing communications between team members can help trace the process of shared
mental model formation, it cannot completely capture the formation of each team
member’s individual model of the situation. Taking measures of team members’
individual mental models and comparing them with the team m
other constructs, and refined to explain the relationships between existing constructs
w
97
all
n
the
independent and this is not a representation of a statistically significant interaction.
In Figure 9, the solid line represents the distributed teams and the dotted line, the
collocated teams. The mean performance scores for collocated teams across RSVP
conditions are very similar (for RSVP, M = 4.64, SD = 3.42; for no-RSVP, M = 4.86, SD
across sites. Before looking at these comparisons, though, we need to revisit H7. Rec
that this hypothesis assumed that collocated teams would outperform distributed teams,
and predicted that if RSVP had an effect on performance, the differences between
collocated and distributed team performance would be smaller in the RSVP conditio
than in the no-RSVP condition. To establish a start point for the discussion, the mean
performance scores for the four combinations of experimental conditions are presented in
Figure 9. It is important to note that these are not independent groups, and the lines in
figure do not represent a statistically significant interaction.
Figure 9. Mean performance scores plotted by location and use of RSVP. Scores are not
98
M = 2.30, SD = 1.99). The wide variance for
all four with
the
o-
y had
RSVP. The collocated teams with no-RSVP performed slightly better than the distributed
teams without RSVP, but not by much. Looking next at NJTF-1 (lower graph), this time
the results followed the pattern predicted in the location hypotheses: collocated teams had
better performance than distributed teams regardless of whether they had RSVP or not.
The results for the RSVP hypotheses, however, are contradictory: RSVP helped teams in
the distributed condition, but in the collocated condition, they actually performed better
without it. At both sites, something else seems to have influenced the performance of the
= 3.98). The means for the distributed teams, in contrast, are markedly different (for
RSVP, M = 5.07, SD = 2.04; for no-RSVP,
mean scores underscores the nature of the data—these are repeated measures
teams having scores in more than one condition. If these were independent groups,
however, the visual impact of the interaction is obvious: distributed teams with RSVP
performed as well as collocated teams. The fact that there was no main effect for location
points to something else making this effect occur: and so we must address the likely
culprit, site effects. Figure 10 presents graphs of mean performance scores according to
location (distributed or collocated) and use of RSVP at the two sites, NASA-Ames in
California, and NJTF-1 in New Jersey. Again, all teams completed runs in 2 of the 4
conditions, so these data are dependent; they are used here to tease apart the nature of
differences between sites.
Looking first at NASA-Ames (upper graph), the results followed the pattern
predicted in RSVP hypotheses: RSVP teams had better performance scores than n
RSVP teams in both conditions. The results for the location hypotheses, however, were
not as expected: the distributed teams outperformed the collocated teams when the
99
teams. There are two possibilities that come to mind based on what is known about the
two sites: experience, and team cohesion.
Experience (firefighting, US&R, and technology) was included in the study
analyses because of its potential effect on study outcomes of interest. In the multilevel
regression analyses for shared mental models, team process, and team performance,
however, experience did not prove to be a significant predictor. However, comparisons of
firefighting experience between the two sites (Table 5) reveal that while both NASA-
Ames and NJTF-1 have a significant percentage of highly experienced participants with
15+ years of experience (46% and 59%, respectively), at NASA-Ames there were also a
significant number of participants with very little time on the job. There, 40% of the
participants had 0-3 years of firefighting experience; at NJTF-1, that percentage was
much s
NJTF-1 id so
much better than the no-RSVP teams at NASA-Ames: it could be that those less
experienced participants were more receptive to using a new type of search tool, as they
did not have a backlog of prior, more traditional search experiences to overcome.
maller (18%). It may be that having fewer participants with less experience at
made a difference. This could also be part of the reason the RSVP teams d
100
Figure 10. Bar graphs showing mean performance scores at NASA-Ames and NJTF-1 sites according to location (distributed or collocated) and use of RSVP. Means are dependent.
101
The second possible explanation is differences in team cohesion across sites.
Team cohesion is the degree to which team members are attracted to their team and
desire to remain in it. Components of team cohesion include interpersonal attraction,
group pride, and task commitment (Driskell et al., 2003). The NASA-Ames teams had a
mix of DART responders that worked together regularly onsite and task force members
from other teams across the country who came to participate in the training exercise. Of
the 14 teams, only 5 consisted of two DART responders; the rest were paired with
visiting responders from other units. The NJTF-1 teams, in contrast, were all from the
same Task Force and had many years of experience working with each other. This could
explain to some degree why the collocated teams there performed best in the
collocated/no-RSVP condition: it most closely resembled their normal pattern of work.
There are certainly other factors that may have contributed to these patterns of
performance, ranging from environmental conditions to individual differences in
technology acceptance. The observations regarding experience and team cohesion seem
to be the most defensible, as they are supported by the demographic details in the study.
Do they offer any support for the existence of the location x RSVP interaction predicted
in H7? That is a matter of speculation. It seems that RSVP can help distributed teams be
more like collocated teams in technical search in terms of performance—but it cannot
replace the “human factors” of individual experience and team strength (cohesion) that
comes from team members knowing and working with each other over time. In any case,
it is safe to say that there was most definitely a site x location x RSVP interaction.
102
odel
ssisted
e
s
he
ect the
Theoretical and Practical Implications
Theoretical Implications
One of the proposed theoretical contributions of this study was to test a portion of
Klimoski & Mohammed’s model, and to extend it to show how the shared mental m
is formed through communication, as posited in the more specific model of robot-a
team performance described in Chapter 3. The findings of the supplemental analyses
support the relationship between team processes and performance, and validate the
reciprocal link between the shared mental model and team process, thereby providing
support for that portion of the Klimoski & Mohammed framework. As a further
theoretical contribution, using this study’s model of robot-assisted team performance, th
process of shared mental model formation, and its influence on team process and
performance was traced through communication via the RASAR-CS, providing support
for the concept of using communication as a measure of the mental model that emerge
through the interaction between team members. Neither model truly expresses all of t
constructs and interrelationships that characterize team performance in extreme
environments such as US&R. Klimoski and Mohammed’s model does not adequately
capture the influence of various constructs on team mental models, and the model of
robot-assisted team performance neglects the broader influences that aff
individuals and the team in incident response. As a thought exercise, how would one
model these influences?
To begin with, Klimoski and Mohammed say that team mental models reflect
team processes (which I agree with). In fact, I would go so far as to say that the
development of team mental models is a team process. They also say that team mental
103
it
team processes which include the
develop ds to
nt
uld
nd
.
mmunity as well as those brought to bear by the organization,
am and individuals. The interaction of these three broad factors (environment, time, and
models are a force with which to harness a team’s capacity, or readiness. In this we are
also in agreement. However, the authors don’t discuss team capacity other than to say
is the team’s latent potential for effective process and performance. I think that this (team
capacity) is a key element in how well teams can form shared mental models, and
deserves to be looked at as an antecedent of the
ment of shared mental models. Moreover, I think the concept of capacity nee
be expanded to levels above and below that of the team. Potential influences on team
capacity could include the team’s current work-life and past work history, cohesion,
group tensions, and relationships with other groups both within and outside its pare
organization. Other important influences that need to be acknowledged and defined are
individual capacity and organizational capacity. Individual capacity, or readiness, wo
include not only levels of training and experience, but other variables that could impact
performance in extreme environments, such as personal morale, emotional state, and
cognitive readiness (Wood, Lugg, Hysong, & Harm, 1999). Organizational capacity
might include leadership, command structure, resource allocation, and both inter- a
intra-organizational coordination and cooperation. All three of these capacity levels
(individual, team, and organization) are impacted by the environment, time, and available
resources. Environmental considerations include the weather, extent of
damage/disruption caused by the incident, and current level of danger/continued risk
Temporal factors include the time elapsed since the incident occurred, time since
mobilization of response, etc. The resources available include considering those existing
in the environment and co
te
104
e capacity (both perceived and real) of the individual, team, and
organiz
hat
ared
l
resources) determines th
ation. These factors in turn all play a role in the development of team mental
models, and other team processes that ultimately affect team performance. Note that this
multilevel model could be further expanded to include constructs of individual and
organizational effectiveness and performance.
Using a map constructed by both team members as a measure of the shared
mental model is another contribution that has implications for future research in shared
mental models. The fact that it showed convergence with elements in the RASAR-CS
shows that we’re getting at the same underlying construct. Maps are a particularly
valuable measure in this domain, more illustrative of the type of shared mental model t
is needed and formed during the task. Still, it is a static representation of a dynamic
process. Maybe there is a way to capture the development of the shared mental model
over time by looking at how the map is created at different points during the search
process…when are details of spatial dimension added, how does the search path form,
and do temporal elements play a role in the mental model as captured on the map
constructed by the team members? Too, it behooves us to investigate the influence of the
individual mental models held by team members; perhaps, as suggested earlier, by having
the two team members construct their maps separately, and compare them with the sh
version prepared after the fact. Looking to see what elements are merged, what gets
dropped or corrected, and observing how team members come to agreement over the fina
correct version of the map would offer an intriguing window into the process of shared
mental model formation.
105
The practical implications of this study are numerous. The fact that RSVP helped
US&R
he
iving
ay
hink about
naviga
d
be
he
t
Practical Implications
technical search teams perform more effectively is encouraging, even though we
haven’t quite figured out how it helps. There are some troubling aspects, however, that
need to be taken into consideration. By giving the tether-handler the shared view from t
robot in the search environment, we inadvertently distracted him from his role by g
him a window into the operator’s task of navigation. The visual attraction of RSVP m
have an attentional tunneling effect, where team members react to what they are seeing
(in terms of navigation) instead of focusing on the primary task of helping search for
victims. Clear roles and task goals need to be identified for team members working in
human-robot teams that optimize their strengths and available resources. Moreover, we
need to train them to function effectively in these roles. The operator has to t
tion and search simultaneously, and it’s the search that suffers—this is the task the
tether-handler can help take on because he has the luxury of not having to drive…an
therefore as the “passenger” can pay far more attention to what’s being seen in the
environment. In the future, when advances in wireless technology make the tether itself
unnecessary, this role could become completely cognitive and mission-oriented. While it
is feasible (and in many ways desirable) that this role could be fulfilled by someone
outside the Hotzone, in reality US&R responders work in teams—no one is going to
out there alone, so it makes sense to utilize that fact. It takes a NASA team to handle t
Mars Rover, a military team to operate the Predator UAV, and the same is true here: i
takes a US&R team to fully use a rescue robot. What is needed is training at the team
106
team
creased.
s
rsonnel. Once a victim has been located, the rescue and extrication
process
level—how to coordinate, what information to share, and what roles and tasks to focus
on.
In terms of design, what can be done to make RSVP better? What roles can the
robot take on as it gains more autonomy? If we can make navigation autonomous,
members can concentrate on the real task at hand (locating victims). The key question
then becomes, what are the salient elements of RSVP that need to be emphasized?
Indicators of the robot’s state and situatedness relative to the search environment are
clearly of interest. If team members had a way of visually annotating their search
(perhaps with a tablet-pc type of interface), and sharing that with other team members,
the power of RSVP as a builder of mutual knowledge would be dramatically in
Finally, having the ability to “play back” earlier portions of their search for comparison
would help team members synthesize information obtained from the robot.
The use of RSVP has applications not only in other functions with US&R teams,
but beyond US&R as well. Within the US&R system, RSVP has potential uses for
coordination of actions across operations - technical search, structural evaluation, rescue
and victim extrication, medical reachback and victim management, safety monitoring,
logistics and resource management. Many of these operations could be expanded to
include offsite pe
can take from 4-10 hours. During the extrication process, a robot providing
shared visual presence can be used as a communication channel not only for medical
personnel, but also for a counselor or family member at a remote location to provide
comfort/reassurance/encouragement to the victim during the prolonged extrication
period. Safety monitoring is another potential domain application. US&R operations
107
e team
during work process and the ongoing condi oid, allowing an objective
viewpo r
nse.
cue
eeds
o
is
require the presence of a safety officer onsite during rescue operations. If a robot goes
into a remote work area (deep into a void or confined space) with a rescue team, it can
provide the rescue squad leader outside the void space with a visual image of th
tion of the v
int (that is, not in the void) from which to monitor the work process and look fo
signs of fatigue, physical danger, or unnoticed errors. This simultaneously opens new
opportunities for logistics and resource management during an emergency respo
Having the robot with a rescue team in a remote, confined work space provides the res
squad leader or incident commander with a more efficient communication channel than
radio, as the robot’s view can provide common ground for more implicit team
coordination. For example, the leader may conduct an onsite assessment of the team’s
progress, estimated completion time of the task, and the team’s ongoing logistical n
by observing the activities via the shared visual presence of the robot. In summary, using
mobile robots as RSVP in US&R environments can serve as a conduit for the transfer of
information horizontally (to other team members involved in common tasks), vertically t
higher/lower levels (squad leaders, incident command, victims) and through diffusion to
specialists (structural engineers, medical personnel) and others (family members, task
force liaisons).
The search and safety monitoring capabilities offered by RSVP are a natural fit
for the military and homeland security domains, and the recent advances in telemedicine
bode well for extending its usage into that area as well. Remote assisted care-giving
not too far away, as evidenced by a recent article in Newsweek where two sons rescued
their elderly mother from her apartment where she had suffered a stroke. They kept a
108
t
pan.
has its
difficulties in trying to conduct quasi-experimental studies in field
setting t of
e
es
that
e
windy,
ere not told that it was an assessment tool. It may be
necessary in future studies t e measure—
an
e
video-camera in her home for communicating via the internet, and noticed she was no
walking around the apartment. Imagine being able to help her remotely through a care-
giver robot. Creepy? Perhaps, but already underway in research labs in Ja
Limitations
Before closing, certain limitations should be acknowledged when considering
these findings. A wise man once told me never to apologize for field research-- it
place, its value is immeasurable, and not meeting the strict requirements of the
experimentalist model is what gives it power in terms of external validity. That being
said, there are always
s. Creating a distributed condition when participants had to stay within earsho
each other was difficult: though participants faced away from each other and were told to
“pretend there’s a wall between you”, this manipulation may not have been strong
enough to find differences between distributed and collocated teams (though there wer
differences in map scores that suggest it was effective). The map measure used to ass
the team mental model of the search has some vulnerabilities of its own. It is likely
some teams had better drawing ability than others, which may have led to differences in
ratings for reasons other than the quality of the shared mental model. It is also possibl
that some teams took more care in constructing their maps than others, for various
reasons: the circumstances in which they had to draw were not convenient (cold,
on top of a rubble pile), and they w
o train participants on how to respond to th
getting them to “tell the story” of their search process on paper. However, I do see it as
appropriate measure in this domain, and think it shows promise as a team-level measur
109
ed by
lly-
of shared mental models. Other measures, such as ranking lists of relevant factors in
order of importance, do not lend themselves easily to this application. Insertion of a new
technology changes the list of relevant factors, and creates new factors.
The team process ratings used in this study were based limited to 4 certain
dimensions (communication effectiveness, support, leadership, and situation awareness).
There are many other ratings categories that can be explored. For example, team
orientation, monitoring, and feedback are other dimensions of team process propos
Dickinson & McIntyre (1997) that might provide insight into how RSVP’s influence on
performance occurs.
Finally, the RASAR-CS is a complex communication coding scheme. Most
coding analyses classify statements into 4 or 5 categories at best; the RASAR-CS has 4
dimensions with a total of 30 possible coding categories. There are many statements that
fall into more than one category, particularly in terms of content and function. This is
reflected in the lower estimates of rater agreement for these dimensions. The rate of
agreement is acceptable for the purpose of the study, and the findings point to the
RASAR-CS as a valuable technique for getting at the details which give insight into the
results. However, other techniques can and should be explored. For example, coding
categories from the RASAR-CS could serve as the basis for construction of behaviora
anchored rating scales or frequency checklists, offering another avenue into the
observation of team functioning.
Parting Thoughts and Future Directions
This study was motivated by the desire to see whether using mobile robots as a
shared visual presence in remote environments could help distributed teams in US&R
110
getting
there’s an obvious gain for the first part; if it worked with team
embe nment
be
t what if it breaks down? We need to know what the
salient aspects of RSVP are so that we can improve robot RSVP design—and we need to
know what the shortcomings and pitfalls of using RSVP are so that we can deal with
them, either through design, or more likely, through training teams how to use it most
effectively. In a training program for rescue robot teams developed and piloted by the
Center for Robot-Assisted Search and Rescue (Burke, Murphy, & Kalyadin, 2005), initial
measures of participant reaction and learning yielded encouraging results. Following
Kirkpatrick’s (1994) model of training evaluation, future developments in training should
include criteria for behavior (performance on the job) and results (impact on
organizational objectives). Other future directions for this research include continuing the
search for the elusive shared mental model. Results from this study suggest investigation
of the influence of individual mental models on the development of a shared mental
perform more effectively. The reasoning behind this was twofold. Effective performance
in this domain consists of two main elements: successfully locating victims and
them out of a dangerous environment, and doing it without getting hurt. If RSVP could
help teams locate victims,
m rs in different locations, then people safely away from the dangerous enviro
could help search for victims—people with fresh minds, clear eyes and no cognitive
fatigue from being onsite for 48 hours straight. The import of this cannot be over-
emphasized. US&R teams operate in extreme environments and are under tremendous
emotional strain in addition to the physical and cognitive stress. The findings from the
study seem to point to RSVP as an idea worth pursuing—but further research needs to
performed to figure out how it works. It’s analogous to driving a car: you don’t have to
know how it works to drive one, bu
111
odel. In addition, the use of maps as a measure of shared mental models merits further
xploration. Finally, I have introduced a multilevel conceptual model of team
performance in extreme environments t the egg of an idea. This model needs
huge potential as a way of allowing
unexpe e
chance to use it. Robots with RSVP took the US&R search teams into an environment
they co fer an
in
hysical environments without being physically present—through the contributions of
their ne
m
e
hat is merely
to be refined and tested.
Because of their mobility, robots offer
humans a presence in remote environments. As with any new technology, new and
cted uses and applications will emerge as the technology matures and people hav
a
uldn’t have reached to search before. In like fashion, robots with RSVP of
entry into a whole new world of teaming—where team members can work together
p
w team member, the robot.
112
References
Ambro er, s, IEEE
[see also IEEE Expert], 15(4), 57-63.
Brannic rement. & C. Prince (Eds.), Team Performance Assessment
and Measurement: Theory, Methods and Applications (pp. 3-16). Mahwah, NJ:
Brenna uses, and
alternatives. Educational and Psychological Measurement, 41, 687-699.
Brooks :
-504.
h rant (No. CRASAR TR2005-14).
Tampa, FL: Center for Robot-Assisted Search and Rescue.
Burke, than One. Paper presented at the
IEEE RO-MAN 13th International Workshop on Robot and Human Interactive
Burke, ion Awareness and Task Performance in
Robot-Assisted Technical Search: Bujold Goes to Bridgeport (No. CRASAR-
Burke, ddle, D. L. (2004). Moonlight in
Miami: A field study of human-robot interaction in the context of an urban search n,
se, R. O., Aldridge, H., Askew, R. S., Burridge, R. R., Bluethmann, W., DiftlM., et al. (2000). Robonaut: NASA's space humanoid. Intelligent System
k, M. T., & Prince, C. (1997). An overview of team performance measuIn M. T. Brannick, E. Salas
Erlbaum.
n, R. L., & Prediger, D. J. (1981). Coefficient kappa: Some uses, mis
, R. (2002). Flesh and Machines: How Robots Will Change Us. New YorkPantheon.
Bruemmer, D. J., Few, D. A., Boring, R. L., Marble, J. L., Walton, M. C., & Nielsen, C.
W. (2005). Shared understanding for collaborative control. Systems, Man and Cybernetics, Part A, IEEE Transactions on, 35(4), 494
Burke, J. L., Murphy, R., & Kalyadin, D. (2005). Annual Report for the Florida Hig
Tech Corridor Workforce Enhancement G
J. L., & Murphy, R. R. (2004a, Sept. 20-22, 2004). Human-Robot Interaction in USAR Technical Search: Two Heads are Better
Communication, Kurashiki, Okayama, JAPAN.
J. L., & Murphy, R. R. (2004b). Situat
TR2004-23). Tampa, FL: Center for Robot-Assisted Search and Rescue.
J. L., Murphy, R. R., Coovert, M. D., & Ri
and rescue disaster response training exercise. Human-Computer Interactio19(1-2), 85-116.
113
s, 34(2), 103-112.
up
asper, J., & Murphy, R. (2002). Workflow Study on Human-Robot Interaction in USAR.
d Cybernetics Part B-Cybernetics, 33(3), 367-385.
l
. ing
ambridge: Cambridge University Press.
, 37-46.
J. (2003).
Coovert, M. D., & Burke, J. L. (2005). Leadership and decision making. In Y. Amichai-
Hamburger (Ed.), The social net: Human behavior in cyberspace (pp. 219-246).
oovert, M. D., & Foster Thompson, L. (2001). Computer Supported Cooperative Work:
Cramton, C. D. (2001). The mutual knowledge problem and its consequences for
dispersed collaboration. Organization Science, 12(3), 346.
Burke, J. L., Murphy, R. R., Rogers, E., Lumelsky, V. J., & Scholtz, J. (2004). Final report for the DARPA/NSF interdisciplinary study on human-robot interaction. Ieee Transactions on Systems Man and Cybernetics Part C-Applications and Review
Cannon-Bowers, J. A., Salas, E., & Converse, S. A. (1993). Shared mental models in
expert team decision making. In N. J. Castellan (Ed.), Individual and grodecision making: Current issues (pp. 221-246). Hillsdale, NJ: Erlbaum.
CPaper presented at the IEEE International Conference on Robotics and Automation (ICRA 2002).
Casper, J., & Murphy, R. R. (2003). Human-robot interactions during the robot-assisted
urban search and rescue response at the World Trade Center. Ieee Transactions on Systems Man an
Clancey, W. J. (2004). Roles for agent assistants in field science: understanding persona
projects and collaboration. Systems, Man and Cybernetics, Part C, IEEE Transactions on, 34(2), 125-137.
Clark, H. H., & Marshall, C. E. (1981). Definite reference and mutual knowledge. In A
K. Joshi, B. L. Webber & I. A. Sag (Eds.), Elements of discourse understand(pp. 10-63). C
Cohen, J. A. (1960). Coefficient of agreement for nominal scales. Educational and
Psychological Measurement, 20 Cooke, N. J., Kiekel, P. A., Salas, E., Stout, R., Bowers, C., & Cannon-Bowers,
Measuring team knowledge: A window to the cognitive underpinnings of team performance. Group Dynamics, 7(3), 179-199.
New York: Oxford University Press.
CIssues and implications for workers, organizations, and human resource management. Thousand Oaks, CA: Sage.
114
. Mahwah, NJ: Erlbaum.
Diez-R w
gical ,
s,
ndsley, M. R. (1988). Design and evaluation for situation awareness enhancement.
Teamwork and the development of shared problem models. In E. Salas & S. M.
l
Foushee, H. C., Lauber, J. K., Baetge, M. M., & Acomb, D. B. (1986). Crew factors in
flight operations: III. The operational significance of exposure to short-haul air
DeShon, R. P., Kozlowski, S. W. J., Schmidt, A. M., Milner, K. R., & Wiechmann, D. (2004). A multiple-goal,multilevel model of feedback effects on the regulation ofindividual and team performance. Journal of Applied Psychology, 89(6), 1035-1056.
Dickinson, T. L., & McIntyre, R. M. (1997). A conceptual framework for teamwork
measurement. In M. T. Brannick, C. W. Prince & E. Salas (Eds.), Team Performance Assessment and Measurement: Theory, methods and applications
oux, A. V. (2000). Multilevel Analysis in Public Health Research. Annual Revieof Public Health, 21(1), 171-192.
Driskell, J. E., Radtke, P. H., & Salas, E. (2003). Virtual teams: Effects of technolo
mediation on team performance. Group Dynamics-Theory Research and Practice7(4), 297-323.
Endo, Y., MacKenzie, D. C., & Arkin, R. C. (2004). Usability evaluation of high-level
user assistance for robot mission specification. Systems, Man and CyberneticPart C, IEEE Transactions on, 34(2), 168-180.
EPaper presented at the Human Factors Society 32nd Annual Meeting, Santa Monica, CA.
Endsley, M. R. (1995). Toward a theory of situation awareness in dynamic systems.
Human Factors, 37(1), 32-64. Fiore, S. M., & Schooler, J. W. (2004). Process mapping and shared cognition:
Fiore (Eds.), Team Cognition: Understanding the factors that drive process and performance (pp. 133-152). Washington, D.C.: American PsychologicaAssociation.
Fleiss, J. L. (1981). Statistical methods for rates and proportions (2nd ed.). New York:
Wiley.
transport operations (No. NASA Technical Memo No. 88322). Mountain View, CA: NASA.
115
ussell, S. R., Setlock, L. D., & Kraut, R. E. (2003, April 5-10, 2003). Effects of head-
),
ssell, S. R., Setlock, L. D., & Parker, E. M. (2003, April 5-10, 2003). Where do helpers
mputer-Human Interaction (CHI 2003), Ft. Lauderdale, FL.
l n Computer Supported Cooperative
Work, Chicago, IL.
Gergle,al of
utwin, C., & Greenberg, S. (2004). The importance of awareness for team cognition in
on:
ackman, J. R. (1987). The design of work teams. In J. W. Lorsch (Ed.), Handbook of
inds, P. J., & Bailey, D. E. (2003). Out of sight, out of sync: Understanding conflict in
Hox, J. Erlbaum.
gen, D. R., Hollenbeck, J. R., Johnson, M., & Jundt, D. (2005). Teams in organizations:
Kanki, B. G., Lozito, S., & Foushee, H. C. (1989). Communication indices of crew coordination. Aviation, Space, & Environmental Medicine, 60(1), 56-60.
Fmounted and scene-oriented video systems on remote collaboration on physical tasks. Paper presented at the Conference on Computer Human Interaction (CHIFt. Lauderdale, Florida.
Fulook? Gaze targets during collaborative physical tasks. Paper presented at the Conference on Co
Gergle, D., Kraut, R. E., & Fussell, S. R. (2004a). Action as language in a shared visua
space. Paper presented at the Conference o
D., Kraut, R. E., & Fussell, S. R. (2004b). Language efficiency and visual technology: Minimizing collaborative effort with visual information. JournLanguage and Social Psychology, 23(4), 491-517.
Gdistributed collaboration. In E. Salas & S. M. Fiore (Eds.), Team CognitiUnderstanding the factors that drive process and performance (pp. 177-202). Washington, D.C.: American Psychological Association.
HOrganizational Behavior (pp. 315-342). Englewood Cliffs, NJ: Prentice-Hall.
Hdistributed teams. Organization Science, 14(6), 615-632.
(2002). Multilevel Analysis: Techniques and Applications. Mahwah, NJ:
IlFrom input-process-output models to IMOI models. Annual Review of Psychology, 56(1), 517-543.
Jiang, X., Hong, J. I., Takayama, L. A., & Landay, J. A. (2004). Ubiquitous computing
for firefighters: Field studies and prototypes of large displays for incident command. Paper presented at the Conference on Computer-Human Interaction 2004, Vienna, Austria.
Jones, H., & Hinds, P. (2002). Extreme work groups: Using SWAT teams as a model for
coordinating distributed robots. Paper presented at the CSCW 2002 Conference on Computer Supported Cooperative Work, New York.
116
Kiekel, m cognition through analysis of communication data. In M. J. Smith, G.
Salvendy, D. Harris & R. J. Koubek (Eds.), Usability evaluation and interface
. Kirkpa rograms: The four levels. San Francisco:
Berrett-Koehler.
Klimoski, R., & Mohammed, S. (1994). Team mental model: Construct or metaphor? Journal of Management, 20(2), 403-437.
Kraiger
Brannick, E. Salas & C. Prince (Eds.), Team performance assessment and h, NJ:
Erlbaum.
Krauss nicative effectiveness. In J. Galegher & R. E. Kraut (Eds.), Intellectual teamwork: Social
Kraut, ation as a conversational
resource in collaborative physical tasks. Human-Computer Interaction, 18, 13-49.
Kraut, formation in shared visual spaces: Informing the development of virtual
co-presence. Paper presented at the Conference on Computer Supported
andis, J. R., & Koch, G. G. (1977). The measurement of observer agreement for
MacMillan, J., Entin, E. E., & Serfaty, D. (2004). Communication overhead: The hidden
cost of team cognition. In E. Salas & S. M. Fiore (Eds.), Team Cognition:
arble, J. L., Bruemmer, D. J., & Few, D. A. (2003). Lessons learned from usability tests
P. A., Cooke, N. J., Foltz, P., & Shope, S. M. (2001). Automating measurementof tea
design (pp. 1382-1386). Mahwah, NJ: Lawrence Erlbaum Associates
trick, D. L. (1994). Evaluating training p
, K., & Wenzel, L. (1997). Conceptual development and empirical evaluation of measures of shared mental models as indicators of team effectiveness. In M.
measurement: Theories, methods and applications (pp. 63-84). Mahwa
, R. M., & Fussell, S. R. (1990). Mutual knowledge and commu
and technological foundations of cooperative work. (pp. 111-145): Lawrence Erlbaum Associates, Inc.
R. E., Fussell, S. R., & Siegel, J. (2003). Visual inform
R. E., Gergle, D., & Fussell, S. R. (2002, November 16-20, 2002). The use of visual in
Cooperative Work 2002, New Orleans, LA.
Lcategorical data. Biometrics, 33(1), 159-174.
Understanding the Factors That Drive Process and Performance (pp. 61-82). Washington, D.C.: American Psychological Association.
Mwith a collaborative cognitive workspace for human-robot teams. Paper presentedat the IEEE International Conference on Systems, Man and Cybernetics.
117
. Journal
of Applied Psychology, 85(2), 273-283.
McGra rmance. Englewood Cliffs, NJ: Prentice Hall.
Morris, N. M., Rouse, W. B., & Zee, T. A. (1987). Adaptive aiding for human-computer control: An enhanced task environment for the study of human problem solving in
Murph
Noldus, L., Trienes, R., Hendriksen, A., Jansen, H., & Jansen, R. (2000). The Observer
Video-Pro: New software for the collection, management, and presentation of search
mputers, 32, 197-206.
lson, G. M., & Olson, J. S. (2003). Distance matters. Human Computer Interaction, 15,
Orasan
). Princeton, NJ: Princeton University, Cognitive Sciences Laboratory.
Orasan reness through communication. Paper presented at the International Conference on Experimental Analysis and
Peterso 001). Controller-to-controller communication
and coordination taxonomy (C T) (No. DOT/FAA/AM-01/19). Washington, D.C.:
Marks, M. A., Mathieu, J. E., & Zaccaro, S. J. (2001). A temporally based framework and taxonomy of team processes. Academy of Management Review, 26(3), 356-376.
Mathieu, J. E., Heffner, T. S., Goodwin, G. F., Salas, E., & Cannon-Bowers, J. A. (2000)
The influence of shared mental models on team process and performance.
th, J. E. (1984). Groups: Interaction and Perfo
complex situations (No. TR-87-011). Norcross, GA: Search Technology, Inc.
y, R. (2000). Introduction to AI Robotics. Cambridge, MA: MIT Press. Murphy, R. R., Blitch, J. G., & Casper, J. L. (2002). RoboCup/AAAI Urban Search and
Rescue events: Reality and competition. AI Magazine, 37-42.
time-structured data from videotapes and digital media files. Behavior ReMethods, Instruments, and Co
Nourbakhsh, I. R., Sycara, K., Koes, M., Yong, M., Lewis, M., & Burion, S. (2005).
Human-robot teaming for search and rescue. Pervasive Computing, IEEE, 4(1), 72-79.
O139-179.
u, J. (1990). Shared mental models and crew decision making (No. Technical Report No. 46
u, J. M. (1995). Evaluating team situation awa
Measurement of Situation Awareness, Daytona Beach, FL.
n, L., Bailey, L., & Willems, B. (24
Department of Transportation, Federal Aviation Administration, Office of Aerospace Medicine.
118
tic nges and results. Robotics and Autonomous
Systems, 42(3-4), 271-281.
Rasbas .0. re for Multilevel Modelling, University of Bristol.
achs, J. (2000). Using a small sample Q sort to identify item groups. Psychological
Salas, E
tanding of team performance and training. In R. W. Swezey & E. Salas (Eds.), Teams: Their training and performance. (pp. 3-29): Ablex Publishing.
Scholtz awareness in search and rescue. Paper presented at the Robotics and
Automation, 2004. Proceedings. ICRA '04. 2004 IEEE International Conference
Sherida control.
Cambridge, MA: MIT Press.
Smith-J d J. A. Cannon-Bowers & E. Salas (Eds.),
Making decisions under stress (pp. 61-88). Washington, D.C.: American Psychological Association.
Sonnenwald, D., & Pierce, L. (2000). Information behavior in dynamic group work
contexts: Interwoven situational awareness, dense social networks and contested collaboration in command and control. Information Processing and Management, 36, 461-479.
Stevens, J. P. (2002). Applied Multivariate Statistics for the Social Sciences (4th ed.).
Mahwah, NJ: Lawrence Erlbaum Associates. Thompson, L. F., & Coovert, M. D. (2003). Teamwork online: The effects of computer
conferencing on perceived confusion, satisfaction and postdiscussion accuracy. Group Dynamics, 7(2), 135-151.
Thrun, S. (2004). Toward a framework for human-robot interaction. Human-Computer
Interaction, 19(1-2), 9-24.
Pineau, J., Montemerlo, M., Pollack, M., Roy, N., & Thrun, S. (2003). Towards roboassistants in nursing homes: Challe
h, J., Steele, F., Browne, W., & Prosser, B. (2005). A User's Gude to MLwiN 2United Kingdom: Cent
Robinson, W. S. (1950). Ecological correlations and the behavior of individuals.
American Sociological Review, 15, 351-357.
SReports, 86, 1287-1294.
., Dickinson, T. L., Converse, S. A., & Tannenbaum, S. I. (1992). Toward an unders
, J., Young, J., Drury, J. L., & Yanco, H. A. (2004). Evaluation of human-robot interaction
on Robotics and Automation.
n, T. (1992). Telerobotics, automation, and human supervisory
entsch, K. A., Johnston, J. H., & Payne, S. C. (1998). Measuring team-relateexpertise in complex environments. In
119
on Eye, A., & Mun, E. Y. (2005). Analyzing Rater Agreement: Manifest Variable Methods. Mahwah, NJ: Erlbaum.
ald, A. (1943). Tests of statistical hypotheses concerning several parameters when the number of observations is large. Transactions of the American Mathematical Society(54), 426-482.
ellens, A. R. (1993). Group situation awareness and distributed decision making: From military to civilian applications. In N. J. Castellan (Ed.), Individual and group decision making: Current issues (pp. 267-291). Hillsdale, NJ: Erlbaum.
est, M., Borrill, C., & Unsworth, K. (1998). Team effectiveness in organizations. In C. L. Cooper & I. T. Robertson (Eds.), International Review of Industrial and Organizational Psychology (Vol. 13, pp. 1-48). Chichester, UK: John Wiley & Sons.
ood, J., Lugg, D. J., Hysong, S. J., & Harm, D. L. (1999). Psychological Changes In Hundred-Day Remote Antarctic Field Groups. Environment and Behavior, 31(3), 299-337.
oods, D. D., Tittle, J., Feil, M., & Roesler, A. (2004). Envisioning human-robot coordination in future operation actions on Systems Man and Cybernetics Part C-Applications and Reviews, 34(2), 210-218.
Yanco, H. A., Drury, J. L., & Scholtz, J. (2004). Beyond usability evaluation: Analysis of
human-robot interaction at a major robotics competition. Human-Computer Interaction, 19(1-2), 117-149.
V
W
W
W
W
W
s. Ieee Trans
120
Appendices
121
Appendix A – Task Scenario Instructions
sk the team if this is their first run or second, follow script 1 or 2 accordingly.
cript for Search Task Scenario #1
earch this void using the robot, looking for victims or signs of ictims (clothing, etc.). We want you use the LOVR technique you learned in the
,SV: You will both have access to the robot’s eye view, but you cannot look at each you.
t’s eye view.
voidspace you search. You can do it any way you ant—as you search, or when you finish. We are recording this on video, so please speak
ell
s the task and completes the map, thank them for participating and ll them where to go next (either the next station or back to the BoO).
A S Begin by introducing yourself and the others on your research team. Verbally confirm that each participant has signed both informed consent and video consent forms. Ask the team to decide who will act as robot operator and tether-handler on this run, explaining that they will exchange roles for the second scenario. Ask if they would like to review the Operator Control Unit before continuing. Then say: Your team’s task is to svawareness training as you work, so talk to your team mate as you search. Depending on the CONDITION, say: Rother—pretend there is a wall betweenR, Not: You cannot look at each other-- pretend there is a wall between you. C, SV: You will both have access to the roboC, Not: ----(nothing extra to say). We’d like you to draw a map of thewclearly and loud enough for us to hear. You’ll have 15 minutes to search this void. I’ll tyou when you’ve been in 10 minutes, and when 15 minutes have passed, I’ll ask you to being the robot back to the start point. If you have questions about how to operate the robot I can answer those for you, but I can’t answer any questions about the search task or help you with that. Do you have any questions before we start? When the team finishete
122
rio #2
yourself and the others on your research team. Verbally confirm at each participant has signed both informed consent and video consent forms. Ask the
team robot operator and tether-handler on the first run, explaining that they wil roles for the second scenario. Also ask them whether they both had the SV in the first run, and if they were told not to look at each other. Be sure to not to administer the sam tions on this run! (You should have this information already, but doublecheck). Ask if they would like to review the Operator Control Unit before ontinuing. Then say:
our team’s task is to search this void using the robot, looking for victims or signs of vict e the LOVR technique you learned in the awareness training as you work, so talk to your team mate as you search.
,SV: You will both have access to the robot’s eye view, but you cannot look at each other—pretend there is a wall between you. R, Not: ot look at each other-- pretend there is a wall between you. C, S You will both have access to the robot’s eye view. C, Not: extra to say).
e’d like you to draw a map of the voidspace you search. You can do it any way you speak
learly and loud enough for us to hear. You’ll have 15 minutes to search this void. I’ll tell you when you’ve been in 10 minutes, and when 15 minutes have passed, I’ll ask you to being the robot back to the start point. If you have questions about how to operate the robot I can answer those for you, but I can’t answer any questions about the search task or help you with that.
hen the team finishes the task and completes the map, thank them for participating and ey).
Appendix A (Continued) Script for Search Task Scena Begin by introducing th
who acted asl exchange
e condi
c Y
ims (clothing, etc.). We want you us
Depending on the CONDITION, say: R
You cannV:
----(nothing
Wwant—as you search, or when you finish. We are recording this on video, so pleasec
Wtell them where to go next (back to the BoO to complete the posttest surv
123
Appendix B: Dem
lease answer the following questions. All information is confidential, and will be used
1. W e?
4. 45-54 5. 55 and up
. Please indicate your gender:
. How many years experience do you have on the job as a firefighter?
1. 0-3 years
3. 8-11 years
5. over 15 years
4. How many years experience do you have on the job as a USAR team member?
1. 0-3 years 2. 4-7 years 3. 8-11 years
5. over 15 years 5. D ave any military experience? (If answer is no, go to #8.)
2. No
so, what branch of the military?
1. Air Force
3. Marines 4. Navy
ographic Survey Questionnaire Pfor research purposes only.
hat is your ag
1. Under 25 2. 25-34 3. 35-44
2
1. M 2. F
3
2. 4-7 years
4. 12-15 years
4. 12-15 years
o you h
1. Yes
6. If
2. Army
124
Appendix B (Continued) 7. How many years of military experience?
1. 0-3 years 4-7 years
3. 8-11 years 4. 12-15 years 5. over 15 years
In the nex skthe scale below: 1 2 3 4 5 very little som 8. Remote-controlled cars, planes, boats
9. Handheld or joy-stick type vid
eras
r
Have yo ver operated a robot before?
1. 2. No
If pl
experience? If so, please describe.
2.
t section, please rate your degree of ill with the following items according to
e moderate above average expert
eo games
10. Video cam
11. Search-cam
12.
13. 14. Do you have any other relevant
s or othe technical rescue equipment
u e
Yes
so, ease describe (what kind of robot, where, when, why, etc.)
App
endi
x C
Com
plet
e C
orre
latio
n Ta
ble
for S
tudy
Var
iabl
es
M
2 3
4 5
6 7
8 9
10
1
13
Var
iabl
e
SD
1 1
1214
15
16
1. P
erfo
rman
ce
4.35
3.06
1
438
0.33
9
2.43
03
-0.
0.00
40.
9
73
0.1
-0
.2
0.79
20.
2
0.04
859
0.1
0.
13
0.
0.01
00.
4
0.34
0.
1625
0.1
0.
06
0.
0.86
20.
4
0.64
0.
11
.1
-0.1
0.
0.
0.0.
3.02
40.
1
-0.0
0.
*
0.
0.
2.86
0
0.1
-0.2
0.
**
04*
0.
0.13
0
04
3.18
640.
0.*
*6*
.196
0.
0.
.172
3.
0843
0.0.
**2*
600
0.7
0.
0.
0
2.99
860.
-0.0
0.
6*
.
1
0.
0.
37
.
80.
-0.
0.
* **
46*
597
0.2
0.
0.
080
0
0.0.
0
0
3.10
600.
52-0
48
0.
0.2
0
053
0
0.
2. M
apSc
ore
2.5
0.
96
0.1
1
3. C
om1
1.
20
0.4
**01
11
41
4. C
om2
2.9
1.
26
-0.0
8 57
87*
1
764
5. C
omTo
tal
3.1
0.
77
0.3
* 01
519
51
859
175
6. S
upp1
3.
0.
58
0.0
16
724
90.
264
1
224
082
0.06
4
7. S
upp2
2.
4
1.05
-0
.14
071
4742
6**
-0.0
05
0.41
6**
1
430
0.23
4 31
00
2 0.
974
0.00
3
8. S
uppT
otal
0.71
-0
.01
1158
317*
0.31
0*
0.64
1**
0.63
0*
1
779
0.44
4 69
1 0.
025
0.02
90
0
9. L
dr1
1.14
-0
.10
73
15
640*
* 0.
297*
0.
279*
0.50
00.
4*
1
492
0.22
9 4
0.
036
0.05
00.
0
10. L
dr2
0.
66
0.0
19
8 15
7 0.
007
0.37
7*0.
373*
0.
245
0.57
* 0
1
658
0.16
8 27
5 0.
964
0.00
70.
008
0.08
6 0
0
11. L
drTo
tal
0.
64
0.0
15
2 03
4 0.
248
0.35
3*0.
305*
0.48
1
0.64
* 0.
**
36**
1
769
0.29
1 81
6 0.
082
0.01
20.
031
00
0
12. S
A1
0.
77
0.0
22
1 5
366*
*0.
347*
0.23
0.15
60.
29
033
3*
0.32
4*
0.36
2**
554
0.12
3 72
9 0.
009
0.01
40.
109
0.27
8 0.
0
001
8 0.
022
0.01
13. S
A2
2.86
1.11
-0
.1
0.
4
200
0.
179
214
245 6
756*
*
0.16
2
.26
0.38
3*
0.00
6
0.71
9
0
0.
5
0
* 0. 0
**
58
07
447*
*
01
0.29
7*
.036
1
14. S
A3
0.
80
0.2
0.
068
0.
287*
0.
043
0
0**
.153
88
0.12
8
.377
0.32
5*
0.02
1
0.23
6
0.1
0.
1
0.3
0.
0.
082
0.
569
0.
158
0.
273
0.
133 58
0.24
9
.082
062
668
1
*
p=0.
05, *
* p=
0.01
125
App
endi
x C
(con
tinue
d)
Var
iabl
e
SD
1 2
4 5
7 8
9 M
3 6
10
11
12
13
14
15
16
15. S
ATo
tal
3.08
0.
0.0.
320.
220.
100.
30.
40.
20.
0.0.
50.
4.4
626
0.5
63
05
4 7*
4
6 94
**
25**
6
465*
* 33
0*
08**
79
**
06*
* 0.
7 94
**
1
0.0.
0.0.
0.0.
0.0
00.
000.
060
0.
-0-0
.10.
17-0
.-0
.2-0
.3-0
.0.
-0
-0.
-0.1
-0.1
-0.4
-0.
-0.0
6
0.
0.0.
0.0.
0.0.
262
0.0.
0.0.
0.67
her
0.0.
0.26
0.01
0.02
0.3
0.0
0.2
0.0.
0.3
0.3
0.38
220.
2.4
60.
0
0.
0.0.
0.0.
60.
0.0.
0.0.
0.0.
000.
her-
Ope
rato
r
0.-0
-0.1
0.39
0.0.
0-0
.2-0
.-0
.-0
-0.
-0.
-0.2
0.1
-0.
0.06
0.
0.
0.0.
0.0.
0.0.
0.0.
0.0.
0.66
0
her-
Teth
er
0.
-0-0
.0-0
.0-0
.00.
3-0
.1-0
.-0
.-0
-0.
-0.2
-0.1
-0.1
-0.
0.08
0.
0.0.
550.
0.0.
0.0.
0.07
0.0.
0.56
0.
pera
tor
0.0.
-0.1
-0.1
0.06
0.1
0.0
-0.
-0.
-0-0
.-0
.2-0
.20.
1-0
.-0
.-0
.
0.0.
0.0.
0.0.
0.0.
0.07
0.0.
000.
esea
rche
r
0.-0
-0.1
-0.0
0.1
-0.3
-0.1
-0.
-0.
-0-0
.-0
.2-0
.10.
0-0
.0.
120.
0.0.
0.0.
0.0.
0.0.
0.46
0.0.
400.
0.-0
0.07
-0.0
-0.0
0.3
-0.1
-0.
-0.
-0-0
.-0
.00.
19-0
.00.
0-0
.0.
1
0.0.
0.0.
0.0.
0.0.
70.
0.17
0.0.
910.
0.
0.
0.-0
.0.
000.
0.0.
0.0.
170.
-0.
0.0.
0.0.
0.1
0.3
0.0.
0.21
0.0.
040.
0.
0.-0
.-0
.00.
120.
00.
1-0
.-0
.
0.-0
.0.
-0.2
-0.
-0.
0.-0
.
0.0.
0.0.
0.0.
0.49
7 0.
0.0.
100.
0.04
0.
n
0.-0
-0.0
0.23
-0.
0.2
-0.
-0.
-0.
-0-0
.-0
.-0
.1-0
.10.
-0.0
80.
4
0.51
0.0.
0.0
0.0.
0.0.
0.24
0.24
0.42
0.6
0.56
0.
0.
-00.
20-0
.0.
100.
1-0
.-0
.0.
0.0
0.0.
1511
0.18
3 0.
17-0
.
0.0.
0.0.
0.0.
0.0.
0.28
0.0.
204
0.23
0.
mat
ion
Synt
hesi
s
0.
-00.
22-0
.20.
04-0
.10.
00.
10.
0.0.
0-0
.00.
080.
130.
00.
08-0
.
0.0.
0.0.
0.0.
0.0.
0.53
0.0.
570.
22
0.0
-045
-0
.379
**
0.-0
.-0
.0.
0.0.
-009
8 -0
.-0
.-0
.-0
.00.
-0
.229
-0
.216
0.
754
0.00
7 0.
05
0.52
4 0.
872
0.17
3 0.
371
0.53
8 0.
5 0.
973
0.76
4 0.
106
0.83
7 0.
658
0.11
0.
132
70
7 0.
021
117
462
0.00
5 00
2 06
8 00
1 01
9
1
1
16. O
pera
tor-
Res
earc
her
0.02
03
.0
25
11
6 34
8*
3 38
* 35
1*
-16
2.3
03*
132
33
94
39**
03
7 1
1
864
441
221
0.01
3 10
8 01
6 01
2 03
3 36
35
9 0.
176
0.00
1 79
9 2
17. O
pera
tor-
Tet
0.47
13
04
9 8
7 1
51*
7 11
28
7*
215
11*
82**
0*
* 0.
8 57
0
7**
16
734
059
907
0.88
4 01
3 3
141
043
134
028
006
0.00
6 0.
112
072
1 91
18. R
esea
rc0.
0302
.0
05
16
5**
-00
2 -
84
3 26
4 18
8 .2
2 25
8 20
8 99
* -
12
079
-3
543*
975
421
005
0.98
8 56
4 10
8 06
4 19
12
4 07
14
8 0.
035
0.43
9 58
7 4
19. R
esea
rc0.
01
02
.094
87
78
46
-
37*
47
188
142
.081
18
4 58
13
38
05
7 -
3 06
7
51
5
588
753
017
308
0.19
2 32
6 0.
574
201
0.
436
338
696
0.6
643
20. T
ethe
r-O
0.45
13
02
5 78
06
5
-86
97
06
7 18
9 .0
95
19
57
55
-07
22
1 41
7**
333*
86
1 21
7 46
5 65
3 19
5 50
3 0.
646
188
0.51
1 18
6 0.
071
4 46
1 12
3 0.
3 01
8
21. T
ethe
r-R
0.02
02
.2
79*
28
09
-09
70
**
98
106
127
.13
174
25
06
-6
066
-1
124
04
9 37
7 95
3 45
3 00
8 16
9 0.
464
379
0.36
9 22
6 0.
116
2 67
9 64
9 0.
1 39
1
22. A
nsw
er
0.16
06
.2
61
2 01
73
-
25*
42
078
206
.109
05
6 94
7
3 99
01
5 7
06
7 62
1 99
3 61
3 02
1 32
6 0.
589
152
0.45
516
83
8 49
4 0.
9 23
8
23. C
omm
ent
0.53
1303
6 19
5 07
1 9
229
125
0.14
9 25
5 0.
081
228
0.27
6 9
0.19
9 09
9 0.
287*
10
6
80
5 17
5 62
6 95
3 1
389
0.
074
0.57
7 11
2 0.
052
5 16
5 49
5 0.
3 46
4
24. I
nstru
ctio
n 0.
1511
23
0 21
7 3
9 6
06
125
098
036
124
-18
6 35
15
4 20
7 -
289*
18
10
8 13
1 83
7 37
4 68
1 46
2 0.
387
803
392
0.19
5 1
287
149
0.2
212
25. Q
uest
io0.
1705
.2
44
95
2 18
6 -
64
329*
00
3 14
6 .1
57
209
169
67
16
058
3 25
**
08
7 0.
10
5 19
5 0.
064
2 98
4 31
2 27
7 14
4
6 2
9 5
002
26. E
nviro
nmen
t 0.
1207
.0
80
9 29
0*
7 23
01
7 0.
013
055
261
25
046
4 0.
8
199
58
0 14
4 04
1 46
1 39
3 90
9 0.
926
703
0.06
7 86
3 0.
752
6 41
3 8
165
27. I
nfor
0.01
02
.1
35
8 4
9 33
41
32
10
2 06
4 56
14
9
2 35
2
05
35
0 11
1 09
3 73
4 35
8 77
8 0.
361
479
0.65
8 70
2 0.
922
7 36
2 81
1 0.
3 72
9
28. N
avig
atio
n
0.1
.027
9*
092
023
196
129
089
.00
5 04
4 23
2 3
064
* p=
0.05
, **
p=0.
01
126
App
endi
x C
(con
tinue
d)
Var
iabl
e M
SD
1
2 3
4 5
6 7
8 9
10
11
12
13
14
15
16
29. O
ffta
sk
0.
07
.04
-8
1 -0
-0
.3-0
.1-0
.07
-0.0
33
.284
* 0.
332*
-0
.327
-0.1
-0-0
04
3 35
5*
00.
20-0
.23
0.03
.2
1306
* 84
-0
-*
69
.16
.075
-0.
0.
0.14
8 0.
106
0.83
8 0.
137
031
202
0.63
1 0.
822
0.0.
0.02
1 0.
1 .2
66
0.60
3 0.
0. R
obot
Situ
ated
ness
0.
0.05
-0
.257
0.
113
0.04
9 0.
025
.052
-
058
-0.
3 0
-0.0
89
0.05
8 0.
0.2
.067
0.
03
0.11
2 0.
355*
0.
0.43
5 0.
735
0.86
6 71
9 68
8 0.
358
1 0.
0.84
7 0.
2 .6
44
0.83
6 0.
0.
1. R
obot
Sta
te
0.0.
09
-0.1
10
0.04
3 -0
.179
0.
149
.002
-
225
0.-0
.0.
071
0.00
1 0.
-093
.1
18
0.44
0**
-0.1
32
0.09
2
0.
769
0.21
3 0.
301
99
116
0.87
9 0.
0.0.
0.67
6 0.
8 .4
13
0.00
1 0.
2. S
earc
h 0.
0.05
0.
429*
-0.1
52
0.08
7 -
.052
17
3 0.
092
0.09
3 0.
0.0.
024
0.6*
.0
02
0.12
6 0.
0.
145
0.00
2 0.
291
0.55
0.
72
231
0.52
7 0.
52
0.0.
0.87
0.
03
991
0.38
4 0.
0.
33. V
ictim
0.
07
0.06
05
3 0.
266
-0.1
82
0.23
8 03
5 -0
.216
-0
.083
-0
.0.
0.0.
18
.253
0.
229
0.0.
0
0.
062
0.20
7 09
6 81
1 0.
132
0.56
8 0.
0.0.
407
0.
.076
0.
11
0.23
34. C
larif
y 0.
0.03
6 0.
043
-0.1
1 -0
.063
-
.334
* -
138
-0.
5 -0
.156
-0
.-0
.-0
.113
-0
11
.048
0.
034
-0.
0.14
3
68
9 0.
664
018
338
0.51
1 0.
28
0.0.
435
0.4
.741
0.
816
0.0.
0.
63
7 03
.1
72
-04
8 -0
.2
-0.0
91
-0.
0.0.
035
-065
.0
41
0.06
8 -0
.04
8 4
.2
34
742
0.72
2 0.
0.0.
0.81
0.
3 .7
76
0.64
0.
certa
inty
0.
03
7 41
65
--
.078
.
0.10
60.
053
0.12
5 0.
205
0.27
0.
041
0.08
1 0.
21
0.38
1*
3 52
059
1 0.
753
0.46
5 0.
717
0.38
6 0.
154
0.79
.7
77
0.57
6 0.
0.
rr
or
0.0
.01
53
-
00.
23
41
0.11
6 0.
0.15
5 -0
046
.159
-0
.033
0.
001
6
0.
055
0.09
2 0.
0.0.
282
0.4
.27
0.82
2 0.
atio
n 0.
06
6
-
0.15
0.
-0.
0.00
1 0.
6 .2
02
0.00
6 0.
0.03
3
7
7
0.0.
0.0.
994
0.3
.16
0.96
4 0.
0.
0.11
08
-0.
0.00
0.04
0.
0.10
7 0.
5 .0
02
0.09
5 0.
0.
4
7
0.0.
0.45
9 0.
3 .9
91
0.51
1 0.
0.
8
7 5*
-0.
1-0
.022
0.21
4 1
-004
.0
05
0 -0
.106
0.
1
5
0.0.
0.0.
448
0.5
.974
1
0.0.
0.11
-0.
0.06
804
40.
116
02
.131
0.
091
-0.
0.
0.
981
0.70
9
0.63
7.7
590.
423
0.1
.365
0.
529
0.
0.
0.04
6 01
8 24
076
6 0.
011
311
-0
0.13
028
170
072
0.0.
538
0.69
1 23
043
9 01
1
330
-0
0.02
2 04
6 06
1 .1
0-
0.
447
0.
0.75
62
2 99
3 17
036
1 0.
527
310
0.20
9*
00.
121
101
300
173
-23
8
0.
0.40
1 48
6 0.
229
096
0.54
1**
-0.
0.12
1 06
2 12
-0
155
173
0.00
0.71
7
0.0.
402
671
210
281
0.
04
-0
.21
00.
0908
6 01
7 .1
-008
0.13
10.
70.
44
0.0.
553
0.90
5 44
058
2 32
1
35. C
onfir
m
18
0.0
-0
.20
0.02
0.85
-0.2
0.03
4
0.81
3
-00.
0506
3 03
6 .0
0-
075
0.
0.15
-0.2
0
0.15
80
0.53
66
2 80
5 65
060
5 0.
783
36. C
onve
y U
n
3 0.
0
0.1
0.0
0.6
-0.0
24
0.87
0.20
4
0.15
5
00
046
1
-*
0.14
90.
3. .
30
143
006
37. C
orre
ctE
1 0
-0
.260 8
0.2
-0.1
26 83
-0.0
260.
16
13
70.
214
4 .
014
8 0.
0.
060.
070.
30.
858
0.26
80.
369
423
318
750
306
0.99
6
38. P
rovi
de In
form
10
0.
0.06
80.
23-0
.09
0.17
1-0
.232
0.00
123
8 0.
104
188
170
012
-
0.
639
0.10
0.50
0.23
50.
105
0.99
709
6 3
471
0.19
1 22
093
3 82
3
39. R
epor
t 30
0.
-0
.018
-0.1
0.12
8-0
.174
0.28
1*0.
008
095
1 -
09
050
102
055
0.
900
0.45
0.37
0.22
70.
048
0.95
451
1 0.
992
0.78
1 53
5 70
047
9 0.
704
40. S
eek
Info
rmat
ion
11
0.05
-0.1
6-0
.21
0.28
-0.1
12-0
.147
0.
207
08
37
-0.1
-
-0.
1 .2
-024
9
0.
244
0.13
0.04
0.44
0.30
80.
1545
5 79
6 40
1 0.
136
150
463
082
41. P
lan
21
0.
0.30
8*0.
004
-0.0
540.
145
0.14
1 0.
157
079
-02
8 0.
0.
-
-.0
-0-
072
-27
2
0.
030
0.31
50.
327
0.27
658
5 0.
847
0.
0
890
621
0.05
6
* p=
0.05
, **
p=0.
01
127
App
endi
x C
(con
tinue
d)
Var
iabl
es
17
24
25
26
27
28
29
30
35
18
19
21
22
23
31
32
33
34
17. O
pera
tor-
Teth
er
1
pe
rato
r 1
2
he
r-Te
ther
22*
5
2
O
pera
tor
0*-0
.290
*51
0.
21
. Tet
her-
Res
earc
her
07
*09
85
5
22. A
nsw
er
041
.061
.0
1 0.
066
80.
64
3.
Com
men
t 8*
*.2
4-0
.1
-0.5
0.0
4.
Inst
ruct
ion
71*
.2
27-0
.0
-0.
-0
1
112
0.22
0
2229
0*22
10.
438
0.39
-0.
-0
1
88
0.0.
091
0..
0.0.
-
03
70.
0.0.
tion
Synt
hesi
s
74
.218
04
9 -0
.00.
3-0
-1
912
90.
0.
1*
0.
0202
-0.
-00.
*
01.3
330.
0
5
29
. Off
task
67
306*
544
0.66
00.
03-0
.-0
217
*
1
0
0.0.
077
18. R
esea
rche
r-O
0.00
1
0.99
19. R
esea
rc-0
.4*
-0.0
3
1
0.
00
0.81
1
20. T
ethe
r-
-0.9
00
*
0.1
041
0.
294
-0.3
0.
0.**
1
0.03
0.
536
0
-0.
-0-0
1
0.
77
0.67
2 0.
943
7
20.
60
0.11
-0
7 84
11**
1
00.
446
0.08
4 2
2-0
.60
*-0 0.
0.
0.
176
222
417
5 5 .6
69**
25
. Que
stio
n 78
2 0.
**
-0
.0
0.
0.
3**
408*
*.2
41
0.0.
041
0.12
4 00
1 0.
005
0.00
3
26. E
nviro
nmen
t 0.
296*
-
205
-005
7 -
159
-01
7 0.
243
0.25
8 -0
.059
1
0.
0.15
3 69
4 26
9 0.
907
0.08
9 07
0.
682
27
. Inf
orm
a-0
.0-0
0.01
00
* .0
89
0.06
1 -0
.03
0.07
8
0.60
0.
0.
737
996
0.03
4 0.
538
675
0.83
80.
589
28. N
avig
atio
n
-0.4
7*
-0.1
4-
8 0.
8 07
3 .4
40**
54
4**
0.01
-0
.451
*-0
.03
1
0.
0-0
.0
0 0.
0.
0.84
8 **
845
0.**
61
2 0.
3 00
1 02
1 0.
947
0.45
5*0.
001
-0.2
530.
83-0
.146
-0
.113
.
0.
646
0.03
1 0
821
0.88
5 13
0.
001
0.0.
311
0.43
6
*
p=0.
05, *
* p=
0.01
128
129
App
endi
x C
(Con
tinue
d)
Var
iabl
es
17
18
19
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
30. R
obot
Situ
ated
ness
0.
165
0.28
2*
0.14
0.
345*
0.
077
0.27
9 -0
.449
**
0.2
-0.2
27
-0.1
99
-0.2
96*
0.46
7**
1
0.
252
0.04
7 0.
333
0.01
4 0.
597
0.05
0.
001
0.16
3 0.
113
0.16
7 0.
037
0.00
1
31. R
obot
Sta
te
-0.0
3 0.
278
-0.1
-0
.099
-0
.133
0.
084
0 -0
.019
-0
.199
-0
.074
-0
.303
* -0
.117
-0
.05
1
0.
839
0.05
0.
488
0.49
4 0.
359
0.56
2 1
0.89
8 0.
167
0.60
8 0.
033
0.41
8 0.
718
32. S
earc
h 0.
103
-0.3
68**
0.
075
-0.0
53
0.09
4 -0
.023
0.
064
-0.1
81
0.23
3 0.
333*
-0
.24
-0.1
73
-0.1
9 -0
.348
* 1
0.
477
0.00
9 0.
606
0.71
7 0.
518
0.87
3 0.
661
0.20
8 0.
104
0.01
8 0.
093
0.23
1 0.
185
0.01
3
33. V
ictim
0.
341*
0.
01
-0.2
88*
-0.3
78**
0.
082
0.17
8 -0
.16
-0.2
08
0.00
7 -0
.251
-0
.194
-0
.183
-0
.06
-0.2
71
-0.0
36
1
0.
015
0.94
7 0.
043
0.00
7 0.
571
0.21
6 0.
268
0.14
7 0.
963
0.07
9 0.
178
0.20
3 0.
661
0.05
7 0.
804
34. C
larif
y 0.
016
-0.0
25
0.28
5*
0.16
7 0.
465*
* -0
.246
-0
.03
0.11
8 0.
105
0.19
9 -0
.143
0.
136
0.06
9 -0
.088
0.
167
-0.1
3 1
0.
913
0.86
4 0.
045
0.24
8 0.
001
0.08
5 0.
837
0.41
5 0.
466
0.16
5 0.
323
0.34
5 0.
632
0.54
5 0.
246
0.36
4
35. C
onfir
m
0.12
6 -0
.043
-0
.139
-0
.155
0.
096
0.29
5*
-0.3
13*
-0.1
73
0.40
4**
0.08
8 -0
.307
* -0
.071
0.
235
-0.1
5 0.
11
-0.0
2 0.
301*
1
0.
384
0.76
6 0.
336
0.28
4 0.
506
0.03
8 0.
027
0.23
0.
004
0.54
3 0.
03
0.62
5 0.
1 0.
3 0.
445
0.89
6 0.
033
36. C
onve
y U
ncer
tain
ty
0.13
3 0.
096
-0.1
03
-0.1
76
0.53
3**
-0.1
58
-0.2
47
0.24
9 -0
.035
0.
247
-0.2
27
0.04
9 0.
169
-0.0
38
0.11
3 0.
135
0.33
0*
0.2
0.
356
0.50
9 0.
477
0.22
1 0
0.27
3 0.
084
0.08
1 0.
811
0.08
3 0.
113
0.73
6 0.
242
0.79
2 0.
433
0.34
9 0.
019
0.16
4
37. C
orre
ct E
rror
0.
142
-0.0
44
0.02
8 0.
08
0.37
6**
-0.1
33
-0.1
28
0.13
4 -0
.046
0.
408*
* -0
.093
0.
128
-0.0
8 -0
.017
0.
322*
-0
.17
0.34
7*
-0.0
26
0.
324
0.76
2 0.
845
0.58
1 0.
007
0.35
8 0.
374
0.35
4 0.
753
0.00
3 0.
52
0.37
7 0.
582
0.90
8 0.
023
0.23
2 0.
014
0.85
8
38. P
rovi
de In
form
atio
n 0.
018
0.05
-0
.05
-0.0
06
0.05
7 0.
167
-0.1
54
-0.1
49
-0.2
42
0.37
6**
-0.0
87
0.05
3 0.
014
0.02
8 0.
360*
-0
.07
-0.1
3 -0
.068
0.
9 0.
732
0.73
0.
969
0.69
6 0.
247
0.28
6 0.
303
0.09
1 0.
007
0.54
8 0.
716
0.92
5 0.
849
0.01
0.
627
0.36
7 0.
64
39. R
epor
t 0.
221
0.26
1 -0
.109
0.
037
-0.2
61
0.46
0**
-0.4
14**
0.
049
0.07
6 -0
.449
**
-0.1
51
0.08
2 0.
254
0.23
9 -0
.550
**
0.18
8 -0
.448
**
-0.2
52
0.
124
0.06
7 0.
452
0.79
9 0.
067
0.00
1 0.
003
0.73
4 0.
601
0.00
1 0.
297
0.57
0.
075
0.09
4 0
0.19
0.
001
0.07
8
40. S
eek
Info
rmat
ion
-0.0
08
0.37
4**
0.18
0.
469*
* -0
.056
-0
.138
-0
.176
0.
756*
* -0
.259
-0
.273
0.
166
0.44
2**
0.17
0.
163
-0.3
91**
-0
.21
-0.2
77
-0.3
89**
0.
958
0.00
7 0.
211
0.00
1 0.
699
0.34
0.
222
0 0.
07
0.05
5 0.
248
0.00
1 0.
239
0.25
9 0.
005
0.15
0.
052
0.00
5
41. P
lan
-0.3
46*
-0.4
14**
0.
068
-0.1
8 -0
.077
-0
.523
**
0.81
0**
-0.3
13*
-0.0
72
0.15
7 0.
375*
* -0
.329
* -0
.500
**
-0.1
98
0.36
6**
-0.0
2 0.
114
-0.1
85
0.
014
0.00
3 0.
638
0.21
0.
597
0 0
0.02
7 0.
619
0.27
5 0.
007
0.02
0
0.16
8 0.
009
0.87
7 0.
432
0.19
9 *
p=0.
05, *
* p=
0.01
130
App
endi
x C
(con
tinue
d)
Var
iabl
es
36
37
38
39
40
41
36. C
onve
y U
ncer
tain
ty
1
37
. Cor
rect
Err
or
0.54
5**
1
0
38. P
rovi
de In
form
atio
n 0.
073
0.19
6 1
0.61
4 0.
172
39. R
epor
t -0
.339
* -0
.362
**
-0.2
62
1
0.01
6 0.
01
0.06
6
40. S
eek
Info
rmat
ion
-0.2
19
-0.1
72
-0.0
75
0.35
4*
1
0.
127
0.23
1 0.
605
0.01
2
41
. Pla
n -0
.076
0.
015
-0.1
96
-0.6
44**
-0
.407
**
1
0.59
8 0.
919
0.17
3 0
0.00
3
* p=
0.05
, **
p=0.
01
131
About the Author
Jennifer Burke received a Bachelor’s Degree in Business from Florida State University in
1982, an M.A. in Counseling from the University of North Florida in 1990, and a M.A. in
Industrial-Organizational Psychology from the University of South Florida in 2003. She taught
mathematics in Florida public schools for 14 years.
Since entering the Ph.D. program at the University of South Florida, Jenny has taught
courses in psychology and conducted training for technologists and first responders through the
Center for Robot-Assisted Search and Rescue. She has co-authored two journal articles, a book
chapter, and numerous technical reports, and has presented her research at national and
international conferences.
Ms. Burke is a member of the Association for Computing Machinery (ACM), the Human
Factors and Ergonomics Society (HFES), the Society for Industrial and Organizational
Psychology (SIOP), and the American Psychological Society (APS).
