1

AI Challenges in Human-Robot Cognitive TeamingTathagata Chakraborti1, Subbarao Kambhampati1, Matthias Scheutz2, Yu Zhang1

1 Department of Computer Science, Arizona State University, Tempe, AZ 85281 USA{ tchakra2, rao, yzhan442 } @ asu.edu

2 Department of Computer Science, Tufts University, Medford, MA 02155 USAmatthias.scheutz@tufts.edu

Abstract—Among the many anticipated roles for robots in thefuture is that of being a human teammate. Aside from all thetechnological hurdles that have to be overcome with respect tohardware and control to make robots fit to work with humans,the added complication here is that humans have many consciousand subconscious expectations of their teammates – indeed, weargue that teaming is mostly a cognitive rather than physicalcoordination activity. This introduces new challenges for the AIand robotics community and requires fundamental changes tothe traditional approach to the design of autonomy. With thisin mind, we propose an update to the classical view of theintelligent agent architecture, highlighting the requirements formental modeling of the human in the deliberative process of theautonomous agent. In this article, we outline briefly the recentefforts of ours, and others in the community, towards developingcognitive teammates along these guidelines.

I. INTRODUCTION

An increasing number of applications demand that humansand robots work together. Although a few of these applicationscan be handled through “teleoperation”, technologies that actin concert with the humans in a teaming relationship withincreasing levels of autonomy are often desirable if not re-quired. Even with a sufficiently robust human-robot interface,robots will still need to exhibit characteristics common inhuman-human teams in order to be good team players. Thisincludes the ability to recognize the intentions of the humanteammates, and to interact in a way that is comprehensibleand relevant to them - autonomous robots need to understandand adapt to human behavior in an efficient manner, much likehumans adapt to the behavior of other humans. Humans areoften able to produce such teaming behavior proactively dueto their ability (developed through centuries of evolution of avariety of implicit or explicit visual, auditory and contextualcues) to quickly (1) recognize teaming context in terms of thecurrent status of the team task and states of the teammates,(2) anticipate next team behavior under the current context todecide individual subgoals to be achieved for the team, and(3) take proper actions to support the advancement of thosesubgoals with the consideration of the other teammates.

The three steps above form a tightly coupled integrated loopduring the coordination process, which is constantly evolvingduring teaming experience. Critically, humans will likely ex-pect all of the above capabilities from a robotic teammate,as otherwise team dynamics will suffer. As such, the chal-lenge in human-robot teaming is primarily cognitive, ratherthan physical. Cognitive teaming allows the robots to adaptmore proactively to the many conscious and subconscious

expectations of their human teammates. At the same time,improper design of such robot autonomy could increase thehuman’s cognitive load, leading to the loss of teaming situationawareness, misaligned coordination, poorly calibrated trust,and ultimately slow decision making, deteriorated teamingperformance, and even safety risks to the humans. As designersof robotic control architectures, we thus have to first isolate thenecessary functional capabilities that are common to realizingsuch autonomy for teaming robots. The aim of the article isto do just that and thus provide a framework that can serve asthe basis for the development of cognitive robotic teammates.

II. RELATED WORK

In human-human teams, it is well understood that ev-ery team member maintains a cognitive model of the otherteammates they interact with [22]. These models not onlycaptures their physical states, but also mental states such as theteammate intentions and preferences, which can significantlyinfluence how an agent interacts with the other agents inthe team. Although such modeling has been identified as animportant characteristic of effective teaming [21], [23], [37],it is less clear how it is maintained at the individual level.Furthermore, the relative importance of different aspects ofsuch models cannot be easily isolated in experiments with hu-man teammates, but must be separately considered for robotssince they often require very different modeling technologies.Such forms of modeling allows the robots to understandtheir human partners, and in turn use this knowledge to plantheir coordination to improve teaming experience. However,although there exists work that has investigated the variousaspects of this modeling [29], [75], [86], a systematic summaryof the important challenges is still missing.

Next, we provide a review of the related work in terms ofagent types (Figure 1) that can be used to implement roboticteammates, following the categorization in [82]. The first twotypes correspond to classical agent architectures in roboticsand artificial intelligence. We list them to facilitate the com-parison of earlier teaming agents and cognitive teaming agents.We show that all of them fall within a spectrum of agents thatdiffer based on the extent to which the agent interaction withthe external world and other agents is modeled.

A. Behavior-based agent

Behavior-based agents [8] have been an important designparadigm for embodied agents (especially robots), in which

arX

iv:1

707.

0477

5v2

[cs

.AI]

13

Aug

201

7

2

Fig. 1. An categorical view of different types of agents. Each type is deeper than the previous types in terms of modeling complexity (from left to right).

complex behaviors result from a collection of basic behav-iors interacting with each other. These basic behaviors oftenoperate in parallel via cooperative or competitive arbitrationschemes [70]. Behavior-based agents have been applied tovarious tasks such as formation control [4], [5], box pushing[58], [34], navigation [59], and surveillance [83]. One issuewith behavior-based agent is that the interactions betweenbasic behaviors often have to be provided manually. Thiscan quickly become impractical when complex interactionsare desired. Furthermore, since this type of agent does notmaintain a model of the world, it cannot reason about itsdynamics and hence is often purely reactive.

B. Goal-based agentIn contrast to behavior-based agents, a goal-based agent

maintains a model of the world (Fig. 2) and how its actions canchange the state of the world. As a result, it can predict how theworld will respond before it executes any action. The earliestagent of this type is Shakey [56]. The model that is maintainedis often specified at a factored level using planning languages,such as STRIPS [30], PDDL [31] or its probabilistic extensions[67], or at the atomic level using MDP specification [62], [29].A goal-based agent can also maintain its own epistemic state[32], such as beliefs and desires [64], [33]. Goal-based agentstypically assume that the given model is complete, which maynot be realistic in open world domains [76].

Both behavior and goal based agent can handle multi-agentcoordination [83], [59], [4], [57], [38]. However, it is oftenassumed that the team is given a specific goal, and the teammembers either are provided information about each other apriori, or can explicitly exchange such information. As a result,agents can readily maintain a model of the others in teaming.While this assumption may be true for robots teaming withrobots, we would definitely not observe such convenience inhuman-robot teams (e.g., requiring humans to provide the in-formation constantly can significantly increase their cognitiveload); furthermore, the goal is often spontaneous rather thangiven. As a result, a robotic teammate that is solely behavioror goal based can only handle specific tasks and will rely onhuman inputs for task assignments.

C. Proactive agentA proactive agent, on the other hand, is supposed to

maintain a model of the others through both observations and

communication (if available and necessary). This model isnot only about the others’ physical state (e.g., location), butalso mental state which includes their goals [77], capabilities[85] (which include the consideration of physical capabilities),preferences [54], and knowledge [6].

Given that none of these are directly available, they mustbe inferred [63] or learned [85], [1], [9] from observations.As a result, the model of the other agents is often subject to ahigh-level of incompleteness and uncertainty. This is especiallytrue when human teammates are involved. Nevertheless, evensuch an approximate model of other agents can be importantfor efficient teaming when used properly. For example, it canbe used by the agents to plan for coordination to exploitopportunities to help the humans in a proactive way [11], [29]while avoiding conflicts [18], [17].

In addition to using the model of the others to plan forcoordination, a proactive agent can also act proactively tochange the others’ modeling of itself when necessary [6]. Forexample, a robot can explicitly convey its intention throughnatural languages [79] or gestures [61] to let the humanunderstand its intention to help or request for help.

D. Social agent

A deeper level of modeling is not only about the otheragents, but also the other agents’ modeling of the agent itself[86]. This includes, for example, the others’ expectation andtrust of the agent itself. Such modeling allows the robot, forexample, to infer the human expectation of its own behaviorand in turn choose behaviors that are consistent with thisexpectation. Expectation and trust, in particular, represent thesocial aspects of agent interactions since they are particularlyrelevant when agents form groups or teams together. An agentthat behaves socially [47], [27], [86] allows the other agents tobetter understand and anticipate its behavior, thus contributingto the maintenance of teaming situation awareness [21]. Inhuman-human teams, social behaviors contribute significantlyto fluent teaming [69].

Similar to a proactive agent, a social agent often has to learnand maintain a model about the various social aspects fromobservations [86]. In addition to using these social aspectsto guide its behavior generation, a social agent can also actto change these aspects (via informing the others about thediscrepancies in their modeling about itself and updating the

3

Fig. 2. Traditional view of the goal-based intelligent agent architecture[66] that describes how the agent models the world, senses changes in theenvironment, plans to achieve goals and acts to execute the plans.

same in its model of the others). For example, a robot canmaintain its trust from the human by constructing excuseswhen a task cannot be achieved [35] to provide explanationsto the human from the robot’s own perspective, while takinginto account the human’s understanding of itself.

Although various types of agents can be used to realize arobotic teammate, based on the above discussion, the chal-lenges that are introduced by the humans in the loop lie inparticular in the implementation of a proactive and socialagents. A common characteristics among these two types ofagents is that both require the notion of mental modeling ofthe other teammates, which cannot be directly observed andmust be inferred cognitively. This is the key requirement of acognitive teaming capability.

III. TRANSITIONING TO A COGNITIVE TEAMING AGENT

In this section, we characterize how each step in the “Sense-Model-Plan-Act” (SMPA) cycle of the classical goal-basedagent view in [66] (shown in Figure 2) has to be updatedto facilitate the mental modeling of the human in the loopin order to enable a truly cognitive teaming agent (shown inFigure 3). Specifically, we introduce the Human Model (HuM)and the Human Mental Model (HuMM) as key components inthe agent’s deliberative process. Changes to Model in Figure3 is a direct result of the requirement of human mentalmodeling. Coarsely speaking, changes to Sense contribute tothe recognition of teaming context, changes to Plan contributeto the anticipation of team behavior and Act, and changesto Act contribute to the determination of proper actions atboth the action and motion levels. In practice, these fourfunctionalities are tightly integrated in the behavior loop.

Sense – The agent can no longer sense passively to checkthat the preconditions of an action are satisfied, or after itapplies an action to the world to confirm that it is updatedaccordingly (“what the world is like now” in Figure 2).In teaming scenarios, the agent needs to proactively makecomplex sensing plans that interact closely with other func-tionalities – Model and Plan – to maintain the correct mental

Fig. 3. An updated view of the architecture of a cognitive teaming agentacknowledging the need to account for the human’s mental state by means ofwhat we refer to as Human Mental Modeling or HuMM.

state (such as intentions, knowledge and beliefs) of its humanteammates in order to infer their needs. For example, howthe robot should behave is dependent on how much and whattype of help the human requires, which in turn depends onthe observations about the human teammates such as theirbehavior and workload. Furthermore, the inference about thehuman mental state should be informed by the human modelthat the robot maintains about the human’s capabilities andpreferences. Note that directly asking humans (i.e. explicitcommunication) is a specific form of sensing.

Model – Correspondingly, the state, i.e. “what the world islike now”, needs to include not only environmental states, butalso mental states of the team members which may not onlyinclude cognitive and affective states such as the human’stask-relevant beliefs, goals, preferences, and intentions, butalso, more generally, emotions, workload, expectations, trustand etc. “What my actions do to the world” then needs toinclude the effects of the robot’s actions on the team member’smental state, in addition to the effects on their physiologicaland physical states and the observable environment; “Howthe world evolves” now also requires rules that govern theevolution of agent mental states based on their interactionswith the world (including information exchange through com-munication); “What it will be like” will thus be an updatedstate representation that not only captures the world state,agent physiological and physical state changes based on theiractions and current states, but also those mental state changescaused by the agent itself and other team members.

Plan – “What action I should do” now involves more com-plex decision-making that must again also consider humanmental state. Furthermore, since the robot actions now caninfluence not only the state of the world but also the mentalstate of the humans, the planning process must also considerhow the actions may influence their mental state and evenhow to affect/manipulate such mental state. For example, inteaming scenarios, it is important to maintain a shared mentalstate between the teammates. This may require the robots togenerate behavior that is expected or predictable to the human

4

teammates such that they would be able to understand therobot’s intention. This can, in fact, be considered an implicitform of signaling or communication. On the other hand, ashared mental state does not necessarily mean that everypiece of information needs to be synchronized. Given thelimitation on human cognitive load, sharing only necessaryinformation is more practical between different teammatesworking on different parts of the team task. A properlymaintained shared mental state between the teammates cancontribute significantly to the efficiency of teaming since itcan reduce the necessity of explicit communication.

Act – In addition to physical actions, we now also havecommunication actions that can change the mental state ofthe humans by changing their beliefs, intents, etc. Actions toaffect the human’s mental state do not have to be linguistic(direct); stigmergic actions to instrument the environment canalso inform the humans such that their mental states can bechanged. Given that an action plan is eventually realized viathe activation of effectors by providing motor commands, Actmust be tightly integrated with Plan. While Plan generatesthe sequence of actions to be realized, motor commands cancreate different motion trajectories to implement each actionand can in turn impact how the plan would be interpretedsince different realizations can exert different influences onthe human’s mental states based on the context.

An Exemplary Human-Robot Teaming Scenario

To better illustrate how mental modeling of teammates cancontribute to the different capabilities needed for cognitiveteaming agents, we will now consider scenarios from a human-robot team performing an USAR task where each subteam iconsists of one human Hi and one robot Ri.

For subteam 1: Based on the floor plan of the buildingin its search area, R1 realizes that the team needs to use anentrance to a hallway to start the exploration. R1 notices thata heavy object blocks the entrance to the hallway. Based on itscapability model of H1 (i.e., what H1 can and cannot lift) andH1’s goal, R1 decides to interrupt its current activity and movethe block out of the way. H1 and R1 then continue exploringdifferent parts of the area independently when H1 discovereda victim and informs R1. R1 understands that H1 needs toget a medical kit to be able conduct triage on this victim assoon as possible but knows that H1 does not know where amedical kit is located. Since R1 has a medical kit already,but cannot deliver it due to other commitments, it places itsmedical kit along the hallway that it expects H1 to go through,and informs H1 of the presence of the kit.

For subteam 2: Based on the floor plan of the building inits search area, R2 finds that all the entrances are automaticdoors that are controlled from the inside. Since the connectioncannot be established due to power lost, the team needs tobreak a door open first. R2 infers that H2 is about to break adoor open based on the teaming context and its observations.Since it knows that breaking the door open may cause a boardto fall on H2, R2 moves to catch the board preventatively.Once H2 and R2 are inside, however, H2 is uncertain about

the structural integrity and has no information on which partsmay easily collapse. R2 has access to the building structureinformation and proposes a plan to split the search in a waythat minimizes human risk.

For both subteams: As both teams are searching their areas,they receive information about a third area to be explored.Since neither H1 nor H2 are finished with their current searchtask, they assume that the other will take care of the thirdarea. Since R1 understands H1 and H2’s current situation,and expects itself to be done with its part of the task soon,R1 decides to work on the third area since it does not expectH1 to need any help. R1 informs H1. H1 is OK with it andinforms H2 that team 1 is working on the third area. WhenR1 arrives at the third area, it notices new situations whichrequire certain equipment from team 2. R1 communicates withR2 about the availability of the missing items. R2 quicklypredicts equipment needs and anticipates that those items arenot needed for a while. After getting the OK from H2 to lendthe equipment to R1, R2 drives off to meet R1 half-way, handover the equipment, and R1 returns to the third area with thenewly acquired equipment. H1 was not informed during thisprocess since R1 understands that H1 has a high workload.Once the equipment is no longer needed, R1 meets up withR2 again, returning the equipment in time for use by H2.

Based on the above scenario, we can see that the mentalmodeling of the others on a cognitive robotic teammate iscritical to the fluent operation of the team For example, R1

needs to understand the capabilities of H1 (i.e., what H1 canand cannot lift); both R1 and R2 need to be able to infer aboutthe intention of the human teammates. The modeling mayalso include the human’s knowledge, belief, mental workload,trust, etc. This human mental modeling for cognitive teamingbetween humans and robots connect with the three capabilitieswe introduced in Section I as critical to the functioning ofhuman-human teams and form the basis of the updated agentarchitecture in Fig. 3 as follows -

C1. Recognizing teaming context to identify the statusof the team task and states of the teammates: For example,based on the floor plan of the building, R1 realizes thatthe team needs to use an entrance to a hallway to start theexploration. R2 finds that all the entrances are automatic doorsthat are controlled from the inside. Consequently, it infers thatthe team needs to break a door open first. This inferenceprocess takes into account the modeling of the teammate’sstate (e.g., the intention to enter the building).

C2. Anticipate team behavior under the current context:For example, given that a heavy object blocks the entrance tothe hallway, R1 infers that the human will be finding a wayto clear the object. R2 infers that H2 is going to break a dooropen based on the teaming context and its observations. Thisprediction takes into account of the modeling of the human’scapabilities and knowledge about the teaming context.

C3. Take proper actions to advance the team goalwhile taking into account the teammates: For example, afteranticipating the human’s plan, the robots should proactivelyhelp the humans (e.g., R1 helps H1 move the block away and

5

Fig. 4. Figure [73] illustrating the expanding scope of the “human-aware”decision making process of an autonomous agent to account for the humanmodel (HuM) and the human mental model (HuMM).

R2 catches the board preventively that can potentially hurt thehuman), while taking the account the modeling of the human’scapabilities, mental workload, and expectation.

Remark: C3 above not only includes actions that con-tribute to the team goal, but also actions for maintainingteaming situation awareness (e.g., making explanations). Assuch, C3 feeds back to C1 and the three capabilities in turnform a loop that should be constantly exercised to achievefluent teaming. Furthermore, although we have been focusingon implicit communication (e.g., through observing behaviors)to emphasize the importance of mental modeling, explicit com-munication (e.g., using natural language) is also an importantpart of the loop. Another note is that since both implicit andexplicit communication can update the modeling of the otherteammates’ mental states as discussed, they are anticipated toevolve the teaming process in the long term.

IV. CHALLENGES

The capabilities reflected in the updated agent architecturepresent several challenges for the design of cognitive roboticteammates – at the core of these issues is the need for anautonomous agent to consider not only its own model butalso the human teammate’s mental model in its deliberativeprocess. In the following discussion, we will outline a few ofour recent works in this direction, outline processes by whichthe agent can deal with such models, and end with a discussionon our work on learning and evaluating these models.

A. Human Aware Planning

Most traditional approaches to planning focus on one-shotplanning in closed worlds given complete domain models.While even this problem is quite challenging, and significant

Fig. 5. Figure [73] illustrating explanation generation [14] via the modelreconciliation process, and explicable plan generation [87] by sacrificingoptimality in the robot’s own model.

strides have been made in taming its combinatorics, plannersfor robots in human-robot teaming scenarios require the abilityto be human-aware. We postulate then a departure fromtraditional notions of automated planning to account for withhumans – many of the challenges that arise from it aresummarized in our work [14] under the umbrella of “Multi-Model Planning”. The term alludes to the fact that a cognitiveagent, in course of its deliberative process, must now considernot only the model M that it has on its own but also themodel HuM of the human in the loop, including the (oftenmisaligned) mental model HuMM of the same task that thehuman might have. This setting is illustrated in Figure 4. Here,by the term “model” of an robot, we include its action ordomain model as well as its state information or beliefs andits goals or intentions. The human model allows the robot toaccount for the human’s participation in the consumption ofa plan, while the human mental model enables the robot toanticipate how the the plan will be perceived by the human aswell as interactions that arise thereof.

Human-Robot Teaming / Cohabitation: Incorporation ofthe human model HuM in the planning process allows therobot to take into consideration possible human participationin the task and thus identify its appropriate role in it. Thiscan relevant both when the robot is explicitly teaming [77],[52], [84], [78] with the human, or when it is just sharingor cohabiting [11], [17], [12], [16] the same workspacewithouts shared goals and commitments. We have exploredthe typical roles of the robot in each of these scenarios –e.g. in planning for serendipity [11] and in planning withresource conflicts [17] we looked at how a robot can planfor passive coordination with minimal prior communication,while in [52], [84] we explored the effects of proactive supporton the human teammate. Indeed, much of existing literature

6

Fig. 6. Figure [80] showing a schematic view of different classes of incomplete models and relationships between them in the spectrum of incompleteness.

on human-aware planning [2], [3], [20], [41], [81], [19], [11],[11], [12] has focused on this setting; we will now exploreadditional challenges to the human-aware planning problemin the context of the human mental model HuMM .

Explicable Task Planning: One immediate effect of modeldifferences between the robot and the human is that a roboteven when optimal on its own can be suboptimal, and henceinexplicable, in the model of the human. This situation isillustrated in Figure 5. When faced with such a situation therobot can choose to produce plans π that are likely to bemore comprehensible to the human by being closer to thehuman expectations πHuMM . This is referred to as explicableplanning – here the robot thus sacrifices optimality in its ownmodel in order to produce more human-aware plans. Thereexists some recent work on motion planning while consideringthe human expectation both while computing trajectories [27],[25], [26]. In recent work [87], [43], [86] we have exploredhow this can be achieved in the context of task both when thehuman model is perfectly known, or when it has to be learnedin the course of interactions. The latter work introduces aplan explicability measure [87] learned approximately fromlabeled plan traces as a proxy to the human model. Thiscaptures the human’s expectation of the robot, which can beused by the robot to proactively choose, or directly incorporateinto the planning process to generate, plans that are morecomprehensible without significantly affecting its quality.

Explanation Generation: Such plans, of course, may notalways be desirable – e.g. if the plan expected by the humanis too costly (or even unsafe or infeasible) in the robot’smodel. Then the robot can choose to be optimal (πM ) in itself,and explain [14], [74] its decisions to the human in terms ofthe model differences. This process of model reconciliationensures that the human and the planner remain on the samepage in course of prolonged interactions. At the end of themodel reconciliation process, the optimal plan in the agent’smodel become optimal in the update human model HuMMas well, as shown in Figure 5. The ability to explain itself is acrucial part of the design of a cognitive teammate, especiallyfor developing trust and transparency among teammates. Weargue [14] that such explanations cannot be a soliloquy – i.e.the planner must base its explanations on the human mentalmodel. This is usually an implicit assumption in the explana-

tion generation process; e.g. imagine a teacher explaining toa student - this is done in a manner so that the student canmake sense of the information in their model.

Human-Aware Planning Revisited: Sometimes the processof explanation, i.e. the cost of communication overhead, mightbe too high. However, at the same time, for reasons weexplained above, there might not be any explicable plansavailable either. An ideal middle ground then is to strike abalance between the explicable planning and explanations.We attempt to do this by employing model-space search [73]during the planning process. From the perspective of designof autonomy, this has two important implications - (1) asmentioned before, an agent can now not only explain but alsoplan in the multi-model setting with the trade-off betweencompromise on its optimality and possible explanations inmind; and (2) the argumentation process is known [51] to bea crucial function of the reasoning capabilities of humans, andnow by extension of autonomous agents as well, as a result ofalgorithms that incorporate the explanation generation processinto the decision making process of an agent itself.

B. Learning Human (Mental) Models

Of course, both the previous challenges were built on thepremise that the human (mental) models are available or atleast learned so as to facilitate the decision making processwith these model in mind. Acquiring of such models, takenfor granted among human teammates through centuries ofevolution, is perhaps the hardest challenge to be overcome torealize truly cognitive teaming. The difficulty of this problemis exacerbated by the fact that much (specifically, HuMM )of these models cannot be learned from observations directlybut only from continued interactions with the human.

However, while much of the work on planning has till nowfocused on complete world models, most real-world scenarios,especially when they involve humans, are open-ended in thatplanning agents typically do not have sufficient knowledgeabout all task-relevant information (e.g., human models) atplanning time – in other words, the planning models wouldbe incomplete. Despite being incomplete, such models mustalso support reasoning as well as be improvable from sensing,i.e. learnable. Hence, an important challenge is to developrepresentations of approximate and incomplete models that are

7

Fig. 7. Testbeds developed to study the dynamics of trust and teamwork between autonomous agents and their human teammates.

easy to learn (for human mental modeling) and can supportplanning/decision-making (for anticipating human behavior).

Existing work on incomplete models (Figure 6) differ inthe information that is available for model learning, as well ashow planning is performed. Some of them start with completeaction models and annotate them with possible conditions tosupport incompleteness [53], [88], [89]. Although these mod-els support principled approaches for robust planning, they arestill quite difficult to learn. On the other end of the spectrum,are very shallow models [80] that assume no structured infor-mation at all which are used mainly in short-term planningsupport such as action recommendation. Partial models, thatare somewhere in between, having more structured informationwhile still being easy to learn [85] can prove to be powerfulsupport for goal recognition. However, planning under suchmodels is incomplete. In our work on explanation generation[74] we demonstrated how annotated models such as above canbe used to deal with model uncertainty, while for explicableplanning [87], [43] we showed how CRF/Regression/LSTM-based models can be used to learn human preferences in termsof plan similarity metrics.

This is, however, only a start in this research direction.Performing human-aware planning with incomplete modelsremains an important challenge in human-aware planning,especially given that we do not yet understand how thesedifferent human models interact. For example, different humanmodels capture different aspects of the human (e.g., capabili-ties [85], intentions [75] and emotions [68]) which are closelyinter-related, and it is not clear how they can be combined.

C. Communication and Evolution of Mental Models

All these processes are aimed at bringing the robot’s modeland the human expectation of it closer over continued interac-tions. We have so far only discussed how a robot can maintain

the human mental models. However, in teaming, this modelingis bi-directional. When robots do not have certain information,the robot can plan to sense and communicate with the human.In cases when the human is suspected to have insufficientinformation about the robot, the robot needs to proactivelycommunicate its model to the human. This communicationcan be, for example, about the intention, plan, explanationsand excuses of behavior [35] for the robot; or explanationsthat can not only make sense of the plan generation processitself [40], [72], [44], but must also make sense from the(robot’s understanding of the) human’s perspective [14]. Thisis especially relevant in the case of human-robot teams, wherethe human’s understanding of the robot may not be accurate.Such explanations must not only be able to justify failure [36],[39], [50], [28] but also the rationale behind a successful planin order for the human to be able to reason about the situationand contrast among alternative hypotheses [14], [46], [45].

Further, explanations must be communicated at a levelunderstandable to the human [60], [65]. Communication canthus involve different modalities such as visual projection,natural languages, gesture signaling, and a mixture of them[13]. This capability is important for the human and robot toevolve their mental models to improve teaming in the longterm. Note that the models (i.e., the human actual model andits representation on the robot) are not required to be aligned,which is often only applicable for repetitive tasks [55]. Muchexisting work on robots communicating with humans usingdifferent modalities can be utilized [42], [79]. However, a morecritical challenge for the robot is to compute when, what, andhow to communicate for model adaptation - communicatingtoo much information can increase the cognitive load of thehuman teammates while communicating too little can decreasethe teaming situation awareness. Existing literature on decisionsupport and human-in-the-loop planning [71], [48], [15] can

8

provide insightful clues to dealing with such challenges incommunication on information among teammates.

It must also be realized that many human-robot teamingtasks are not only complex, but can also span multiple episodesfor an extended period of time. In such scenarios, the system’sperformance is dependent on how the teams perform in thecurrent task, as well as how they perform in future tasks.A prerequisite to consider long-term teaming is to maintainmental states of the agents (e.g., trust) that influence theirinteractions, and analyze how these states dynamically affectthe teaming performance and how they evolve over time.

D. Evaluation / Microworlds

The design of human-machine systems is, of course, largelyincomplete unless tested and validated in the proper settings.Although the existing teamwork literature [24] on human-human and human-animal teams has identified characteristicsof effective teams – in terms of shared mental models [10],[49], team situational awareness [37], and interaction [22] – itis less clear how much of those lessons carry over to human-robot teams. To this end, we have developed a suite of testbedsor microworlds for generation and testing of hypothesis andrapid prototyping of solutions. Figure 7 illustrate some of themicroworlds we have used so far – anticlockwise from the left,this includes [1-2] a shared workspace [13] between humansand semi-autonomous agents that supports communicationacross various modalities such as speech, brainwaves (EEG)and augmented reality (AR) [video: https://goo.gl/KdsEgr]; [4-5] simulated Urban Search and Rescue scenarios [52], [84],[73], [74] with internal semi-autonomous agents (humans androbots) supervised by external human teammates [video: https://goo.gl/BKHnSZ]; and [3,6] simulated (such as autonomousdriving and collaborative assembly) domains for the studyof multi-model planning [87], [43], [84], [52] with humansin the loop [video: https://goo.gl/UpEmzG]. The aim hereis to conduct human-human or Wizard of Oz studies [7] incontrolled settings and replicate desired behavior in the designof cognitive teammates.

V. CONCLUSION

In this paper, we discussed the challenges in design ofautonomous robots that are cognizant of the cognitive aspectsof working with human teammates. We argued that tradi-tional goal-based and behavior-based agent architectures areinsufficient for building robotic teammates. Starting with thetraditional view of a goal-based agent, we expand it to includea critical missing component: human mental modeling. Wediscussed the various tasks that are involved when such modelsare present, along with the challenges that need to be addressedto achieve these tasks. We hope that this article can serveas guidance for the development of robotic systems that canenable more natural teaming with humans.

REFERENCES

[1] Pieter Abbeel and Andrew Y. Ng. Apprenticeship learning via inversereinforcement learning. In Proceedings of the Twenty-first InternationalConference on Machine Learning, ICML ’04, pages 1–, New York, NY,USA, 2004. ACM.

[2] Rachid Alami, Aurelie Clodic, Vincent Montreuil, Emrah Akin Sisbot,and Raja Chatila. Toward Human-Aware Robot Task Planning. In AAAISpring Symposium: To Boldly Go Where No Human-Robot Team HasGone Before, 2006.

[3] Rachid Alami, Mamoun Gharbi, Benjamin Vadant, Raphael Lallement,and Adolfo Suarez. On human-aware task and motion planning abilitiesfor a teammate robot. In Human-Robot Collaboration for IndustrialManufacturing Workshop, RSS, 2014.

[4] R.C. Arkin. Motor Schema – Based Mobile Robot Navigation. Inter-national Journal of Robotics Research, 8(4):92–112, Aug. 1989.

[5] T. Balch and R.C. Arkin. Behavior-based formation control formultirobot teams. IEEE Transactions on Robotics and Automation,14(6):926–939, Dec. 1998.

[6] Chitta Baral, Gregory Gelfond, Enrico Pontelli, and Tran Cao Son. Anaction language for multi-agent domains: Foundations. under submissionto Artificial Intelligence, 2015.

[7] Cade Earl Bartlett. Communication between Teammates in Urban Searchand Rescue. Thesis, 2015.

[8] Rodney Brooks. Intelligence without representation. Artificial Intelli-gence, 47:139–159, 1991.

[9] Benjamin Burchfiel, Carlo Tomasi, and Ronald Parr. Distance minimiza-tion for reward learning from scored trajectories. In National Conferenceon Artificial Intelligence.

[10] J.A. Cannon-Bowers, E. Salas, and S. Converse. Shared mental modelsin expert team decision making. Current issues in individual and groupdecision making.

[11] Tathagata Chakraborti, Gordon Briggs, Kartik Talamadupula, Yu Zhang,Matthias Scheutz, David Smith, and Subbarao Kambhampati. Planningfor serendipity. In IROS, 2015.

[12] Tathagata Chakraborti, Vivek Dondeti, Venkata Vamsikrishna Meduri,and Subbarao Kambhampati. A game theoretic approach to ad-hoccoalition formation in human-robot societies. In AAAI Workshop onMulti-Agent Interaction without Prior Coordination (MIPC), 2016.

[13] Tathagata Chakraborti, Sarath Sreedharan, Anagha Kulkarni, and Sub-barao Kambhampati. Alternative Modes of Interaction in ProximalHuman-in-the-Loop Operation of Robots. CoRR, abs/1703.08930, 2017.

[14] Tathagata Chakraborti, Sarath Sreedharan, Yu Zhang, and SubbaraoKambhampati. Plan explanations as model reconciliation: Movingbeyond explanation as soliloquy. In IJCAI, 2017.

[15] Tathagata Chakraborti, Kartik Talamadupula, Kshitij P Fadnis, MurrayCampbell, and Subbarao Kambhampati. UbuntuWorld 1.0 LTS-APlatform for Automated Problem Solving & Troubleshooting in theUbuntu OS. In AAAI, 2017.

[16] Tathagata Chakraborti, Kartik Talamadupula, Yu Zhang, and SubbaraoKambhampati. A formal framework for studying interaction in human-robot societies. In AAAI Workshop on Symbiotic Cognitive Systems(SCS), 2016.

[17] Tathagata Chakraborti, Yu Zhang, David Smith, and Subbarao Kamb-hampati. Planning with resource conflicts in human-robot cohabitation.In AAMAS, 2016.

[18] M. Cirillo, L. Karlsson, and A. Saffiotti. Human-aware task planning formobile robots. In Advanced Robotics, 2009. ICAR 2009. InternationalConference on, pages 1–7, June 2009.

[19] Marcello Cirillo. Planning in inhabited environments: human-aware taskplanning and activity recognition. PhD thesis, Orebro university, 2010.

[20] Marcello Cirillo, Lars Karlsson, and Alessandro Saffiotti. Human-awaretask planning: An application to mobile robots. ACM Trans. Intell. Syst.Technol., 1(2):15:1–15:26, December 2010.

[21] N. J. Cooke. Team cognition as interaction. Current Directions inPsychological Science, 2015.

[22] N. J. Cooke, J. C. Gorman, C. W. Myers, and J.L. Duran. Interactiveteam cognition. Cognitive Science, 2013.

[23] N. J. Cooke and M. L. Hilton. Enhancing the effectiveness of teamscience. National Academies Press, 2015.

[24] Nancy J Cooke, Jamie C Gorman, Christopher W Myers, and Jasmine LDuran. Interactive team cognition. Cognitive science, 37(2):255–285,2013.

[25] Anca Dragan, Shira Bauman, Jodi Forlizzi, and Siddhartha Srinivasa.Effects of robot motion on human-robot collaboration. In Human-RobotInteraction, March 2015.

[26] Anca Dragan, Rachel Holladay , and Siddhartha Srinivasa. Deceptiverobot motion: Synthesis, analysis and experiments. Autonomous Robots,July 2015.

[27] Anca Dragan and Siddhartha Srinivasa. Generating legible motion. InProceedings of Robotics: Science and Systems, 2013.

[28] Thomas Eiter, Esra Erdem, Michael Fink, and Jan Senko. Updatingaction domain descriptions. Artificial intelligence, 2010.

9

[29] Alan Fern, Sriraam Natarajan, Kshitij Judah, and Prasad Tadepalli. Adecision-theoretic model of assistance. J. Artif. Int. Res., 50(1):71–104,May 2014.

[30] Richard E. Fikes and Nils J. Nilsson. Strips: A new approach to theapplication of theorem proving to problem solving. In Proceedings of the2Nd International Joint Conference on Artificial Intelligence, IJCAI’71,pages 608–620, San Francisco, CA, USA, 1971. Morgan KaufmannPublishers Inc.

[31] M. Fox and D. Long. PDDL2. 1: An extension to PDDL for expressingtemporal planning domains. Journal of Artificial Intelligence Research,20(2003):61–124, 2003.

[32] Andr Fuhrmann. The Journal of Symbolic Logic, 57(4):1479–1481,1992.

[33] Michael Georgeff, Barney Pell, Martha Pollack, Milind Tambe, andMichael Wooldridge. The Belief-Desire-Intention Model of Agency,pages 1–10. Springer Berlin Heidelberg, Berlin, Heidelberg, 1999.

[34] B.P. Gerkey and M.J. Mataric. Sold!: Auction methods for multi-robotcoordination. IEEE Transactions on Robotics and Automation, SpecialIssue on Multi-robot Systems, 18(5):758–768, 2001.

[35] Moritz Gobelbecker, Thomas Keller, Patrick Eyerich, Michael Brenner,and Bernhard Nebel. Coming up with good excuses: What to do whenno plan can be found. In ICAPS, pages 81–88, 2010.

[36] M. Goebelbecker, T. Keller, P. Eyerich, M. Brenner, and B. Nebel.Coming up With Good Excuses: What to do When no Plan Can beFound. 2010.

[37] J.C. Gorman, N.J. Cooke, and J.L. Winner. Measuring team situationawareness in decentralized command and control environments. Er-gonomics, 49:1312–1325, 2006.

[38] Barbara J. Grosz and Sarit Kraus. Collaborative plans for complex groupaction. Artif. Intell., 86(2):269–357, October 1996.

[39] Andreas Herzig, Viviane Menezes, Leliane Nunes de Barros, and RenataWassermann. On the revision of planning tasks. In Proceedings of theTwenty-first European Conference on Artificial Intelligence, ECAI, 2014.

[40] Subbarao Kambhampati. A classification of plan modification strategiesbased on coverage and information requirements. In AAAI 1990 SpringSymposium on Case Based Reasoning, 1990.

[41] Uwe Koeckemann, Federico Pecora, and Lars Karlsson. Grandpa hatesrobots - interaction constraints for planning in inhabited environments.In Proc. AAAI-2010, 2014.

[42] Thomas Kollar, Stefanie Tellex, Deb Roy, and Nick Roy. Towardunderstanding natural language directions. In International IEEE/ACMConference on Human-Robot Interaction, 2010.

[43] Anagha Kulkarni, Tathagata Chakraborti, Yantian Zha, Satya GautamVadlamudi, Yu Zhang, and Subbarao Kambhampati. Explicable robotplanning as minimizing distance from expected behavior. CoRR,abs/1611.05497, 2016.

[44] Pat Langley. Explainable agency in human-robot interaction. In AAAIFall Symposium Series, 2016.

[45] Tania Lombrozo. The structure and function of explanations. Trends inCognitive Sciences, 10(10):464 – 470, 2006.

[46] Tania Lombrozo. Explanation and abductive inference. Oxford handbookof thinking and reasoning, pages 260–276, 2012.

[47] Jim Mainprice, E Akin Sisbot, Thierry Simeon, and Rachid Alami. Plan-ning safe and legible hand-over motions for human-robot interaction.IARP workshop on technical challenges for dependable robots in humanenvironments, 2(6):7, 2010.

[48] Lydia Manikonda, Tathagata Chakraborti, Kartik Talamadupula, andSubbarao Kambhampati. Herding the Crowd: Using Automated Planningfor Better Crowdsourced Planning. Journal of Human Computation,2017.

[49] J. E. Mathieu, T. S. Heffner, G. F. Goodwin, E. Salas, and J. A. Cannon-Bowers. The influence of shared mental models on team process andperformance. Journal of Applied Psychology, 2000.

[50] M Viviane Menezes, Leliane N de Barros, and Silvio do Lago Pereira.Planning task validation. In Proc. of the ICAPS Workshop on Schedulingand Planning Applications, pages 48–55, 2012.

[51] Hugo Mercier and Dan Sperber. Why Do Humans Reason? Argumentsfor an Argumentative Theory. Behavioral and Brain Sciences, 2010.

[52] Vignesh Narayanan, Yu Zhang, Nathaniel Mendoza, and SubbaraoKambhampati. Automated planning for peer-to-peer teaming and itsevaluation in remote human-robot interaction. In HRI, 2015.

[53] Tuan Nguyen, Subbarao Kambhampati, and Sarath Sreedharan. Robustplanning with incomplete domain models. Artificial Intelligence, 2017.

[54] Tuan Anh Nguyen, Minh Do, Alfonso Emilio Gerevini, Ivan Serina,Biplav Srivastava, and Subbarao Kambhampati. Generating diverse plansto handle unknown and partially known user preferences. ArtificialIntelligence, 190(0):1 – 31, 2012.

[55] Stefanos Nikolaidis and Julie Shah. Human-robot cross-training: Com-putational formulation, modeling and evaluation of a human teamtraining strategy. In Proceedings of the 8th ACM/IEEE InternationalConference on Human-robot Interaction, HRI ’13, pages 33–40, Piscat-away, NJ, USA, 2013. IEEE Press.

[56] Nils J. Nilsson. Shakey the robot. Technical report.[57] Raz Nissim, Ronen I. Brafman, and Carmel Domshlak. A general, fully

distributed multi-agent planning algorithm. In AAMAS, pages 1323–1330, Richland, SC, 2010. International Foundation for AutonomousAgents and Multiagent Systems.

[58] L.E. Parker. ALLIANCE: an architecture for fault tolerant multirobot co-operation. IEEE Transactions on Robotics and Automation, 14(2):220–240, 1998.

[59] L.E. Parker and F. Tang. Building multirobot coalitions throughautomated task solution synthesis. Proceedings of the IEEE, 94(7):1289–1305, Jul. 2006.

[60] Vittorio Perera, Sai P. Selvaraj, Stephanie Rosenthal, and ManuelaVeloso. Dynamic Generation and Refinement of Robot Verbalization.In Proceedings of RO-MAN’16, the IEEE International Symposium onRobot and Human Interactive Communication, Columbia University,NY, August 2016.

[61] Dennis Perzanowski, Alan C. Schultz, and William Adams. Integratingnatural language and gesture in a robotics domain. In Proceedings ofthe 1998 IEEE International Symposium on Intelligent Control, 1998.

[62] Martin L. Puterman. Markov Decision Processes: Discrete StochasticDynamic Programming. John Wiley & Sons, Inc., New York, NY, USA,1st edition, 1994.

[63] Miquel Ramırez and Hector Geffner. Plan recognition as planning. InIJCAI, pages 1778–1783, 2009.

[64] Anand S. Rao and Michael P. Georgeff. BDI Agents: From Theoryto Practice. In Proceedings of the First International Conference onMulti-Agent Systems (ICMAS-95, pages 312–319, 1995.

[65] Stephanie Rosenthal, Sai P. Selvaraj, and Manuela Veloso. Verbalization:Narration of Autonomous Mobile Robot Experience. In Proceedingsof IJCAI’16, the 26th International Joint Conference on ArtificialIntelligence, New York City, NY, July 2016.

[66] Stuart J Russell, Peter Norvig, and Ernest Davis. Artificial intelligence:a modern approach. Prentice Hall, 3 edition, 2010.

[67] Scott Sanner. Relational dynamic influence diagram language (rddl):Language description, 2011.

[68] M. Scheutz and P. Schermerhorn. Affective goal and task selection forsocial robots. In J. Vallverdu and D. Casacuberta, editors, Handbookof Research on Synthetic Emotions and Sociable Robotics: New Appli-cations in Affective Computing and Artificial Intelligence. Idea GroupInc., 2009.

[69] Matthias Scheutz, Julie Adams, and Scott DeLoach. A frameworkfor developing and using shared mental models in human-agent teams.JCEDM, forthcoming.

[70] Matthias Scheutz and Virgil Andronache. Architectural mechanismsfor dynamic changes of behavior selection strategies in behavior-basedsystems. IEEE Transactions of System, Man, and Cybernetics Part B:Cybernetics, 34(6):2377–2395, 2004.

[71] Sailik Sengupta, Tathagata Chakraborti, Sarath Sreedharan, and Sub-barao Kambhampati. RADAR - A Proactive Decision Support Systemfor Human-in-the-Loop Planning. In ICAPS Workshop on User Inter-faces for Scheduling and Planning, 2017.

[72] Shirin Sohrabi, Jorge A. Baier, and Sheila A. McIlraith. Preferredexplanations: Theory and generation via planning. In Proceedings ofthe 25th Conference on Artificial Intelligence (AAAI-11), pages 261–267, San Francisco, USA, August 2011.

[73] Sarath Sreedharan, Tathagata Chakraborti, and Subbarao Kambhampati.Balancing Explicability and Explanation in Human-Aware Planning. InAAAI Fall Symposium on AI for HRI, 2017.

[74] Sarath Sreedharan, Tathagata Chakraborti, and Subbarao Kambhampati.Explanations as Model Reconciliation - A Mutli-Agent Perspective. InAAAI Fall Symposium on Human-Agent Groups, 2017.

[75] K. Talamadupula, G. Briggs, T. Chakraborti, M. Scheutz, and S. Kamb-hampati. Coordination in human-robot teams using mental modeling andplan recognition. In Intelligent Robots and Systems (IROS 2014), 2014IEEE/RSJ International Conference on, pages 2957–2962, Sept 2014.

[76] Kartik Talamadupula, J. Benton, Subbarao Kambhampati, Paul Scher-merhorn, and Matthias Scheutz. Planning for human-robot teaming inopen worlds. ACM Transactions on Intelligent Systems and Technology.(Special Issue on Applications of Automated Planning), 1(2), 2010.

[77] Kartik Talamadupula, Gordon Briggs, Tathagata Chakraborti, MatthiasScheutz, and Subbarao Kambhampati. Coordination in human-robotteams using mental modeling and plan recognition. In IROS, 2014.

10

[78] Kartik Talamadupula, Gordon Briggs, Matthias Scheutz, and SubbaraoKambhampati. Architectural mechanisms for handling human instruc-tions in open-world mixed-initiative team tasks. To Appear in Advancesin Cognitive Systems, 6, 2017.

[79] Stefanie Tellex, Adrian Li, Daniela Rus, and Nicholas Roy. Asking forhelp using inverse semantics. In In RSS, 2014.

[80] Xin Tian, Hankz Hankui Zhuo, and Subbarao kambhampati. DiscoveringUnderlying Plans Based on Distributed Representations of Actions. InAAMAS, 2016.

[81] Stevan Tomic, Federico Pecora, and Alessandro Saffiotti. Too cool forschool ??? adding social constraints in human aware planning. In Procof the International Workshop on Cognitive Robotics (CogRob), 2014.

[82] Michael Wooldridge. An Introduction to MultiAgent Systems. WileyPublishing, 2nd edition, 2009.

[83] Y. Zhang and Parker, L.E. IQ-ASyMTRe: Forming executable coalitionsfor tightly coupled multirobot tasks. IEEE Transactions on Robotics,29(2):400–416, 2013.

[84] Yu Zhang, Vignesh Narayanan, Tathagata Chakraborti, and SubbaraoKambhampati. A human factors analysis of proactive assistance inhuman-robot teaming. In IROS, 2015.

[85] Yu Zhang, Sarath Sreedharan, and Subbarao Kambhampati. Capabilitymodels and their applications in planning. In AAMAS, 2015.

[86] Yu Zhang, Sarath Sreedharan, Anagha Kulkarni, Tathagata Chakraborti,Hankz Hankui Zhuo, and Subbarao Kambhampati. Plan explicability forrobot task planning. In Proceedings of the RSS Workshop on Planningfor Human-Robot Interaction: Shared Autonomy and CollaborativeRobotics, 2016.

[87] Yu Zhang, Sarath Sreedharan, Anagha Kulkarni, Tathagata Chakraborti,Hankz Hankui Zhuo, and Subbarao Kambhampati. Plan explicabilityand predictability for robot task planning. In ICRA, 2017.

[88] Hankz Hankui Zhuo and Subbarao Kambhampati. Model-Lite Planning:Case-Based vs. Model-Based Approaches. Artificial Intelligence, 2017.

[89] Hankz Hankui Zhuo, Tuan Anh Nguyen, and Subbarao Kambhampati.Refining incomplete planning domain models through plan traces. InIJCAI, 2013.