Manuela Veloso, Anthony Stentz, Alexander RudnickyBrett Browning, M. Bernardine Dias

Faculty

Thomas Harris, Brenna Argall, Gil JonesSatanjeev Banerjee

Students

Human-Robot “Pickup” Teams with Language-Based Interaction

Sponsored by The Boeing Company

Boeing Human-Robot Teams Project2Carnegie Mellon School of Computer Science

Project Goals

• Robots that discover and understand each other’s capabilities

• Robots that can team together and coordinate their activities

• Human-robot teams that collaborate to accomplish tasks

Boeing Human-Robot Teams Project3Carnegie Mellon School of Computer Science

Domain: Treasure Hunt

Human-robot teams competing to locate “treasure” in an unknown environment

Team 3, report your location

Boeing Human-Robot Teams Project4Carnegie Mellon School of Computer Science

Treasure Hunt scenarios• One human and two robots search for a

treasure and return it to base [Y1] stationary human; Pioneer/Segway

close-coupled team Semi-mobile human (partially accessible

zones) [Y2]

• Human and two teams search for treasure(s) [Y2-3]

• Two teams of human/robots compete to locate and retrieve treasure(s) [Y4]

Boeing Human-Robot Teams Project5Carnegie Mellon School of Computer Science

Language Interface Integration• MAP GUI integrated

Graphical display of information from robots Mixed speech / gesture inputs

• Flexible architecture Dynamic incorporation of additional robots Improved communications protocols

• Language and Dialog Navigation and search language Clarification and confirmation keyed to robot

capability

Boeing Human-Robot Teams Project6Carnegie Mellon School of Computer Science

Architecture

Sphinx Phoenix

Tablet

Rosetta

Raven Claw 1

Raven Claw 2

Helios

Helios

Kalliope

Bac

kend

Clie

nt

Robot 1

Robot 2

User interface Dialog control Robot system

OpTrader

Map Server

SpeechGUI

OlympusBoingLib

Boeing Human-Robot Teams Project7Carnegie Mellon School of Computer Science

Multi-modal interface• Some classes of

information are communicated more effectively by gesture than by language

• User can specify a search path using stylus, ask robots to identify themselves

• Components Java GUI TeamTalk dialog

system Fujitsu Stylistic

5500 tablet PC

Boeing Human-Robot Teams Project8Carnegie Mellon School of Computer Science

Video of Human-Robot interaction

Boeing Human-Robot Teams Project9Carnegie Mellon School of Computer Science

Near term goals Dynamic updating of map information Access to robot capability data Goal conflict resolution (play / direct

command) Better Op Trader state transparency

Boeing Human-Robot Teams Project10Carnegie Mellon School of Computer Science

Publications• T. K. Harris, S. Banerjee, and A. I. Rudnicky. Heterogeneous Multi-Robot

Dialogues for Search Tasks. (2005) AAAI Spring Symposium: Dialogical Robots, Palo Alto, California.

• T. K. Harris, S. Banerjee, A. Rudnicky, J. Sison, Kerry B., and A. Black. A Research Platform for Multi-Agent Dialogue Dynamics. (2004) Proceedings of The IEEE International Workshop on Robotics and Human Interactive Communications, Kurashiki, Japan.

Boeing Human-Robot Teams Project11Carnegie Mellon School of Computer Science

Demonstration Videos

• Robot Coordination Video

• Human-Robot multi-modal interaction Video

Boeing Human-Robot Teams Project12Carnegie Mellon School of Computer Science

Ongoing challenges• Search in cluttered environments• Learn to dynamically select of tactics, integrating

information from team members (human, robot) and knowledge of environment

• Extend play coordination to include complete state machines with additional synchronization primitives and human input

• Develop multiple model-based object and teammate tracking

• Extend grounding interactions between human and robot• Improve robot infrastructure; system operations made

fully routine.

Boeing Human-Robot Teams Project13Carnegie Mellon School of Computer Science

Y2 Plan

• Develop techniques to enable robots to form pickup teams with dynamic sub-team formation and execute coordinated actions Formalize requirements for robot team

participation Incorporate a new (Boeing) robot into the

team Explore different human-robot team

compositions (team size, robot types)

Boeing Human-Robot Teams Project14Carnegie Mellon School of Computer Science

Y2 Plan

• Extend language and agent interfaces to allow humans to interact efficiently with pickup robot teams Incorporate visual feedback from robots Access richer robot state information Extend domain ontology Extend capability for clarification sub-dialogs Introduce simple landmark grounding

capability

Boeing Human-Robot Teams Project15Carnegie Mellon School of Computer Science

Y2 Plan

• Investigate extensions to the Y1 scenario Size and complexity of the environment Dynamic environments

• Identify a final Y2 demo scenario

Boeing Human-Robot Teams Project16Carnegie Mellon School of Computer Science

Technology Transfer to Boeing TeamTalk spoken language interface

Includes Sphinx recognition system, Phoenix semantic parser, RavenClaw dialogue manager, Rosetta language generator; Boeing has synthesizer (Theta)

Updates provided over time Working with Boeing to adapt system

Boeing Human-Robot Teams Project17Carnegie Mellon School of Computer Science

Questions?