NSF/NASA Workshop on

Autonomous Mobile Manipulation (AMM)

Houston, Texas, March 10/11, 2005

Final ReportNovember, 2005

prepared by:

Oliver Brock and Roderic GrupenLaboratory for Perceptual Robotics

University of Massachusetts Amherst

Executive Summary

This report presents the findings of a panel of experts from the robotics industry, academia, andfederal R&D agencies regarding the potential socioeconomic value of Autonomous Mobile Ma-nipulation (AMM) systems and makes programmatic recommendations aimed at assuring that theUS capitalizes on large emerging markets that depend on this technology.

Mobile manipulation systems are capable of moving about and interacting mechanically withlarge-scale environments. They are surrogates for people in dangerous and/or remote environ-ments, and they assist human beings in factories and in homes. Autonomous mobile manipulationsystems are capable of doing work without constant oversight and intervention by human oper-ators. These technologies address important and timely social and economic concerns, amongthem:

1. civilian and urban infrastructure, manufacturing, logistic and supply chain support,

2. defense, security and surveillance, search and rescue,

3. satellite rendezvous, capture, re-boost, maintenance, in-orbit maintenance of spacecraft,planetary exploration and habitation,

4. undersea and deep ocean science, oil and gas industry applications, and,

5. personal, home service, elder care, and rehabilitation applications.

Worldwide Capital Investment Service robotics applications are projected to grow from ap-proximately $5.4B now to $52B in 2025 (UN Economic Commission study). In Japan, the DentsuCommunications Institute estimates that networks of robots will create a $20B market by 2013,5.7 times the existing manufacturing market for robotics. The Japanese Ministry of Economy,Trade, and Industry (METI) has earmarked $33M this year for robotics research, and in Europe anAdvanced Robotics program budgets $100M for the next three years. In Korea, robotics has beenidentified as one of ten “engines for economic growth” and funding for “robots for a silver society”totals about $80M per year. In contrast, DARPA has become highly applied, NASA basic researchhas been drastically reduced, and the NSF funds about $10M per year in basic research.

Critical Technological Milestones To complement research and development world-wide, itis recommended that R&D in this area focus on unique strengths in US research laboratories andindustry. To capture critical intellectual capital, the participants recommend creating technologicalfoci:

• dexterous manipulation and physical interaction,

• multi-sensor perception in unstructured environments,

• control and safety near human beings,

• technologies for human-robot interaction, and

• architectures that support fully integrated AMM systems.

Specific Programmatic Recommendations Integrated AMM systems require best practicesfrom several disciplines, including mobility, manipulation, perception, packaging, power, and com-munication. The portfolio of basic research must engage a broad segment of the R&D infrastruc-ture to build on existing strengths in the United States in a manner that complements comparableinitiatives in other countries.

One and two investigator grant applications should be encouraged to develop and evaluate newtechnology in a manner that facilitates sharing and reuse. Center-scale funding mechanisms shouldbe used to foster the development of integrated systems and form partnerships between industryand academia. The goal of these consortia is to draw on expertise in the community to constructsophisticated AMM systems for targeted applications.

Common infrastructure will accelerate technological progress. Examples of such infrastructureinclude a commodity dexterous robot hand, configured for important AMM tasks, with integratedwrist, arm, and whole-surface tactile sensing. Today, a system such as this is very rare and there-fore, too expensive. Another example is a simple integrated mobile manipulator aimed at a muchlower price point that could be disseminated very liberally. Healthy user communities with com-mon infrastructure will incubate new technologies, foster the exchange of software, and help toidentify and evaluate best practices. Center Scale research funding could support the implementa-tion of highly integrated combinations of these simpler platforms.

2

Contents

Executive Summary 1

Preface 4

1 Introduction 4

2 Worldwide Capitalization and State of the Art 5

3 Enabling Technologies 63.1 Power . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73.2 Sensing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73.3 Signal Distribution and Distributed Processing . . . . . . . . . . . . . . . . . . . . 73.4 Mobility . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83.5 Actuation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8

4 Scientific Challenges 94.1 Perception . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 94.2 Safe Manipulators . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 104.3 Grasping and Dexterous Manipulation . . . . . . . . . . . . . . . . . . . . . . . . 104.4 Human-Robot Interaction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 114.5 Knowledge Representation: Objects and Environments . . . . . . . . . . . . . . . 114.6 Programming . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 124.7 Integrated Architectures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12

5 Programmatic Recommendations 135.1 Recommended Funding Mechanisms . . . . . . . . . . . . . . . . . . . . . . . . . 13

5.1.1 Centers for Autonomous Mobile Manipulation . . . . . . . . . . . . . . . 135.1.2 Single Investigator Programs of Research . . . . . . . . . . . . . . . . . . 145.1.3 Infrastructure Support . . . . . . . . . . . . . . . . . . . . . . . . . . . . 145.1.4 AMM User Community . . . . . . . . . . . . . . . . . . . . . . . . . . . 15

5.2 Recommended Research Initiatives . . . . . . . . . . . . . . . . . . . . . . . . . . 165.3 Intermediate Goals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17

6 Acknowledgments 17

References 19

Appendix 20

A List of Participants 20A.1 Steering Committee . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20A.2 Focus Groups . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20A.3 All Participants . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21

B Whitepaper: Integrating Manual Dexterity with Mobility for Human-Scale ServiceRobotics 24B.1 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24B.2 Commercial Potential . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25B.3 Technological and Economic Risk . . . . . . . . . . . . . . . . . . . . . . . . . . 28B.4 Research and Technical Challenges . . . . . . . . . . . . . . . . . . . . . . . . . . 28B.5 Action Items . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29B.6 Affiliations of the Signatories . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30B.7 The US Moon-Mars Initiative . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32B.8 American Demographic Trends . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32

4

Preface

Planning for the Workshop on Autonomous Mobile Manipulation began in mid-2004, when sev-eral members of the technical community started a discussion about the scientific and commercialpotential for autonomous mobile manipulators. Such devices are capable of moving about andperforming mechanical work in unstructured environments without continuous intervention fromhuman operators. Integrated machines of this type will impact the health-care industry, the ser-vicing of orbiting spacecraft and satellites, security and disaster relief, exploration and interactionwith hazardous environments, military applications, manufacturing, supply chain support, and lo-gistics. Appendix B presents a summary of these discussions in the form of a white paper, endorsedby a significant subset of the workshop participants.

The Workshop on Autonomous Mobile Manipulation took place in Houston on March 10 and11, 2005. Forty-two experts in robot design, control and representation, grasping and manip-ulation, perception, and machine learning were assembled to make recommendations about thetechnical feasibility, scientific hurdles, and commercial potential of Autonomous Mobile Manipu-lators (AMM). This report summarizes the discussions and findings of the workshop. It attempts tocapture diverse opinions and to translate them into recommendations for funding agents and policymakers as well as for the R&D community.

The AMM workshop was sponsored by the National Science Foundation (NSF) and by theNational Aeronautics and Space Administration (NASA) but the opinions contained herein arethose of the participants and do not necessarily reflect the opinions of the sponsors.

1 Introduction

Autonomous Mobile Manipulation (AMM) requires the creation of technologies that are capableof projecting mechanical work through communication networks to remote locations via roboticsystems. An integrated AMM system must successfully marry technologies that are often treatedindependently.

Autonomy in AMM endows the robot with some capacity for independent decision-making.This is important to avoid continuous human oversight and intervention especially in inhospitableenvironments where user guidance may be delayed or absent altogether. Even short delays canrender AMM systems ineffective if they depend on detailed human supervision. For example,force feedback must likely be interpreted locally to assure that objects will not escape the grasp ofthe robot—even milliseconds of delay can be too much. Mechanisms for extracting informationfrom sensory feedback, producing corrective actions, learning re-usable patterns of activity, andmodifying plans are central challenges. Importantly, AMM systems may be required to co-existand interact with humans in human environments.

Mobility concerns the mapping and traversal of relatively large scale environments. It wasgenerally agreed that technologies for mobility are relatively mature compared to other aspects ofAMM. The remarkable performance of systems in the 2005 DARPA Grand Challenge testify tothe advanced capabilities available today. In the context of AMM, mobility increases the exposure

5

to variety of objects and environmental conditions that serve as the backdrop for dexterous manip-ulation tasks. This variety must be met with corresponding resourcefulness and variability in robotbehavior.

Manipulation entails the coordination of contact forces between bodies to control movementsand forces in the environment. This spans a range of tasks from earth moving to dexterous assem-bly tasks. Specific AMM systems will address some portion of this range. Research in this area isrelatively mature in structured, industrial settings where geometries, and positions are completelyknown. In this context, techniques for describing the mechanics of interacting rigid bodies are nowpart of standard engineering practice. However, manipulating non-rigid material and performingdexterous manual interactions remain extremely challenging and are critical technical milestonesat the heart of future AMM applications. Sensor-based approaches for controlling manual skillsare in their infancy and must be the subject of considerable research.

Many of the target applications could be realized by autonomous AMM systems with the visualcapacity of a two-year-old human infant, the manual dexterity of a six-year-old, and the ability tomove about in human-scale environments. Moreover, if integrated AMM systems can demonstratethis level of competence, then the community will have the foundational technologies for higherlevels of performance. Specific applications will place different relative emphasis on these capa-bilities and skills derived from them. Some applications will require significant collaboration withhuman clients, necessitating new approaches to programming and the ability to engage in socialinteractions.

The goal of the workshop was to elaborate on the state-of-the-art and to review the dimensionsof a research program that could overcome the perceived technical barriers. We begin by sur-veying the state-of-the-art in important related technologies (Section 2). Following this summary,Section 3 reviews enabling technologies. These technologies are not specific to AMM and willbenefit from development in associated fields. Workshop participants also identified a number ofcritical research challenges described in Section 4. Based on these findings, Section 5 describesprogrammatic recommendations to funding agents and policy makers. These recommendationsaim to create a national research environment, to build on existing strengths in the US, and toassume scientific leadership in the area of Autonomous Mobile Manipulation.

2 Worldwide Capitalization and State of the Art

Despite the fact that the first industrial robots were invented and commercialized in the US, noneof the major industrial robots are built by US companies today. Japan, Korea, and Europe arecurrently investing significantly more in private sector robotics concerns than does the US [2].Moreover, in Japan robotics is one of seven areas of emphasis in the national strategy for develop-ing new industries, and in Korea robotics is designated one of the ten “economic growth engines”with total robotics funding of about $80M per year.

In Asia and Europe, governments and funding agencies are making significant financial in-vestments in research activities associated with humanoid robotics and mobile manipulation. TheJapanese Ministry of Economy, Trade, and Industry has allocated $33M per year for robotics, com-

6

bined with other robotics programs, Japan’s investment in basic robotics research is estimated to beclose to $100M per year including technologies for in-home health care, many of which are robottechnologies. The European Advanced Robotics program is funded at about $33M per year. Incontrast to these aggressive investments, the National Science Foundation currently invests about$10M in basic robotics research per year in the United States. These imbalances are even morestriking when one considers research investment in industry. Foreign corporations, such as Toyota,Honda, and Sony, have underwritten substantial research in robotics over the last 10 years. In theUS, no comparable industrial initiatives exist.

Much of this investment is in service to the development of humanoid robots—robots that havethe same form factor as human beings and thus, can go where humans go, reach what humans canreach, and can use tools and objects engineered for human use. This area differs from AMM in thefocus on anthropomorphic robots, but otherwise shares many of the scientific challenges. Abouta dozen anthropomorphic, humanoid robots have been developed in Japan and Korea to serveas experimental testbeds for ongoing research activities. In most cases, these platforms are forexperimentation in bipedal locomotion and postural control. Among the notable efforts, Honda’sP Series robots (through Asimo) that began in 1991, the humanoid series developed at WasedaUniversity (since 1970), and the H series at the University of Tokyo (since 1991). Sony began theSDR series in 1997 that is in its forth generation robot called the QRIO and the Fujitsu corporationis in the second generation of HOAP platforms. In Korea, the KIST NBH-1 series, and at KAISTthe KHR-1, 2, and 3 robots incorporate 2-five fingered hands. Japan and Korea are clearly aheadin the area of humanoid packages and mobility, and some aspects of service and personal robotsincluding entertainment robots. Europe shows strength in mobility for structured environmentsincluding urban transportation) as well as elder care and home service robots.

Experimental platforms in the US, by way of contrast, focus on bi-manual, humanoid upperbodies with multi-fingered dexterous hands. Although Japan and Korea have national strategiesaimed at building better hands and manipulation controls, and Germany has significant examplesof dexterous robot hands (DLR hand and the commercial Shadow hand), the United States stillenjoys a discernible technological lead in dexterous limbs (hands and arms) and in manipulationcontrol. These domestic technology efforts include: Robonaut (NASA Johnson Space Center—Robonaut has demonstrated untethered mobility as well), the SARCOS humanoid, Domo (MIT),and Dexter (UMass Amherst) [1]. Other strengths of US technologies include mobility in large-scale outdoor environments (DARPA Grand challenge 2005), architectures for integrated control,sensing, and computation, and in space and defense applications.

This report recommends aggressive investments in research leading to integrated manual skillsand dexterous limbs for AMM systems. Moreover, the report also recommends a national pol-icy for encouraging significant participation in US industries in this effort is necessary to buildperceived advantages into commercial potential in emerging markets.

3 Enabling Technologies

Workshop participants differentiated between enabling technologies and scientific challenges (Sec-tion 4). Enabling technologies accrue benefits to society that are not necessarily exclusive to AMM,

7

they relate to other devices with sensors and controls as well—automotive systems, prosthetics,medical equipment, and smart consumer electronics—and will benefit from significant investmentby other technology sectors as well. A good example is research into fuel cells that may well helpto make AMM systems practical, but are the subject of large existing research programs. Scientificchallenges for AMM, on the other hand, describe areas in which new capabilities specific to AMMmust be created to achieve working systems.

3.1 Power

Power density and operational life between charges is a critical technology supporting AMM andmany other important applications. This is a large and thriving research area including basic andapplied efforts in both academia and industry. A great deal of progress has been made over the pastseveral years. Lithium polymer batteries are surpassing the performance of NiMH and lead acidbatteries. Better battery chemistries, dielectric materials, and fuel cells are continuously improvingand will likely prove important in future AMM systems.

3.2 Sensing

In the realm of mobility, effective sensing modalities (such as laser range finders and ladar) haveresulted in significant scientific progress. The state of the art in other transducers and sensor sys-tems relevant to autonomous manipulation has not progressed nearly as far. In addition to sensorsfor mobility, sensing systems relevant to AMM include: joint encoders, load cells, accelerometry,thermal sensing, tactile sensing, acoustic sensors, and CCD arrays (computer vision). Of these,computer vision sensors are relatively cheap, with standardized hardware and interfaces, and withlibraries of signal processing and filtering software. Load cells, accelerometry, and thermal trans-ducer technologies are quite mature, but require effort on a case-by-case basis to configure, am-plify, filter, interpret, and package in order to create robust sensor systems for AMM applications.The companies that manufacture these devices are either very small or see robotics as a very smallpart of their market. The miniaturization and standardization of these sensors would facilitate theuse of multi-modal sensing in AMM.

In contrast, tactile sensing technologies to date are inadequate. Several transduction techniquesare promising, but integrated commercial systems for tactile sensing that are robust, that conformto generic surface geometries, and packaged in arrays with adequate spatial resolution have not yetbeen demonstrated.

3.3 Signal Distribution and Distributed Processing

Wiring sounds prosaic but it remains perhaps the #1 problem in making an advanced hand for ma-nipulation. Bus structures help, but raise their own problems regarding a lack of suitable standardsfor distributed processing and the difficulty of addressing many sensors with different bandwidthrequirements (from 10

1 to 103 Hertz). Wireless systems may well be suitable for low bandwidth

tasks, such as monitoring temperatures, but are not well suited for conveying accelerometry or

8

force information. These signals require significantly higher bandwidth and temporal precision.Research on conductive polymers that can be integrated with three dimensional parts is promisingand may provide practical alternatives to wires and flex circuits. Research on ways to fabricate sen-sors in-situ, directly deposited on parts, with local multiplexing, communications will ultimatelyhelp. Beyond the robotics community, automotive companies like GM are very interested in elim-inating the need for wiring harnesses. A particular challenge is to minimize the amount of powerwiring, as well as signal wiring. Self-powered sensors and ways of turning the entire car body intoa power bus have been considered.

Related to the wiring and sensing problems is the need for better, embedded processing envi-ronments. This issue will clearly impact many industries beyond AMM. Advances in low powerprocessors, I/O, communication, fault tolerance, and software development environments will ac-celerate application development and contribute more standard libraries of embedded computingsolutions.

3.4 Mobility

Mobility is fundamental to AMM systems, however, the state-of-the-art in mobility is significantlymore advanced than technology supporting manual dexterity. The 2005 DARPA Grand Challengehas demonstrated the autonomous navigation of a 131.2 mile off-road course in the Mojave desertin just under 7 hours. This context for autonomous mobility is similar to a variety of potentialAMM applications. Many others require vehicles with different form factors, will require othersensors (instead of, or in addition to GPS and ladar), may exploit inertial loads in the upper body,and must be cognizant of the safety of adjacent human beings. They must be capable of navigatingindoor, outdoor, in micro-gravity, or underwater as the application requires. Several approachesto mobility have been deployed, many mechanisms for R&D exist in the military, industry, andacademia. Moreover, common frameworks and tools are emerging. AMM research and develop-ment should leverage momentum in this area.

3.5 Actuation

Examples of manipulation systems employing pneumatic and hydraulic actuation exist, but in-troduce challenges related to transportable power sources that are compatible with many AMMenvironments. The leading technology in this sector relies on DC motors and gear trains. Hybridactuators that include passive as well as active control elements reduce the energy required in theform of “isometric work” while providing artificial damping via velocity feedback. Technologiesincluding passive energy storage and dissipation elements can better manage work.

New technologies (shape memory alloys, polymers, biological proteins, and bucky tubes) arestill the subject of basic research and it remains unclear whether they will impact the developmentof practical AMM systems. Novel actuation mechanisms may also play an important role in thecreation of inherently safe manipulators (see Section 4.2).

9

4 Scientific Challenges

Workshop participants prepared briefings (http://www-robotics.cs.umass.edu/amm/) focused oncharacterizing the state-of-the-art and scientific challenges underlying working AMM systems andthe applications that depend on them. The five briefs were:

1. Grasping and Manipulation - Rod Brooks, Mark Cutkosky, Rod Grupen, Andrew Miller,Nancy Pollard, Jeff Trinkle

2. Control and Representation - John Hollerbach, Oussama Khatib, Vijay Kumar, Al Rizzi,Daniela Rus

3. Perception - Narendra Ahuja, Yiannis Aliomonos, Dana Ballard, Chris Brown, Greg Hager,Martial Hebert, Martin Jagersand, Jean Ponce

4. Embodiment - Kenneth Salisbury, Akhil Madhani, John Morrell, Bill Townsend, John Evans

5. Teaching and Learning - Andrew Ng, Brian Scasselati, Chris Atkeson, Goeff Gordon, JamesBagnell, Jessica Hodgins, Maja Mataric, Stefan Schaal

Following these briefings, workshop participants discussed these focal areas in a variety ofvenues (plenary, and break-out) and identified the following scientific and technological challengesfor AMM. These summaries assume strengths in the US regarding information technologies, mo-bility, navigation, dexterous robot hands, and manipulation planning and control technologies.

4.1 Perception

Perception refers to processes that interpret signals derived from computer vision, tactile sensing,force feedback, acoustic sensors, proprioceptive information, and other sensory signals that canbe used to differentiate operational contexts [3]. In AMM applications, sensor geometries andperceptual skills will be customized to address required tasks and environments. The computervision community has developed a number of standardized techniques for addressing a variety ofreal-world problems. However, workshop participants believed that these techniques alone can-not yet fully address the perceptual requirements of AMM tasks. Tactile sensing technologies arecritical and are currently inadequate, although several transduction techniques are promising. In-tegrated tactile arrays with sufficient spatial resolution, I/O, and embedded signal processing mustbe available to realize most AMM applications. These devices must be applicable to many surfacegeometries.

Multi-sensor systems can combine sources of information to resolve ambiguity. To date, thevast majority of sensor interpretation work is exclusive to a single sensor modality. We must en-courage these communities to remove barriers to a more comprehensive approach. New approachesto perceptual tasks for AMM must make it possible for robots to behave appropriately in the ab-sence of complete world knowledge and must bias behavior toward activities that recover missingstate. Active sensing strategies can create conditions in which important events are detectable and

10

configure sensor systems to observe them. These techniques hold significant promise to recoverhidden state information and to discriminate between important operating contexts. Active infor-mation extraction using multi-modal percepts are derived from interaction at several time scalessimultaneously. This is an important aspect of controlling interactions with the world that incor-porate real-world effects, such as occlusion, background separation, and sensor faults. Relationalstructure in feedback variables at multiple time scales inform applications including hand-eye co-ordination, preshaping, reaching, grasping, positioning, and manipulating objects.

4.2 Safe Manipulators

Technology for industrial robots has progressed over the past 20 years. Initially automation waslimited to multi-robot work cells (conveyor belts, fixtures, and synchronous work flow) that min-imize forceful interactions by subscribing to structure imposed at design time. Technologies forrobot-robot cooperation were developed characterized by controllable transfer devices, fixtures,and continuous work flow. Modern factory automation has developed recently to support human-robot cooperation in which robots interact closely with human beings to amplify strength andconform to virtual constraints.

State-of-the-art in current industrial practice has demonstrated robots that respond gracefullyto anticipated forces by instrumenting the human-robot interface. However, these interfaces arehighly structured in current manufacturing tasks and are not yet adequate for the unstructured ap-plication domains anticipated in AMM. New work aimed at producing safe manipulators involvesmechanism and actuator design, operating systems, software verification, multi-modal sensing andperception, sensor-based control, and human-robot interaction technologies. Safety is a ubiquitousconcern in AMM systems and will be achieved in a comprehensive systems framework. These is-sues are of current interest in the development of manufacturing technology as well and significantco-development is anticipated.

4.3 Grasping and Dexterous Manipulation

Traditions in robot grasping and manipulation use prior geometric knowledge exclusively to reasonabout sequences of contact forces. In structured environments, like manufacturing, this perspectivehas led to useful applications like fixturing and parts feeders. AMM applications require that theseresults be extended to minimize their sensitivity to incomplete or imprecise prior information.

New sensor-based control techniques are required to orchestrate informative contact interac-tions with the world using closed-loop responses to stimuli. Feedback from computer vision sys-tems, auditory signals, proprioceptive signals in the limbs, force information at multiple contactsites and control dynamics can be used to recover hidden and/or missing state information.

Basic research required in this area includes:

I grasp controllers representing independent objectives in grasping behavior (force closure,collision avoidance, mobility, kinematic conditioning, etc) that can be combined to formintegrated manual behavior,

11

I robust, domain-general manual skills with unknown objects based on patterns of feedbackfrom vision, load cells, tactile sensors, and control dynamics,

I non-rigid bodies including soil and fabric,

I skills supporting the use of tools designed for humans,

I “whole-body” manual behavior, and

I coordinated multi-robot manipulation policies.

4.4 Human-Robot Interaction

Autonomous mobile manipulators are intended to operate in human environments. Therefore,the ability to interact with humans represents a critical component of AMM systems. Humanscommunicate with gestures (manual and vocal), language, through shared knowledge, and throughbehavior. Cooperating agents depend on implicit assumptions about how a particular task must beaccomplished. Effective human-robot interactions will share technologies for user interfaces, andwill incorporate strategies underlying effective human-human collaborations.

Assistive robotics [5] is an example of a short-term AMM application. It provides for tech-nologies for robotic systems that interact closely with human clients, such as: assistance for reha-bilitation from stroke, injury, or disease; guidance and crowd control in disaster areas; therapy forthe cognitively disabled or people with developmental disorders; and assistance for people withspecial needs. These applications depend to varying degrees on physical contact with the environ-ment and interaction with humans using voice and gesture. The subject of human-robot interactionwas addressed in detail by a previous DARPA/NSF workshop held in September 2001 [4].

4.5 Knowledge Representation: Objects and Environments

Unlike current approaches to robot control, AMM systems must address a broad array of tasksand must possess policies for setting up and reconfiguring for new tasks automatically. In manu-facturing applications, the process of reconfiguring for new task is extremely expensive and timeconsuming for human operators. To make robots more autonomous, they must learn and manipu-late models describing events and transitions that occur in response to control activities. Sharableand extensible representations that can be used for manipulating things in the world are required.They should represent system dynamics in a natural way in order to capture mobility and manipu-lation dynamics and provide a language of manual tasks with formal semantics. Human-robot orrobot-robot coordination happens in the context of shared activities. Therefore, knowledge repre-sentations must support recognizing activities as well as generating them. Representations like thisare important aspects of human-robot interaction technologies and research that aims to programby demonstration.

Implicit behavioral models can capture salience and attentional policies, can support predictionand planning, and can create the empirical basis for accumulating explicit world knowledge. Forexample, some properties are observable only in the context of extended interaction, including

12

complete geometrical information, object mass, friction, coefficient of restitution, bulk materialproperties, etc. Techniques must be developed for deciding which domains of interaction shouldbe used to generate implicit world models and how such knowledge can be reapplied to span arange of associated tasks.

4.6 Programming

AMM systems are complex dynamical systems—interactions between body parts, sensors, archivalinformation, other robots, human collaborators, and unstructured environments challenge our abil-ity to anticipate the global dynamics of such systems. As a consequence, we expect that it willbecome increasingly difficult to specify programs a priori. Instead, users will program such sys-tems incrementally in a manner that reflects the kinds of skills required and constraints imposedby the target application.

New techniques are required for conveying programming inputs to robots for mobile manipu-lation tasks. Programmers can be adjacent to the robot in which case vocalization and gesture areconvenient parts of an interface, or they can be remote, with programming exchanges via telepres-ence and teleoperation. These interfaces must convey goals and sequences of control tasks derivedfrom programming interactions, from which the robot assembles new programs in situ. Interfaceabstraction should increase as the robot becomes more competent. The robot will learn new pro-grams by imitating instructors, by autonomous exploration subject to structural guidance, and byusing skills gained in one context in related situations.

Programming AMM applications should focus on:

action research - composing movement primitives into motor skills, movement recognition andimitation,

interaction research - communicative actions between robots and between humans and robots.

engagement research - intention and user modeling, shared attention and motivation, activitymodeling, social skills

4.7 Integrated Architectures

“Architecture” concerns the manner in which the components of an AMM system are organized,composed, integrated, and archived. The architecture facilitates resource allocation, coordinatesmultiple objectives (including knowledge discovery), and supervises the composition of new be-havior. Architectural support includes mechanisms for identifying and exploiting structure auto-matically in the interaction between embodiment, control, learning, planning, and techniques forautomated reasoning. This aspect of AMM systems has received very little attention and is anindispensable part of leveraging existing technology to create functioning systems.

A number of architectural research topics are part of a comprehensive research portfolio, in-cluding:

13

1. the composition of basic behavior into larger behavioral units,

2. life-long learning architectures to accumulate control knowledge and to coordinate explo-ration according to developmental milestones,

3. mechanisms for exploiting constructive intra-action coupling in light of environmental dy-namics,

4. mechanisms for acquiring control knowledge empirically from a set of examples in collabo-ration with a human instructor/client,

5. hierarchical control knowledge representations, in which, behavior can be composed of ab-stract actions that are themselves coordinated skills and activities, and

6. shared libraries for running devices and representing programs for common tasks and robotconfigurations.

5 Programmatic Recommendations

With respect to anthropomorphic AMM hardware, many Asian research labs, in particular in Japanand Korea, have a strong lead. The WTEC Report [2] presents the findings of an independent panelof experts charged with summarizing the state of robotics research worldwide and understandingthe social and economic motivations for investment. On the basis of approximately 25 site visitsto leading research laboratories in Asia, the report recommends that a complementary focus forUS-lead research should be in the upper body, including bi-manual dexterity.

In other research sectors, US strengths were seen in the areas of information technologies formobility (as distinct from integrated systems), navigation, hands, manipulation planning, controltechnologies, and computer vision. These strengths are consistent with the aforementioned focuson upper bodies in the hardware sector. The WTEC report concludes by recommending a policythat nurtures the core strengths of our research communities to prepare for emerging world market.

5.1 Recommended Funding Mechanisms

To remove institutionalized impediments to progress in AMM systems, workshop participants pro-posed several funding initiatives.

5.1.1 Centers for Autonomous Mobile Manipulation

It is a significant undertaking to operate a full-scale bi-manual mobile manipulator. The platformitself can be quite expensive and needs staff support and regular maintenance. Moreover, an inte-grated AMM platform requires coordinating multiple disciplines with expertise ranging from sys-tems engineering to basic research. Center-scale grants are needed to bring a consortia of researchinstitutions and industry together to manage the multi-disciplinary efforts. Workshop participants

14

felt that this is an important mechanism for cross fertilization among research groups and disci-plines. The goal of AMM Centers is the creation of a prototype platform for an appropriate AMMtarget application and provides members of the consortia access to experimental systems. Centerswill also emphasize evaluation and identification of best practices and will drive the creation ofstandard practices and libraries for AMM systems.

5.1.2 Single Investigator Programs of Research

Some of the scientific challenges described above can be supported via existing programs and rela-tively small, 1-2 PI efforts. A broad and integrated AMM initiative, however, cannot be supportedexclusively by the existing Robotics and Computer Vision programs at NSF. To ensure a produc-tive AMM initiative, broader programmatic support for research in AMM are required includingNSF, NIH, DARPA, NASA, and industry.

Workshop participants suggested that these funding mechanisms focus on basic research, novelapplications, the creation of critical infrastructure including data sets, evaluation metrics, andshared libraries to facilitate the deployment of such systems and accelerate new development.

5.1.3 Infrastructure Support

In order to engage the nation’s research infrastructure effectively, the per-unit cost of researchplatforms must be lowered. Currently, existing platforms are very expensive and customized foraddressing subsets of the AMM program scope. A common and cost effective platform would doa great deal toward creating a user community and new standardized technology. This was thesubject of a great deal of discussion at the workshop and is identified as a contributing factor forsuccess. Much of this task is related to systems engineering and represents an important role forautomotive, aircraft, and heavy equipment industries, which are uniquely capable in this regard. AUS strategy for engaging these industries (as has been done in Japan, Korea, and to some degreein Europe) could significantly accelerate progress towards projected AMM markets.

Hardware Infrastructure

Progress in this field is hampered by the paucity of experimental facilities, their special purposenature, maintenance requirements, and expense. Many practical AMM applications can not beaddressed without also providing access to integrated mobile manipulators. Three non-exclusiveapproaches to providing hardware infrastructure were proposed.

The first approach proposes investment in the development and dissemination of a cost effec-tive and easily customized mobile manipulator platform. It should support broad experimentalstudies into the critical scientific challenges posed in Section 4 and be delivered at a price pointthat makes it available widely across the research community. Broad distribution among industrial,research, and educational institutions will help to galvanize a domestic research community.

The second hardware infrastructure concept focuses on creating commodity multi-fingeredrobot hands with integrated tactile sensing that are robust, reliable, and dexterous—engaging

15

multiple fingers and palm during interactions with objects. It should be equipped with dense tactilefeedback from the surfaces of the fingers and palm and packaged together with an integrated wristand arm. Its strength-to-weight ratio should support realistic payloads.

The third proposal aims to create a number of sophisticated and highly integrated bi-manualmobile manipulators should be created by collaboration between industry and academia, operatedand maintained by contract in order to evaluate best practices as they are developed in the largerresearch community. These high-end platforms will serve as prototypes for commercial AMMapplications.

Software Infrastructure

Successful research in AMM has to combine techniques and components from different labora-tories and across different scientific disciplines. Such an integration is beyond the scope of mostongoing research efforts. It will therefore be necessary to develop a broad range of methods andprinciples to support such integration. This will involve standards, protocols, shared architec-tures, common interfaces, representations and file formats, libraries, data sets, and many additionalmeans of facilitating integration and sharing of experiences and resources.

The importance of software infrastructure and software architecture is generally underesti-mated in the academic world, in spite of many “best practices” that have proven extremely suc-cessful in industry over the course of several decades. In this domain, researchers should turn tothese practices and adopt them to accommodate a level of integration among homogeneous systemsthat is commonly performed among practitioners in industry.

5.1.4 AMM User Community

Autonomous Mobile Manipulation requires a more significant integration effort than is typical inmost US robotics research. A dexterous mobile robot carries power, communication, and on-boardprocessing for perception, control, learning, reasoning, and interfaces. It is therefore importantto bring researchers together in the form of multi-disciplinary research programs and integrativeresearch issues.

Currently, the research infrastructure (professional societies, conferences, research institutes,and funding mechanisms) are not designed to cross boundaries. For example, over the past severalyears, there has been rapid progress in the development of computer vision techniques applicableto robotics. However, the scientific interests of the vision and robotics communities have divergedto the point that few major journals or conferences publish new and innovative results at the con-fluence of vision and robotics. A positive outcome of this workshop is the announcement of aJoint Issue of IJCV and IJRR on Vision and Robotics to provide a forum for the communication ofnew ideas at the interface between the vision and robotics communities on topics relevant to AMMresearch (http://www.vision-based-control.org/JointIssue).

Mechanisms for maintaining this dialog and encouraging cross-disciplinary interactions, in-cluding educational initiatives, and partnerships with industry are very important to develop vibrantuser community around AMM systems. This includes continued funding for workshops, specialtracks at conferences, and special issues of journals.

16

In addition to integrating researchers, program managers should look for ways to coordinateprograms across agencies (NSF, DARPA, NIH, NASA, DOE) in service to critical applications andtechnologies. In particular, workshop participants suggested that it is especially important to createvenues that integrate expertise in social and behavioral sciences, biology, information technology,machine learning, and robotics. Finally, given the non-uniform distribution of expertise and in-vestment worldwide, it is very important to look for international collaborations that are mutuallyvaluable.

5.2 Recommended Research Initiatives

Section 4 describes the most important scientific challenges to deployable AMM systems identifiedby workshop participants. We recommend four main research thrusts.

I Behavior: AMM systems must respond continuously to variation in the environment.Controlled behavior, such as grasping, must capture domain general strategies as well asspecialized skills. New techniques for creating behavior are required to address real-worldscenarios: including robustness to occlusion and signal integrity, mechanisms for generaliz-ing behavior to new situations, hierarchical organizations, and multi-modal feedback. Someexamples of such work include:

• motor primitives and motor libraries

• force feedback and admittance control

• learning algorithms for control

• integrated reach, preshape, grasp, manipulate

• dexterity

• haptic exploration

• modeling the mechanics and dynamics of interaction

• non-rigid bodies

• whole-body multi-contact control - resource allocation for multi-functional appendages(feet, legs, hands, arms)

• self organizing, self-assembling, self-repairing robots and control

I Perception: Percepts differentiate task-specific operating contexts, and thus, inform controldecisions. Important state can be extracted from streams of multi-modal feedback signals.Some examples of such work include:

• category-level recognition - visual and haptic,

• unsupervised discovery, modeling, and understanding in natural environments,

• large, integrated sensor systems,

• biologically plausible architectures for sensor interpretation, and

• datasets and metrics.

17

I Programming: AMM systems are complex, distributed, interacting processes that willchallenge traditional programming methods. Programming may take the form of well struc-tured training episodes conducted by a human user/client that serve to expose importantperceptual information and motor policies. Some examples of such work include:

• haptics and teleoperation,

• human-robot cooperation,

• imitation of movement and activity; skill transfer,

• intent and user modeling,

• shared attention and user engagement, and

• task description languages w/formal semantics.

I Architectures: Organizational principles for integrated AMM systems facilitate planning,resource allocation, goal specification (including knowledge discovery), and the combinationof actions that lead to useful behavior. Some examples of research activities in this areainclude:

• primitives for closed-loop behavior,

• composition of sequential, concurrent, and hierarchical behavior,

• accommodating heterogeneous “systems-of-systems;”

• uncertainty and data-driven approaches, and

• knowledge representation.

5.3 Intermediate Goals

In addition to basic research, participants discussed goals that are of interest today and would beappropriate to guide ongoing research activities over the short term (18 month time-frame).

I logistics, supply chain applications, loading pallets/trucks, WalMart-scale distribution sys-tems, re-fueling operations

I battlefield support: deployment of sensors, handling explosive ordnance, rescue operations

I agriculture and construction

I exploration - assisting astronauts, taking samples, assembling, repairing, inspecting, rescu-ing

I elder care - assistive/service robotics, cleaning, companion, go-fer, medical interfaces, cog-nitive and physical prosthesis

6 Acknowledgments

The organizers would like to express their gratitude to Junku Yuh (National Science Foundation,NSF) and Bob Savely and Robert Ambrose (National Aeronautics and Space Administration,

18

NASA). Without their financial support and organizational assistance this workshop would nothave been possible. Funding from the National Science Foundation and grant NSF IIS 0515874 isalso gratefully acknowledged.

The members of the steering committee were instrumental in the preparation of the technicalcontent for this workshop. The organizers would like acknowledge their contributions. The steer-ing committee consisted of Rob Ambrose (Platforms and International Initiatives), Rod Grupen(Grasping and Manipulation), Vijay Kumar (Control and Representation), Greg Hager (Percep-tion), Kenneth Salisbury (Embodiment), and Stefan Schaal (Teaching and Learning). We wouldlike to thank all participants (see Appendix A) for their insightful contributions to the discussionsand the report.

Finally, the organizers are indebted to Priscilla Coe from UMass Amherst and to Melita Stub-blefield from NASA Johnson Space Center for their help in organizing, administering, and execut-ing this event.

19

References

[1] R. Ambrose. Platforms and international initiatives. Presentation at the NSF/NASA Workshopon Autonomous Mobile Manipulation, March 2005.http://www-robotics.cs.umass.edu/amm/presentations/ambrose.ppt.

[2] G. Bekey, R. Ambrose, V. Kumar, A. Sanderson, B. Wilcox, and Y. Zheng. Review of interna-tional research in robotics. Proceedings of the World Technology Evaluation Center (WTEC)Workshop, September 2005.

[3] G. Hager. Perception. Presentation at the NSF/NASA Workshop on Autonomous MobileManipulation, March 2005.http://www-robotics.cs.umass.edu/amm/presentations/hager.ppt.

[4] R. Murphy and E. Rogers. Human-robot interaction. Proceedings of the DARPA/NSF Studyon Human-Robot Interaction, September 2001. http://www.cse.calpoly.edu/ erogers/HRI/HRI-report-final.html.

[5] S. Schaal. Teaching, learning, and developmental programming. Presentation at theNSF/NASA Workshop on Autonomous Mobile Manipulation, March 2005.http://www-robotics.cs.umass.edu/amm/presentations/schaal.pdf.

20

A List of Participants

A.1 Steering Committee

1. Robert Ambrose, Johnson Space Center, NASAPlatforms and International Initiatives

2. Rod Grupen, University of Massachusetts AmherstGrasping and Manipulation

3. Greg Hager, Johns Hopkins UniversityPerception

4. Vijay Kumar, University of PennsylvaniaControl and Representation

5. Kenneth Salisbury, Stanford UniversityEmbodiment

6. Stefan Schaal, University of Southern CaliforniaTeaching, Learning, and Programming

A.2 Focus Groups

1. Platforms and International InitiativesRobert Ambrose, Johnson Space Center, NASA

2. Grasping and ManipulationRodney Brooks, Massachusetts Institute of TechnologyMark Cutkosky, Stanford UniversityRod Grupen, University of Massachusetts AmherstAndrew Miller, Columbia UniversityNancy Pollard, Carnegie Mellon UniversityJeff Trinkle, Rensselaer Polytechnic Institute

3. PerceptionNarendra Ahuja, University of Illinois at Urbana-ChampaignYiannis Aloimonos, University of MarylandDana Ballard, University of RochesterChris Brown, University of RochesterGreg Hager, Johns Hopkins UniversityMartial Hebert, Carnegie Mellon UniversityMartin Jagersand, University of AlbertaJean Ponce, University of Illinois at Urbana-Champaign

4. Control and RepresentationJohn Hollerbach, University of UtahOussama Khatib, Stanford UniversityVijay Kumar, University of PennsylvaniaAl Rizzi, Carnegie Mellon UniversityDaniela Rus, Massachusetts Institute of Technology

5. EmbodimentJohn Evans, President, John M Evans LLCAkhil Madhani, Disney Imagineering R&DJohn Morrell, Segway LLCKenneth Salisbury, Stanford University

6. Teaching, Learning, and ProgrammingChristopher Atkeson, Carnegie Mellon UniversityDrew Bagnell, Carnegie Mellon UniversityGeoff Gordon, Carnegie Mellon UniversityJessica Hodgins, Carnegie Mellon UniversityMaja Mataric, University of Southern CaliforniaAndrew Ng, Stanford UniversityBrian Scasselatti, Yale UniversityStefan Schaal, University of Southern California

A.3 All Participants

1. Narendra Ahuja, University of Illinois at Urbana-Champaign

2. Yiannis Aloimonos, University of Maryland

3. Robert Ambrose, Johnson Space Center, NASA

4. Dana Ballard, University of Rochester

5. Drew Bagnell, Carnegie Mellon University

6. Oliver Brock, University of Massachusetts Amherst

7. Rodney Brooks, Massachusetts Institute of Technology

8. Chris Brown, University of Rochester

9. Mark Cutkosky, Stanford University

10. John Evans, President, John M Evans LLC

11. Andrew Fagg, University of Oklahoma

12. Doug Gage, XPM Technologies

22

13. Rod Grupen, University of Massachusetts Amherst

14. Greg Hager, Johns Hopkins University

15. Martial Hebert, Carnegie Mellon University

16. John Hollerbach, University of Utah

17. Martin Jagersand, University of Alberta

18. Oussama Khatib, Stanford University

19. Vijay Kumar, University of Pennsylvania

20. Akhil Madhani, Disney Imagineering R&D

21. Giacomo Marani, University of Hawaii

22. Maja Mataric, University of Southern California

23. Andrew Miller, Columbia University

24. Mark Moll, Rice University

25. John Morrell, Segway LLC

26. Andrew Ng, Stanford University

27. Una-May O’Reilly, Massachusetts Institute of Technology

28. Alan Peters, Vanderbilt University

29. Nancy Pollard, Carnegie Mellon University

30. Jean Ponce, University of Illinois at Urbana-Champaign

31. Al Rizzi, Carnegie Mellon University

32. Stephen Rock, Stanford University

33. Daniela Rus, Massachusetts Institute of Technology

34. Kenneth Salisbury, Stanford University

35. Bob Savely, Johnson Space Center, NASA

36. Brian Scassellati, Yale University

37. Stefan Schaal, University of Southern California

38. William Townsend, Barrett Technology Inc.

39. Jeff Trinkle, Rensselaer Polytechnic Institute

23

40. Ben Wegbreit, Strider Labs, Inc.

41. Thomas Willeke, NASA Aimes

42. Junku Yuh, National Science Foundation

24

B Whitepaper: Integrating Manual Dexterity with Mobilityfor Human-Scale Service Robotics

B.1 Summary

This position paper argues that a concerted national effort to develop technologies for roboticservice applications is critical and timely—targeting research on integrated systems for mobilityand manual dexterity. This technology provides critical support for several important emergingmarkets, including: health care, service and repair of orbiting spacecraft and satellites, planetaryexploration, military applications, logistics, and supply chain support. Moreover, we argue thatthis research will contribute to basic science that changes the relationship between humans andcomputational systems in general.

This is the right time to act. New science and key technologies for creating manual skills inrobots using machine learning and haptic feedback, coupled with exciting new dexterous machinesand actuator designs, and new solutions for mobility and humanoid robots are now available. Aconcentrated program of research and development engaging federal research agencies, industry,and universities is necessary to capitalize on these technologies and to capture these markets.

Investment in the US lags other industrialized countries in this area partly because initial mar-kets will probably serve health care and will likely appear first outside the borders of the UnitedStates. It is the considered opinion of the signatories of this document that this situation must bereversed. To capitalize on domestic research investment over the past two decades, and to realizethis commercial potential inside of the United States, we must transform critical intellectual capitalinto integrated technology now. Our goal is to ensure that the US economy and scientific commu-nities benefit as this nascent market blossoms and we will outline the economic risks of allowingother nations to continue to go it alone.

Robert Ambrose

Christopher Atkeson

Oliver Brock

Rodney Brooks

Chris Brown

Joel Burdick

Mark Cutkosky

Andrew Fagg

Roderic Grupen

Jeffrey Hoffman

Robert Howe

Manfred Huber

Oussama Khatib

Pradeep Khosla

Vijay Kumar

Lawrence Leifer

Maja Mataric

Randal Nelson

Alan Peters

Kenneth Salisbury

Shankar Sastry

Robert (Bob) Savely

Stefan Schaal

25

B.2 Commercial Potential

The service robotics industry is projected to be a huge commercial opportunity with products goingto market at an accelerating rate over the next 20 years. Service robotics currently shares someimportant characteristics with the automobile industry in the early twentieth century [2] and thehome computer market in the 1980’s [6]. We argue that many of the new opportunities that existrely on technologies supporting manual dexterity. Specifically, the marriage of new research onmanual dexterity involving grasping and manipulation with more mature technologies for signalprocessing and mobility can yield new integrated behavior that supports applications heretoforeunattainable.

We are advancing this argument now because new developments regarding actuation and sens-ing promise to make robots more responsive to unexpected events in their immediate surroundings.This is a boon to mobility technology and is the “missing link” to producing integrated manipu-lation systems. New, bio-inspired robots are demonstrating impressive performance and betterrobustness than their traditional robotic forbears. We have discovered that legged locomotion neednot be as difficult and complex as we had thought. Therefore, we can afford to add new capabili-ties, and complexity, on top of legged platforms. Grasping and dexterous manipulation still awaitcomparable insights and the technological foundations are now in place.

Health Care: Health care is the largest segment of the US economy and is becoming too expen-sive to deliver. We follow closely on the heels of Asia and Europe where demographic pressureis forcing technology to meet the demand for a more efficient means of “in-place” elder care now.This pressure is due largely to a precipitous decline in the ratio of wage earners to retirees andthe prospects are very nearly the same in much of the industrialized world. So far, the US is fore-stalling the problem by holding the birth rate slightly above 2.0 (2.07 in 2004 [1]) but inevitably,the same challenge faces the US health care system in the future [5]. The prospects for large scaleinstitutionalization of the elderly population are daunting, both in terms of the investment in in-frastructure required and in the quality of life issues as older people are moved out of their homesand into centralized facilities.

The answer is technology for “aging in place.” The centralized “mainframe” approach to healthcare for baby boomers around the globe must be augmented with information technology andassistive devices that promise to be the health care equivalent of low cost personal computers [6].In the long term, we must “consumerize” health and wellness technologies and make it practicaland affordable to push them into existing homes. The goal must be to provide cognitive andphysical assistance to the elderly and infirm. Dexterous machines are an important facet of thisarmamentarium. In the shorter term, mobile manipulators can make significant contributions tohealth care in existing hospitals for services to convalescence and recovery operations before theymake it into people’s homes. There are millions of stroke and heart attack patients that are notcurrently getting adequate post-surgical followup and quality-of-life support. These systems mustshare critical geometry with humans to co-exist in human environments and to serve as assistantsto clients across a wide spectrum of service specifications.

26

In-Orbit Servicing of Satellites: The dream of a re-usable space shuttle that can service theInternational Space Station (ISS) and important and unique orbiting platforms like the Hubblespace telescope is waning as administrators at NASA struggle with the exposure of humans toa 1:50 risk of catastrophic failure of the spacecraft. The design of the shuttle is driven by theconfiguration required for a space plane with the significant overhead of human life support. Thesupport chain of maintenance and supply at the threshold of space should not expose humans tounnecessary risk. Robotic maintenance missions are the answer. Collaboration between robots andhumans in such missions is facilitated when both humans and robots can operate the same toolsand have overlapping sensory viewpoints, accessible workspace, and force and velocity capacity.This argues for anthropomorphic robot design to achieve these specifications and once again, robothands and dexterous manipulation are an important key to success.

Planetary Exploration: On January 14, 2004, the President outlined his goals to return to theMoon and then push onto Mars. These goals will require the construction of habitat, and the main-tenance and operation of science labs, geological exploration crews, chemical processing plants,etc. The human pioneers that first undertake this mission will be exposed to tremendous risk whileoutside of protected habitat and yet such activity cannot be avoided. There is a clear role for robotsthat can both navigate and manipulate with some degree of autonomy. Non-dexterous mobilemanipulators capable of excavation and resource extraction will partner with dexterous mobilemanipulators to mine raw materials and to dig trenches, install habitat modules, and then coverthem with regolith to protect them from radiation. The same machines will transition over timeto assist humans that occupy these habitats, and will also serve as caretakers in between humancrews.

Computer systems that act as cognitive and physical prosthetics for astronauts in these hostileenvironments are feasible and necessary to reach these ambitious goals. The round trip commu-nication latency can vary between 2 seconds (low Earth orbit) to upwards of 30 minutes (Marsdepending on where the planets are in their orbits) making it impossible for controllers on Earthto react to problems on the space vehicle or in Martian habitats in a timely manner. Intelligentsystems with the capacity for collaborative and independent problem solving become critical tomission success. Rather than simply follow preprogrammed commands, robots must be able toassess a situation and recommend a course of action without human intervention every step of theway and then effect a solution involving a spectrum of human-robot collaborations.

Manual dexterity and autonomous mobility are key elements of this vision. The couplingbetween systems that are designed to avoid some forms of contact with the environment whileseeking others—often simultaneously and in service to multiple objectives—is critical to missionsuccess.

Military Applications: The Pentagon spent $3 billion on unmanned aerial vehicles between1991 and 1999 and is reportedly prepared to spend $10 billion by 2010 under a Congressionalmandate that one third of its fleet of ground vehicles should be unmanned by 2015 (NationalDefense Authorization Act for Fiscal Year 2001, S. 2549, Sec. 217). The same impact is expectedfor pilotless air and water vehicles, where drone aircraft for reconnaissance and air to ground

27

missile deployment is already becoming accepted military doctrine. Boeing, Northrop Grummond,and Intel (among many others) are currently assembling infrastructure to support these significantmarkets.

A similar revolution in military technology, one that exploits new technology for manual dex-terity, finally promises to replace human hands in dangerous environments as well. With the abilityto manipulate, autonomous machines may one day serve to reduce the exposure of human soldiersin combat, in the supply chain (re-fueling, ordnance), in BSL4 facilities for handling dangeroussubstances. Moreover, this new technology can provide mobile information gathering agents withthe ability to probe environments, dig, and sample soil.

Logistics and Supply Chain Support Almost every aspect of product distribution is automatedwith two notable exceptions: transportation and load/unload at distribution centers. Loading andunloading shipping containers, and inventory control in warehouse and distribution operationscan be automated in the near term by mobile manipulation systems. For example, if there issufficient demand volume, there is a significant cost advantage to using shipping containers totransport materials by land and sea. As a result, large distribution systems, like Walmart, canreduce costs significantly by making inventory management and distribution more autonomous.Mobile manipulation technologies can support automating the rest of distribution, logistics, andmaterial supply chain reducing costs, enhancing inventory tracking and supply chain security.

Contributions to Basic Science: Many of the traits we consider uniquely human stem not fromgreat capacity for strength, speed, or precision, but instead from our adaptability and ingenuity—our dexterity. When we move from laboratories and simulation into the real world, the merits offlexibility and adaptation and the cognitive representations that support these processes are clearlyjustified [7].

The human hand and the neuroanatomy that co-evolved to support it are critical to the successof human beings on earth and our distinctive cognitive ability. In addition to creating integratedmobile manipulators and an array of autonomous manual skills, research on mechanisms, control,and representations for robot hands has the potential to advance our understanding of the com-putational processes underlying cognition. Specifically, the process of grounding knowledge hasimportant implications in language, human development, and man-machine interfaces.

The result will be practical implementations of machines and computational decision makingthat responds to changing situations and complicated environments. Mobile manipulators exploitstructure in the form of Newtonian mechanics. We may exploit rules governing other domainsas well: in bioinformatics, molecular forces and reaction dynamics govern behavior, in enterprisesystems, business rules form categories of transactions and documents. A focused and integratedresearch initiative in this area will prepare for emerging commercial markets, lead to new kindsof adaptable machines, and influence the future relationship between networks of machines andhuman societies.

28

B.3 Technological and Economic Risk

Technological Leadership Commercial versions of mobile manipulation systems will supportservice robotics, health care, military and space applications—markets that can transform economies.Moreover, virtually any computational system that interacts with complex and open environmentsor datasets will benefit.

Europe and Japan are investing tremendous resources in the development of this technologywith $30 billion dollars of investment planned over the next 5 years in Japan alone to prepare forthe nascent service robotics market aimed at elder care. By way of comparison, the total budgetfor the National Science Foundation, including operations and all areas of supported research isapproximately $5.6B/year in 2004 (NSF PR 04-12 - February 2, 2004). Honda, Sony, and Toyotaare making significant investments into humanoid robotic technology. Toyota is launching a servicerobotics division to respond to the R&D challenges posed in this new domain [3].

So far, most efforts in humanoid robotics have focused largely on walking. It stands to reasonthat new technologies for manipulation and manual dexterity are next. With stakes of this magni-tude, the US must take measures to mobilize its resources by actively building consortia of industryand academia to meet this challenge. If this is not afforded the priority it deserves, then we willhave squandered our technological advantage.

Educational US academic institutions have been the torch bearer for high technology training forthe entire world community for several decades. Sadly, the United States is losing that distinction.Applications for graduate school in the US from Europe and Asia are down starkly in the pastcouple of years. This is due in part to restricted access of foreign-born students to our educationalmarket since 9/11/01, and also partly to the massive investment by these nations into research,development, and education. US educational institutions are an effective pipeline for creativeyoung researchers that can be emulated in other parts of the world to serve their economies. Weare being outspent and it will take much less time to lose our advantage in education and trainingthan it took to create it. We argue that a concerted investment in technologies for mobility andmachine dexterity involving all branches of engineering, materials science, computer science, andcognitive science will serve to shore up this slowly eroding infrastructure and attract the world’sbest young minds into areas of critical future economic value to our nation.

B.4 Research and Technical Challenges

• Embodiment - Power, actuation, packaging, mobility, mechanisms, sensors

– Reliable integrated packages for sensing (tactile, visual, auditory, and proprioceptive)and actuation (power source, power-to-weight ratio, volume, controllability) systemsmust be developed to meet these goals.

– Simple, robust, cost effective mechanical systems - combining safety, load carrying ca-pacity and speed, dexterity and power. Hands are an essential sensorimotor componentfor achieving the applications cited.

29

– A new approach encompassing embodiment, control, and cognitive organization is nec-essary to fuel critical future applications.

• Grasping and Manipulation

– New control techniques are required for robots to interact purposefully with the envi-ronment at scales representing the human niche (ranging approximately from 10

−2m

to 101

m, from 0.01 N to 102

N , and over durations ranging from milliseconds tohours).

– New techniques are required to model and reason about complex systems and “systemof systems” ranging from coordinating multiple limbs, large scale mobility, multiplerobots, and human-robot teams.

• Control/Perception/Representation/Cognition

– New approaches to representing sensorimotor interaction are needed at several levels(feature, object, context) and at several spatial and temporal scales.

– Incomplete world state must be addressed with intelligent, active information gatheringtechnologies that recover critical context on a task-by-task basis.

– New approaches are needed for modeling “activity” in sensor data and discrete eventfeedback.

– Representations employed by robots must be grounded in natural phenomena accessi-ble directly to humans and robots alike.

• Teaching, Learning, and Developmental Programming -Interactions between body parts, sensors, archival information, other robots, human collabo-rators, and an unstructured environment form hierarchies of complex systems that challengetraditional approaches to programming.

– New approaches to instruction, imitation, and exploration must be incorporated intomachine learning techniques to acquire the building blocks of cognitive systems.

– Formal models of generalization, and processes of assimilation and accommodation.

– New programming techniques are needed that incorporate lifelong training and instruc-tion.

– Methods for transferring experience earned by one agent (human or robot) into mean-ingful and actionable knowledge by another agent or agents.

B.5 Action Items

The immediate agenda involves using this document to address the community, including fundingagencies, industry, and academia, in order to direct attention to this critical technical, economic,and scientific challenge. The signatories of this document would be happy serve help this role.

30

This introduction will be followed by presentations at workshops, symposia, and panels toelaborate on the critical technical challenges and opportunities, to create a more detailed researchagenda, and to create an organized advocacy group and fund-raising strategy A joint NSF/NASAWorkshop on Autonomous Mobile Manipulation is in the planning stages. and will lead to aproposal soon to kick this process off. We invite outside participants to help in the organization ofsuch events and we look forward to serving the information needs of industry and federal agenciesin realizing this important milestone in service robotics applications.

References

[1] Central Intelligence Agency. The world factbook.http://www.cia.gov/cia/publications/factbook/geos/us.html, 2004.

[2] Uutinen Julkaistu. Service robotics defines future of man-machine interaction. Tekes Tech-nical Programmes, http://akseli.tekes.fi/Resource.phx/tuma/kone2015/en/robotics-uutinen.htx,2004.

[3] D. Kara. Sizing and seizing the robotics opportunity. In COMDEX, 2003.

[4] R.W. Pew and editors Van Hemel, S.B. Technology for Adaptive Aging. The NationalAcademies Press, Washington, DC, 2003.

[5] Nicholas Roy, Gregory Baltus, Dieter Fox, Francine Gemperle, Jennifer Goetz, Tad Hirsch,Dimitris Margaritis, Mike Montelermo, Joelle Pineau, Jamie Schulte, and Sebastian Thrun.Towards personal service robots for the elderly. In Workshop on Interactive Robots and Enter-tainment (WIRE), 2000. http://web.mit.edu/nickroy/www/papers/wire2000.pdf.

[6] B. Schlender. Intel’s Andy Grove: The next battles in tech. Fortune, pages 80–81, May 2003.

[7] M. Swinson and D. Bruemmer. Expanding frontiers of humanoid robotics. Intelligent Systems,12(17), 2000.

B.6 Affiliations of the Signatories

This document was prepared by Professors Rod Grupen and Oliver Brock at the Laboratory forPerceptual Robotics, University of Massachusetts Amherst.

Ambrose, Robert - Robonaut Team Leader, Automation, Robotics and Simulation Division, NASAJohnson Space Center

Atkeson, Christopher - Professor, Robotics Institute, Carnegie Mellon University

Brock, Oliver - Assistant Professor of Computer Science and Co-Director of the Laboratory ofPerceptual Robotics, University of Massachusetts Amherst

31

Brooks, Rodney - Professor of Computer Science and Engineering and Director of the ComputerScience and Artificial Intelligence Laboratory (CSAIL) at the Massachusetts Institute ofTechnology, and co-founder and Chief Technology Officer of iRobot Corporation.

Brown, Chris - Professor, Computer Science, University of Rochester

Burdick, Joel - Professor of Mechanical Engineering and Bioengineering; Deputy Director, Cen-ter for Neuromorphic Systems Engineering, California Institute of Technology

Cutkosky, Mark - Professor and Associate Chair Design Division, Department of MechanicalEngineering, Stanford University

Fagg, Andrew - Associate Professor of Computer Science, University of Oklahoma

Grupen, Roderic - Professor of Computer Science and Director of the Laboratory of PerceptualRobotics, University of Massachusetts Amherst

Hoffman, Jeffrey - Professor, Department of Aeronautics and Astronautics, Massachusetts Insti-tute of Technology

Howe, Robert - Professor of Engineering, Director of the BioRobotics Laboratory, Harvard Uni-versity

Huber, Manfred - Assistant Professor, Department of Computer Science and Engineering, Uni-versity of Texas at Arlington

Khatib, Oussama - Professor of Computer Science, Stanford University

Khosla, Pradeep - Professor and Head, Department of Electrical and Computer Engineering,Philip and Marsha Dowd Professor of Electrical and Computer Engineering and Robotics,Carnegie Mellon University

Kumar, Vijay - Professor, Mechanical Engineering and Applied Mechanics, Neuroscience - mir-ror neurons, activity recognition, teaching and learning Psychology and Computer and In-formation Science, University of Pennsylvania

Leifer, Lawrence - Professor, Department of Mechanical Engineering, Director, Stanford Centerfor Design Research Director, Stanford Learning Laboratory, Stanford University

Mataric, Maja - Associate Professor, Computer Science Department, Director, USC Center forRobotics and Embedded Systems (CRES), University of Southern California

Nelson, Randal - Associate Professor, Department of Computer Science, University of Rochester

Peters, Alan - Associate Professor of Electrical Engineering, Department of Electrical and Com-puter Science, Vanderbilt University

Salisbury, Kenneth - Departments of Computer Science and Surgery, Stanford University.

32

Sastry, Shankar - NEC Distinguished Professor of Electrical Engineering and Computer Sci-ences and Bioengineering, Berkeley University

Savely, Robert (Bob) - Senior Scientist for Advanced Software Technology, Chief Scientist Au-tomation, Robotics and Simulation Division, NASA Johnson Space Center

B.7 The US Moon-Mars Initiative

The Moon-Mars initiative includes a new space vehicle to return astronauts to the Moon as earlyas 2015. Highlights of President Bush’s space exploration goals include:

• completing work on the International Space Station by 2015;

• developing and testing a new manned space vehicle(the crew exploration vehicle) by 2008and conducting the first manned mission by 2014;

• returning astronauts to the moon as early a 2015 and no later than 2020;

• using the Moon as a stepping stone for human missions to Mars and worlds beyond; and

• allocating $11 billion in funding for exploration over the next five years, which includesrequesting an additional $1 billion in fiscal 2005 (Congress responded in July by recom-mending a $220 million reduction)

B.8 American Demographic Trends

The United States has seen a rapid growth in its elderly population during the 20th century. Thenumber of Americans aged 65 and older climbed to 35 million in 2000, compared with 3.1 millionin 1900. For the same years, the ratio of elderly Americans to the total population jumped from onein 25 to one in eight. The trend is guaranteed to continue in the coming century as the baby-boomgeneration grows older. Between 1990 and 2020, the population aged 65 to 74 is projected to grow74 percent.

The elderly population explosion is a result of impressive increases in life expectancy. Whenthe nation was founded, the average American could expect to live to the age of 35. Life expectancyat birth had increased to 47.3 by 1900 and the average American born in 2000 can expect to live tothe age of 77.

Because these older age groups are growing so quickly, the median age reached 35.3 years in2000, the highest it has ever been. West Virginia’s population is the oldest, with a median age of38.6 years. Utah is the youngest, with a median age of 26.7 years.

33