RoboBrain: Large-Scale Knowledge Engine for Robotsasaxena/papers/robobrain2014.pdf · RoboBrain: Large-Scale Knowledge Engine for Robots Ashutosh Saxena, Ashesh Jain, Ozan Sener,

RoboBrainLarge-Scale Knowledge Engine for Robots

Ashutosh Saxena Ashesh Jain Ozan Sener Aditya Jami Dipendra K Misra Hema S KoppulaDepartment of Computer Science Cornell University and Stanford University

Email asaxenaasheshozanseneradityajdkmhemacsstanfordedu

AbstractmdashIn this paper we introduce a knowledge enginewhich learns and shares knowledge representations for robotsto carry out a variety of tasks Building such an engine bringswith it the challenge of dealing with multiple data modalitiesincluding symbols natural language haptic senses robot trajec-tories visual features and many others The knowledge storedin the engine comes from multiple sources including physicalinteractions that robots have while performing tasks (perceptionplanning and control) knowledge bases from WWW and learnedrepresentations from leading robotics research groups

We discuss various technical aspects and associated challengessuch as modeling the correctness of knowledge inferring latentinformation and formulating different robotic tasks as queriesto the knowledge engine We describe the system architectureand how it supports different mechanisms for users and robotsto interact with the engine Finally we demonstrate its use inthree important research areas grounding natural languageperception and planning which are the key building blocks formany robotic tasks This knowledge engine is a collaborativeeffort and we call it RoboBrain

I INTRODUCTION

Over the last decade we have seen many applications thathave redefined machine intelligence by successfully combininginformation at a large-scale Examples include Google knowl-edge graph [15] IBM Watson [19] Wikipedia Apple Siri andmany others The very fact they know answers to many of ourday-to-day questions and not crafted for a specific task makesthem valuable for humans Inspired by them researchers haveaggregated domain specific knowledge by mining data [4 6]processing natural language [10] images [13] and speech [45]The knowledge available from these sources is centered aroundhumans however their symbolic nature makes them of limiteduse for robotsmdashfor example imagine a robot querying asearch engine for how to ldquobring sweet tea from the kitchenrdquo(Figure 1)

Contrary to humans for whom incomplete and ambiguousinstructions may suffice robots require access to a large varietyof information with finer details for performing perceptionplanning control and natural language understanding Specifi-cally the robot would need access to knowledge for groundingthe language symbols into physical entities knowledge thatsweet tea can either be on table or in fridge and knowledge forinferring the appropriate plans for grasping and manipulatingobjects Efficiently handling this joint knowledge represen-tation across different tasks and modalities is still an openproblem in robotics

sweet_tea

looks_like

grasp_feature

pouring

has_trajectory

looks_like

subject_of

object_o

object_of

first_order_collection

type_of

refrigerator looks_like

object_of

PerceptionModule Environment

PerceptionQuery

PlanningModule

PlanningQuery

Representation

Fig 1 An example application showing a robot using RoboBrainfor performing tasks The robot is asked ldquoBring me sweet teafrom the kitchenrdquo where it needs to translate the instruction into theperceived state of the environment Our RoboBrain provides usefulknowledge to the robot for performing the task (a) sweet tea can bekept on a table or inside a refrigerator (b) bottle can be grasped incertain ways (c) opened sweet tea bottle needs to be kept upright(d) the pouring trajectory should obey user preferences of movingslowly to pour and so on

In this paper we present RoboBrain a knowledge enginethat allows robots to learn and share such knowledge We learnthese knowledge representations from a variety of sourcesincluding interactions that robots may have while performingperception planning and control as well as natural languageand other unstructured knowledge and visual data from theInternet Our representation considers several modalities in-cluding symbols natural language visual or shape featureshaptic properties and so on We believe that by learningand sharing such large-scale knowledge different robots canbecome adept faster at performing a variety of tasks in newsituations

We mention a few challenges that we need to address whiledesigning RoboBrain

bull Multi-modal data Robots have a variety of sensorsranging from visual (such as images 3D point-cloudsand videos) to haptic Our representations should be

Fig 2 A visualization of the RoboBrain graph on Nov 2014 showing about 50K nodes and 100K directed edges The left inset showsa zoomed-in view of a small region of the graph with rendered media This illustrates the relations between multiple modalities namelyimages heatmaps words and human poses For high-definition graph visualization see http robobrainmegraphNov26 2014pdf

capable of encoding these different modalitiesbull Beliefs in the concepts The concepts learned from the

data come from a variety of noisy sources RoboBrainshould be able to represent the belief in the correctnessof learned concepts

bull Never ending learning Our knowledge engine should beable to incorporate new knowledge and update beliefsThe system should be scalable allow real-time updates toexisting knowledge and allow knowledge sharing acrossrobots and users

bull Ability to answer queries Many robotic tasks such as per-ception reasoning and grounding natural language shouldbe able to make use of the knowledge in RoboBrain Weprovide an interface to RoboBrain as a query library forbuilding various applications

bull Knowledge acquisition RoboBrain acquires knowledgefrom different sources including physical robot interac-tions and partner projects Thus we need an architecturethat is compatible with most of the sources Currentlythe partners include Tell Me Dave [44] Tellexrsquos group[56] PlanIt [26] and affordances [29]

bull Crowd-sourcing Incorporating knowledge from differentsources may lead to significant inconsistencies There-fore we need to have some supervisory signal for learn-ing correct concepts in RoboBrain Hence our systemcomprises several forms of feedback that allows users togive feedback at different levels of engagement

In order to address the aforementioned challenges wepresent an architecture that can store the large-scale knowl-edge in a way that is efficient to retrieve and update duringlearning Our knowledge is multi-modal and primarily containsrelational information therefore we use a graph representationof the knowledge (see Figure 1 and 2) We also present a querylibrary that robots can use for performing different tasks

We present use of RoboBrain on three applications in thearea of grounding natural language perception and planningIn first we show how we can ground natural language com-mands into an action plan for robot controllers In second weconsider human activity anticipation Many perception tasksrequire different contextual information such as spatial contextaffordances and others In third we consider path planningfor mobile manipulators in context-rich environments Whenplanning a path a robot needs to have the knowledge of theusage of different objects and human preferences We showhow we use RoboBrain to formulate a cost function for theplanner to plan paths

One nice property that emerges because RoboBrain beingan integrated knowledge engine is that representations are nowshared across research areas For example knowledge abouthow cups are placed on a table (upright especially when theycontain liquids) could have been learned from perception butit would help in manipulation as well

As more and more researchers contribute knowledge toRoboBrain it will not only make their robots perform betterbut we also believe this will be beneficial for the roboticscommunity at large RoboBrain is under open Creative Com-mons Attribution license (aka CC-BY)1 and is avalable athttprobobrainme

II RELATED WORK

Knowledge bases Collecting and representing a large amountof information in the form of a machine-readable knowledgebase (KB) has been widely studied in the areas of artifi-cial intelligence data mining natural language processingand machine learning Early seminal works have manuallycreated knowledge bases (KBs) with the study of common

1httpcreativecommonsorglicensesby25 Extra conditions may applydepending on the data sources see robobrainme for detailed information

sense knowledge (Cyc [40]) and lexical knowledge of englishlanguage (WordNet [18]) With the rapid growth of WikipediaKBs started to use crowdsourcing (DBPedia [4] Freebase [6])and automatic information extraction methods (Yago [55 24])for mining knowledge

Recently several approaches to KB generation from webdata and other unstructured sources have been developedExamples include NELL [10] and NEIL [11] One of thelimitations of these KBs is their strong dependence on asingle modality that is the text modality There have been fewsuccessful attempts to combine multiple modalities Imagenet[13] enriched the wordnet synsets with images obtained fromlarge-scale internet search and a crowdsourced approach wasused to obtain the object labels These object labels werefurther extended to object affordances [61]

In industry we have seen many successful applicationsof the existing KBs within the modalities they covered forexample Google Knowledge Graph and IBM Watson JeopardyChallenge [20] However the existing KBs do not applydirectly to robotics problems where the knowledge needs to beabout entities in the physical world and of various modalities

Robotics Works For robots to operate autonomously theyshould perceive our environments plan paths manipulateobjects and interact with humans This is very challengingbecause solution to each sub-problem varies with the taskhuman preference and the environment context We nowbriefly describe previous work in each of these areasPerceiving the environment Perception is a key element ofmany robotic tasks It has been applied to object labeling [372 60] scene segmentation [23] robot localization [43 46]feature extraction for planning and manipulation [30] under-standing environment constraints [28] and object affordances[12 34] Sharing representations from visual information notonly improve the performance of each of the perception tasksbut also significantly help various applications such as au-tonomous or assistive driving [14 8] anticipation [31 35 59]planning sociable paths [36] and for various household choressuch as grasping and cutting [41] Sharing representations fromother modalities such as sound [52] and haptics [21] wouldalso improve perceptionPath planning and manipulation There exist a large class ofalgorithms which allow robots to move around and modifythe environment Broadly the planning algorithms can becategorized as motion planning [62 53] task planning [1 7]and symbolic planning [17 51] Bakebot [7] towel-folding[54] IkeaBot [32] and robots preparing pancakes [5] are few ofthe many successful planning applications In order to executecomplex manipulation tasks cost functions have been learnedin a data-driven manner [50 26] Most planning algorithmsabstract out the perception details however as explainedearlier access to perception and manipulation knowledge canallow robots to plan in dynamic real world environmentsInteracting with humans Another important aspect is human-robot interaction Previous works have focused on variousaspects such as human-robot collaboration for task com-

pletion [48 33] generating safe robot motion near humans[42 39] obeying user preferences [9] generating humanlike and legible motions [22 16] interaction through naturallanguage [57 44] etc All these applications require access toperception manipulation language understanding etc furtherdemonstrating the need for large scale multi-modal data

Previous efforts on connecting robots range from creatinga common operating system (ROS) for developing robotapplications [49] to sharing data acquired by various robotsin the cloud [58 3] For example RoboEarth [58] provides aplatform for the robots to store and off-load computation tothe cloud and communicate to other robots and KIVA systems[3] use the cloud to coordinate motion and update trackingdata for hundreds of mobile platforms However RoboBrainknowledge engine provides the knowledge representation layeron top of data storing sharing and communication

III OVERVIEW

We represent the knowledge in RoboBrain as a graph (seeSection IV Figure 1 and Figure 2) The concepts are the nodesin the graph and the edges represent relations between them

RoboBrain is a never ending learning system in that it con-tinuously incorporates new knowledge from different sourcesand modifies the graph In other words we look at everyadditional knowledge acquisition as a measurement event thatdynamically modifies the graph In the next section we willdescribe the formal definition of the RoboBrain graph howwe update it and the concept of latent graph

Before we describe the technical specification of RoboBrainwe mention a few of the technical components

Knowledge acquisition RoboBrain acquires knowledge fromvarious sources including physical robot interactions knowl-edge bases from WWW and from partner projects One keyknowledge resource for RoboBrain is by crawling knowledgesources on the Internet such as WordNet ImageNet Free-base and OpenCyc These knowledge sources provide lexicalknowledge grounding of concepts into images and commonsense facts about the world In detail

bull WordNet provides the required lexical knowledge thatallows us to group nouns and verbs into distinct cognitiveconcepts

bull ImageNet grounds the noun category from WordNet intoimages This allows our knowledge graph to associate animage with every noun

bull OpenCyc is an ontology for everyday common senseknowledge

We are also crawling many other sources such as Wikipediato add knowledge to RoboBrain The technical details ofadding knowledge to RoboBrain is described in Section IV

Disambiguation Incorporating new knowledge into the graphis challenging We need to decide whether the incomingknowledge should create new nodes and edges or insteadadd information to the existing nodes and edges Since ourknowledge graph carries semantic meaning it should resolve

polysemy using the context associated with nodes For ex-ample a lsquoplantrsquo could mean a lsquotreersquo or an lsquoindustrial plantrsquoand merging them together will create errors in the graphTherefore RoboBrain needs to disambiguate the incominginformation using the associated context and decide how toadd it to the existing knowledge graph

Beliefs Knowledge that we incorporate in the RoboBrain maynot be fully reliable and there could be inconsistencies in theknowledge coming from different sources We need a way toindicate beliefs in the correctness of the concepts and relationsin the RoboBrain

Online crowd-sourced interaction We take inspiration fromlarge-scale crowd-sourced projects such as Wikipedia Whileprevious works have focussed on using crowd-sourcing fordata collection and labeling (eg [13]) our work focuses ontaking crowd-sourced feedback at many levelsbull Weak feedback We have found that it is easiest for users

to provide a binary ldquoApproverdquoldquoDisapproverdquo feedbackwhile they are browsing online Our RoboBrain systemencourages users to provide such feedback which is thenused for several purposes such as estimating belief on thenodes and edges

bull Feedback on graph We present the graph to the userson the Internet and smartphones where they can givefeedback on the correctness of a node or an edge in thegraph Such feedback is stored for inferring the latentgraph The key novelty here is that the system shows theproposed ldquolearnedrdquo concepts to the users mdash thus gettingfeedback not at the data level but at the knowledge level

bull Feedback from partner projects There are several partnerprojects to RoboBrain such as Tell Me Dave [44] PlanIt[26] and so on These projects have a crowd-sourcedsystem where they take feedback demonstrations orinteraction data from users Such data as well as thelearned knowledge is shared with the RoboBrain

We not only need to figure out how to present knowledgein a consumable form to users but also need to figure out howto store and use the user feedback in improving the qualityand correctness of knowledge in the RoboBrain

Queries The primary purpose of the RoboBrain is to allowqueries by the users and robots We present a RoboBrainQuery Library (RaQueL) that is used to interact with theRoboBrain knowledge base It comprises of functionalitiesfor retrieving sub-graph ranking relations between nodesusing beliefs functions to work with multi-modal media andother application tailored functionalities that allow its use bythe users and robots Details of RaQueL along with usageexamples are provided in Section VI

IV KNOWLEDGE ENGINE FORMAL DEFINITION

In this section we present the formal definition ofRoboBrain (RB) Knowledge Engine

Our RB is expressed as a directed graph G = (VE) seeFigure 2 for an example The vertices V can be of a varietyof types such as image text video haptic data or a learned

entity such as affordances deep features parameters etc Theedges E sube V times V timesC link two nodes with a direction and atype where C is the set of edge types See Table I and II fora few examples

We further define a set of auxiliary functionsXv(middot) Xe(middot) bv(middot) and be(middot) where Xv(v) and Xe(e)is the raw data associated with the vertex v and the edgee and bv(v) and be(e) are the beliefs that indicate thecorrectness of the node v and the edge e respectively

An edge (v1 v2 `) is an ordered set of two nodesv1 and v2 of type ` Few examples of such edges are(StandingHuman Shoe CanUse) (StandingHuman N (microΣ)SpatiallyDistributedAs) and (Grasping DeepFeature23 Uses-Feature) These edges can be considered as (subject objectpredicate) triplets Although we do not impose any constraintsover the nodes in terms of content or modality we require theedges to be consistent with the RB edge set

A Creating the Graph

Graph creation consists of never ending cycle of twostages namely knowledge acquisition and inference Withinthe knowledge acquisition stage we collect data from varioussources and during the inference stage we apply statisticaltechniques to infer the latent information in the aggregateddata These two stages are explained in more detail below

Knowledge acquisition RoboBrain accepts new informationin the form of an edge set This information can either be froman automated algorithm or from our research collaborators Wecall this edge set a feed

We add the new information to the existing graph througha graph union of the existing graph and the feed followedby an inference Specifically given a new feed consisting ofN edges (v11 v

1) (vN1 vN2 `

N ) and an existing graphG = (VE) graph union stage gives us Gprime = (V prime Eprime) as

V prime = v11 cup v12 cup cup vN1 cup vN2 cup VEprime = (v11 v

1) cup cup (vN1 vN2 `

N ) cup E(1)

Inferring the Latent Graph We treat the acquired knowl-edge as a noisy observation of the real knowledge-base thatshould contain the proper information about the real physicalworld We may not be able to observe the real graph directlytherefore we call it the latent graph For example latent infor-mation ldquocoffee is typically in a containerrdquo is partially observedthrough many images of a container in media correspondingto coffee This construction can be explained in a generativesetting as having a latent graph G of all the knowledge aboutphysical word The measurements through various channelsof limited expressiveness (eg natural language trajectoriesetc) in the form of feeds create an ambiguity over theunderlying latent graph

In order to obtain the latent graph G(V E) we canapply several operations over the aggregated data We describetwo of them here split and merge Split operation is definedas splitting a node into a set of two nodes Edges havingend point in the split node are connected to one of the

Bowl CanUse N(micro 1 Σ1 )

CupHasMed

ediaHasMedia

SpatiallyDist

CanUse

SittingHumanN(micro 1 Σ1 )

(a) original graph

CanUse

CupHasMed

ediaHasMedia

SpatiallyDist

CanUse

(b) feed insertion

N(micro 1 Σ1 )

HasMedia Mug

SittingHumanBowl

TypeOf

edia HasMedia

SpatiallyDist

CanUseSpatiallyDistCanUse N(micro 1 Σ1 )

CanUse

(c) after merge(MugMugprime) rarr Mug split(Cup)rarr (CupMugprime)

Fig 3 Visualization of inserting new information lsquoSitting human can use a mugrsquo and the inferred split and merge operations on thegraph (a) shows the original sub-graph (b) information about a Mug is seen for the first time and the corresponding node and edge areinserted (c) inference algorithm infers that previously connected cup node and cup images are not valid anymore and it splits the Cup nodeinto two nodes as Cup and Mugprime and then merges Mugprime and Mug nodes

TABLE I Some examples of different node types in our RoboBraingraph For full-list please see the code documentation

Word An english word represented as an ASCII stringDeepFeature Feature function trained with a Deep Neural NetworkImage 2D RGB ImagePointCloud 3D point cloudHeatmap Heatmap parameter vector

TABLE II Some examples of different edge types in our RoboBraingraph For full-list please see the code documentation

IsTypeOf human IsTypeOf a mammalHasAppearance floor HasAppearance as follows (this image)CanPerformAction human CanPerformAction cuttingSpatiallyDistributedAs location of human is SpatiallyDistributedAsIsHolonym tree IsHolonym of leaf

resultant nodes using the disambiguation algorithm A mergeoperation is defined as merging two nodes into a single nodewhile updated the connected edges with the merged nodeThese two operations create latent concepts by splitting nodesto semantically different subsets or inferring common latentconcepts that exist in multiple nodes via merge operation Anexample of such an update is shown in Figure 3 When a newinformation ldquositting human can use a mugrdquo is added it causesthe split of Cup node into Cup and Mug nodes

Thus we can define the inferred latent RoboBrain graph as

G = splitvs1 mergevm1vm2 splitvsM G

Resulting Latent Graph Since we do not expect the acquiredknowledge to be complete we treat the information proba-bilistically In other words we express the uncertainty over themissing information as a set of beliefs These beliefs representthe confidence over the created nodes and edges In otherwords we apply an update operation on the belief functions asnew knowledge is added to the RoboBrain Ideally the beliefsshould converge to the latent graph of the physical world withmore information and feedback Practically however we neverexpect full convergence and we design probabilistic queryalgorithms that use the beliefs for answering the input queries

V SYSTEM ARCHITECTURE

We now describe the system architecture of RoboBrainshown in Figure 4 The system consists of four interconnected

Knowledge Inference

Knowledge acquisition

Knowledge Parser

Knowledge Storage

User SessionLogs

Serving Data

Public APIsKnowledge Parser

Internet Knowledge Bases

(Wikipedia ImageNet etc)

Robo BrainProject

Partners

Robo Brain Crawler

RB Knowledge Base

Feeds User Feedback Machine Learning

parameters

Large Media Storage(Images

Videos etc)

Robots Cars and

Smart devices

Robo BrainQuery Language

Distributed QueueLearning and Inference

Learning algorithms Disambiguation Ranking

Graph builder etc

Robo Brain Graph database

httprobobrainme

GraphVisualizer

Fig 4 RoboBrain system architecture It consists of four intercon-nected knowledge layers and supports various mechanisms for usersand robots to interact with the RoboBrain

layers (a) knowledge acquisition (b) knowledge parser (c)knowledge storage and (d) knowledge inference The principlebehind our design is to efficiently process large amount ofunstructured multi-modal knowledge and represent it usingstructured RoboBrain graph In addition our design also sup-ports various mechanisms for users and robots to interact withthe brain Through these interactions users retrieve knowledgeand also improve RoboBrain through feedback Below wediscuss each of the components

Knowledge acquisition layer is the interface between theRoboBrain and the different sources of multi-modal dataThrough this layer RoboBrain gets access to new informationwhich the upper layers process (see Figure 4) RoboBrainprimarily collects knowledge through its partner projects andby crawling the existing knowledge bases such as FreebaseImageNet and WordNet etc as well as unstructured sourcessuch as Wikipedia and Youtube

Knowledge parser layer of RoboBrain processes the dataacquired by the acquisition layer and converts it to a consistentformat for the storage layer It also marks the incomingdata with appropriate meta-data such as timestamps source

version number etc for scheduling and managing future dataprocessing Further since knowledge bases might change withtime it adds a back pointer to the original source

Knowledge storage layer of RoboBrain is responsible forstoring different representations of the data In particular itconsists of a NoSQL document storage database cluster ndash RBKnowledge Base (RB-KB) ndash to store ldquofeedsrdquo parsed by theknowledge parser crowd-sourcing feedback from users andparameters of different machine learning algorithms providedby RoboBrain project partners RB-KB offloads large mediacontent such as images videos and 3D point clouds to adistributed object storage system built using Amazon SimpleStorage Service (S3) The real power of RoboBrain comesthrough its graph database (RB-GD) which stores the struc-tured knowledge The data from RB-KB is refined throughmultiple learning algorithms and its graph representation isstored in RB-GD One of the primary goal behind this design isto keep RB-KB as RoboBrainrsquos single source of truth (SSOT)SSOT centric design allow us to re-build RB-GD in case offailures or malicious knowledge sources

Knowledge inference layer contains the key processing andmachine learning components of RoboBrain All the new andrecently updated feeds go through a persistent replicated dis-tributed queuing system which are consumed by some of ourmachine learning plugins (disambiguation graph builder etc)and populates the graph database These plugins along withother learning algorithms (that operates on entire knowledgegraph) constitutes our Learning and Inference frameworkThis framework comprises a bunch of fault tolerant restfulapplication servers that are scaled automatically based onload Every application server also bookmarks the progressinformation (metadata) so that a new server can catch upinstantly if one of the systems die

RoboBrain also support various interaction mechanisms toenable robots cars and users to communicate with RoboBrainknowledge base RoboBrain query library is the primarymethod for robots to interact with the RoboBrain A set ofpublic APIs allow information to be presented on WWW forenabling online learning mechanisms (crowd-sourcing)

Our RoboBrain system is designed with several other de-sired properties such as security overall throughput moni-toring and alerting Our front end and API servers (queryingand crowd-sourcing systems) are auto-scaled based on loadand most of the data is served using a commercial contentdelivery network to reduce the end user latency The detaileddescription of the system design is beyond the scope of thispaper and will be described in future publications

VI ROBOBRAIN QUERY LIBRARY (RaQueL)

For robots to perform tasks (see Section VII for someexamples) they not only need to retrieve the stored knowl-edge (eg trajectories heatmaps object affordance naturallanguage lexicons etc) but also need to perform certain typesof operations specific to the applications This situation is notusually encountered in traditional databases and thus we need

to design a RoboBrain Query Library (RaQueL) for makingRoboBrain easily usableRaQueL can be used for diverse tasks such as semantic

labeling cost functions and features for grasping and manipu-lation grounding language anticipating activities and fetchingtrajectory representations Typically the queries involve find-ing a sub-graph matching a given pattern retrieving beliefsover nodes and edges and then performing reasoning on theextracted information The structure of RaQueL admits thefollowing three types of functionsbull Graph retrieval function fetch function for querying

a sub-graph of RoboBrain that satisfies a given pattern Apattern defines relations between node and edge variablesEg the pattern given below defines two node variablesuv in parenthesis and one edge variable e in squarebrackets The arrows represent the edge direction

(u)rarr [e]rarr (v)

This pattern is satisfied by all paths of length 1 andfetch function can then be used to retrieve all of them

bull Integrating programming constructs RaQueL inte-grates programming language constructs such as mapfilter find etc

bull Task specific functions Functions for performing aspecific type of task such as ranking trajectories se-mantic labeling of an image grounding natural languagefunctions to work with heatmaps etc

We describe these functions in detail in the following

A Graph retrieval function

The fetch function is used to retrieve a sub-graph ofRoboBrain that matches an input pattern The function fetch

takes as input a pattern Pattern(arg1 arg2 middot middot middot ) wherearg1 arg2 middot middot middot are the pattern variables and returns a listof tuples with instantiated values of variables

fetch Pattern(arg1 arg2 middot middot middot )rarr [(arg(i)1 arg

(i)2 middot middot middot )]i

This pattern is matched against sub-graphs of RoboBrain andthese matched sub-graphs are used to create initialized entriesfor the pattern variables RaQueL uses Neo4j Cypher [47] toimplement the fetch functionality We now explain the usageof fetch in the following examples

Example 1 Retrieve all the activity nodes connected to nodeswith name ldquocuprdquo

fetch(name lsquocupprime)rarr [istype lsquoIsActivityprime]rarr (v)

The parenthesis describes a node variable whose attributespecifications are given inside the curly brackets In this querythe name attribute of the node should be lsquocuprsquo The edgevariable also has a istype attribute specification The patternwill match against paths of length one u e v where nodeu has name lsquocuprsquo and edge e has type lsquoIsActivityrsquo Sincethere is only one exposed variable v the fetch will return alist of tuple with instantiated values of variable v only

For brevity we will directly write the attribute value foredges when attribute name is istype

We can also fetch paths between nodes using the extensionof the above example These paths can be unbounded orbounded by a given length

Example 2 Return the path between the nodes with nameldquocuprdquo and ldquochairrdquo

fetch(name lsquocupprime)rarr [rlowast]rarr (name lsquochairprime)fetch(name lsquocupprime)rarr [r lowast 5]rarr (name lsquochairprime)

The first query returns edges of all the paths between nodeswith name lsquocuprsquo and lsquochairrsquo The second one only returnsedges of all paths of length less than or equal to 5

The fetch function in general can accept any pattern stringallowed by Cypher For more details we refer the readers toCypherrsquos documentation [47]

B Integrating Programming Constructs

While fetch function is sufficient for queries where thepattern is known in advance For other type of queries wewant to make a sequence of fetch queries depending upon theresults from the previous queries This is specially useful forgreedy exploration of the graph eg by fetching only on thosenodes returned by a fetch query which have a high beliefFor this reason RaQueL supports functional programmingconstructs such as map filter find etc2

We now present how these constructs can be combined withfetch to do a greedy exploration of RoboBrain Consider theproblem of exploring the sub-graph rooted at a given nodeu and finding its relations to all other nodes such that theserelations have a certain minimum belief

Formally by relation Ru of a given node u we refer to atuple (u P v) where P is a path from node u to node v

In such cases exploring the entire graph is impracticalHence several greedy search heuristics can be employed Wenow take up a specific example to explain this further

Example 3 Find at least N number of relations of nodesnamed lsquocupprime which have a belief of atleast b

query t = fetch (uname lsquocupprime)rarr [P lowast middot middot middot t]rarr (v)

trustedPath t = filter (λ (u P v) rarr belief P ge b) query t

find (λrelrarr len rel ge N) map (λ trarr trustedPath t) [1 middot middot middot ]

In this query we begin by defining two functions query andtrustedPath The query function takes a parameter t andreturns all relations (u P v) such that u is a node with namelsquocuprsquo and P is a path of length less than t The trustedPath

function takes a parameter t and filters in only those relations(u P v) returned by calling query t such that path P hasa belief of at least b Here the function belief P is anuser-defined function which returns a belief for path P Thiscould be as simple as additionmultiplication of beliefs overnodes and edges in path P The last line gradually calls thetrustedPath function on increasing values of t isin 1 2 middot middot middot and each time returning a set of relations whose belief isat least b The find function then returns the first element

2map applies a function to list elements filter returns list elements whichsatisfy a predicate find returns the first element that satisfies the predicate

(which itself is a list) in this list whose length is at least N Handling the corner case where there does not exist a set ofN relations of belief at least b is not shown in this examplefor brevity Note that since these programming constructs arelazy-evaluated the last line terminates as soon as it exploresthe sub-graphs rooted at lsquocuprsquo nodes until the depth requiredfor finding N relations with belief at least b

C Task Specific Functions

RaQueL provides functions for special tasks such as se-mantic labeling of an image ranking a trajectory based onhuman preference finding a set of affordances for a givenobject grounding natural language expressions etc RaQueLalso provides functionalities to work with media files ofRoboBrain such as functions to generate and work withheatmaps trajectories videos etc For full list of functionssupported by RaQueL and more examples please refer tothe RoboBrain website

VII APPLICATIONS

In this section we show use of RoboBrain in following threeapplications (a) grounding natural language into controllers(b) anticipating human actions and (c) planning in presenceof humans

A Grounding Natural Language

In the context of a given environment and natural languagediscourse we define the problem of grounding a naturallanguage command as coming up with an action plan for arobot that accomplishes close to the given command

It is challenging to ground natural language for an openset of tasks For example grounding the sentence ldquoget mea cup of hot waterrdquo requires that the robot must know theappearance of cup it should be able to infer that cup has thecontainable affordance The robot must be able to classify ifthe cup has water or not and in case the cup is empty then therobot must know that objects such as tap fridge dispenser etccan potentially fill it with water It must also know that objectssuch as stove microwave can heat an object Finally robotshould be able to manipulate these devices to accomplish therequired sub-goals Specifically this consists of knowledge ofenvironment such as manipulation features of objects objectaffordances prior constraints over state-action space etc oflanguage such as synset relations POS tags and knowledgethat requires both such as grounding of noun-phrases like ldquoredmugrdquo We now take specific examples to show how some ofthese queries are translated to a graph-query on RoboBrain

Use of RoboBrain A common approach to language ground-ing problem is to maximize the likelihood P (I|EL) of thegrounded instruction sequence I given the environment Eand the natural language command L Figure 5 shows sucha setting for the task of making recipes The instructionsequences typically consist of action primitives such as pourgrasp keep etc that are suitable for high-level planning Forexample the instruction sequence corresponding the commandldquofill a cup with waterrdquo could be

cereal

ldquoadd ice-cream to cup and drizzle syrup over itrdquoargmaxI P(Ι | EL)

squeezeable (syrup1)on (syrup1pot1)getTraj(pour cup1)

True belief 06False belief 1

[traj1traj2]

Learned Concepts

Grounded Sequence

moveto(spoon1)grasp(spoon1)scoop(spoon1ice-cream1)place(ice-cream1pot1)place(spoon1table1)grasp(syrup1)squeeze(syrup1pot1)

cup rarrcup1vanilla syruprarrsyrup2drizzle syruprarrgrasp(syrup1) hellip

Fig 5 Grounding a natural language sentence for a given environment requires several pieces of knowledge Robot should know theappearance of each object it should know the affordances and possible action related to each object and it should know the trajectories andmanipulation features of those actions These are all translated as queries to the RoboBrain graph

moveto(cup01) grasp(cup01) moveto(sink01) keep(cup01on sink01) toggle(sink knob01) wait() toggle(sink knob01)

However not all instructions are meaningful (trying to pourfrom a book) and not all instruction sequences are valid(keeping cup inside microwave without opening the door)In order to come up with meaningful instruction sequence Irequires knowing what actions can be performed with a givenobject and what sequences of instructions are permitted Misraet al [44] used strips language to represent pre-conditions fora given instruction

For example in Figure 5 the action primitivesqueeze(syrup1pot1) is meaningful only if it satisfiesthe following predicatebull grasping pr2 syrup1 Robot is grasping the syrupbull squeezeable syrup1 Syrup bottle should be squeezeablebull on syrup1 pot1 Syrup bottle should be placed directly

above the potRoboBrain is used to compute satisfiability of these predicateswhich are challenging to compute for an arbitrary object Forexample the satisfiability of the second predicate for textmodality is written as the following RaQueL query

squeezeable syrup1 = len fetch (uname lsquosyrupprime)rarr[lsquoHasAffordanceprime]rarr (vname lsquosqueezeableprime) gt 0

Once the robot finds the optimum instruction sequence itneeds to know how to translate action primitives such aspour(cup01) into trajectories This is challenging since thetrajectory would depend upon the object parameter (considerpour(cup01) and pour(kettle01)) These are translated to queriesto RoboBrain as follows

pour cup01 = fetch (name lsquopourprime)rarr(vname lsquoHasTrajectoryprime)larr (name lsquocupprime)

The above example returns all the trajectory nodes(containingtrajectory parameters) that are connected to nodes with namecup and nodes with name pour Note that since RoboBrainreturns the result with beliefs that are continuously updatedby user feedback we further order them by rank and usethe highest belief node or apply more specific filters suchas safety

B Anticipating Human actionsAssistive robots working with humans need to reactively

respond to the changes in their environments In addition tounderstanding what can be done in the environment and detectthe human activities the robots also need to anticipate whichactivities will a human do next (and how) This enables anassistive robot to plan ahead for reactive responses [35]

In this application our goal is to predict the future activitiesas well as the details of how a human is going to performthem in short-term (eg 1-10 seconds) For example if arobot has seen a person move his hand to a coffee mug itis possible he would move the coffee mug to a few potentialplaces such as his mouth to a kitchen sink or to a differentlocation on the table In order to predict this we model theactivities as a conditional random field which captures therich spatial-temporal relations through object affordances [35]However the challenge is that there are a large number ofobjects and affordances that need to be stored and updated asnew affordances are discoveredUse of RoboBrain In this work we use RoboBrain to storethe learned activity affordance and trajectory parameters andquery for the appropriate knowledge as and when requiredby the anticipation algorithm More specifically there are twosets of queries issued to the RoboBrain We first query for theRoboBrain for node and edge parameters of the conditionalrandom field model as shown below

parents n = fetch (v)rarr [lsquoHasParametersprime]rarr (handle n)parameters n = fetch (name n)rarr [lsquoHasParametersprime]rarr

(vsrc lsquoActivityprime)node parameters a = filter(λurarr len parents u = 1)parameters aedge parameters a1 a2 = filter(λurarr len parents u = 2 and

u in parameters a2) parameters a1

The queries node parameters and edge parameters are issuedwith all activities of interest and pairs of them as parametersrespectively The returned parameters are then used by therobot to infer activities and affordances in its environmentOnce the activities of the past are predicted the second setof queries shown below are issued to the RoboBrain forgenerating possible futures

$QWLFLSDWLRQ

(QYLURQPHQW2EMHFWVDQG$FWLYLWLHV

$IIRUGDQFHV7UDMHFWRUSDUDPHWHUV

)HHGEDFNWR5RERUDLQ$QWLFLSDWLRQ6FRUHV

PRYHDEOH

VWDWLRQDUPRYLQJ UHDFKLQJ

SODFLQJ

7RS$QWLFLSDWLRQV

^UHDFKDEOH`

^SODFHDEOHUHDFKDEOH`Fig 6 RoboBrain for anticipating human activities Robot queries RoboBrain for the activity affordance and trajectory parameters inorder to generate and rank the possible future activities in a given environment Feedback from the scored anticipations are then providedto the RoboBrain for updating the beliefs over the affordance and activity nodes

affordances n = fetch (name n)rarr [lsquoHasAffordanceprime]rarr(vsrc lsquoAffordanceprime)

trajectories a = fetch (handle a)rarr [lsquoHasParametersprime]rarr(vsrc lsquoAffordanceprime type lsquoTrajectoryprime)

trajectory parameters o = map(λararr trajectories a) affordances o

The affordances query is issued for each object of interest inthe environment in order to obtain the possible future objectaffordances and hence the actions that can be performed withthem For each retrieved object affordance the trajectoriesquery is issued to fetch the corresponding motion trajectoryparameters using which the anticipation algorithm generatesthe most likely futures and ranks them (see Figure 6)

Querying the RoboBrain for activity and affordance parame-ters allows the anticipation algorithm to scale to new activitiesand affordances For example as other works on learningactivities and affordances [38 61] use RoboBrain these willbecome available to the anticipation application as well

Feedback to RoboBrain In addition to querying RoboBrainfor affordance and trajectory parameters for predicting thefuture the anticipation algorithm can also provide feedbackon the quality of the information returned by the query Forexample when queried for the possible affordances of theobject cup if an incorrect affordance is returned say writableThe anticipation algorithm gives the possibility of writing witha cup a very low score This information is communicated backto RoboBrain which is useful for updating the beliefs in theknowledge graph

C Planning in presence of humans

One key problem robots face in performing tasks in humanenvironments is identifying trajectories desirable to the usersAn appropriate trajectory not only needs to be valid froma geometric standpoint (ie feasible and obstacle-free) butit also needs to satisfy the user preferences [26 27] For

Feedback

Fig 8 Feedback to RoboBrain RoboBrain also improves withfeedback provided on physical robots In this figure user providesfeedback on robot by improving the trajectory proposed by theRoboBrain As feedback user moves the knife away from himself

example a household robot should move a glass of water in anupright position while maintaining a safe distance from nearbyelectronic devices Similarly robot should move sharp objectssuch as knife strictly away from nearby humans [25]

These preferences are commonly represented as cost func-tions parameters of which are learned in a data-driven mannerThe cost functions for generating good trajectories jointlymodel the environment task and trajectories This joint mod-eling is expensive in terms of resource requirement and datacollection and typically research groups have independentlylearned different cost functions over the years [26 36 31] thatare not shared across the research groupsUse of RoboBrain We now demonstrate use of RoboBrain forplanning Through RoboBrain robots can share cost functionparameters for different planning problems such as indoor2D-navigation manipulation planning high-level symbolicplanning etc as shown in Figure 7 In addition to this robotscan also query RoboBrain for expressive trajectories featuresto instantiate cost functions for predicting good trajectoriesThis kind of sharing will allow research groups to share theirlearned cost functions and trajectory features

Figure 7 shows how PlanIt planning system [27] queriesRoboBrain for trajectory parameters for the task at hand Therobotrsquos goal is to move an egg carton to the other end of

vasecup egg cereal

Task Move the egg cartonPlanIt

EnvironmentObjects and

Trajectoryparameters

argmax

User Feedback

Co-active Feedback to RoboBrain

Top three trajectories

Fig 7 RoboBrain for planning trajectory Robot queries RoboBrain through PlanIt planning system [27] for trajectory parameters forplanning fragile objects such as egg cartons (Right) If robot is uncertain about optimal behaviour it elicits feedback from users Feedbackfrom users are directly registered in RoboBrain and trajectory parameters are updated for future iterations

table Since eggs are fragile users would prefer to move themslowly and close to table surface In order to complete the taskrobot first queries RoboBrain for labeling of objects in theenvironment For this it uses task specific semanticLabelingfunction described in Section VI-C

objects = semanticLabeling ldquoenvironmentpngrdquo

After object labels are obtained it locates the egg cartonand queries for its attributes which the RoboBrain returnsas fragile Finally it queries for the trajectory parameters offragile objects Below we show the RaQueL queries

attributes n = fetch (name n)rarr [lsquoHasAttributeprime]rarr (v)

trajectories a = fetch (handle a)rarr [lsquoHasTrajectoryprime]rarr (v)

trajectory parameters = map(λararr trajectories a) attributes lsquoeggprime

After getting the cost function parameters robot samplestrajectories and executes the top-ranked trajectory

Feedback to RoboBrain Sometimes the robot may not becertain about the optimal trajectory when it can ask the usersfor feedback This feedback is directly registered in RoboBrainand trajectory parameters are updated for future queries asshown in Figure 7 A user can provide various kinds oftrajectory feedback For example in Figure 7 user provides co-active feedback the robot displays top three ranked trajectoriesand user selects the best one [26] In another example Robo-Brain also improves with feedback provided physically on therobot In Figure 8 the user directly improves a trajectory bycorrecting one of the trajectory waypoints Such feedback canthus help improve all the robots that use RoboBrain

VIII DISCUSSION AND CONCLUSION

RoboBrain is a collaborative effort to create a knowledgeengine that acquires and integrates knowledge about thephysical world the robots live in from several sources withmultiple modalities We described some technical challengeswe have addressed such as the multi-modality of knowledgenever-ending learning large-scale nature of the system useof crowd-sourcing and allowing queries with RaQueL for

building applications with RoboBrain We have shown how touse RoboBrain in three different areas perception planningand natural language grounding We showed that sharingknowledge across different robotic tasks allows them to scaletheir learning to new situations

RoboBrain is an ongoing effort where we are constantlyimproving different aspects of the work We are improving thesystem architecture and expanding RaQueL to support scalingto even larger knowledge sources (eg millions of videos)Furthermore we have several ongoing research efforts thatinclude achieving better disambiguation and improving never-ending learning abilities More importantly we are constantlyexpanding the set of our RoboBrain research partners Thiswill not only improve the abilities of their robots but alsotheir contribution of knowledge to RoboBrain will help otherresearchers in the robotics community at large

ACKNOWLEDGMENTS

We thank Arzav Jain for his help in building the graphinfrastructure for the RoboBrain We thank Bart SelmanJitendra Malik and Ken Goldberg for their visionary ideasWe thank Emin Gun Sirer for help with databases and thankSilvio Savarese Stefanie Tellex Fei-Fei Li and ThorstenJoachims for useful discussions We also thank Yun Jiang IanLenz Jaeyong Sung Chenxia Wu and Ayush Dubey for theircontributions to the knowledge for the RoboBrain We alsothank Michela Meister Hope Casey-Allen and Gabriel Khofor their help and involvement with the RoboBrain project Weare also very thankful to Debarghya Das and Kevin Lee fortheir help with developing the front-end for the RoboBrain

This work was supported in part by Army Research Office(ARO) award W911NF-12-1-0267 Office of Naval Research(ONR) award N00014-14-1-0156 NSF National RoboticsInitiative (NRI) award IIS-1426744 Google Faculty Researchaward (to Saxena) and Qualcomm research award This wasalso supported in part by Google PhD Fellowship to Koppulaand by Microsoft Faculty Fellowship NSF CAREER Awardand Sloan Fellowship to Saxena

REFERENCES[1] R Alami A Clodic V Montreuil E A Sisbot and R Chatila Toward

human-aware robot task planning In AAAI Spring Symposium To BoldlyGo Where No Human-Robot Team Has Gone Before 2006

[2] A Anand H S Koppula T Joachims and A Saxena Contextuallyguided semantic labeling and search for 3d point clouds IJRR 2012

[3] R D Andrea Guest editorial A revolution in the warehouse Aretrospective on kiva systems and the grand challenges ahead IEEETran on Automation Science and Engineering (T-ASE) 9(4) 2012

[4] S Auer C Bizer G Kobilarov J Lehmann R Cyganiak and Z IvesDbpedia A nucleus for a web of open data Springer 2007

[5] M Beetz U Klank I Kresse A Maldonado L MosenlechnerD Pangercic T Ruhr and M Tenorth Robotic roommates makingpancakes In Humanoids 2011

[6] K Bollacker C Evans P Paritosh T Sturge and J Taylor Freebase acollaboratively created graph database for structuring human knowledgeIn Proc ACM SIGMOD pages 1247ndash1250 2008

[7] M Bollini S Tellex T Thompson M Roy and D Rus Interpretingand executing recipes with a cooking robot In ISER 2012

[8] W Burgard D Fox and S Thrun Probabilistic state estimationtechniques for autonomous and decision support systems InformatikSpektrum 34(5) 2011

[9] M Cakmak S S Srinivasa M K Lee J Forlizzi and SB KieslerHuman preferences for robot-human hand-over configurations In IROS2011

[10] A Carlson J Betteridge B Kisiel B Settles E R Hruschka Jrand T M Mitchell Toward an architecture for never-ending languagelearning In AAAI volume 5 page 3 2010

[11] X Chen A Shrivastava and A Gupta NEIL Extracting VisualKnowledge from Web Data In ICCV 2013

[12] V Delaitre D Fouhey I Laptev J Sivic A Gupta and A Efros Scenesemantics from long-term observation of people In Proc ECCV 2012

[13] J Deng W Dong R Socher L-J Li K Li and Li Fei-Fei ImagenetA large-scale hierarchical image database In CVPR 2009

[14] D Dolgov S Thrun M Montemerlo and J Diebel Path planning forautonomous vehicles in unknown semi-structured environments IJRR29(5) 2010

[15] XL Dong T Strohmann S Sun and W Zhang Knowledge vault Aweb-scale approach to probabilistic knowledge fusion In KDD 2014

[16] A Dragan and S Srinivasa Generating legible motion In RSS June2013

[17] S Edelkamp and P Kissmann Optimal symbolic planning with actioncosts and preferences In IJCAI 2009

[18] C Fellbaum WordNet Wiley Online Library 1998[19] D Ferrucci E Brown J Chu-Carroll J Fan D Gondek A Kalyanpur

A Lally J W Murdock E Nyberg J Prager et al Building watsonAn overview of the deepqa project AI magazine 31(3)59ndash79 2010

[20] D A Ferrucci Introduction to this is watson IBM J of RnD 56(34)1ndash1 2012

[21] M Gemici and A Saxena Learning haptic representation for manipu-lating deformable food objects In IROS 2014

[22] M J Gielniak C Karen Liu and A L Thomaz Generating human-likemotion for robots IJRR 32(11) 2013

[23] S Gupta R Girshick P Arbelaez and J Malik Learning rich featuresfrom RGB-D images for object detection and segmentation In ProcECCV 2014

[24] J Hoffart F M Suchanek K Berberich and G Weikum Yago2a spatially and temporally enhanced knowledge base from wikipediaArtificial Intelligence 19428ndash61 2013

[25] A Jain S Sharma and A Saxena Beyond geometric path planningLearning context-driven trajectory preferences via sub-optimal feedbackIn ISRR 2013

[26] A Jain B Wojcik T Joachims and A Saxena Learning trajectorypreferences for manipulators via iterative improvement In NIPS 2013

[27] A Jain D Das J K Gupta and A Saxena Planit A crowdsourcedapproach for learning to plan paths from large scale preference feedbackarXiv preprint arXiv14062616 2014

[28] Z Jia A Gallagher A Saxena and T Chen 3d reasoning from blocksto stability IEEE PAMI 2014

[29] Y Jiang H Koppula and A Saxena Hallucinated humans as the hiddencontext for labeling 3d scenes In CVPR 2013

[30] D Katz A Venkatraman M Kazemi J A Bagnell and A StentzPerceiving learning and exploiting object affordances for autonomouspile manipulation Autonomous Robots 37(4) 2014

[31] K Kitani B D Ziebart J A Bagnell and M Hebert Activityforecasting In Proc ECCV 2012

[32] R A Knepper T Layton J Romanishin and D Rus Ikeabot Anautonomous multi-robot coordinated furniture assembly system InICRA 2013

[33] H Koppula A Jain and A Saxena Anticipatory planning for human-robot teams ISER 2014

[34] HS Koppula and A Saxena Physically grounded spatio-temporalobject affordances In Proc ECCV 2013

[35] HS Koppula and A Saxena Anticipating human activities using objectaffordances for reactive robotic response In RSS 2013

[36] M Kuderer H Kretzschmar C Sprunk and W Burgard Feature-basedprediction of trajectories for socially compliant navigation In RSS 2012

[37] K Lai L Bo X Ren and D Fox A Large-Scale Hierarchical Multi-View RGB-D Object Dataset In ICRA 2011

[38] T Lan TC Chen and S Savarese A hierarchical representation forfuture action prediction In Proc ECCV 2014

[39] P A Lasota G F Rossano and J A Shah Toward safe close-proximityhuman-robot interaction with standard industrial robots In CASE 2014

[40] D B Lenat Cyc A large-scale investment in knowledge infrastructureCommunications of the ACM 38(11)33ndash38 1995

[41] I Lenz H Lee and A Saxena Deep learning for detecting roboticgrasps In RSS 2013

[42] J Mainprice and D Berenson Human-robot collaborative manipulationplanning using early prediction of human motion In IROS 2013

[43] C McManus B Upcroft and P Newmann Scene signatures Localisedand point-less features for localisation In RSS 2014

[44] DK Misra J Sung K Lee and A Saxena Tell me dave Context-sensitive grounding of natural language to mobile manipulation instruc-tions In RSS 2014

[45] A R Mohamed T N Sainath G Dahl B Ramabhadran G E Hintonand M A Picheny Deep belief networks using discriminative featuresfor phone recognition In (ICASSP) pages 5060ndash5063 2011

[46] T Naseer L Spinello W Burgard and C Stachniss Robust visualrobot localization across seasons using network flows In AAAI 2014

[47] neo4j Cypher lsquolsquohttpneo4jcomdocsstablecypher-query-langhtmlrdquo[48] S Nikolaidis P A Lasota G F Rossano C Martinez T A Fuhlbrigge

and J A Shah Human-robot collaboration in manufacturing Quanti-tative evaluation of predictable convergent joint action In ISR 2013

[49] M Quigley B Gerkey K Conley J Faust T Foote J Leibs E BergerR Wheeler and A Ng Ros an open-source robot operating systemIn ICRA Workshop on Open Source Software 2009

[50] N D Ratliff J A Bagnell and S S Srinivasa Imitation learning forlocomotion and manipulation In Int Conf on Humanoid Robots 2007

[51] J Rintanen Planning as satisfiability Heuristics Artificial Intelligence19345ndash86 2012

[52] A Saxena and AY Ng Learning sound location from a singlemicrophone In ICRA 2009

[53] J Schulman J Ho A X Lee I Awwal H Bradlow and P AbbeelFinding locally optimal collision-free trajectories with sequential convexoptimization In RSS 2013

[54] JM Shepard MC Towner J Lei and P Abbeel Cloth grasp pointdetection based on multiple-view geometric cues with application torobotic towel folding In ICRA 2010

[55] F M Suchanek G Kasneci and G Weikum Yago a core of semanticknowledge In WWW pages 697ndash706 2007

[56] S Tellex T Kollar S Dickerson M R Walter A G Banerjee S JTeller and N Roy Understanding natural language commands forrobotic navigation and mobile manipulation In AAAI 2011

[57] S Tellex R Knepper A Li D Rus and N Roy Asking for help usinginverse semantics In RSS 2014

[58] M Waibel M Beetz R DrsquoAndrea R Janssen M Tenorth J CiveraJ Elfring D Galvez-Lopez K Haussermann J Montiel A PerzyloB Schieszligle A Zweigle and R van de Molengraft Roboearth A worldwide web for robots IEEE Robotics amp Automation Magazine 2011

[59] J Walker A Gupta and M Hebert Patch to the future Unsupervisedvisual prediction In CVPR 2014

[60] C Wu I Lenz and A Saxena Hierarchical semantic labeling for task-relevant rgb-d perception In RSS 2014

[61] Y Zhu A Fathi and Li Fei-Fei Reasoning about object affordances ina knowledge base representation In Proc ECCV 2014

[62] M Zucker N D Ratliff A D Dragan M Pivtoraiko M K C MDellin J A Bagnell and S S Srinivasa CHOMP covariant hamilto-nian optimization for motion planning IJRR 32(9-10) 2013

Introduction

Related Work

Overview

Knowledge Engine Formal Definition

Creating the Graph

System Architecture

RoboBrain Query Library (RaQueL)

Graph retrieval function

Integrating Programming Constructs

Task Specific Functions

Applications

Grounding Natural Language

Anticipating Human actions

Planning in presence of humans

Discussion and Conclusion