Technical Cognitive Systems - ias.cs.tum.eduDepartment of Informatics Intelligent Autonomous Systems Technische Universität München Example: Cracking an egg \A cook is cracking a

$Page 1: Technical Cognitive Systems - ias.cs.tum.eduDepartment of Informatics Intelligent Autonomous Systems Technische Universität München Example: Cracking an egg \A cook is cracking a$
Department of InformaticsIntelligent Autonomous Systems Technische Universität München

Technical Cognitive Systems

Michael Beetz

Intelligent Autonomous SystemsTechnische Universitat Munchen

Summer Term 2012

Part X

Reasoning


Outline

Motion Planning

Motion Planning

Michael BeetzSummer Term 2012

Technical Cognitive Systems204


Outline

Motion Planning

Motion Planning




Making pancakes from web instructions

From web-instructions:

• Mix flour, sugar, egg and milk together in a largebowl

• Pour mixture into frying pan

• Flip your pancake using the long handled spatula

Challenge: how to infer missing knowledge?

• underspecified actions like mixing, pouring, andflipping?

• where to stand?

• where to put objects?

• . . .

Motion Planning




Outline

Motion Planning

Motion Planning




Example: Cracking an egg

“A cook is cracking a raw egg against a glass bowl. Properly performed, theimpact of the egg against the edge of the bowl will crack the eggshell in half.Holding the egg over the bowl, the cook will then separate the two halves ofthe shell with his fingers, enlarging the crack, and the contents of the eggwill fall gently into the bowl. The end result is that the entire contents ofthe egg will be in the bowl, with the yolk unbroken, and that the two halvesof the shell are held in the cook’s fingers.” (Common Sense Problem Page)

Variants: What happens if. . .

• the cook brings the egg to impact very quickly? Very slowly?

• the bowl is made of looseleaf paper? of soft clay?

• the bowl is smaller than the egg? bowl is upside down?

• the cook tries this procedure with a hard-boiled egg? a coconut?

Motion Planning




Example: Cracking an egg

“A cook is cracking a raw egg against a glass bowl. Properly performed, theimpact of the egg against the edge of the bowl will crack the eggshell in half.Holding the egg over the bowl, the cook will then separate the two halves ofthe shell with his fingers, enlarging the crack, and the contents of the eggwill fall gently into the bowl. The end result is that the entire contents ofthe egg will be in the bowl, with the yolk unbroken, and that the two halvesof the shell are held in the cook’s fingers.” (Common Sense Problem Page)

Variants: What happens if. . .

• the cook brings the egg to impact very quickly? Very slowly?

• the bowl is made of looseleaf paper? of soft clay?

• the bowl is smaller than the egg? bowl is upside down?

• the cook tries this procedure with a hard-boiled egg? a coconut?

Motion Planning




Logical Axiomatization by (Morgenstern 2001)

Motion Planning





Motion Planning





Theory is based on the event calculus (Kowalski 86)

courtesy of (Shanahan 99)

Motion Planning





Axiom ECF1 (InitiallyP(f) ∧¬Clipped(0,f,t2)) ∨(Happens(a,t1) ∧Initiates(a,f,t1) ∧t1 ¡t2 ∧¬Clipped(t1,f,t2)) ⇒Holds(f,t2)Axiom ECF2 (InitiallyN(f) ∧¬Declipped(0,f,t2)) ∨(Happens(a,t1) ∧Terminates(a,f,t1)∧t1 ¡ t2 ∧¬Declipped(t1,f,t2)) ⇒¬Holds(f,t2)Axiom ECF3 Clipped(t1,f,t2)⇔∃a,t (Happens(a,t) ∧(Terminates(a,f,t) ∨Releases(a,f,t))∧t1 ¡ t ∧t ¡ t2)Axiom ECF4 Declipped(t1,f,t2)⇔∃a,t (Happens(a,t) ∧(Initiates(a,f,t) ∨Releases(a,f,t))∧t1 ¡ t ∧t ¡ t2)

Liquid and Solid Objects, Materials:Axiom 1 Holds(Solid(o), t) ∨ Holds(Liquid(o), t)Axiom 2 (Mutual Exclusivity of Material Properties) Hard(m) ∨Soft(m) ∨ Fragile(m)∨Hard-to-Break(m) ∨Unbreakable(m)Axiom 3 Unbreakable(m1) ∧Hard-to-Break(m2) ⇒ HardertoBreak(m1,m2)Hard-to-Break(m1) ∧ Fragile(m2) ⇒ HardertoBreak(m1,m2)Axiom 4 HardertoBreak(m1,m2) ∧ HardertoBreak(m2,m3) ⇒ HardertoBreak(m1,m3)Axiom 5 Glass(m) ⇒ Hard-to-Break(m)Axiom 6 Glass(m) ⇒ Hard(m)

Containers:Definition 1 ClosedContainerShape(r) is true iff r is an ordinary point set and thecomplement of r has exactly two connected components.Definition 2 insideof(r) is the closure of the finite component of the complement of r.Definition 3 outsideof(r) is the closure of the infinite component of the complement of r.Definition 4 Holds(ClosedContainer(c),t) ⇔(Holds(shape(c)=r,t) ∧ ClosedContainer-Shape(r))

. . .

Motion Planning





In total, the theory includes > 60 axioms, > 20 definitions and lemmas.

Example:

Axiom 48: Holds(HardertoBreak(MaterialOf(o2),MaterialOf(o1)),t) ∧Holds(Hard(MaterialOf(o2)),t) ∧ Holds(Hard(MaterialOf(o1)),t) ∧Holds(ObjPart(pt1,o1),t) ∧ Holds(ObjPart(pt2,o2),t) ⇒Initiates(Hit(pt1,pt2,crackforce(o1,pt1,pt2)),CrackedContainer(o1),t)

Motion Planning





• Simplifications:

– ignore role and position of hands– shape of the crack is ignored– a container does not separate into two (or more) fragments– only 180-degree rotations are considered– . . .

• Variants the theory cannot handle:

– cooks brings the egg to impact very quickly/slow– bowl is upside down– cook uses a hard-boiled-egg, a coconut, or M&M– cook puts egg in bowl and exerts steady pressure with his hand– . . .

Important for robot manipulation: abstraction level, variants,non-deterministic and concurrent actions (not isolated problems)

Motion Planning










Motion Planning










Motion Planning




Animation Movies / Computer Games

Motion Planning




Physics Engines (ODE, Bullet, . . . )

Physics engines don’t have problems with:

• abstraction level (fine level of granularity)

• variants (alternative configurations)

• non-deterministic actions (noise)

• concurrent actions

Example: Dropping an Egg from 0.5 meters

Falling egg Hitting ground Breaking of joints Broken egg

time

Motion Planning




Outline

Motion Planning

Motion Planning




Robots Learning from InstructionsMaking a pancake

From the web-instructions:

• Mix flour, sugar, egg and milktogether in a large bowl


• Flip your pancake using the longhandled spatula

Challenge:

How to infer missing knowledge forunderspecified actions like mixing,pouring, and flipping?

Motion Planning




Robots Learning from InstructionsMaking a pancake

From the web-instructions:

• Mix flour, sugar, egg and milktogether in a large bowl


• Flip your pancake using the longhandled spatula

Challenge:

How to infer missing knowledge forunderspecified actions like mixing,pouring, and flipping?

Motion Planning




Bridging the Knowledge Gapbetween abstract instructions and robot control parameters

Learning from observations:

• Problem: almost only successful and goodperformances

; How can we derive flexible, robust, and failure-aware robot controlprograms?

Claim

Robots need an understanding of Naive Physics in order to choose controlparameters that lead to desired effects

Motion Planning








Claim

Robots need an understanding of Naive Physics in order to choose controlparameters that lead to desired effects

Motion Planning








Claim

Robots need an understanding of Naive Physics in order to choose controlparameters that lead to desired effectsMotion Planning




Proposed ApproachBasic idea

worldsituation

simulationperceptionlogging

logicalaxiomatization

axiomatization+ timelines

proof

worldsituation′

translatesparametrizes

represents/formalizes

naive physics inference(including mental

simulation)

answers

deduces

???

Motion Planning




Framework Overview

parametrizes

translates Logged Simulations

monitors

Augmented Simulation

breaking

detailed 3D models

+controllers

+monitors

rigid body dynamicsfor everyday objects

absorbing

Simulation (partial)

PROLOG

Timeline

Representations

Domain

Knowledge

Description LogicOWL

Fact-basePROLOG

First-order Representations Subsymbolic Representations

Motion Planning




Domain KnowledgePhysical aspects about Eggs

Egg

Liquid Solid Container

EggWhiteYolk EggShell

Viscosity Mass Extensions Rigidity Fragility Shape

hasPart hasPart

isType isType isType

hasProp hasProp hasProp hasProp hasProp hasProp hasProp

Motion Planning




Physics-based SimulationModelling Shape and Properties of an Egg

Body id: #42x: -0.082 y: 0.297 z: 0.265type: {solid, eggshell} mass: 0.1

Joint id: #88body1: #42 body2: #18x: -0.123 y: 0.322 z: 0.199roll: 0.0 pitch: -54.092 yaw: -31.626type: {solid, eggshell} forcelimit: 150 broken: false

Body id: #71x: -0.053 y: 0.305 z: 0.082type: {liquid, egg-yolk} mass: 0.1

Object modelling similar to work by Johnston and Williams [?]

Motion Planning




Augmenting Simulationwith Everyday Phenomena

We need more than rigid body dynamics for:

• breaking

• absorbing liquids

• melting

Augmenting object descriptions with:

• graph-based shape models

• controllers that compute physical effects

– perform computation in each simulation step– e.g.: breaking, absorbing, etc

• monitors that log physical properties:

– position, velocity, acceleration, forces, contact points, etc

Motion Planning




Translating Logged Simulationsinto Timelines

Fluents and events at points in time / time intervals

• Holds(f,t) / Holds-tt(f,[t1,t2])

• Occurs(ev,t) / Occurs-tt(f,[t1,t2])

How predicates are grounded in simulations:

• Holds(supports(o1,o2),t) ⇔Holds(Contacts(o1,o2), t) ∧p1 = SimulatorValueAt(Pose(o1),t) ∧p2 = SimulatorValueAt(Pose(o2),t) ∧isDirectlyBelow(p1,p2) ∧gravityForceIsCancelledOut(o2)

Motion Planning




Predicates Grounded in Simulations

Static configurations:

fluent intuitive descriptioncontacts(o1, o2) object o1 and object o2 contact each otherattached(o1, o2) object o2 is attached to object o1

supports(o1, o2) object o1 supports object o2

contains(o1, o2) container o1 contains object (or stuff) o2

broken(o1) object o1 is brokenspilled(o1) object o1 is spilled

Events:

events intuitive descriptioncolliding(o1, o2) object o1 and object o2 are collidingfalling(o1) object o1 is fallingmoving(o1) object o1 is movingopeningGripper(o1) robot is opening gripper o1

closingGripper(o1) robot is closing gripper o1

breaking(o1) object o1 is breakingspilling(o1) object o1 is spilling over a surface

Motion Planning




Temporal Projection Language

Asserting a scenario:

?- create scenario(egg scenario),

add to scenario(egg scenario,kitchen env),add to scenario(egg scenario,pr2 robot),add to scenario(egg scenario,egg1).

Motion Planning






?- create scenario(egg scenario),add to scenario(egg scenario,kitchen env),

add to scenario(egg scenario,pr2 robot),add to scenario(egg scenario,egg1).

Motion Planning






?- create scenario(egg scenario),add to scenario(egg scenario,kitchen env),add to scenario(egg scenario,pr2 robot),

add to scenario(egg scenario,egg1).

Motion Planning






?- create scenario(egg scenario),add to scenario(egg scenario,kitchen env),add to scenario(egg scenario,pr2 robot),add to scenario(egg scenario,egg1).

Motion Planning




Temporal Projection Language (cont’d)

Making temporal projections:

?- make projection(egg scenario,grasp obj,[(P1, force, [1, 3, 5, 7, 9, 11]),(P2, position, [-0.02, 0.00, 0.02])],

Timeline),not(occurs(break, Time, Timeline)),not(occurs(slip, Time, Timeline)).

P1 = 5,P2 = 0.00

Motion Planning




Temporal Projection Process

• make projection:sample parameters

• simulate:setup simulatorrun simulation

• translate:ground predicates inlogged simulations

• evaluate:events/fluentsspecialized predicates

make_projection(scenario,pogram,params,Timeline)

occurs(event,Time,Timeline)

egg slips

away

egg is hold

in gripper

egg is

broken

simulate(scenario,program,params,Log)

translate(Log,Timeline)

occurs(moving(egg), 12)

occurs(moving(egg), 13).


occurs(falling(egg), 15).




occurs(moving(egg), 19)..


























Motion Planning




Example: Pouring pancake mix

• Parameters: position, time, angle• Outcomes: number of particles on pan (spilled on table)

• Specialized predicates on particle sets: round/centered

Motion Planning




Example: Flipping a pancake

• Parameters: angle of spatula• Outcomes: turned, not turned

• Common failures:

break push off fold stick onMotion Planning




Summary

We propose simulation-based temporal projection:

worldsituation

simulationperceptionlogging

logicalaxiomatization

axiomatization+ timelines

proof

worldsituation′

translatesparametrizes

represents/formalizes

naive physics inference(including mental

simulation)

answers

deduces

• Framework for making predictions about manipulation actions

• Temporal projection language, e.g. make projection

• Extended rigid-body simulation with everyday phenomena

• Grounding predicates in data structures of simulation

Motion Planning




Outline

Motion Planning

Motion Planning




Motivation: Put the pancake mix on the counter

Whereshould Iput it???

Motion Planning




Motivation

• Infer plan parameters at run-time.

• Integrate state of the current world.

• Don’t abstract away from important geometric properties.

• Generation of parameters under geometric constraints vs. verification.

Motion Planning




Plan parameterizations: Designators

• Symbolic descriptions of Objects, Locations and Actions

• Examples:

– (a location (on table) (reachable-by Rosie))– (an object (type cup))– (an action (to reach) (the object (name Cup1)))

• Example plan:

(with-designators

((cup-loc (a location (on counter-top)))

(cup (an object (type cup) (at cup-loc)))

(pick-up-loc (a location (to reach) (obj cup))))

(at-location pick-up-loc

(achieve (obj-grasped cup))))

Motion Planning




Example: Location to put-down pancake mix

(a location(for pancake-mix)(on counter)(reachable-for Rosie)(visible-for James)(not-hindering (the activity (type pancake-making))))

Concepts

• Generative model

• Stability reasoning

• Reachability reasoning

• Visibility reasoning and perspective taking

• Light-weight temporal projection

Motion Planning






Concepts






Motion Planning






Concepts






Motion Planning






Concepts






Motion Planning






Concepts






Motion Planning




First-order representation of Designators


⇓

posesOn(Counter, PancakeMix, ?Poses) ∧ member(?P, ?Poses)∧ assertPose(PancakeMix, ?P) ∧ stable(PancakeMix)∧ reachable(Rosie, PancakeMix) ∧ blockingObjects(Rosie, PancakeMix, ∅)∧ visible(James, PancakeMix)∧ projectPlan(MakePancakes, ?Tl) ∧ bagof(?F, flawsInTimeline(?Tl, ?F), ∅)

Motion Planning






⇓


Motion Planning






⇓


Motion Planning






⇓


Motion Planning






⇓


Motion Planning




Inference algorithm

posesOn(Counter, Cup, ?Poses) ∧ member(?P, ?Poses)∧ assertPose(Cup, ?P) ∧ stable(Cup)

1. posesOn(Counter, Cup, ?Poses)

2. member(?P, ?Poses)

3. assertPose(Cup, ?P)

4. stable(Cup)

Motion Planning




Inference algorithm





4. stable(Cup)

Create distribution for sampling poses

Motion Planning




Inference algorithm





4. stable(Cup)

Draw a pose sample

Motion Planning




Inference algorithm





4. stable(Cup)

Place the mug

Motion Planning




Inference algorithm





4. stable(Cup)

Simulate for 5s, fail!

Motion Planning




Inference algorithm





4. stable(Cup)

Backtrack, draw another pose sample

Motion Planning




Inference algorithm





4. stable(Cup)

Place the mug

Motion Planning




Inference algorithm





4. stable(Cup)

Simulate for 5s, succeed!

Motion Planning




Built-in Predicates of CRAM Reasoning

Stabilitycontact(O1, O2) Contact between objectsstable(O) Stability of object

Visibilityvisible(P, O) Object visible from pose Poccluding(P, O1, O2) Object O2 occludes object O1

Reachabilityreachable(R, O) Object O is reachable by robot RblockingObjects(R, O, B) B is the list of objects that “block” O

Motion Planning








Stability reasoning

• Bullet physics engine

• Infer collisions and stability of scenes

• Simulate for a short period of time (5s)

• Example: mugs can be placed on plates but not the other way around

5s

5s

Motion Planning








Visibility reasoning

• OpenGL to infer visibility of objects

• Render scene from camera frame, each object in a different color and countpixels

• Infer visibility of objects and occluding objects

See also “Geometric Tools for Perspective Taking for Human-Robot Interaction” [Marin-Urias 2008]

Motion Planning








Reachability

• Use inverse kinematics to infer reachability

• No grasp planning, reach for object center

• Blocking objects = objects in collision with IK solution

Motion Planning




Examples

Find a pose for the cup...

... on the counter.

posesOn(Counter, Cup, ?Poses) ∧ member(?P, ?Poses)

∧ assertPose(Cup, ?P) ∧ visible(Pr2, Cup)∧ reachable(Pr2, Cup)∧ blockingObjects(Rosie, PancakeMix, ∅)

Motion Planning




Examples


... on the counter that is visible for the robot.

posesOn(Counter, Cup, ?Poses) ∧ member(?P, ?Poses)∧ assertPose(Cup, ?P) ∧ visible(Pr2, Cup)

∧ reachable(Pr2, Cup)∧ blockingObjects(Rosie, PancakeMix, ∅)

Motion Planning




Examples


... on the counter that is visible and reachable for the robot.

posesOn(Counter, Cup, ?Poses) ∧ member(?P, ?Poses)∧ assertPose(Cup, ?P) ∧ visible(Pr2, Cup)∧ reachable(Pr2, Cup)

∧ blockingObjects(Rosie, PancakeMix, ∅)

Motion Planning




Examples


... on the counter that is visible and reachable for the robot and there are noblocking objects.

posesOn(Counter, Cup, ?Poses) ∧ member(?P, ?Poses)∧ assertPose(Cup, ?P) ∧ visible(Pr2, Cup)∧ reachable(Pr2, Cup)∧ blockingObjects(Rosie, PancakeMix, ∅)

Motion Planning




Temporal projection

(a location ...(not-hindering (the activity (type (pick-up Cup1)))))

⇓

projectPlan(PickUp(Cup1), ?Tl) ∧ bagof(?F, flawsInTimeline(?F), ∅)

1. Execute plan in projection mode

2. Projection generates a timeline

3. Match pre-defined flaws on the timeline

Motion Planning




Temporal projection

(a location ...(not-hindering (the activity (type (pick-up Cup1)))))

⇓

projectPlan(PickUp(Cup1), ?Tl) ∧ bagof(?F, flawsInTimeline(?F), ∅)

1. Execute plan in projection mode

2. Projection generates a timeline

3. Match pre-defined flaws on the timeline

Motion Planning




Reasoning based on Execution Traces

• Why did you leave the cup on thetable while clearing it?

• Where did you stand whileperforming a task?

• What did you see?

• How did you move?

• How did you move the arm whilegrasping the bottle?

• . . .

Motion Planning




Our approach

1. Record execution trace

– Belief state– State of plan execution, tasks, activation, deactivation, results

2. Provide an interface to the execution trace through a first-orderrepresentation

– Symbolic annotations of plans– Causal relations through plan hierarchy– Symbolic representation of objects in plans

Motion Planning




Our approach





Motion Planning




Our approach





Motion Planning




System Overview

Logging

Beliefstate

SensorsTaskTree

...

......

Interpreter Plan Actuators

Plan Execution

Motion Planning




Plan projection and Timeline generation

Achieve(ObjInHand(Cup1))

Perceive(Cup1)

......

AtLocation(ReachLoc)

ObjectGrasped(Cup1) ObjectLifted(Cup1)

t

Motion Planning






Perceive(Cup1)

......



t

Motion Planning






Perceive(Cup1)

......



t

Motion Planning






Perceive(Cup1)

......



t

Motion Planning






Perceive(Cup1)

......



t

Motion Planning




Concepts for Reasoning

To reason about plans and their effects, we define

• Events

• Occasions

• Plan representation

• Intentions and the purpose of tasks.

• Helper predicates to access the execution trace.

Motion Planning




System Design

High LevelPlans

Belief stateOccasions

Perception Navigation ManipulationProcessModules

Designators

Events

Motion Planning




Occasions and Events

• Events

– Occur at points in time.– We assume that they are caused by actions of the robot.– Describe changes in the (believed) world state.– Examples: PickUp(obj), PerceiveObject(obj)

⇒ Occurs(ev , t)

• Occasions

– Describe states.– Can hold at intervals or points in time:

At(t), During(t1, t2), Throughout(t1, t2)– Inferred based on events– Examples: Loc(obj, loc), ObjectInHand(obj)

⇒ Holds(occ, t)

Motion Planning





• Events


⇒ Occurs(ev , t)

• Occasions



⇒ Holds(occ, t)

LocChange(obj) An object changed its locationLocChange(Robot) The robot changed its locationCollision(obj1, obj2) obj1 and obj2 started collidingCollisionEnd(obj1, obj2) obj1 and obj2 stopped collidingPickUp(obj) obj has been picked upPutDown(obj) obj has been put downObjectPerceived(obj) The object has been perceived

Motion Planning





• Events


⇒ Occurs(ev , t)

• Occasions



⇒ Holds(occ, t)

Motion Planning





• Events


⇒ Occurs(ev , t)

• Occasions



⇒ Holds(occ, t)

Loc(obj, loc) The location of an objectLoc(Robot, loc) The location of the robotObjectVisible(obj) The object is visible to the robotObjectInHand(obj) The object is carried by the robotObjectPlacedAt(obj, loc) The object has been placed at locationTaskStatus(task, status) The status of a task

Motion Planning




Plan Representation

Achieve(Loc(Obj, Loc))

Perceive(Loc(Obj, Loc))

Achieve(ObjInHand(Obj))

......

Achieve(ObjPlacedAt(Obj, Loc))

......

...

• Semantic annotation of sub-plans: Achieve(Loc(Obj, Loc))• Representation of objects, locations and action parametrized by designators

table = LocationDesig(On(Table))mug = ObjectDesig(Type(Mug), Color(White), At(table))grasp = TrajectoryDesig(To(Grasp), Obj(mug))

Motion Planning




Intentions

Achieve(Loc(bottle, table))

Achieve(ObjectOpened(fridge))

Achieve(ObjInHand(bottle))

Achieve(ObjectClosed(fridge))

Achieve(ObjPlacedAt(bottle, table))

• Plan interpretation creates a task tree.

• The intention of a task is to achieve a state in the world, i.e. an occasion.

• Intention of task tsk is to achieve ObjectInHand(Obj)⇔ TaskGoal(tsk,Achieve(ObjectInHand(Obj)))

Motion Planning




Examples (1)

Which object did you see in the drawer?

Query for the object designator at the end of a perceive goal.

Task(tsk) ∧ TaskGoal(tsk, Perceive(obj))∧ TaskEnd(tsk, t)∧ DesignatorTypeAt(obj, type, t)⇒ type = Plate

Motion Planning




Examples (1)

Which object did you see in the drawer?

Query for the object designator at the end of a perceive goal.

Task(tsk) ∧ TaskGoal(tsk, Perceive(obj))∧ TaskEnd(tsk, t)∧ DesignatorTypeAt(obj, type, t)⇒ type = Plate

Motion Planning




Examples (2)

Are there unachieved plans?

Query for top-level plans where occasions do not hold at the end.

UnachievedGoalError(o) ⇔TopLevel(tt) ∧ Task(tsk)∧ Subtask(tt, tsk) ∧ TaskGoal(tsk, o)∧ TaskEnd(tt, t) ∧ ¬Holds(o, t)

Motion Planning




Examples (2)

Are there unachieved plans?

Query for top-level plans where occasions do not hold at the end.

UnachievedGoalError(o) ⇔TopLevel(tt) ∧ Task(tsk)∧ Subtask(tt, tsk) ∧ TaskGoal(tsk, o)∧ TaskEnd(tt, t) ∧ ¬Holds(o, t)

Motion Planning




Questions?

Motion Planning



Documents

Technical Cognitive Systems - ias.cs.tum.eduDepartment of Informatics Intelligent Autonomous Systems Technische Universität München Example: Cracking an egg \A cook is cracking a