Motion Planning for Robotic Manipulation of

Deformable Linear Objects (DLOs)

Mitul Saha and Pekka Isto

Research supported by NSF

Artificial Intelligence LabStanford University

Research Institute for TechnologyUniversity of Vaasa, Finland

• The ability to autonomously manipulate objects is one of most desirable features in a robot. Hence manipulation planning has been an active area of research for the last many decades

–it is difficult to model and predict the deforming nature of deformable objects–struggle in basic motion planning

• There has not been much development in manipulation planning for deformable objects because

Manipulation Planning Research so far…

• So far, manipulation planning research has mainly focused on manipulating rigid objects

• We have been interested in manipulation planning for deformable objects, because a large number of objects that we handle in our daily lives are deformable to some extent

• The ability to autonomously manipulate objects is one of most desirable features in a robot. Hence manipulation planning has been an active area of research for the last many decades

–it is difficult to model and predict the deforming nature of deformable objects–struggle in basic motion planning

• There has not been much development in manipulation planning for deformable objects because

Manipulation Planning Research so far…

• So far, manipulation planning research has mainly focused on manipulating rigid objects

• We have been interested in manipulation planning for deformable objects, because a large number of objects that we handle in our daily lives are deformable to some extent

• The ability to autonomously manipulate objects is one of most desirable features in a robot. Hence manipulation planning has been an active area of research for the last many decades

–it is difficult to model and predict the deforming nature of deformable objects–struggle in basic motion planning

• There has not been much development in manipulation planning for deformable objects because

Manipulation Planning Research so far…

• So far, manipulation planning research has mainly focused on manipulating rigid objects

• We have been interested in manipulation planning for deformable objects, because a large number of objects that we handle in our daily lives are deformable to some extent

• The ability to autonomously manipulate objects is one of most desirable features in a robot. Hence manipulation planning has been an active area of research for the last many decades

–it is difficult to model and predict the deforming nature of deformable objects–struggle in basic motion planning

• There has not been much development in manipulation planning for deformable objects because

Manipulation Planning Research so far…

• So far, manipulation planning research has mainly focused on manipulating rigid objects

• We have been interested in manipulation planning for deformable objects, because a large number of objects that we handle in our daily lives are deformable to some extent

Manipulation Planning for Deformable Linear Objects (DLOs)

GOAL: to develop a motion planner that would enable robots to autonomously manipulate Deformable Linear Objects (ropes, cables, sutures) in various settings.

bowline knot

figure-8 knot

sailing knot

autonomous robotic DLO manipulation

knot tying in daily/recreational life laying/loading cables in industrial settings

suturing in medical surgery

robotdress

Manipulation Planning for Deformable Linear Objects

(DLOs)

• The DLO manipulation problem is extremely challenging for robotics because

o being highly deformable, they can exhibit a much greater diversity of behaviors, which are hard to model and predict

o identifying topological states of DLOs is coupled with some unsolved problems in knot-theory/ mathematics

Interesting

Challenging

• The DLO manipulation problem has a nice structure. It brings together robotics, knot theory, and computational mechanics.

Previous Related Work

“Planning of One-Handed Knotting/Raveling Manipulation of Linear Objects”,IEEE ICRA 2004, Wakamatsu, et. al.

- knot simplified using Reidemeister moves (RM) from knot theory-one robot used to execute the RMs-assumes DLO resting on a plane

Previous Related Work

“Planning of One-Handed Knotting/Raveling Manipulation of Linear Objects”,IEEE ICRA 2004, Wakamatsu, et. al.

Our contribution:

-DLO need not be in a plane-We use more than one robot in coordination-We consider collision constraints (robot-DLO, robot-obstacle)-We consider the physical behavior of the DLO while planning-We consider interaction of the DLO with other objects

- knot simplified using Reidemeister moves (RM) from knot theory-one robot used to execute the RMs-assumes DLO resting on a plane

The Manipulation Problem

How do we define goal configurations?

available robot arms

• Goal configurations are defined in terms of topology instead of exact geometry

Geometrically differentbut

topologically same: Bowline knot

Defining Goal Configurations

while winding, number of wounds

more important

• In knot theory, crossing configuration of a curve is used to characterize its topology

Defining Goal Configurations

planar projection of the DLO central axis

• In knot theory, crossing configuration of a curve is used to characterize its topology

Crossing Configuration: (C1, C2, C3, C4): ((1,-6)-, (-2,5)-, (3,-8)-, (-4,7)-)

crossing:local self-intersections

Defining Goal Configurations

C1: (1,-6)-

C2: (-2,5)-

C3: (3,-8)-

C4: (-4,7)- sign of a crossing

planar projection of the DLO central axis

how to account forinteractions with other objects?

make them partthe DLO

semi-deformable linear object (sDLO)

We take as input the physical model of the DLO in the form of a state transition function f:

Physical modeling of the DLO

Suture model: [Brown, et al., 04] Elastic thread model: [Wang, et al., 05] Nylon thread model: [Dhanik, 05]

Recent successes in computational mechanics:

• Manipulation using 2 cooperating robot arms

Manipulation Tools

• Manipulation using 2 cooperating robot arms

• Use of static sliding supports (“tri-needles”) to provide structural support

Manipulation Tools

• Defining “Forming Sequence”

Forming Sequence: C2, C1, C4, C3

Basis of our Planning Approach

walk along the DLO;crossing “formed” when encountered the second time

• Defining “Forming Sequence”

Forming Sequence: C2, C1, C4, C3

Basis of our Planning Approach

walk along the DLO;crossing “formed” when encountered the second time

C2

C1C4

C3

A DLO topology or knot can be tied, crossing-by-crossing, in the order defined by its “forming sequence”

• Defining “Forming Sequence”

Forming Sequence: C2, C1, C4, C3

Basis of our Planning Approach

• Defining “loop hierarchy”used to determine the placementof

static sliding supports (“tri-needles”)

walk along the DLO;crossing “formed” when encountered the second time

C2

C1C4

C3

A DLO topology or knot can be tied, crossing-by-crossing, in the order defined by its “forming sequence”

Our Manipulation Planning Algorithm

-search the configuration-space using a sampling-based tree

-use forming sequence to bias search

-use physical model to sample new DLO shapes

-use the loop hierarchy to place static sliding supports (tri-needles)

search tree

forbidden region

Our Manipulation Planning Algorithm

-search the configuration-space using a sampling-based tree

-use forming sequence to bias search

-use physical model to sample new DLO shapes

-use the loop hierarchy to place static sliding supports (tri-needles)

search tree

forbidden region

grasping robot fails

Robot A

DLO

Robot A

Robot B

Our Manipulation Planning Algorithm

-search the configuration-space using a sampling-based tree

-use forming sequence to bias search

-use physical model to sample new DLO shapes

-use the loop hierarchy to place static sliding supports (tri-needles)

search tree

forbidden region

Our Manipulation Planning Algorithm

-search the configuration-space using a sampling-based tree

-use forming sequence to bias search

-use physical model to sample new DLO shapes

-use the loop hierarchy to place static sliding supports (tri-needles)

search tree

forbidden region

tri-needles

loop hierarchy

Results

bowline knot

sailing knot

bow

neck-tie

-Planner implemented in C++-Took 15-20 minutes on a 1GB, 1GHz processor to generate manipulation plans for tying popular knots: bowline, neck-tie, bow (shoe-lace), and stunsail-Videos: http://ai.stanford.edu/~mitul/dlo

Results

Results

neck-tie

In the real-life, we have tested the ability of the planner to generate robust plans by tying the popular Bowline knot with various household ropes on a hardware platform with two PUMA robots, using the manipulation plan generated by the planner.

Results

bowline knot

robustness dues to tri-needles

Conclusion

• We have developed a motion planner for manipulating deformable linear objects (such as ropes, cables, sutures) in 3D using cooperating robots.

- it can tie self-knots and knots around rigid objects - unlike in traditional motion planning, goals are topological and not geometric

- we account for the physical behavior of the DLO - it is robust to imperfections in the physical model of the DLO - it is first of its kind (we not aware of any other planner for computing collision-free

robot motions to manipulate a DLO in environments with obstacles) - the implemented planner has been tested both in graphic simulation and in real-life on a dual-PUMA-560 hardware platform

suturing in medical surgery collaboration with General Motors

• Future Plans

Motion Planning for Robotic Manipulation of Deformable Linear Objects (DLOs)

Acknowledgement: Advisory: Jean-Claude Latombe PUMA experiments: Oussama Khatib, Irena, Jaehueng Park, Jin SungPhysical models of ropes: Etienne Burdet, Wang Fei (EPFL)Useful comments: anonymous reviewers

- Tight knots - Semi-tight knots

We focus on two types of common knots:

Crossing Configuration:((1,-6)-, (-2,5)-, (3,-8)-, (-4,7)-)

over

under over

Needle Placement