Background subtraction Video imageBackground Human area -
Slide 8
Obtained 3D Sequence
Slide 9
Stick image unfortunately still, unstable Motion capture
system
Slide 10
Observation: Motion capture system Joint angles obtained
Theoretically, a robot can imitate the same dance???
Slide 11
But, . AIST dynamic simulator
Slide 12
Worse with steps!!!
Slide 13
Learning from observation Observation Performance
Relation-1Relation-2 Action Representation Not direct imitation
Top-down approach
Slide 14
Three issues Representation What does the dancer perform? How
is the dancer performing? Demonstration How does a robot perform
using his/her body?
Slide 15
History: learning from observation 1988
Slide 16
Learning from observation Top-down approach Ikeuchi, Reddy,
Tanguy 89
Slide 17
Object Recognition
Slide 18
Task Recognition
Slide 19
Relation transition = Task AB BA A B B A Ikeuchi, Reddy, Tanguy
89 Put A on top B Put A side of B
Slide 20
Later system 1988 1990 1995 2002
Slide 21
Possible contact relations in polyhedral world From Kuhn Tucker
Theory
Slide 22
Relation transition = Tasks
Slide 23
Task and skill parameters object start configuration object
approach configuration object approach direction gripper start
configuration gripper approach configuration gripper approach
direction 3d-s3d-a Move-to-contact Skill parameters
Slide 24
Observation
Slide 25
Real-time stereo hardware
Slide 26
Observation in CAD world
Slide 27
Task Recognition based on contact transition Make-contact in
translation Slide in translation
(20010000)(11010010)(20010020)(11010010)
(02010020)(20010021)(10100111)(01100121) Make-contact in
translation Make-contact in translation Slide in rotation
Slide 28
Demonstration Takamatsu, Kimura, Ikeuchi 2002
Slide 29
How about other domains? Task models in contact operation
Polyhedral objects Mechanical parts Flexible objects (Rope) Task
models in non-contact operation Hand motion Whole body with
dynamics
Slide 30
Task models primitive1primitive2 movement StateS1S2S3
Slide 31
Mathematical background How to describe a state of a knot What
kind of motion primitives to be used? 1. State: P-data 2. Motion
primitives: Reidemeister moves+ Cross Knot theory
Slide 32
P-data 1 2 3 4 5 6 4 5 6 1 2 3 over 1 under 2 over 3 under 4
value 3 1 2 4 2 1 1 2 3 4 5 6 sign vertical OOUUUO
Slide 33
Necessary & Sufficient From a line drawing of a rope, we
can obtain a unique P-data representation. [Inverse]: From a P-data
representation, we can reconstruct a line drawing of a rope.
Slide 34
Three Types of Reidemeister moves Reidemeister move
Reidemeister move Reidemeister move
Slide 35
Sufficiency to cover all possible moves Two equivalent knots
convert to each other by a finite number of Reidemeister moves
Proof provided by Reidemeister [Reidemeister 32]
Slide 36
Observation Observation Convert to P-Data Transition of P-Data
-> Reidemeister move
Slide 37
Task models One P-data transition One Reidemeister move
Dancing robot Through observing human dance Can a robot
dance
Slide 42
Task model design Foot supporting upper body Upper body
representing a dance
Slide 43
Foot Task models: what a human does? Left step Right step
standing Foot contact Squat Waist position
Slide 44
Recognizer Step contact states speed of foot Squat speed of
waist
Slide 45
Motion-capture dataResults:what a human does Recognition
results
Slide 46
Task: What a dancer does Skill: How a dancer is doing? Standing
Squat Step Period Foot width depth Foot width Highest point What
How From motion-capture data
Slide 47
Skill reconstruction Skill prototype Skill parameters from
observation New trajectory
Slide 48
Start point End point Generated trajectory Foot Width Highest
points Skill parameters observed
Slide 49
Foot trajectory Whole leg motion
Slide 50
Upper body tasks: Teacher s sketch on how to dance What is
this? How can we extract?
Slide 51
Key pose extraction Assumption brief stops of body parts z x y
Body centered coordinates time Vel. Brief stops
Slide 52
Key poses extracted (based on only motion) Segmentation based
on the assumption Over-segmentation New assumption rhythm + brief
stop
Slide 53
Rhythm analysis Estimated Beat Interval 0.704 [sec] Music with
inserted Beep = 84
Slide 54
Key poses extracted (rhythm and motion) R. Hand L. Foot R. Foot
L. Hand
Slide 55
Comparison Teachers key-poses from her sketch Extracted
key-poses from motion capture data
Slide 56
Interpolation Hierarchical B-spline Key point Teachers
motion
Slide 57
Adjustment of whole body ZMP = Zero Moment Point
Slide 58
ZMP control Calculate current ZMP Compare with desired ZMP
Adjust waist position to reduce the difference
Slide 59
Costarring with the dance teacher Nakaoka 2006, Shiratori 2007
With cooperation of AIST, Kawata
Slide 60
Beyond current dancing robot able to imitate dance motions
Need: Listening capability Self-dancing
Slide 61
Synthesis of New Dance from Music Motion reservoir: motion
segments with code and intensity Analyze current music: music code
and intensity Search motion segments to match music code and
intensity Connect segments seamlessly for generating a new
dance
Slide 62
Generated new dance (Shiratori 07) Input Music: Kansho Motion
reservoir: Six Japanese folk dances
Slide 63
Beyond current dancing robot able to imitate dance motions
Need: Listening capability Self-dancing Adjusting to music
Tempo
Slide 64
Motion difference due to tempi Faster tempo: detailed motions
omitted Green Original Tempo Yellow 1.3 faster (Synchronized)
Slide 65
Hierarchical B-spline Input t B-splin t B-spline t Half knot
intervals t Difference
Slide 66
Remove higher layers within speed limit constraint 1 2 3 4 w
limit
Slide 67
Generated 1.5 faster motion Human Generated Simple fast
forward
Slide 68
Beyond current dance robot able to mimic dance motions Need:
listening capability Need: ego-desire of a robot dance to perform
to improve cannot observe its own dance painting robot
Slide 69
Toward a robot painter System design Self-judgment on painting
results beautiful satisfy ugly dissatisfy
Structure of painting Representation for painting
Representation for painting Model acquisition Painting by a
robot
Slide 72
Observation 3D model of an apple
Slide 73
Why 3D model? Represent as we do (assumption) Obtain painting
features from the representation Arbitrary view Superimposed views
Abstract view
Slide 74
How to paint? Acquired model Representation Painting by a
robot
Slide 75
Arbitrary view
Slide 76
Contour features from an apple model 11 segments
Slide 77
How to paint? Acquired model Representation Painting by a robot
Do we need a robot body for painting?
Slide 78
Painting with a brush Grasp a brush Paint a line with the brush
Paint a line with the brush Verify painting results Verify painting
results Re-painting
Slide 79
Grasping a brush
Slide 80
Verifying the results Green OK Purple need re-painting
Slide 81
Re-painting through visual feedback
Slide 82
Painting an apple
Slide 83
Paintings by the robot Due to three- finger grasping
Slide 84
Future Plan: hierarchical painting
Slide 85
Future plan: motion representation Marcel Duchamp
Slide 86
Remaining Issues Self-judgment on painting results beautiful
satisfy ugly dissatisfy Robot s ego-desire to paint Mind and
desire
Slide 87
Science in dancing and painting Learning from observation Task
recognition what he/she does Skill recognition how he/she does Body
recognition: what and how to do Robot artist through
learning-from-observation embodying artist mind ???
Slide 88
Art and Science Middle ages: University = Art + Science 20 th
century Divorce between Art and Science 21 st century Let s
remarriage between art and science through Robot Artist based
Learning-from-observation paradigm
Slide 89
Misc Info Creation of 21 st Century Digital Art Under JST-CREST
program Web: http://www.cvl.iis.u-tokyo.ac.jp cvl: Computer Vision
Lab iis: Institute of Industrial Science u-tokyo: The University of
Tokyo
