Robotic Vision - Vision for Robotics #IEEE #QLD #CIS #Colloquium

Robotic Vision

Jürgen ‘Juxi’ Leitner arc centre of excellence for robotic vision

queensland university of technology

a vision for robotics

j.leitner@qut.edu.au - http://Juxi.net

http://roboticvision.org/2

2013

http://roboticvision.org/3

recognising  objects  &  stuff

recognising  places

detec4ng  mo4on move  to  see

see  to  move

context  for  seeing

seeing  for  context

seeing  creates  memories

memory  helps  seeing

paying  a;en4on

recognizing  humans,  their  ac4vi4es  and  intent

Seeing

http://roboticvision.org/4

10   96%eyedesigns

animalshave eyes

Nature Reviews | Neuroscience

Protostomes

Bilateria

Ecdysozoa

Lophotrochozoa

~580 Mya

~550 Mya

~530 Mya

~500 Mya

~430 Mya

Deuterostomes

Chordates*

Craniates*

Vertebrates*

Gnathostomes*

Arthropods

Annelids

Molluscs

Hemichordates

Echinoderms

Cephalochordates*

Tunicates*

Myxiniformes*

Petromyzoniformes*

Last fossil jawless fish

Stages of interest in vertebrate eye evolution Cambrian

Mya 600 550 500 450 400 0

2 1 3 4 5 6

Lampreys

Jawed vertebrates

Hagfish

Lancelets

Sea squirts

Ocellus

Eye patch

ProtostomeAn animal belonging to the protostome super-phylum, which is characterized by its members’ embryonic development, in which the first opening (the blastopore) becomes the mouth (protostome is Greek for ‘first mouth’). All protostomes are invertebrates.

DeuterostomeAn animal belonging to the deuterostome super-phylum of the animal kingdom, which is characterized by its members’ embryonic development, in which the first opening (the blastopore) becomes the anus (deuterostome is Greek for ‘second mouth’). In addition to the chordate phylum (which includeds vertebrates), the other two main phyla are the echinoderm phylum and the hemichordate phylum.

ChordateAn animal belonging to the chordate phylum, which comprises vertebrates, tunicates and cephalochordates. These animals are characterized by the presence of a notochord, a dorsal-nerve cord and pharyngeal slits or pouches.

AgnathanA jawless fish within the chordate phylum (agnatha is Greek for ‘no jaw’). The two extant groups are hagfish and lampreys.

GnathostomeThe jawed vertebrates (gnathostome is Greek for ‘jaw mouth’), comprising fish and tetrapods (including birds and mammals).

craniates, hagfish have the most basal body-plan. They possess neither jaws nor vertebrae and are therefore usually regarded not as vertebrates but rather as a sister group. The vertebrates comprise an early jaw-less (agnathan) division, of which the only living examples are lampreys, and a later jawed division, the gnathostomes, which includes fish and tetrapods.

Controversy has long surrounded the interrelation-ship between hagfish, lampreys and jawed vertebrates. BOX 1 summarizes current views, and in FIG. 1 we show hagfish diverging either before the divergence of lam-preys or else after lampreys separated from the line that would become the jawed vertebrates.

Not only has extensive gene duplication occurred throughout the evolution of animals22, but in addition it is widely accepted that two rounds of whole-genome duplication occurred early in vertebrate evolu-tion23–29; most likely, one duplication occurred before the agnathans split from the vertebrate line and one occurred after (FIG. 1; for reviews, see REFS 30–32). It is also clear that the vertebrate organizer, which deter-mines the body plan of developing embryos, arose in early chordates33–35. These genetic developments are likely to have been of crucial importance in early vertebrate evolution, but they are beyond the scope of this Review.

Figure 1 | The origin of vertebrates. The evolution of jawed vertebrates is illustrated against an approximate time-scale of millions of years ago (Mya). The taxa considered in this Review are indicated with an asterisk and are accompanied by schematics and diagrams of the ‘eye’ region. The earliest chordates, represented by extant cephalochordates and tunicates, are thought to have appeared around 550 Mya. Jawless craniates (agnathans) were present in the early Cambrian, by 525 Mya, and a time of 530 Mya has been indicated for their presumed first appearance. As elaborated on in BOX 1, there is considerable controversy as to whether myxiniformes (solely represented by extant hagfish) diverged before or after the separation of lampreys from jawed vertebrates (shown as dashed black and grey lines). Numerous lines of jawless fish evolved between 500 and 430 Mya ago, although none have survived to the present day. The first jawed vertebrate arose around 430 Mya, and this line is represented today by cartilagenous fish, bony fish and tetrapods. Six ‘stages of interest’ in vertebrate eye evolution correspond to the time intervals between the divergence of important surviving taxa. This diagram does not include the evolutionary changes that have occurred in the last 400 million years. The presented timeline is based primarily on evidence from the fossil record; see REFS 2,13,15,17,18,144,160–163. The schematics are modified, with permission, from REF. 11 (1996) Oxford University Press (lancelet, sea squirt, hagfish and lamprey) and REF. 164 (2004) Academic Press (jawed vertebrate). The eye images are reproduced, with permission, from the following references: lancelet, REF. 165 BIODIDAC (1996) University of California Museum of Paleontology; sea squirt, REF. 63 (2006) Blackwell Publishing; hagfish, REF. 166 (2006) Australian Museum. Lamprey and jawed vertebrate eye images are courtesy of G. Westhoff and S. P. Collin).

R E V I E W S

NATURE REVIEWS | NEUROSCIENCE VOLUME 8 | DECEMBER 2007 | 961

http://roboticvision.org/

http://roboticvision.org/6

tinyurl.com/QUTRobotics

roboticvision.org

http://roboticvision.org/7

http://roboticvision.org/

http://roboticvision.org/9

Juxi Leitner

h"p://Juxi.net

http://roboticvision.org/

Dalle Molle Institute for AI (IDSIA)

10

WorkJuxi

Leitner

PhD Informatics / Intelligent Systems

MSc Space Robotics & Automation

BSc Information & Software Engineering

Intelligent (Space) Robots European Space Agency (ESA)

Erasmus Intelligent Systems

Work (Humanoid) Robot VisionInstituto Superior Técnico (IST)

Mobility Intelligent Space Systems Laboratory

About Me

Work Robotic Vision and Actions

h"p://Juxi.net

Queensland University of Technology (QUT)

controlling spacecraft with

ANNs

[Leitner et al, iSAIRAS 2010]

position, velocity, orientation

thruster

[Leitner et al, iSAIRAS 2010]

trajectory

robustness

[Leitner et al, iSAIRAS 2010]

http://roboticvision.org/14

projectIM-CLeVeR

http://robotics.idsia.ch/im-clever/

vision  and  ac5ons

http://Juxi.net/projects

perceptionvisual

thanks to G. Metta and IIT for this picture

object  manipula5ontowards  learned

http://Juxi.net/projects

objectsdetecting

objectsdetecting

Harding, Leitner, Schmidhuber, 2013Leitner et al., ICDL 2012, IJARS 2012, BICA 2012, CEC 2013

embedding domain knowledge

+ min dilate avg INP INP INP OpenCV  func5ons

full  images

+ min dilate avg INP INP INP diff

using building blocksOpenCV

+ min dilate avg INP INP INP thresh+ min dilate avg INP INP INP blur+ min dilate avg INP INP INP normalize+ min dilate avg INP INP INP input

icVision

icImage* BlueCupFilter::runFilter() { icImage* node43 = InputImages[4]; icImage* node49 = node43->LocalAvg(15);

icImage* out = node49->threshold(81.532f); return out; }

framework

cartesian genetic programming

+ min dilate avg INP INP INP

Harding, Leitner, Schmidhuber, 2013Leitner et al., ICDL 2012, IJARS 2012, BICA 2012, CEC 2013

learningapproach

detection

icImage GreenTeaBoxDetector::runFilter() { icImage node0 = InputImages[6]; icImage node1 = InputImages[1]; icImage node2 = node0.absdiff(node1); icImage node5 = node2.SmoothBilateral(11); icImage node12 = InputImages[0]; icImage node16 = node12.Sqrt(); icImage node33 = node16.erode(6); icImage node34 = node33.log(); icImage node36 = node34.min(node5); icImage node49 = node36.Normalize();

//cleanup ... icImage out = node49.threshold(230.7218f); return out; }

detect

detect

detection

approachcgp

[Leitner et al, iSAIRAS 2012]

visualhand detection

[Leitner et al, CEC 2013]

handsdetecting

[Leitner et al, CEC 2013]

approachsupervised learning

BUT

clusteringfeature

saliencymap

Autonomous Approach

[Leitner et al, ICDL/EpiRob 2012]

presegmentation

resultscomparing

[Leitner et al, ICDL/EpiRob 2012]

features

CGP-­‐IP

towards  learning

http://Juxi.net/projects

object  manipula5on

learningspatial perception

trainingset

9DOF

iCubpositio

n in the frame

2/6 per eye

Carte

sian

Coor

dinate

s

transferringspatial perception

[Leitner et al, IROS 2012]

setuplearning

trainingset

9DOF

iCubpositio

n in the frame

2/6 per eye

Carte

sian

Coor

dinate

s

.

.

.~1000

spatial perception neural network

...

9DO

FiC

ubpo

sitio

n in

the

imag

e2/

6 pe

r eye

Cart

esian

Coor

dina

tes

fully

con

nect

ed

fully

con

nect

ed

...

results ANN

-700

-600

-500

-400

-300

-200

-100

0

100

200

300

400

0 200 400 600 800 1000

Y (

mm

)

Sample Index

PredictedExpected

100

150

200

250

300

350

400

450

500

550

600

0 200 400 600 800 1000

X (

mm

)

Sample Index

-1300

-1200

-1100

-1000

-900

-800

-700

-600

-500

-400

-300

-200

0 200 400 600 800 1000

Z (

mm

)

Sample Index

-700

-600

-500

-400

-300

-200

-100

0

100

200

300

400

0 200 400 600 800 1000

Y (

mm

)

Sample Index

PredictedExpected

100

150

200

250

300

350

400

450

500

550

600

0 200 400 600 800 1000

X (

mm

)

Sample Index

-1300

-1200

-1100

-1000

-900

-800

-700

-600

-500

-400

-300

-200

0 200 400 600 800 1000

Z (

mm

)

Sample Index

[Leitner et al, IJCNN 2013]

towards  learning

http://Juxi.net/projects

object  manipula5on

MoBeEv2[Frank, Leitner et al. 2012, 2013]

[Frank, Leitner et al., ICINCO, 2012]

MoBeEframework [Frank, Leitner et al., ICINCO, 2012]

hand/armop-space forcing

CSWorld

CSHand

CSR/CSL

[Leitner et al, in prep]

teleoperation

motion

robot[Stollenga, Leitner et al, IROS 2013]

generationmotion

Shak

ey 2

013

Win

ner

towards  learning

http://Juxi.net/projects

object  manipula5on

coordinationhand-eye

model

http://Juxi.net/projects

manipulation for improved perception

http://roboticvision.org/57

case #1 learning

from supervised to robotic-assisted unsupervised learning

Autonomous Learning Of Robust Visual Object Detection And Identification On A Humanoid. J. Leitner, P. Chandrashekhariah, S. Harding, M. Frank, G. Spina, A. Förster, J. Triesch, J. Schmidhuber. ICDL/EpiRob 2012.

http://roboticvision.org/58

case : poking

segmentation before & after action

http://roboticvision.org/59

deep learning visual control[Zhang et al, ACRA 2015/ICRA2016]

http://roboticvision.org/60

deep learning visual control[Zhang et al, ACRA 2015/ICRA2016]

http://roboticvision.org/61

conclusions

a novel way of object segmentationlearning and teaching perceptionintegration action-perception side

reactive reaching/graspingimproving perception with (inter-)actions

learning neuro-controllers

http://roboticvision.org/

Australian Moonshot

62

tinyurl.com/QUTLunaRoo

http://roboticvision.org/

Australian Deep Learning WS

63

http://Juxi.net/workshop/deeplearning-applications-vision-robotics-2015/

for listeningthank youj.leitner@roboticvision.org http://Juxi.net/projects