1

The coevolution of robot controllers (”brains”) and morphologies (”bodies”) – challenges and opportunities

Stefano Nichele

February 18, 2015

2

Outline •  Evolutionary Robotics

–  The basics –  Why Evolutionary Robotics?

•  Evolution of Controllers •  Co-evolution body/brain

–  Why co-evolve? –  How co-evolve? –  State of the art

•  (Major) Challenges and Opportunities –  Simulation vs Real –  Making it work! –  Artificial Life approach

3

Evolutionary Robotics

4

Evolutionary Robotics

”Evolutionary robotics is a new technique for automatic creation of autonomous robots.”

S. Nolfi and D. Floreano (2000). Evolutionary Robotics: The Biology, Intelligence and Technology of Self-Organizing Machines. The MIT Press

•  Seminal book on the subject •  EvoRobotics ~ 20 years old

5

•  Yet, surprisingly the number of real, physical robots with evolved bodies can be counted on the fingers of one hand

•  95+% [1]: evolution of robot brains (only the controllers)

•  4-%: evolution of robot morphologies in simulation (co-evolution body/brain)

•  <1%: co-evolution of body/brain in real robots

[1] Stefano Nichele, rough estimate.

6

Why Evolutionary Robotics?

7

Because...

•  More complex (dynamic) environments •  More complex robots (morphologies and brains) •  Difficult to design/program (top-down vs bottom-up)

•  Suggest/test hypothesis on behavior and cognition •  Help understand emergence of life/intelligence

8

Why is it difficult?

•  Behavioral physical systems are difficult to design –  Much more than computer programs, they also depend on

interactions with the environment §  Dynamic, unpredictable §  Not fully accessible to robots

•  Robot + environment = dynamic system –  Robot’s sensory state is function of environment and its own

actions –  Not easy to know a priori what internal mechanism results in which

behavior (or the other way around)

9

Evolution of controllers

10

Evolution of robot controllers

M. Waibel, D. Floreano, L. Keller (2011). A Quantitative Teat of Hamilton’s Rule for the Evolution of Altruism. PLoS Biol 9(5): e1000615

What is evolved:

•  ANN weights •  Connections •  Learning rules •  Nodes •  Archietectures •  ...

Nolfi & Floreano, 2004 Evolutionary Robotics

11

Evolutionary substrate •  Evolve network controllers in simulation (artificial agents) •  Co-Evolve morphology and control in simulation

–  Early work by Cliff (1993) and Sims (1994)

•  Evolve HW for control –  Thompson (1995): evolved networks of logic getes on FPGA (simulation)

•  Evolve in simulation, test on real robot –  Firmly the state of the art. –  Why? Simulated robots can easily change & evolution takes time! –  Problem: Reality gap

•  Evolution entirely on robots M, Matatic and D. Cliff (1996) . Challenges in evolving controllers for physical robots. Robotics and Autonomous Systems 19(1996): 67-83

(individuals X generations X evolved period)

12

Co-evolution of controllers and morphologies

13

Why evolving robot morphology?

•  Both control circuit and body plan co-evolve in true evolvable hardware

•  Body/brain = chicken & egg problem (Funes & Pollack, 1997)

•  A brain does not do much without a body, while a body cannot do much without a brain to control it

•  The course of natural evolution shows a history of body, nervous system and environment all evolving simultaneously in cooperation with, and in response to, each other

H. Lund, J. Hallam, W. Lee (1997), Evolving Robot Morphology. Proceeding of the 4th International Conference on Evolutionary Computation

14

Why (and how) co-evolve •  How to co-optimize morphology/control:

1.  Direct methods: 1-to-1 genome to parts mapping 2.  Recursive methods: L-system/grammar (a la Sims) 3.  Growth / development: more biological a. Optimize parameters of a fixed body plan b. Optimize body plan itself

•  Why, beyond the vague argument that together is better: –  Best body plan for a given task not known a priori by designer –  Simpler control if morphology is optimized –  Body parts can specialize (more objectives, e.g. object manupulation) –  Generalization to unseen environmental conditions (robust)

J. Bongard,The Utility of Simulated Robot Morphology Increases with Task Complexity for Object Manipulation. Artificial Life 16(2010): 201-223. MIT Press.

15

Co-evolution, more objectives •  Bongard, 2010

G=grasping, A=active categorical perception, L=lifting LN=phalanges’ lengths, RD=phalanges’ radii, SP=spacing between fingers

16

The GA for robot bodies

A. Winfield and J. Timmis (2015), Evolvable Robot Hardware. Evolvable Hardware, eds M. Trefzer and A. Tyrrell, Springer, in press.

17

Relevant work and State of the art

18

Sims (1994)

•  1st body / brain co-evolution with GA •  3D simulated physical world •  Morphology: shape, size, joints, ... •  Controller: #nodes, connections, type of functions, ... •  L-system for development

K. Sims (1994), Evolving Virtual Creatures. SIGGRAPH 1994 Proceedings.

19

Morphology space (body plan)

•  Robot body plan (Lund, 1997) should adapt to task: –  Type, number, position of sensors –  Body size –  Wheel radius, wheel base –  Motor time constants

•  Small changes in morphology = large changes in performance

•  Some body plans are impossible or impracticable

20

Lund et. al (1997) - cricket

The Khepera robot with ears. The ears have programmable amplifiers, synthesizers, and mixers.

Gryllus Bimaculatus

•  Male produces a species-specific song •  Female has complex delay-based ears •  Recognize sound, navigate towards source •  Co-evolution of controller and ear morphology

•  Include body plan in genome as Hox genes o  to control body plan o  (in nature, growth of body parts)

21

LEGO robots

P. Funes and J. Pollack (1998). Evolutionary Body Building: Adaptive Physical Design for Robots. Artifical Life 4(1998): 337-357

H. Lund (2003). Co-evolving Control and Morphology with LEGO Robots. In Hara and Pfeifer (eds). Morpho-functional Machines, Springer Verlag.

•  Demonstrate that evolving morphology for the real world is possible •  Conservative and efficient simulator

•  GA body (string) •  3 wheels, 25 wheel bases, 11 light sensor positions = 825 morphologies •  GP brain (tree-like) •  linear perceptron (evolution of 6 weights) •  simulate and transfer to reality •  line following task

22

GOLEM (2000)

H. Lipson and J. Pollack (2000). Automatic Design and Manifacture of Robotic Lifeforms. Nature, 406(2000): 974-978

•  Simulated control and mechanics •  Direct encoding of anatomic parts / connectivity •  3D printed (except motors)

Limitations: •  No sensors / interactions with environment •  Not automatic feedback from real world •  Reality gap

23

Morphology Self-Modeling

J. Bongard, V. Zykov and H. Lipson (2006). Resilient Machines Through Continuous Self-Modeling. Science, 314(2006): 1118-1121

•  Indirectly infer own structure •  Create compensatory behavior in case of failure •  Modeling, testing, prediction... •  Development of cognition in machines •  Consciousness

24

Robogen

J. Auerbach, D. Aydin, A. Maesani, P. Kornatowski, T. Cieslewski, G. Heitz, P. Fernando, I. Loshchilov, L. Daler and D. Floreano. RoboGen: Robot Generation through Artificial Evolution. Artificial Life 14: International Conference on the Synthesis and Simulation of Living Systems, New York, NY, USA, July 30-August 2, 2014.

•  Integrated open-source environment

•  Co-evolve robot controllers (fully-connected recurrent ANNs) and bodies (represented as GP trees)

•  Predefined 3D printable body parts

•  Educational project

25

State of the art (co-evolution)

•  Hornby, Pollack (2001) * •  Bongard (2009) •  Auerbach, Bongard (2010) •  Aydin (2010)

Developmental or generative processes *

26

Challenges and opportunities

27

What HW can we evolve ?

•  Sensing: cameras, IR sensors, touch sensors •  Signaling: LEDS, speakers, WiFi •  Actuation: Motors •  Energy: Electrical Power Source •  Control: Microcontrollers •  Physical Structure: metal or plastic structure for chassis, legs,

wheels, grippers etc. •  Interconnect: Wires connecting electronics

...a realistic subset

28

The Reality Gap

•  Difference between real reality and simulated reality •  Consequence: real robot performance does not match evolved

robot in simulation

K. Glette, G. Klaus, J. Cristobal Zagal, J. Tørresen. Evolution of Locomotion in a Simulated Quadruped Robot and Transferral to Reality Proceedings of the 17th International Symposium on Artificial Life and Robotics, 2012

29

Open issues

•  Designer’s influence is still strong: fitness function, genotype-to-phenotpe mapping, environment, population size, robot body, control architecture

–  Active research in all those issues

•  Relevance for the understanding of natural systems –  Evolutionary Robotic models are extremely simplified... –  ...but useful to understand general principles

•  Co-evolution •  Interaction learning / evolution

30

Challenges

•  Hardware design (many degrees of freedom) •  Scalability of controllers •  With brain-body co-evolution, search space quickly

becomes very large (scalability) •  Application (identify specific application domains where

benefits can be demonstrated) •  Real time on real HW takes time

–  Battery lifetime –  Robot lifetime (maintenance, repairs)

•  Noise and errors in simulation –  Interleaving evolution in simulation and testing on-line –  Feedback

31

Grand Challenges

•  ”Making it work” (Winfield, ICES 2014) Limited progress in last 15 years

•  Genotype-Phenotype mapping: ontogenetic developmental principles must be incorporated –  Variable length genomes –  simmetry / modularity / reuse

•  Big systems, e.g. more than 1000 parts •  Automatically synthesize more complex behaviors than

those designed by hand •  Co-evolve morphology and control (not in simulation)

32

The Triangle of Life

A. Eiben, N. Bredeche, M. Hoogendoorn, J. Stradner, J. Timmis, A. Tyrrell and A. Winfield. The Triangle of Life: Evolving Robots in Real-time and Real-space. Proceedings of the European Conference on Artificial Life 2013. The MIT Press.

•  Fully embodied evolution in real-time and real-space •  General conceptual framework •  Co-evolution body/brain •  Partly implemented in EU Symbrion

To date, no system that implements a complete artificial ecology with physically reproducing robots, where robot morphology and controllers co-evolve in an open-ended process

33

Modular Self-Reconfigurable Robot Systems

•  Fixed morphology, but •  Self-assembly •  Co-evolved robot controllers / morphology

(disassemble and reassemble machines to form new morphologies)

•  Versatility, robustness, low cost

Molecube system from Cornell

34

Opportunities

•  Artificial life approach

Here there is no infrastructure: the co-evolving body/brain machinery is completely embodied within the robotic population

35

References S. Nolfi and D. Floreano (2000). Evolutionary Robotics: The Biology, Intelligence and Technology of Self-Organizing Machines. The

MIT Press

M. Waibel, D. Floreano, L. Keller (2011). A Quantitative Teat of Hamilton’s Rule for the Evolution of Altruism. PLoS Biol 9(5): e1000615

M, Matatic and D. Cliff (1996) . Challenges in evolving controllers for physical robots. Robotics and Autonomous Systems 19(1996): 67-83

H. Lund, J. Hallam, W. Lee (1997), Evolving Robot Morphology. Proceeding of the 4th International Conference on Evolutionary Computation

J. Bongard,The Utility of Simulated Robot Morphology Increases with Task Complexity for Object Manipulation. Artificial Life 16(2010): 201-223. MIT Press.

A. Winfield and J. Timmis (2015), Evolvable Robot Hardware. Evolvable Hardware, eds M. Trefzer and A. Tyrrell, Springer, in press.

K. Sims (1994), Evolving Virtual Creatures. SIGGRAPH 1994 Proceedings.

P. Funes and J. Pollack (1998). Evolutionary Body Building: Adaptive Physical Design for Robots. Artifical Life 4(1998): 337-357

H. Lund (2003). Co-evolving Control and Morphology with LEGO Robots. In Hara and Pfeifer (eds). Morpho-functional Machines, Springer Verlag.

H. Lipson and J. Pollack (2000). Automatic Design and Manifacture of Robotic Lifeforms. Nature, 406(2000): 974-978

J. Bongard, V. Zykov and H. Lipson (2006). Resilient Machines Through Continuous Self-Modeling. Science, 314(2006): 1118-1121

J. Auerbach, D. Aydin, A. Maesani, P. Kornatowski, T. Cieslewski, G. Heitz, P. Fernando, I. Loshchilov, L. Daler and D. Floreano. RoboGen: Robot Generation through Artificial Evolution. Artificial Life 14: International Conference on the Synthesis and Simulation of Living Systems, New York, NY, USA, July 30-August 2, 2014.

K. Glette, G. Klaus, J. Cristobal Zagal, J. Tørresen. Evolution of Locomotion in a Simulated Quadruped Robot and Transferral to Reality. Proceedings of the 17th International Symposium on Artificial Life and Robotics, 2012

A. Eiben, N. Bredeche, M. Hoogendoorn, J. Stradner, J. Timmis, A. Tyrrell and A. Winfield. The Triangle of Life: Evolving Robots in Real-time and Real-space. Proceedings of the European Conference on Artificial Life 2013. The MIT Press.

36

37

Competitive Co-evolution

•  CCE (e.g. predator-prey) may enhance artificial co-evolution of body/brain and optimize controller/morphology combination

•  Competing populations can reciprocally drive each other to incrementally increasing levels of behavioral complexity

G. Buason, N. Bergfeldt and T. Ziemke (2005). Brains, Bodies, and Beyond: Competitive Co-Evolution of Robot Controllers, Morphologies and Environments. Genetic Programming and Evolvable Machines, 6, 25-51, 2005

38 Ziemke et al, 2005

Exp. 1 Preconditions: 1- same max speed in both robots 2- speed in predator constrained the view angle

Exp. 2 – adding a constraint Preconditions: 1- same max speed in both robots 2- speed in both robots constrained by the view angle

Morphological space Prey ”choose” speed over vision

39

Artificial neuron (McCullock and Pitts, 1943) Artificial Neural Networks EvoDevo

Prerequisites

Encodings: -  Direct (limited: scalability, no morphogenesis, limited regularity/modularity, no reuse) -  Matrix rewriting -  Cellular encoding -  Axon growth -  HyperNeAT (regularity without development using CPPN) -  Indirect encodings