SASO, FOCAS workshop, 8th Sept. 2014 Payam Zahadat, … ,Thomas Schmickl

Artificial Life Lab, Karl-Franzens University Graz, Austria

SOCIAL ADAPTATION OF ROBOTS FOR MODULATING

SELF-ORGANIZATION IN ANIMAL SOCIETIES

Email: thomas.schmickl@uni-graz.at, Twitter: @thomasschmickl

ASSISIBF

Animal and robot Societies Self-organize and Integrate

by Social Interaction

BF: bees & fish

http://assisi-project.eu twitter:@AssisiEU

BEST OF ANIMAL-ROBOT INTERACTION TODAY

Emergent Collective (mixed) cognition Individual cognition machine

Individual cognition animal

INTERACTION OF 2 INFORMATION-PROCESSING SYSTEMS:

OVERALL CONCEPT

ROBOTS

HONEYBEES BUILD ADAPTIVE SOCIETIES: •  Honeybees build a self-organizing society that

makes decisions collectively to adapt to changes.

OUR PARADIGM

Robot / Individual

Actuation [2]

Environment

Senses

Animal / Individual

Senses [1] Actuation

Self-Organization (social) [3]

(Evolution, Learning (rewire) [4]

many

Operator Collective Adaptive System:

TYPES OF INTERACTION (CHANNELS) Sensing: •  Temperature •  Proximity & Touch •  Vibration Actuation •  Temperature •  Vibration •  Light •  Electric field •  Magnetic field Adaptation •  Self-Organization •  GRN •  ANN •  Artificial Hormone Systems •  Evolutionary Computation •  Machine Learning •  Modelling (incl. automatized model building)

CASU

SENSING 1.)

MEASURING DENSITY OF BRISTLEBOTS BY CASU

SELF-PROGRAMMING ROBOTS

40 experiments (learning phase)

Result: density sensor

64 experiments (validation phase)

Apply those rules

Learns rules

RULES-4 machine learning

algorithm

Dr. Ziad Salem, UNIGRAZ and UNIZAGREB-FER

ACTUATION 2.)

CASU DESIGN

(a) Top-side view(b) Cross-section view

(c) A thermal photo showing two CASUs in which oneemits heat stimulus and the other is inactive

Fig. 2: Bee CASU hardwares and emission of heat stimulus.

are integrated on two sides of a PCB. The electric field hasa maximal intensity up to 10kV/m close to the emitter infrequency range up to 1kHz. The maximal intensity of themagnetic field is about 2 mT at low frequencies (5-100 Hz).The emitter, due to its low weight, can be also used to producefast temperature variations. The emitters are controlled via adedicated high-voltage/high-current board, which is connectedto the main control system.

For detecting bee presence, we use infrared (IR) proximitysensors. These sensors should not influence bee behavioursince bees do not see IR light [8]. Six of these sensors aremounted on the upper side of the body and they can detect beesat distance up to 20 mm which is the same size as diameterof a CASU.

For decentralized inter-CASU communication, we use sev-eral communication and sensing channels. In the envisionedbee arena, every CASU is connected with four neighbouringCASUs via copper. On the end of each copper bridge, everyunit has a temperature sensor and a vibration sensor whichenables monitoring and sensing of the neighbouring CASUstate. Furthermore, there is also a possibility of wireless inter-CASU communication through a low-range IR transceiverlocated on each bridge.

An example setup for experimenting with young honeybees(Apis mellifera), aged from 1 to 24 hours, is represented inFig. 1c. In this setup, two CASUs are embedded in a woodenfloor surrounded by a rectangular arena with semicircularshort sides. The setup is illuminated by infrared light andtherefore the bees are not able to use visual cues for navigation.An overhead acA2040-25gm/NIR monochrome near-infraredenhanced GigE CCD camera (Basler AG, Germany) with amaximum resolution of 2048 px ⇥ 2048 px and equipped witha 12 mm Kowa lens (Kowa, Japan) records the experiments.

III. FEEDBACK MECHANISMS IN DIFFERENT LEVELS

Depending on the adaptation mechanism that is used tomodify CASU controllers and constraints in the experimental

setups, both global and local feedbacks can be beneficial.

A. Global feedback

For providing the system with a global feedback, multi-target tracking software has been developed and already ap-plied to our fish arena (Fig. 3). The features representing theagents’ heads in the dorsal view can be precisely detectedby the software by using a corner detection based technique.The detected spatial locations of the agents in the mixedsociety were used as inputs of individual Kalman filters, wherethe initial speeds of the agents were assumed to be zeroand translational motion models with constant accelerationswere used to predict and correct the posteriori states of theagents. The Kalman filter is a powerful Bayesian estimationtechnique used to track stochastic dynamical systems undernoise conditions and uncertainties [9]. Correspondences be-tween predicted states and new measurements were treated asa combinatorial optimization problem based on the computeddistance cost matrix. A new detected blob was assumed to bea new measurement if its value was below a pre-establishedthreshold criterion based on the Euclidean distance between thei� th agent and the j� th detected feature. The main featuresof the developed software are summarized as: (1) automaticassignment of individual CASUs, (2) automatic detection ofthe features of interest representing the frontal head of the con-sidered fishing lures and living agents (Zebrafish), representedby Ci, and Aj in Fig. 3, respectively where the subscripts i andj denote individual agents in the mixed society, (3) estimationof the CASUs’ poses based on the previously tracked states,(4) collision avoidance mechanism using combined potentialfields, and (5) vision-based PID controller.

B. Local feedback

To provide the system with local feedback, the informationfrom local sensors of the CASUs needs to be processed.This mechanism can be used in our Bee CASUs due to theirability to detect bees via IR sensors. A modified versionof a machine learning algorithm known as RULES-4 (RULeExtraction System version 4) [10], [11], [12] is used to estimatethe density of agents based on the view of the Bee CASUs.The algorithm generates a set of rules based on initial datasetsand can be updated and refined rapidly when new data areavailable.

We have carried out a number of preliminary experimentswith a single CASU and a number of Bristlebots emulatingbees. For the experiments, the agents moved randomly in theenvironment and the inputs form IR sensors of the CASUwere recorded generating samples for the machine learningalgorithm. The number of agents changed between 1 and 9, and109 samples were collected. The dataset was then divided intotwo parts of 70% and 30% for training and testing accordingly.The learning algorithm ran with training data for 65536 time-steps and generated a set of rules consisting of a number ofconditions on the sensor values and a classification decision.

In the experiments, many of the algorithm parametersthat influence the quality of the generated rules, i.e., short-term memory (STM) and the number of conditions, werealtered during the learning process. Regulating the numberof examples in STM is essential for generating good results.

SAMPLE VIDEO OF A BEE-ATTRACTING CASU

SELF-ORGANIZATION 3.)

VIBRATION

ADAPTATION (EVOLUTIONARY COMPUTATION)

4.)

A NEW PARADIGM IN EVOLUTIONARY COMPUTATION:

programmable, evolvable

Not programmable, but predictable

Evolutionary robotics: Natural swarm-systems:

Swarm robotics: Novel Paradigm:

A NEW PARADIGM IN EVOLUTIONARY COMPUTATION:

Novel Paradigm:

THE FIRST PROOF-OF-CONCEPT MODELS

Stimulus Temp. Light Vibration Effect attractive repellent stop-signal Diffusion rate 0,2 0 0,01 Decay rate 0,1 1 0,9 Instantly reachable

no yes no

Blockable by bee no yes no

Three actuators (floor patches) in every cell generate 3 different types of stimuli: A: Temperature B: Light C: Vibration

Bees react differently to different stimuli: - If the cell is vibrating over a given threshold à Stay - Otherwise: Let the strongest stimulus win à Go

1D model

EXAMPLE FOR EVOLUTIONARY CONCEPT

describes processes of natural pattern formation and growth. Ithas been successfully applied in robotic applications [17] andis capable of generating diverse patterns of behaviour [18]. AnAHHS is defined by a set of artificial hormones and rules. Therules define the dynamics of hormone concentrations basedon their current values and sensory inputs. Concentrations ofselected hormones can then be used as outputs of the system.

C. Evolving CA controllers in interaction with animals

Here we evolve CA controllers for bee CASUs in a two-dimensional arena and illustrate that the resulting patterns areable to guide a bee through either attractive or repellent stimulito a specific target zone. The arena is a 9 ⇥ 9 grid withcorners removed to approximate a circle (61 cells). This fairlystraightforward task is intended to establish how the controllersand bee model interact, rather than to represent an ecologicallyrelevant scenario at this stage. We restrict our investigationsto single modalities meaning that all the CASUs are able togenerate only an attractive modality (A) or a repulsive modality(B) in separate cases. Combinations of the two modalities andthe stop signals remain for future work.

Each CASU is modelled as a pulse-coupled oscillatorwhose dynamics are defined through interactions with otherCASUs in a local neighbourhood, N . The ith cell activa-tion, xi, is updated as follows: xi(n + 1) = xi(n) +

1T +

✏�i(n)g(xi(n)), where T is the inter-pulse interval for anisolated cell, �i(n) is the proportion of neighbouring cells thatfired in timestep n, and g(·) is the pulse-coupling function.If xi exceeds its threshold the cell moves to a pulsing state,corresponding to a stimulus mode being activated. (Note thatthis is a discretised equation, based on a continuous versionfrom [14].) Each cell in the arena can sense the states of itsVon Neumann neighbours. Here we use a linear pulse-couplingfunction and update the cell states in a random order.

We use an evolutionary algorithm to find parameter val-ues that enable the oscillators to match a specific activationsequence, placing three parameters for each CASU under evo-lutionary control: x(0), T , ✏. Each genome in the populationhas a total of 61⇥3=183 real-valued variables to encode con-trollers for all the CASUs. We apply three variation operators:Gaussian mutation, one-point crossover, and crossover thatinherits (x(0), T, ✏) tuples from 61 randomly selected parents.A genome is evaluated by advancing all CASUs for neval

timesteps, accruing a point for each CASU that matches thetarget in each timestep. Further details can be found in [19].

To interface the pulsing outputs of the CASU model withthe continuous sensory channels of the bee model, we performa transduction via a leaky integrator for each of the CASUsas defined by: v =

1R (v � vleak) +

Ptt�l P (n), where R is

a notional resistance, P (n) is the presence (or absence) ofpulse at a given timestep, l is a window length, and v is thecontinuous signal strength. l influences how the timescales ofthe CASU and the bee correspond to one another.

The evolved CASU controllers are able to move thesimulated bee to the target zone in the arena, and to do soeither via waves of attractive stimulus (which lead the beeto the target area) or via waves of repulsive stimulus (whichfollow, or ‘chase’ the bee). We find that both forms of controlare successful even operating against a fixed gradient of the

Tim

e-st

eps

(top

tobo

ttom

)

arena cells (one-dimensional)

(a) unevolved

arena cells (one-dimensional)

(b) evolved

Fig. 5: An example behaviour of the bees and CASUscontrolled by a random (unevolved) and an evolved AHHS.The green areas indicate the target zones. The yellow tracesindicate the bee trajectories, showing that the evolved AHHScontrollers successfully guide the bees into the target zones(green area).

opposing sign (repulsive or attractive, respectively). However,the latter is slightly more sensitive to how the timescales ofbee and CASU are matched: the repulsive strategy is successfulonly when the CASU activates the leaky integrator for at least73% of the coupling window, while the attractive strategysucceeds with 63% or greater. Finally, neither strategy is ableto overcome a faulty CASU that does not emit any pulses:the bees are guided towards the target area, but do not movebeyond the fault.

D. Evolving AHHS controllers in interaction with animals

Here we evolved AHHS controllers for CASUs within anarena containing several bees. The goal is to guide the bees toaggregate in two predefined target zones (Fig. 5). The arenain this experiment is one-dimensional consisting of L = 100

cells. Each CASU is controlled by an AHHS controller. All thecontrollers are genetically identical. The genomes are evolvedby Wolfpack Evolutionary Algorithm [20]. Within the arena,10 simulated bees move according to their reaction to thestimuli of the CASUs (see Table I).

The fitness function is defined as fitness =P

j (L� dj) ,

where dj is the distance of bee j from the centre of the closesttarget area.

Fig. 6 shows the results collected from 7 independentevolutionary runs. As the figure shows, the fitness increasesquickly and after 150 evaluations maximum fitness (1000)is reached for at least one of the runs while the other runsalso reached high fitnesses (median 988). Although not locatedexactly in the centre, by the end of each run 10/10 bees are inthe target zones.

E. From abstract to detailed physical simulators

The evolutionary approach pursued in our research canrequire thousands of iterations in order to produce controllers

Fig. 6: Fitness progress in evolution of AHHS controllers.

that can induce complicated animal behaviours. Since it is notfeasible to run such high numbers of iterations in practice, thistype of research calls for a simulator that can run simulationsfeaturing up to a hundred entities on a time scale of seconds orminutes. On the other hand, the simulations should be sophis-ticated enough that the evolved controllers can be applied toa real system, possibly with only minor modifications. We aredeveloping such a tool, as an extension to the Enki open sourcesimulator [21]. Our simulator can currently model several typesof physical interactions, such as proximity, light and heat. Thesource code is available on GitHub [22], licensed under thepermissive LGPL license.

V. CONCLUSION AND FUTURE WORK

In this article we describe a novel scientific concept ofintroducing self-organizing, interacting, autonomous roboticdevices into societies of animals (Fig. 1). We demonstratethat it is possible to do so in two different ways: Oneis to produce a static array of autonomous sensor-actuatorunits (with honeybees); the other is to introduce a group offreely moving units (with fishes). Our concept involves severalprerequisites which we showed to be feasible in this first reportbased on preliminary data of our mixed societies: We describethe mechatronics of our robots, we show that we are able to getfeedback from the experimental setups either globally by usingtracking (Fig. 3) or locally by the robots that detect the animals(Fig. 4a) and run sophisticated algorithms to produce requiredinformation that help self-organization on the collective level(Fig. 4b). We show (Fig. 2c) that our robots are able to emitstimuli that are capable of locally affecting the animals con-cerning their self-organization. Finally, we also demonstratethat this collective system, consisting of animals and machines,which we call a “mixed society”, is able to exhibit societallearning through evolutionary computation algorithms (Fig. 6).In the future, we plan to perform this societal learning on-board, on-line and unsupervised by allowing the robots toadapt their behaviours during the runtime of experiments bysimultaneously observing the behavior of bees to achieve anonline fitness feedback.

ACKNOWLEDGEMENT

This work is supported by: EU-ICT project ‘ASSISI bf’,no. 601074.

REFERENCES

[1] S. Camazine, J.-L. Deneubourg, N. R. Franks, J. Sneyd, G. Theraulaz,and E. Bonabeau, Self-Organizing Biological Systems. Princeton Univ.Press, 2001.

[2] C. Saverino and R. Gerlai, “The social zebrafish: Behavioral responsesto conspecific, heterospecific, and computer animated fish,” BehaviouralBrain Research, vol. 191, no. 1, pp. 77 – 87, 2008.

[3] M. Szopek, T. Schmickl, R. Thenius, G. Radspieler, and K. Crailsheim,“Dynamics of collective decision making of honeybees in complextemperature fields.” PLoS One, vol. 8, no. 10, p. e76250, 2013.

[4] F. Bonnet, P. Retornaz, J. Halloy, A. Gribovskiy, and F. Mondada,“Development of a mobile robot to study the collective behaviorof zebrafish,” 4th IEEE RAS & EMBS International Conference onBiomedical Robotics and Biomechatronics, pp. 437–442, Jun. 2012.

[5] T. Landgraf, H. Nguyen, S. Forgo, J. Schneider, J. Schroer, C. Kruger,H. Matzke, R. O. Clement, J. Krause, and R. Rojas, “Interactive RoboticFish for the Analysis of Swarm Behavior,” in Advances in SwarmIntelligence. Springer, 2013, pp. 1–10.

[6] S. Butail, T. Bartolini, and M. Porfiri, “Collective Response of ZebrafishShoals to a Free-Swimming Robotic Fish,” PLoS ONE, vol. 8, no. 10,2013.

[7] S. Magnenat, P. Retornaz, M. Bonani, V. Longchamp, and F. Mondada,“ASEBA: A modular architecture for event-based control of complexrobots,” IEEE/ASME Transactions on Mechatronics, vol. 16, no. 2, pp.321–329, 2011.

[8] R. Menzel, “Farbensehen bei Insekten - ein rezeptorphysiologischerund neurophysiologischer Problemkreis,” Verhandlungen der DeutschenZoologischen Gesellschaft, pp. 26–40, 1977.

[9] R. E. Kalman, “A New Approach to Linear Filtering and PredictionProblems 1,” Journal Of Basic Engineering, vol. 82, no. Series D, pp.35–45, 1960.

[10] D. T. Pham and S. S. Dimov, “An algorithm for incremental inductivelearning,” Journal of Engineering Manufacture, vol. 211, no. B, pp.239–249, 1997.

[11] Z. Salem, “Enhanced computer algorithms for machine learning,”Ph.D. dissertation, Intelligent Systems Laboratory, Cardiff School ofEngineering, Universiy of Wales Cardiff, UK, Cardiff, 2002.

[12] D. T. Pham and Z. Salem, “Improved inductive learning using trainingdata reorganisation,” in the 1st international conference on Informationand communication Technologies from Theory to Applications, 2004.

[13] C. G. Langton, “Computation at the edge of chaos: phase transitionsand emergent computation,” Physica D: Nonlinear Phenomena, vol. 42,no. 1, pp. 12–37, 1990.

[14] R. E. Mirollo and S. H. Strogatz, “Synchronization of pulse-coupledbiological oscillators,” SIAM Journal on Applied Mathematics, vol. 50,no. 6, pp. 1645–1662, 1990.

[15] T. Schmickl and K. Crailsheim, “Modelling a hormone-based robotcontroller,” in MATHMOD 2009 - 6th Vienna International Conferenceon Mathematical Modelling, 2009.

[16] A. M. Turing, “The chemical basis of morphogenesis,” PhilosophicalTransactions of the Royal Society of London. Series B, BiologicalSciences, vol. B237, no. 641, pp. 37–72, 1952.

[17] T. Schmickl, H. Hamann, J. Stradner, R. Mayet, and K. Crailsheim,“Complex taxis-behaviour in a novel bio-inspired robot controller,” inProc. of the ALife XII Conference. MIT Press, 2010, pp. 648–655.

[18] P. Zahadat and T. Schmickl, “Generation of diversity in a reaction-diffusion-based controller,” Artificial Life, vol. 20, no. 3, p. 319342,2014.

[19] F. Silva, L. Correia, and A. L. Christensen, “Modelling synchronisationin multirobot systems with cellular automata: Analysis of updatemethods and topology perturbations,” in Robots and Lattice Automata,G. Sirakoulis and A. Adamatzky, Eds. Berlin: Springer, forthcoming.

[20] P. Zahadat and T. Schmickl, “Wolfpack-inspired evolutionary algorithmand a reaction-diffusion-based controller are used for pattern formation,”in Genetic and Evolutionary Computation Conference (GECCO), 2014.

[21] S. Magnenat, M. Waibel, and A. Beyeler, “Enki - an open source fast2d robot simulator,” Online, 2014, http://home.gna.org/enki/.

[22] D. Miklic and P. Mariano, “Assisi playground - an open source simu-lator of mixed societies,” Online, 2014, https://github.com/larics/assisi-playground.

fitness = distmax − dist jNbees

j=1

∑

describes processes of natural pattern formation and growth. Ithas been successfully applied in robotic applications [17] andis capable of generating diverse patterns of behaviour [18]. AnAHHS is defined by a set of artificial hormones and rules. Therules define the dynamics of hormone concentrations basedon their current values and sensory inputs. Concentrations ofselected hormones can then be used as outputs of the system.

C. Evolving CA controllers in interaction with animals

Here we evolve CA controllers for bee CASUs in a two-dimensional arena and illustrate that the resulting patterns areable to guide a bee through either attractive or repellent stimulito a specific target zone. The arena is a 9 ⇥ 9 grid withcorners removed to approximate a circle (61 cells). This fairlystraightforward task is intended to establish how the controllersand bee model interact, rather than to represent an ecologicallyrelevant scenario at this stage. We restrict our investigationsto single modalities meaning that all the CASUs are able togenerate only an attractive modality (A) or a repulsive modality(B) in separate cases. Combinations of the two modalities andthe stop signals remain for future work.

Each CASU is modelled as a pulse-coupled oscillatorwhose dynamics are defined through interactions with otherCASUs in a local neighbourhood, N . The ith cell activa-tion, xi, is updated as follows: xi(n + 1) = xi(n) +

1T +

✏�i(n)g(xi(n)), where T is the inter-pulse interval for anisolated cell, �i(n) is the proportion of neighbouring cells thatfired in timestep n, and g(·) is the pulse-coupling function.If xi exceeds its threshold the cell moves to a pulsing state,corresponding to a stimulus mode being activated. (Note thatthis is a discretised equation, based on a continuous versionfrom [14].) Each cell in the arena can sense the states of itsVon Neumann neighbours. Here we use a linear pulse-couplingfunction and update the cell states in a random order.

We use an evolutionary algorithm to find parameter val-ues that enable the oscillators to match a specific activationsequence, placing three parameters for each CASU under evo-lutionary control: x(0), T , ✏. Each genome in the populationhas a total of 61⇥3=183 real-valued variables to encode con-trollers for all the CASUs. We apply three variation operators:Gaussian mutation, one-point crossover, and crossover thatinherits (x(0), T, ✏) tuples from 61 randomly selected parents.A genome is evaluated by advancing all CASUs for neval

timesteps, accruing a point for each CASU that matches thetarget in each timestep. Further details can be found in [19].

To interface the pulsing outputs of the CASU model withthe continuous sensory channels of the bee model, we performa transduction via a leaky integrator for each of the CASUsas defined by: v =

1R (v � vleak) +

Ptt�l P (n), where R is

a notional resistance, P (n) is the presence (or absence) ofpulse at a given timestep, l is a window length, and v is thecontinuous signal strength. l influences how the timescales ofthe CASU and the bee correspond to one another.

The evolved CASU controllers are able to move thesimulated bee to the target zone in the arena, and to do soeither via waves of attractive stimulus (which lead the beeto the target area) or via waves of repulsive stimulus (whichfollow, or ‘chase’ the bee). We find that both forms of controlare successful even operating against a fixed gradient of the

Tim

e-st

eps

(top

tobo

ttom

)

arena cells (one-dimensional)

(a) unevolved

arena cells (one-dimensional)

(b) evolved

Fig. 5: An example behaviour of the bees and CASUscontrolled by a random (unevolved) and an evolved AHHS.The green areas indicate the target zones. The yellow tracesindicate the bee trajectories, showing that the evolved AHHScontrollers successfully guide the bees into the target zones(green area).

opposing sign (repulsive or attractive, respectively). However,the latter is slightly more sensitive to how the timescales ofbee and CASU are matched: the repulsive strategy is successfulonly when the CASU activates the leaky integrator for at least73% of the coupling window, while the attractive strategysucceeds with 63% or greater. Finally, neither strategy is ableto overcome a faulty CASU that does not emit any pulses:the bees are guided towards the target area, but do not movebeyond the fault.

D. Evolving AHHS controllers in interaction with animals

Here we evolved AHHS controllers for CASUs within anarena containing several bees. The goal is to guide the bees toaggregate in two predefined target zones (Fig. 5). The arenain this experiment is one-dimensional consisting of L = 100

cells. Each CASU is controlled by an AHHS controller. All thecontrollers are genetically identical. The genomes are evolvedby Wolfpack Evolutionary Algorithm [20]. Within the arena,10 simulated bees move according to their reaction to thestimuli of the CASUs (see Table I).

The fitness function is defined as fitness =P

j (L� dj) ,

where dj is the distance of bee j from the centre of the closesttarget area.

Fig. 6 shows the results collected from 7 independentevolutionary runs. As the figure shows, the fitness increasesquickly and after 150 evaluations maximum fitness (1000)is reached for at least one of the runs while the other runsalso reached high fitnesses (median 988). Although not locatedexactly in the centre, by the end of each run 10/10 bees are inthe target zones.

E. From abstract to detailed physical simulators

The evolutionary approach pursued in our research canrequire thousands of iterations in order to produce controllers

CONTROL (FROM “OUTSIDE”)

5.)

CONTROLLING BEES Send the swarm around the arena counterclockwise

HAVE FUN !

6.)

FUN WITH BRISTLEBOTS

Email: thomas.schmickl@uni-graz.at, Twitter: @thomasschmickl