51

Co-evolution, Games, and Social Behaviors

Embed Size (px)

DESCRIPTION

Professor Xin Yao gave a lecture on "Co-evolution, games, and social behaviors" in the Distinguished Lecturer Series - Leon The Mathematician. More Information available at: http://goo.gl/G7MdD

Citation preview

Page 1: Co-evolution, Games, and Social Behaviors

�� � � �� � �� �� �� � �� � �� � � �� �� � �� ����

��� � �� �! " #%$& '() " *+ $, " $ ( " -. /

012 � �3 �4 5 �6 798: ;< 01 � ;� 1 ; => 2 ; ?� �@ ;< A�: B �4 C� <6 � � D2 �6

E F DHG � A: � � = C� <6 � � D2 �6 CI JK > > = ?� �: ; FL � � D F �6

?M NO 2 : : 7O P P!Q Q QR 1 AR G 2 �6 R �1R 8S PUT V� �

W XYZ [\] ^Y_ _ `a YZ `b\ Y ] c\ bd e `f_ g `Z ch ] i

Page 2: Co-evolution, Games, and Social Behaviors

�� � �� � � �� �� � � � �� � � � � � �� � � �� � � �� �� � �

� � � � � � �� � � � � �� ��

IR �2 �: D �6 ; A

K R �2 B D �6 ; A

�R �2 B 1 ��� ;@ �3 8: � � �

�R 8: �6 �: � 1 6 � F 8 3 �< � A �: � � � � 4 3 ; �<� � � D A B A: ;6 A

JR > 2 ;! "#$ D �6 ;

%R & 8 3 : � 73 ;3 ;@ ;3 A �4 1 � � 7 ;< �: � � � �� F < ; 798: �: � � �

'R 5 � � 1 3 8 F� � D M ;6 �<S A

Page 3: Co-evolution, Games, and Social Behaviors

�� � �� � � �� �� � � � �� � � � � � �� � � �� � � �� �� � �

� � � � � ��

��

��

��

��

5 � � 7 ;< �: ;

$ ;4 ;1 :

5 � � 7 ;< �: ; $ ;4 ;1 :

�� �� � � �� � ��� ���� � �� � �� ��� � � ���� � � � � �� �� � !�" ��$# !% � � � � � � �� � �& �� �' � � � !

( ) * ),+ ) - � !� !�� � ! � � - . + . * . ( � " % + . / ( 0 - 12 � &

Page 4: Co-evolution, Games, and Social Behaviors

�� � �� � � �� �� � � � �� � � � � � �� � � �� � � �� �� � �

�� � �� �� �� � � � � � � � �� � � �� � � � � � � �

$ �@ � A� �� 1 �6 6 ;� : ; F: 2 �:

> 2 ;� � 73 � B ;< 1 � A ; �! #$ /2 � A D < ; �: ;< D ;� ;< �3 �: B �� F � 7 73 � 1 �G �3 �: B: � < ; �3 � 3 �4 ;

A�: 8 �: � � � AR "� � F F�: � � � : � : 2 ; 7< � G 3 ;6 A �4 ;� ;< D B 1 � � A ;<@ �: � � � = ;1 �3 � D B = �� F

� @ ;< 7 � 798 3 �: � � � =6 �� B �: 2 ;< < ; �3 � 3 �4 ; 7< � G 3 ;6 A 1 �� G ; < ; 7< ; A ;� : ; F G B: 2 ;

! #$ 7 �< � F� D6 R

5 �3 6 �� = �� F� 3 �� 1 ; �� F 8 G ;<6 �� 2 �@ ; �3 A� � � F� 1 �: ; F: 2 �: : 2 ;! "# $ � A� � 8 �3 �: �: �@ ;3 B F�� ;< ;� : �4 < �6 : 2 ;K "#$ �� F: 2 �: �R R R 1 ;<: �� � A: < �: ; D� ; A: 2 �: Q � <S Q ;3 3

4 � < �� F�@ � F 8 �3 A � � : 2 ; # < � A� � ;<�� A$ �3 ;6 6 �4 ��3 � � 3 �< D ; D < � 8 7 AR �

Page 5: Co-evolution, Games, and Social Behaviors

�� � �� � � �� �� � � � �� � � � � � �� � � �� � � �� �� � �

� � �� � � � � � �

> 2 ;! "#$ D �6 ; 1 �� G ; F ;� � ; F G B: 2 ;4 �3 3 � Q � � D: 2 < ; ; 7< � 7 ;<: � ; AO

IR ; �1 2 73 � B ;< 4 �1 ; A: Q � 1 2 � � 1 ; A G ;: Q ; ;� 1 � � 7 ;< �: � � � � 5 / �� F F ;4 ;1: � � � �$ /�

K R : 2 ;$ � 7: � � � � A F �6 � � �� : 4 � < ; �1 2 73 � B ;< =�R ;R = ; �1 2 � A G ;: : ;< � � 1 2 � � A�� D$ : 2 ��

5 � � 6 �: : ;< 2 � Q 6 �� B � 4 : 2 ; �: 2 ;< 73 � B ;< A 1 2 � � A ; 5 �

�R : 2 ; F �6 � � �� : $ A: < �: ; D� ; A � � : ;< A ;1: � � � F ;� 1 � ;� : ; � 8 �3 � G < � 8 6 R "� 7 �<: � 1 8 3 �< =: 2 ;

� 8: 1 �6 ; �4 �3 3 73 � B ;< A 1 2 � � A ;: 2 ;� < � � � � F �6 � � �� : 5 A: < �: ; D� ; A � A 7< ;4 ;< �G 3 ;4 < �6

;@ ;< B 73 � B ;<�� A 7 � � � : �4 @ � ;Q : � : 2 ; � � ; �� Q 2 � 1 2 ;@ ;< B� � ; 1 2 � � A ; A$ = G 8: � � � � ;

� A6 �: �@ �: ; F: � F ;@ � �: ; 8 � �3 �: ;< �3 3 B4 < �6 $ R

Page 6: Co-evolution, Games, and Social Behaviors

�� � �� � � �� �� � � � �� � � � � � �� � � �� � � �� �� � �

� � �� � � � � � � � � ��� � � � ��

73 � B ;< $

5

� � ���

! 8 6 G ;< �4 1 � � 7 ;< �: � < A �6 � � D: 2 ; < ;6 �� � � � D � � 73 � B ;< A

���

���

� � �� � �

�� �� � �

�� �� � �� �� � ��� �� � � �� � �� � �� � � �� � � � � �� �� � !�" ��# !� � � � � � � �� � � ��� � �� �� � ��� � � � � �" ��� �" % �� � �" ! � !�� � !�� � !� �%� / 1 � . �� ��� � �� � �� � /� 1 � � � . �� " % �� � � . �� ��� � � � � � � � / � 1 �� . / � 0 ��� � 12 � ��� � � � � � � & �� �

��� ��� � �� � �� � ! ! � � � �� � � � ��� �� � � � � � � �� &

Page 7: Co-evolution, Games, and Social Behaviors

�� � �� � � �� �� � � � �� � � � � � �� � � �� � � �� �� � �

� � �� � � � � � � � � � � � � � �

73 � B ;< $

5

� � ���

! 8 6 G ;< �4 1 � � 7 ;< �: � < A �6 � � D: 2 ; < ;6 �� � � � D � � 73 � B ;< A

��

��

� � �� � �

� � � � � � �

� � � � �

Page 8: Co-evolution, Games, and Social Behaviors

�� � �� � � �� �� � � � �� � � � � � �� � � �� � � �� �� � �

� � � � � � � � � � �

�> 2 ;! #$ 1 � < < ; A 7 � � F A: � �: < 8 3 B < ;6 �<S �G 3 ; < �� D ; �4 < ; �3 � Q � <3 F A� 1 � �3 7< � G 3 ;6 A = G 8:

�4 ;Q A�6 73 ; ; V �6 73 ; A Q �3 3 A8 � 1 ;R � � R 5 �3 6 �� =� �� � � ��� � � �� �� ��� � ��

� � � �� � # ;< D �6 � � # < ; A A =I �� K R � 7 7R I J '�� I J � /

Page 9: Co-evolution, Games, and Social Behaviors

�� � �� � � �� �� � � � �� � � � � � �� � � �� � � �� �� � �

� � � � � � � �� � � � �� �

�2 �: Q � 8 3 F B� 8 F � � A � < �: � � � �3 �� F�@ � F 8 �3 �

Page 10: Co-evolution, Games, and Social Behaviors

�� � �� � � �� �� � � � �� � � � � � �� � � �� � � �� �� � �

�� � � � � � �� � � � � � � � � �� � � � � � �

ER DR =: 2 ; ;� ;< D B 1 < � A� A =: 2 ; F < � 8 D2 : � � C < �: �� � � � : 2 ; A8 6 6 ;< �4 I � ' % = ;: 1R

� ": � A ;@ � F ;� : 3 B �� ; �1 2 � � F�@ � F 8 �3 � A < �: � � � �3 A ;3 4 � � � : ;< ; A: : � � D� � < ;: 2 ; 1 �3 3 4 � <

< ; A: < �� � : � < < ; A 7 ;1 : �@ ; �4 : 2 ; 1 2 � � 1 ; A �4 : 2 ; �: 2 ;< AR C 8: � �� F: 2 � A � A: 2 ; < 8 G

� �4 ;@ ;< B� � ; 78 < A8 ; A � � F�@ � F 8 �3 < �: � � � �3 �: B � � : 2 � A Q � B =: 2 ; B �< ; Q � < A ; � �

: 2 �� �4 ;@ ;< B� � ; � A6 �: �@ �: ; F G B 1 �3 3 ;1 : �@ ; < �: � � � �3 �: B �� F ; V ;< 1 � A ; A < ; A: < �� � : R

"4 ;@ ;< B� � ;: < � ; A: � G ; � �4 < ; ; < � F ;< �: 2 ;� � ��� � � ; D ;: A � < � F ; �: �3 3 R �

Page 11: Co-evolution, Games, and Social Behaviors

�� � �� � � �� �� � � � �� � � � � � �� � � �� � � �� �� � �

� � �� � � � � � � � � � � �� � � � �

�� < A: F� A1 8 A A ; F G B �< F� � � � I � %�R

� > 2 ;< ; �< ; A� V4 �<6 ;< A Q �: 2 � � ; 1 � Q ; �1 2 Q ;� D2 : ; F �: I � � �3 GR

� > 2 ; B A2 �< ; � 1 �6 6 � � 7 � A: 8 < ; Q 2 � 1 2 1 �� � � 3 B A8 A: �� � A� V 1 � Q AR

� E �12 � F F�: � � � �3 1 � Q Q �3 3 < ; F 8 1 ;: 2 ; Q ;� D2 : �4 ;@ ;< B 1 � Q G B I � �3 GR

� E �12 4 �<6 ;< � A �3 Q � B A G ;: : ;< � � : � 2 �@ ; �� � F F�: � � � �3 1 � Q < ; D �< F3 ; A A � 4 : 2 ;

12 � � 1 ; A � 4 �: 2 ;< 4 �<6 ;< AR

� "4 : 2 ; B �3 3 2 �@ ; � � ; � F F�: � � � �3 1 � Q =: 2 ; B ;� F 8 7 Q �: 2 : Q � � � �� 3 GR 1 � Q AR

0 8 1 2 : < � D ; F� ; A �< ;� �: 8 � 1 �6 6 � � � � � 8 < 2 � D2 3 B 1 �6 7 ;: �: �@ ; Q � <3 FR ": � A A8 D D ; A: ; F

: 2 �: �: 2 ; �6 7 � @ ;< � A2 6 ;� : � 4 A6 �3 3 4 �<6 ;< A � � E� D3 �� F F 8 < � � D: 2 ; 7 ;< � � F �4 : 2 ;

;� 1 3 � A8 < ; A � � : 2 ; ;� D2 : ; ;� : 2 1 ;� : 8 < B6 � B2 �@ ; G ; ;� ; V �1 ;< G �: ; F G B: 2 � A

72 ;� �6 ;� � � R �

Page 12: Co-evolution, Games, and Social Behaviors

�� � �� � � �� �� � � � �� � � � � � �� � � �� � � �� �� � � �

� � � �� � � �� � �� ��

IR 5 �� ;@ �3 8: � � � � 1 1 8 < A4 < ; � 8 ;� : 3 B � � ! �: 8 < ;R

K R 5 �� ;@ �3 8: � � � F � ; A� �: < ; � 8 � < ; �� B � V ; F �: � ; A A4 8 � 1 : � � � R

�R 5 �� ;@ �3 8: � � � 7< � @ � F ; A �� � 1 ; � 7 7< � �1 2 : � 3 ; �<� Q �: 2 � 8: ; V 7 ;<: S � � Q 3 ; F D ; �� F �

: ; �12 ;<R

Page 13: Co-evolution, Games, and Social Behaviors

A Rigorous Theoretical Framework for Measuring Generalisation of Co-evolutionary Learning 2

'

&

$

%

Introduction: Evolutionary Learning

Very straightforward conceptually!

1. Initialise population, X(t = 1).

2. Evaluate fitness of each population member.

3. Select parents from X(t) based on fitness.

4. Generate offspring from parents to obtain X(t + 1)

5. Repeat steps (2-4) until some termination criteria are met.

Page 14: Co-evolution, Games, and Social Behaviors

A Rigorous Theoretical Framework for Measuring Generalisation of Co-evolutionary Learning 3

'

&

$

%

Two Approaches to Evolutionary Learning

Things might get a little trickier.

Michigan Approach: Holland-style learning classifier systems (LCS), whereeach individual is a rule. The whole population is a complete (learning)system.

Pitt Approach: Each individual is a complete system.

This talk deals only with the Pitt-style evolutionary learning since it is morewidely used.

Page 15: Co-evolution, Games, and Social Behaviors

A Rigorous Theoretical Framework for Measuring Generalisation of Co-evolutionary Learning 4

'

&

$

%

Current Practice in Evolutionary Learning

fitness evaluation and selection

"genetic" operators

mutation

. . . . . .

a population of individuals(learning systems, e.g., ANNs orrule−based systems)

best individual

Pitt Style Evolutionary Learning

crossover

Figure 1: A general framework for Pitt style evolutionary learning.

Page 16: Co-evolution, Games, and Social Behaviors

A Rigorous Theoretical Framework for Measuring Generalisation of Co-evolutionary Learning 5

'

&

$

%

Fitness Evaluation

1. Based on the training error.

2. Based on the training error and complexity (regularisation), e.g.,

1fitness

∝ error + α ∗ complexity

Page 17: Co-evolution, Games, and Social Behaviors

A Rigorous Theoretical Framework for Measuring Generalisation of Co-evolutionary Learning 6

'

&

$

%

What If No Error Function Is Available

1. Or, we don’t know how to obtain the fitness function required to evaluatethe fitness of a population member, e.g.,, if we want to evolve game-playingstrategies.

2. In other words, the exact teacher/target information is unavailable.

Page 18: Co-evolution, Games, and Social Behaviors

A Rigorous Theoretical Framework for Measuring Generalisation of Co-evolutionary Learning 7

'

&

$

%

Well ... We have Co-evolutionary Learning

1. Initialise population, X(t = 1).

2. Evaluate fitness through interactions between population members.

3. Select parents from X(t) based on fitness.

4. Generate offspring from parents to obtain X(t + 1)

5. Repeat steps (2-4) until some termination criteria are met.

Page 19: Co-evolution, Games, and Social Behaviors

�� � �� � � �� �� � � � �� � � � � � �� � � �� � � �� �� � � �

# � ' �#

� �" �� �� � �" ! /� � � � !� � � ! 1

������������ ����

��������

������

��� ���

��� �!"�$#

� � �� � �� � � � � � �� ��� &�

�� & %

� & &

� & �

� &��

& %

& &

& �

�� � � � �� ' � ! �� ! �� � � � �� " ! � �� � � � ( �� � �� � � � !� �� � � �" �� � � &

Page 20: Co-evolution, Games, and Social Behaviors

�� � �� � � �� �� � � � �� � � � � � �� � � �� � � �� �� � � �

� �� � � � � �� � � � � � � � � � � �

IR 5 � � 7 ;< �: �@ ; G ;2 �@ � � 8 < A 1 �� G ; ;@ �3 @ ; F4 < �6 � 7 � 798 3 �: � � � �4 < �� F �6 A: < �: ; D� ; A

Q �: 2 � 8: A 7 ;1 �4 B� � D �� ; V 73 � 1 �: �� F � V ; F �: � ; A A4 8 � 1 : � � � R

K R 1 � � 7 ;< �: �@ ; 7 � 798 3 �: � � � 1 �� G ;@ 8 3 � ;< �G 3 ;: � F ;4 ;1 : �@ ; � � : < 8 F ;< AR

�R 5 �� ;@ �3 8: � � � �< B3 ; �<� � � D6 � B� �: G ; �G 3 ;: � 7< � F 8 1 ; A: < �: ; D� ; A Q 2 � 1 2 D ;� ;< �3 � A ;

Q ;3 3 � D �� � A: 8 � A ; ;� � 7 7 � � ;� : AR

Page 21: Co-evolution, Games, and Social Behaviors

�� � �� � � �� �� � � � �� � � � � � �� � � �� � � �� �� � � �

� � � � � � � �� �� � � � �� � � � � �� � � � � � � �

IR E@ �3 @ � � D � A: < �: ; D B: 2 �: � A 1 � 7 �G 3 ; �4 F ; �3 � � D Q �: 2 �3 3 S � � F A �4 � 7 7 � � ;� : A 1 �� G ;

2 �< FR

K R & � BG ; Q ; 1 �� 8 A ; �: ; �6 �4 � � F�@ � F 8 �3 A � A � A: < �: ; D B A� : 2 �: ; �1 2 � � F�@ � F 8 �3 � A

� � 3 B F ; �3 �� D � � � ; � < : Q � : B 7 ; A �4 � 7 7 � � ;� : AR 0 8 1 2 � � F�@ � F 8 �3 A A2 � 8 3 F G ; � 8 �: ;

; � A B: � ;@ �3 @ ;R

�R C 8: 2 � Q : � ;@ �3 @ ; �: ; �6 �4 � � F�@ � F 8 �3 A Q 2 � �< ; 1 �6 73 ;6 ;� : �4 ; �1 2 �: 2 ;< �

�R <: � � 1 � �3 A 7 ;1 � �: � � � � 7 7 ; �< A: � G ; �� ;� ;1 : �@ ;: ;1 2 � � � 8 ;: � �12 � ;@ ;: 2 �: R

Page 22: Co-evolution, Games, and Social Behaviors

�� � �� � � �� �� � � � �� � � � � � �� � � �� � � �� �� � � �

� �� � � � � � � � � � � � �� � �� � � � � � � � � � � � �� �

IR D ;� ;: � 1 �3 D � < �: 2 6 Q � A 8 A ; F: � ;@ �3 @ ; A: < �: ; D� ; A4 � < 73 � B� � D: 2 ;K "#$ R

K R "6 73 � 1 �: �: � ; A A A2 �< � � D Q � A 8 A ; F: � 4 � <6 F�� ;< ;� : A 7 ;1 � ; A � A 7 ;1 � �3 � A: A /R

�R D �: � � D �3 D � < �: 2 6 Q � A 8 A ; F: � F ;1 � F ; Q 2 � 1 2 A 7 ;1 � ; A A2 � 8 3 F < ; A 7 � � F: � ��

8 � S � � Q � � 7 7 � � ;� : R

�R E V 7 ;< �6 ;� : �3 < ; A8 3 : A A2 � Q ; F: 2 �: � 8 < 6 ;: 2 � F Q � <S ; F@ ;< B Q ;3 3 R

Page 23: Co-evolution, Games, and Social Behaviors

�� � �� � � �� �� � � � �� � � � � � �� � � �� � � �� �� � � �

� �� � � � � �� � �� � � � � � � � � � � � � � �� � �� � �

quality strategythe opponent uses, and uses the best

Finds which high-

counter-strategy.

GateStrategy 3

Strategy 3

Module 1

Module 2

Module 3

counter-strategies

Find best

Strategy 1

Strategy 2

speciated GAspecies fromExpert-level

Opponent uses

strategy. an expert-level

�� �� � �� � ��� �" � � � ��� �� � � ��� � � �� �" �" � % �� �� �" � �� � �� � � ! � �" � � � � � �� � �"� � � �� � �� &

Page 24: Co-evolution, Games, and Social Behaviors

�� � �� � � �� �� � � � �� � � � � � �� � � �� � � �� �� � � �

� � � � � �� � � � � � � � �� �

� � �

0: < �: ; D B � � � A > � ; A @ ;< � D ; 01 � < ;

� � � Q � �: 2 ;<�� A

G ; A: R A< �R � % � �R � J � IR �K K IR JI �

D �: ;R A< �R % � � �R � J � IR JK � IR K � �

� �� � � � ' � ! �� ! � �� �" !� � � �� � !� � � !� � �� � �� � ! �� � � � � � ���� � � � � �" � �� �� � ' � ! ��� � � ���� � ( �� � �� � � � � � �� �� �% � � !� �� � � � � & �� �� � ! �� ! � �� � � ' �� � � �% � ' �� � �� " !&

Page 25: Co-evolution, Games, and Social Behaviors

�� � �� � � �� �� � � � �� � � � � � �� � � �� � � �� �� � � �

� � � � � � �� � � � � � � � � �� � � � � � �� � � � � � � � � � �

IR & �S �� D � � F�@ � F 8 �3 A F�� ;< ;� : O

� � �

��

� ��� � /�� � / � 1� / � 1 1 � 0 � � / � 1 �

Q 2 ;< ;

� � / � 1 � / � � / � 1� � / � 1 1 �� � /�� / � 1� � / � 1 1

� / � 1 � A: 2 ; ;� A ;6 G 3 ; � 8: 798: R

K R 3 3 � � F�@ � F 8 �3 A �< ;3 ; �<� ; F� �� � �� �� �� R

�R � < � F� ;� : F ; A1 ;� : �3 D � < �: 2 6 A = A8 1 2 � A C # =1 �� G ; 8 A ; FR

Page 26: Co-evolution, Games, and Social Behaviors

�� � �� � � �� �� � � � �� � � � � � �� � � �� � � �� �� � � �

� � � � � � �� �� � � � � � � � � � � � � � � � � �� �

IR �2 �: � A: 2 ; �6 7 �1 : � 4 2 � A: � < B3 ;� D: 2 �6 ;6 � < B / � � : 2 ; ;@ �3 8: � � � �

K R �2 �: � A: 2 ; �6 7 �1 : � 4 D < � 8 7 A�� ; � � : 2 ; ;@ �3 8: � � � �

Page 27: Co-evolution, Games, and Social Behaviors

�� � �� � � �� �� � � � �� � � � � � �� � � �� � � �� �� � �

� � � � � �� � � �� � � � � � � �� � �

1

1.5

2

2.5

3

3.5

4

0 100 200 300 400 500 600 700 800 900 1000

Ave

rage

pay

off

Generation

3 prisoners, history 2

Page 28: Co-evolution, Games, and Social Behaviors

�� � �� � � �� �� � � � �� � � � � � �� � � �� � � �� �� � � �

� � � � � �� � � �� � � � � � � �� � �

1.5

2

2.5

3

3.5

4

0 100 200 300 400 500 600 700 800 900 1000

Ave

rage

pay

off

Generation

3 prisoners, history 3

Page 29: Co-evolution, Games, and Social Behaviors

�� � �� � � �� �� � � � �� � � � � � �� � � �� � � �� �� � � �

� � � � � �� � � �� � � � � � � �� � �

2.2

2.4

2.6

2.8

3

3.2

3.4

3.6

3.8

4

0 100 200 300 400 500 600 700 800 900 1000

Ave

rage

pay

off

Generation

3 prisoners, history 4

Page 30: Co-evolution, Games, and Social Behaviors

�� � �� � � �� �� � � � �� � � � � � �� � � �� � � �� �� � � �

� � � � � �� � � �� � � � � �

&�

& �

& &

& %�

� � � � � & � % � � � � � � � & � % � � � �

� ' �� � � � ��� ����

� �" �� �� � �" !

� � � � � �� ! �� � !� �� � �# � � !&� � � �#

�� �� � �� � �� � � � � ( �� � �� � � � !� �� � � �� � � � �� �� � �" � � �� � � ! � � !� � �� � �� � � � &

�" � � � � � � � �� " ! �� � � � � � �� � � � � � � � � � �� �� � �" �� " % �" � � � " �� � !� � � � � � !�� � � % � � �� � � �" & �� � � �� � ��" � � � � �" � ! �� � � �� � � !� � � � � � � � � � � � � � � " % % � � � �' �� !� �� � � � �� �� � �" &

Page 31: Co-evolution, Games, and Social Behaviors

�� � �� � � �� �� � � � �� � � � � � �� � � �� � � �� �� � � �

� � � � � �� � � �� � � � � �

�� & �

�� & �

�� & �

�� & �

&� � � � � & � % � � � � � � � & � % � � � �

� ' �� � � � ��� ����

� �" �� �� � �"

� � � � � �� ! �� � !� �� � � / �� � �� ! 1

� � !& � � � �

�� �� � &� � �� � � � � ( �� � �� � � � !� �� � � �� � � � �� �� � �" � � �� � ! � � !� � � � �� � �� � � � &�� � � �� � ��" � � � � �" � ! �� � � �� � � !� � � � � � � � � � � � � �� " % % � � � �' �� ! � �� � � � �� �� � �" &

Page 32: Co-evolution, Games, and Social Behaviors

�� � �� � � �� �� � � � �� � � � � � �� � � �� � � �� �� � � �

� � � � � �� � � �� � � � � �

��

&�

%�

� �

� �� � � � � & � % � � � � � � � & � % � � � �

� ' �� � � � ��� ����

� �" �� �� � �" !

% � � � � �� ! �� � !� �� � � /� &� �� ! 1� � !& % � � �

% � �

� � �

� � �

�� �� � �� � �� � � � % ( �� � �� � � � !� �� � � �� � � � �� �� � �" " �' �� � � �� � � !& �� � � �� � ��" � � �� �" � ! �� � � �� � � !� � � � � � � � � � � � � �� " % % � � � �' �� ! � �� � � � �� �� � �" & � � % � � �" !� � �� �� � �� � � � ! �� " !� � ' � � �" ' �� � �% � �� �� � � !� � � !� � �� � � ! � �� � � � � � � �� � �" !&

Page 33: Co-evolution, Games, and Social Behaviors

�� � �� � � �� �� � � � �� � � � � � �� � � �� � � �� �� � � �

� � � � � � �� � � �� � � � � �

� �

& %

� �� �

� �

� &� %

� �� � � � � & � % � � � � � � � & � % � � � �

� ' �� � � � ��� ����

� �" �� �� � �"

& � � � � �� ! �� � !� �� � �# � � !& & � � �#

� � �

� � �

% � �

�� �� � %� � �� � � � & ( �� � �� � � � !� �� � � �� � � � �� �� � �" " �' �� � � �� � � !& �� �� �� � ��" � � �� �" � ! �� � � �� � � !� � � � � � � � � � � � � ��� " % % � � � �' �� ! � �� � � � �� �� � �" &

Page 34: Co-evolution, Games, and Social Behaviors

�� � �� � � �� �� � � � �� � � � � � �� � � �� � � �� �� � � �

� � � �� � � � � �� � � � �� � � � � �

� & �

� & � �� & &

� & & �� & �

� & � �� & %

� & % �� � � � � & � % � � � � � � � & � % � � � �

� � � !

� �" �� �� � �"

� � � ! �" � � � � �� � �" ! � � � ��" � % ( �� � �� � � � !� �� � �

� � !& % � � �

�� �� � �� (" �� " ! � �� � � % ( �� � �� � � � !� �� � � �� � �� �� � � � �� �� � �" " �' �� � � �� � � ! �� � �� � � ! % � � �" !� � �� � !� � �� � � � � � � � �� � �" ! � � ' � � �" ' �� � �% & � � � ! � !� � � � ' �� � � �

�� � � �� � � �" � �� � � � � !� �� � � �" �" � ' � � � �" �� � � � � !�� � �" & �� � � ! � � � & � � � �� " !� � �� � �" � ' �� � � � � �� � �� �� � � !�� � �" � � !� �" ' �� � �% � � ��� � �� � � �

� ��� �� � � �

� � &

Page 35: Co-evolution, Games, and Social Behaviors

�� � �� � � �� �� � � � �� � � � � � �� � � �� � � �� �� � � �

� �� � � � � �� � � � � � � � � � � � �

IR & ;6 � < B2 ;3 7 A3 ; �<� � � DR N � � D2 � A: � < B3 ;� D: 2 A ;� 1 � 8 < � D ; 1 � � 7 ;< �: � � � R

K R N �< D ; D < � 8 7 A6 �S ;: 2 ; ;@ �3 8: � � � �4 1 � � 7 ;< �: �@ ; A: < �: ; D� ; A6 � < ; F� � 1 8 3 : R

�� �� �� � =

IR � ; < �< ;3 B 1 �6 6 �: � 8 < A ;3 @ ; A: � � � ; Q � B � < : 2 ; �: 2 ;<R � ; 7< ;4 ;< : � G ; A�6 ;Q 2 ;< ;

�� : 2 ;6 � F F3 ;R

K R � ; �< ;6 � < ;3 �S ;3 B: � 2 �@ ; A2 � <: D �6 ; A Q �: 2 A2 � <: 6 ;6 � < BR

�R � ;6 � B� �: G ; �G 3 ;: � � � : ;< �1 : Q �: 2 ;@ ;< B� � ; �3 : 2 � 8 D2 Q ;S � � Q : 2 ;6 G B� �6 ; A

�< ; 798: �: � � � /R

Page 36: Co-evolution, Games, and Social Behaviors

�� � �� � � �� �� � � � �� � � � � � �� � � �� � � �� �� � � �

� � �� � � � � � � � � �� � � �� � � � � � ��

� � �� 0 �� 0 � � �� � �� �� �� �� � � �� �� �

0 �� �� �� � ���

0 � �� � �� �

�� �� � �� �� � ��� ��� � �� � �� ��� � � � ��� � � � � �� �� � !�" ��$# ! % � � � � � � �� � � �� �� � �� �� � � � � � ! � �� � � � �� �� � �" &

> 2 ; 7 � B� � : � 73 � B ;< � A D�@ ;� G BO

�� � � � �� � � �� � 0 � � � ) /� � � � )� � � 1 / 1

Q 2 ;< ; � � �� F � � �< ;: 2 ; 1 � � 7 ;< �: � � � 3 ;@ ;3 A � 4 : 2 ;: Q � 73 � B ;< A = Q 2 � 1 2 �< ; F� A1 < ;: � � ; F

� � : � 4 � 8 < 1 2 � � 1 ; A �4 1 � � 7 ;< �: � � � R

Page 37: Co-evolution, Games, and Social Behaviors

�� � �� � � �� �� � � � �� � � � � � �� � � �� � � �� �� � �

� � � �� � ��

� "� 4 � <6 �: � � � �G � 8: � D �6 ; ��R ;R =� � : ;< �1 : � � � � A6 � F ; �@ ��3 �G 3 ;: � �3 3 �: 2 ;< AR � � 8

1 �� A ; ; A�6 ;� � ;� A < ; 78: �: � � � Q �: 2 � 8: � � : ;< �1 : �� D Q �: 2 2 �6 P2 ;< F� < ;1 : 3 BR

� �� < A: 1 � � A� F ;< ; F G B ! � Q �S �� F 0� D6 8 � FR

� C 8: � � 3 B � � K � 73 � B ;< "# $ D �6 ; A Q �: 2 : Q � 3 ;@ ;3 A �4 1 � � 7 ;< �: � � � R

� ! � ;@ �3 8: � � � �< B �3 D � < �: 2 6 Q � A 8 A ; F � � : 2 ;� < Q � <S R #3 � B ;< A F� F� �: ;@ �3 @ ; �� : 2 ;� <

1 � A ;R

Page 38: Co-evolution, Games, and Social Behaviors

�� � �� � � �� �� � � � �� � � � � � �� � � �� � � �� �� � � �

� � � � � � � � � � � ��

�2 �: � A: 2 ; �6 7 �1 : � 4 6 8 3 : � 73 ;3 ;@ ;3 A �4 1 � � 7 ;< �: � � � � � < ; 798: �: � � � � � : 2 ; ;@ �3 8: � � �

�4 1 � � 7 ;< �: � � � �

Page 39: Co-evolution, Games, and Social Behaviors

�� � �� � � �� �� � � � �� � � � � � �� � � �� � � �� �� � � �

� � � � � �� � � � � �� � �

� E �12 A: < �: ; D B � A < ; 7< ; A ;� : ; F G B �: Q ��� 3 � B ;< 4 ; ; F� 4 � < Q �< F� ;8 < �3 � ;: Q � <S Q �: 2 K �

2 � F F ;� � � F ; AR

� > 2 ;< ; � A � � ; � 8: 798: � � F� 1 �: � � D: 2 ; 1 8 < < ;� : �1 : � � � : �S ;� G B: 2 ; 73 � B ;<R

� > 2 ;< ; �< ; � @ ; � � 78: AO �I /: 2 ; 73 � B ;<�� A � Q � 7< ;@ � � 8 A6 � @ ; � �K /: 2 ; � 7 7 � � ;� : � A

7< ;@ � � 8 A6 � @ ; � � � / Q 2 ;: 2 ;< : 2 ; � 7 7 � � ;� : ; V 73 � �: ; F: 2 ; 73 � B ;< � � � / Q 2 ;: 2 ;< : 2 ;

73 � B ;< ; V 73 � �: ; F: 2 ; � 7 7 � � ;� : � �� F � J / < ; 798: �: � � � �4 : 2 ; � 7 7 � � ;� : R

Page 40: Co-evolution, Games, and Social Behaviors

�� � �� � � �� �� � � � �� � � � � � �� � � �� � � �� �� � � �

� � � �� � � � � � � � �� �� � � � � �� � � � � � �� �� � �� � � � �� �

1

1.5

2

2.5

3

3.5

4

0 50 100 150 200 250 300

Ave

rage

pay

off

Generation

No reputation, pop 100, 10 iterations per game, 2 levels of coop

’p100it10c2norep.a/pop.av’ u 3:9’p100it10c2norep.b/pop.av’ u 3:9’p100it10c2norep.c/pop.av’ u 3:9’p100it10c2norep.d/pop.av’ u 3:9’p100it10c2norep.e/pop.av’ u 3:9’p100it10c2norep.f/pop.av’ u 3:9’p100it10c2norep.g/pop.av’ u 3:9’p100it10c2norep.h/pop.av’ u 3:9’p100it10c2norep.i/pop.av’ u 3:9’p100it10c2norep.j/pop.av’ u 3:9

�� �� � � � � � � � �" �� � � ! �" � � � �� �� �� � �" ! � !� � � �� � � ! � � ! !�" � �" � � ' �� � � � � � �� �� �

� � � " ��� �� � � ! � � � � �� �� �� � �" !&

Page 41: Co-evolution, Games, and Social Behaviors

�� � �� � � �� �� � � � �� � � � � � �� � � �� � � �� �� � � �

� � � �� � � � � �� � �� �� � � � � �� � � � � � �� �� �� � � � �� �

1

1.5

2

2.5

3

3.5

4

0 50 100 150 200 250 300

Ave

rage

pay

off

Generation

No reputation, pop 100, 5 iterations per game, 2 levels of coop

’norep100it5c2.a/pop.av’ u 3:9’norep100it5c2.b/pop.av’ u 3:9’norep100it5c2.c/pop.av’ u 3:9’norep100it5c2.d/pop.av’ u 3:9’norep100it5c2.e/pop.av’ u 3:9’norep100it5c2.f/pop.av’ u 3:9’norep100it5c2.g/pop.av’ u 3:9’norep100it5c2.h/pop.av’ u 3:9’norep100it5c2.i/pop.av’ u 3:9’norep100it5c2.j/pop.av’ u 3:9

�� �� � �� � � � � � �" �� � � ! �" � � � �� �� �� � �" ! � !� � � �� � � ! � � ! ! �" � �" � � ' �� � � � � � �� �� �&

Page 42: Co-evolution, Games, and Social Behaviors

�� � �� � � �� �� � � � �� � � � � � �� � � �� � � �� �� � � �

� � � �� � �� � � � � � �� � � � � � �� �� � �� � � � �� �

1

1.5

2

2.5

3

3.5

4

0 50 100 150 200 250 300

Ave

rage

pay

off

Generation

Reputation, pop 100, 10 iterations per game, 2 levels of coop

’p100it10c2rep.a/pop.av’ u 3:9’p100it10c2rep.b/pop.av’ u 3:9’p100it10c2rep.c/pop.av’ u 3:9’p100it10c2rep.d/pop.av’ u 3:9’p100it10c2rep.e/pop.av’ u 3:9’p100it10c2rep.f/pop.av’ u 3:9

’p100it10c2rep.g/pop.av’ u 3:9’p100it10c2rep.h/pop.av’ u 3:9’p100it10c2rep.i/pop.av’ u 3:9’p100it10c2rep.j/pop.av’ u 3:9

�� �� � �� � � � � � �" �� � � ! �" � � � �� �� �� � �" ! � !� � � �� � � ! � � ! !�" � �" � � ' �� � � � � � �� �� � �� � � � � � �� � �" � � � � !� � � � �� �� � �" � �� � � � � �� � &

Page 43: Co-evolution, Games, and Social Behaviors

�� � �� � � �� �� � � � �� � � � � � �� � � �� � � �� �� � � �

� � � �� � �� � � � � � �� � � � � � �� �� �� � � � ��

1

1.5

2

2.5

3

3.5

0 50 100 150 200 250 300

Ave

rage

pay

off

Generation

Reputation, pop 100, 1 iteration per game, 2 levels of coop

/home/darwen/src/nipd/runs/p100it1c2rep

�� �� � �� � � � � � �" �� � � ! �" � � �� �� �� � �" � !� � � �� � � ! � � ! ! �" � �" � � ' �� � � � � � �� �� �&

� � � � � � �� � �" � � � � ! �� � !� � �� � �� �� � � � � � � ' � � �� � �" &

Page 44: Co-evolution, Games, and Social Behaviors

�� � �� � � �� �� � � � �� � � � � � �� � � �� � � �� �� � � �

� � � � � � � � � � � � � �� � � � � � �� � �� �� � � � � � �� � � � � � ��

1

1.2

1.4

1.6

1.8

2

2.2

2.4

2.6

2.8

0 50 100 150 200 250 300

Ave

rage

pay

off

Generation

No reputation, pop 150, 20 iterations per game, 8 levels of coop

’p150it20c8norep.b/pop.av’ u 3:9’p150it20c8norep.c/pop.av’ u 3:9’p150it20c8norep.d/pop.av’ u 3:9’p150it20c8norep.e/pop.av’ u 3:9’p150it20c8norep.f/pop.av’ u 3:9’p150it20c8norep.g/pop.av’ u 3:9’p150it20c8norep.h/pop.av’ u 3:9’p150it20c8norep.i/pop.av’ u 3:9’p150it20c8norep.j/pop.av’ u 3:9

�� �� � �� � � � � � �" �� � � !� � �� �� �� � �" ! � � �� � % � � � � � � ! � �� � � � �� �� � �" �� " % % � � ���� � �" % � � �" �� � !&

Page 45: Co-evolution, Games, and Social Behaviors

�� � �� � � �� �� � � � �� � � � � � �� � � �� � � �� �� � � �

� � � �� � �� � � � � � �� � � � � � �� � � �

1

1.5

2

2.5

3

3.5

4

0 50 100 150 200 250 300

Ave

rage

pay

off

Generation

Reputation, pop 150, 20 iterations per game, 8 levels of coop

’p150it20c8rep.b/pop.av’ u 3:9’p150it20c8rep.c/pop.av’ u 3:9’p150it20c8rep.d/pop.av’ u 3:9’p150it20c8rep.e/pop.av’ u 3:9’p150it20c8rep.f/pop.av’ u 3:9’p150it20c8rep.g/pop.av’ u 3:9’p150it20c8rep.h/pop.av’ u 3:9’p150it20c8rep.i/pop.av’ u 3:9’p150it20c8rep.j/pop.av’ u 3:9

�� �� � &� � � � � � �" �� � � ! � � �� �� �� � �" ! � � �� � % � � � � � � ! � � � � � � �� �� � �" & � �� �� � � � �� � �" �� � � � �� �� � �" � ! � � � � �� � � � � �� � &

Page 46: Co-evolution, Games, and Social Behaviors

�� � �� � � �� �� � � � �� � � � � � �� � � �� � � �� �� � � �

� � � �� � �� �� �� �� �� �

E@ ;� Q 2 ;� : 2 ;� 8 6 G ;< �4 1 � � 7 ;< �: � � � 3 ;@ ;3 A � A2 � D2 R

1

1.5

2

2.5

3

3.5

4

0 50 100 150 200 250 300

Ave

rage

pay

off

Generation

Reputation, pop 150, 10 iterations, 64 levels of coop

’p150it10c64rep.a/pop.av’ u 3:9’p150it10c64rep.b/pop.av’ u 3:9’p150it10c64rep.c/pop.av’ u 3:9’p150it10c64rep.d/pop.av’ u 3:9’p150it10c64rep.e/pop.av’ u 3:9’p150it10c64rep.f/pop.av’ u 3:9’p150it10c64rep.g/pop.av’ u 3:9’p150it10c64rep.h/pop.av’ u 3:9’p150it10c64rep.i/pop.av’ u 3:9’p150it10c64rep.j/pop.av’ u 3:9

�� �� � �� � � � � � �" �� � � ! � �� �� �� � �" ! � � �� � �� �� �� �� � � �� � � � �� �� � �" & � �� �� � � � �� � �" �� � � � �� �� � �" � ! !� � � � � � � �� � &

Page 47: Co-evolution, Games, and Social Behaviors

�� � �� � � �� �� � � � �� � � � � � �� � � �� � � �� �� � �

�� � � � � �� �

IR "#$ D �6 ; A �< ; 8 A ;4 8 3 6 � F ;3 A � 4 < ; �3 � 3 �4 ; A�: 8 �: � � � AR

K R ! "#$ D �6 ; A �< ;6 � < ; < ; �3 � A: � 1 �� F D ;� ;< �3 : 2 �� : 2 ;K "#$ � � ;R

�R & 8 3 : � 73 ;3 ;@ ;3 A �4 1 � � 7 ;< �: � � � F� A1 � 8 < � D ; 1 � � 7 ;< �: � � � R

�R M ; 798: �: � � � 1 �� 6 �: � D �: ;: 2 � A �� F ;� 1 � 8 < � D ; 1 � � 7 ;< �: � � � R

JR M ; 798: �: � � � 1 �� ;� 1 � 8 < � D ; 1 � � 7 ;< �: � � � ;@ ;� Q 2 ;� : 2 ; D �6 ; �� F2 � A: � < B3 ;� D: 2 A

�< ; ; V: < ;6 ;3 B A2 � <: R

%R "#$ D �6 ; A Q �: 2 6 8 3 : � 73 ;3 ;@ ;3 A � 4 1 � � 7 ;< �: � � � � A Q ;3 3 � A < ; 798: �: � � � G ;2 �@ ;

� 8 �3 �: �: �@ ;3 B F�� ;< ;� : 3 B4 < �6 1 3 � A A� 1 �3 "#$ D �6 ; AR

'R 5 �� ;@ �3 8: � � � �< B3 ; �<� � � D � A �� ;� ;1: �@ ;3 ; �<� � � D: ;1 2 � � � 8 ;R

Page 48: Co-evolution, Games, and Social Behaviors

�� � �� � � �� �� � � � �� � � � � � �� � � �� � � �� �� � � �

� � � � � � � � � �� � �

IR # �3 �: � 1 �3 A B A: ;6 A P A�: 8 �: � � � A = ;R DR =1 �3 F Q �< =6 � A A�3 ; 1 < � A� A = ;: 1R

K R E1 � � �6 � 1 A B A: ;6 A = ;R DR = 8 A: < �3 � �� 7 ;: < �3 6 �<S ;: =� � : ;<� �: � � � �3 1 � � ; ;6 �<S ;: =

;: 1R�R C� �3 � D� 1 �3 �� F ;1 �3 � D� 1 �3 A B A: ;6 AR

�R 0 � 1 � �3 A B A: ;6 A = ;R DR = � 8 < F� � � ;< : � � � D2 : �

Page 49: Co-evolution, Games, and Social Behaviors

A Rigorous Theoretical Framework for Measuring Generalisation of Co-evolutionary Learning 9

'

&

$

%

A Simple Research Question

• If I invent a wonderful co-evolutionary learning algorithm and use it toco-evolve a really intelligent game-playing strategy (e.g., for chess, carracing, iterated prisoner’s dilemma, or others), how do I know it wouldperform well against a new opponent that it has never seen before?

• Can we say anything at all about the ability (performance) of our co-evolvedsolutions in a new and unseen environment?

• Sounds like generalisation to me.

Page 50: Co-evolution, Games, and Social Behaviors

A Rigorous Theoretical Framework for Measuring Generalisation of Co-evolutionary Learning 15

'

&

$

%

True Generalization Performance

1. Given a co-evolved strategy i, let test strategies j be obtained from strategyspace S. The true generalization performance of strategy i, Gi, is:

Gi = EP1(j)[Gi(j)] =∫

S

Gi(j)P1(j)dj, (1)

where Gi is the expectation of strategy i’s performance against j, Gi(j),w.r.t. distribution P1(j) over strategy space S.

2. A simplified form:

Gi =1M

M∑

j

Gi(j), (2)

which is simply its average performance against all strategies j.

Page 51: Co-evolution, Games, and Social Behaviors

More Information

• S. Y. Chong, P. Tino and X. Yao, “Measuring Generalization Performance in Co-evolutionary Learning,” IEEE Transactions on Evolutionary Computation, 12(4):479-505, August 2008. – IEEE Transactions on Evolutionary Computation

Outstanding 2008 Paper Award (bestowed in 2010).