ELeaRNT: Evolutionary Learning of Rich Neural Network Topologies Authors: Slobodan Miletic 3078/2010 [email protected] Nikola Jovanovic 3077/2010 [email protected]

ELeaRNT:ELeaRNT: Evolutionary Learning of Evolutionary Learning of

Rich Neural Network Rich Neural Network TopologiesTopologies

Authors:Authors:

Slobodan Miletic 3078/2010 Slobodan Miletic 3078/2010 [email protected]@gmail.com

Nikola Jovanovic 3077/2010 Nikola Jovanovic 3077/2010 [email protected][email protected]

IntroductionIntroduction

Genetic algorithmGenetic algorithm mimics natural evolutionmimics natural evolution candidate solutioncandidate solution mutationmutation

Neural networkNeural network based on biological neuronsbased on biological neurons network consists of neurons grouped in network consists of neurons grouped in

layerslayers

2/16Slobodan Miletić [email protected] Jovanović [email protected]

Problem definition Problem definition

Current design methods are manual Current design methods are manual and inefficient and inefficient

Hard to define number of neurons and Hard to define number of neurons and connectionsconnections

No automated design methodNo automated design method

for specific optimal topologyfor specific optimal topology


Problem importance Problem importance

Neural networks have a large use Neural networks have a large use areaarea

Creating new neural networksCreating new neural networks

takes money and timetakes money and time


Problem trendProblem trend

Computers are getting more Computers are getting more powerful powerful

New neural network uses are found New neural network uses are found

If not solved, this problem would If not solved, this problem would slow down the evolution of neural slow down the evolution of neural networksnetworks


Existing solutionsExisting solutions

Trial & ErrorTrial & Error Manual algorithmManual algorithm Few parameters for optimizationFew parameters for optimization Long, costly, and not very efficientLong, costly, and not very efficient



Destructive AlgorithmDestructive Algorithm Starts with very big networksStarts with very big networks Gets results by pruning the initial Gets results by pruning the initial

networknetwork A lot of time is spent on unnecessary A lot of time is spent on unnecessary

training training

of big networksof big networks



Constructive AlgorithmConstructive Algorithm Starts with a small neural networkStarts with a small neural network Adds nodes and connectionsAdds nodes and connections Uses input/error rate to form new nodesUses input/error rate to form new nodes Can miss optimal solutionCan miss optimal solution


Proposed solutionProposed solution What’s better?What’s better?

General algorithm – fitness function change General algorithm – fitness function change

enables generation enables generation of different neural networks typesof different neural networks types

Created neural networks outperform Created neural networks outperform neural network models designed by handneural network models designed by hand

Besides the best network, it creates several Besides the best network, it creates several

suboptimal networks suboptimal networks that can that can also be used as a solutionsalso be used as a solutions


Proposed solution Proposed solution

What’s new?What’s new?

No similar general algorithm on the No similar general algorithm on the

marketmarket

Original set of genetic algorithm Original set of genetic algorithm

mutationsmutations


Proposed solution Proposed solution

What’s its future?What’s its future? Computer power and parallelism are Computer power and parallelism are

increasing increasing

which allows more complex neural which allows more complex neural networksnetworks

Automated neural network generation Automated neural network generation algorithmalgorithm

like this will allow creation like this will allow creation

of complex neural networksof complex neural networks


f2

g5

p2

h7

f3

r2

No change

Add node

a4

b3

c7

d2

q5

g5 h

7

f3

a4

b3

d2

f2

c7

p2

r2

f2

p2

r2

g5 h

7

f3

c7a

4 b3

d2

f3

g5

h7

r2

a4 b

3

c7

f2

p2

d2

Drop node

Activation Function

h7

r2

a4 b

3

c7 f

2f5

p2

d2

f2

f2

f2

f2

f2

f2

f2

f2

g5

p2

h7

f3

r2

f2

a3

a3

f2

f2

g2

h2

f1

a3

b4

c5

c3

c4

c5

f2

d2

a3

q3

r2

h1

h1

h1

h1

f3

h2

f3

g3

g3

h2

a4 b

3

c7 d

2

Solution detailsSolution details

Number of Neurons

a4 b

1

c7 d

2

a4 b

3

c7 d

2

3

f2

g5

p2

h7

f3

r2

Drop Connection

a4 b

3

c7 d

2

Add Connection

f2

c7

p2

r2


1 point Crossover2 point Crossover

a4

b3

c7

d2

q5

Solution detailsSolution detailsh7

r2

a4 b

3

c7 f

2f5

p2

d2

g5 h

7

f3

a4

b3

d2

a4 b

1

c7 d

2

f2

g5

p2

h7

f3

r2

a4 b

3

c7 d

2

f2

c7

p2

r2

a4

b3

d2

q5

Add node

No change

a4

b3

c7

d2

q5

Drop connection

a4

b3

c7

d2

q5

h7

f3

d2

a4

g5

a4

b3

b3

c7

d2

q5

a4

c7

d2

g5

h7

b3

f3

a4

d2

b3

q5


1 point Crossover1 point Crossover2 point Crossover

f3

a4

d2

b3

q5

Solution detailsSolution details

a4

b3

c7

d2

q5

a4

b3

c7

d2

q5

a4

c7

d2

g5

h7

b3

f3

a4

d2

b3

q5

h7

f3

d2

a4

g5

a4

b3

b3

c7

d2

q5

a4

b3

d2

q5

g b

Activation function

Activation function

f15 h

7

f3

d2

a4

b3

c7

f3

a4

d2

a3

q5

5

Number of

Neurons

g

10

h7

f3

d2

a4

b3

c7 a

1 f3b

3 q5

h7

f3

d2

a4

b3

c7 a

4d2

g5

Drop node

f3b

3 q5

h7

f3

d2

a4

b3

c7

Add connectio

n

a4

d2

g5

Drop connection

g5 h

7

f3

d2

a4

b3

c7

Add node

f3

a4

d2

b3

q5

Add connection


2 point Crossover

ConclusionConclusion

New way to create neural networksNew way to create neural networks

Results fully comparable Results fully comparable

with hand designed networkswith hand designed networks

Space for further improvementSpace for further improvement

More then one created network can More then one created network can

be used be used 15/16

Slobodan Miletić [email protected] Jovanović [email protected]

QUESTIONS?QUESTIONS?


Documents

ELeaRNT: Evolutionary Learning of Rich Neural Network Topologies Authors: Slobodan Miletic 3078/2010 [email protected] Nikola Jovanovic 3077/2010 [email protected]