View
215
Download
1
Category
Tags:
Preview:
Citation preview
ELeaRNT:ELeaRNT: Evolutionary Learning of Evolutionary Learning of
Rich Neural Network Rich Neural Network TopologiesTopologies
Authors:Authors:
Slobodan Miletic 3078/2010 Slobodan Miletic 3078/2010 sloba10@gmail.comsloba10@gmail.com
Nikola Jovanovic 3077/2010 Nikola Jovanovic 3077/2010 nikolaj_ub@yahoo.co.uknikolaj_ub@yahoo.co.uk
IntroductionIntroduction
Genetic algorithmGenetic algorithm mimics natural evolutionmimics natural evolution candidate solutioncandidate solution mutationmutation
Neural networkNeural network based on biological neuronsbased on biological neurons network consists of neurons grouped in network consists of neurons grouped in
layerslayers
2/16Slobodan Miletić sloba10@gmail.comNikola Jovanović nikolaj_ub@yahoo.co.uk
Problem definition Problem definition
Current design methods are manual Current design methods are manual and inefficient and inefficient
Hard to define number of neurons and Hard to define number of neurons and connectionsconnections
No automated design methodNo automated design method
for specific optimal topologyfor specific optimal topology
3/16Slobodan Miletić sloba10@gmail.comNikola Jovanović nikolaj_ub@yahoo.co.uk
Problem importance Problem importance
Neural networks have a large use Neural networks have a large use areaarea
Creating new neural networksCreating new neural networks
takes money and timetakes money and time
4/16Slobodan Miletić sloba10@gmail.comNikola Jovanović nikolaj_ub@yahoo.co.uk
Problem trendProblem trend
Computers are getting more Computers are getting more powerful powerful
New neural network uses are found New neural network uses are found
If not solved, this problem would If not solved, this problem would slow down the evolution of neural slow down the evolution of neural networksnetworks
5/16Slobodan Miletić sloba10@gmail.comNikola Jovanović nikolaj_ub@yahoo.co.uk
Existing solutionsExisting solutions
Trial & ErrorTrial & Error Manual algorithmManual algorithm Few parameters for optimizationFew parameters for optimization Long, costly, and not very efficientLong, costly, and not very efficient
6/16Slobodan Miletić sloba10@gmail.comNikola Jovanović nikolaj_ub@yahoo.co.uk
Existing solutionsExisting solutions
Destructive AlgorithmDestructive Algorithm Starts with very big networksStarts with very big networks Gets results by pruning the initial Gets results by pruning the initial
networknetwork A lot of time is spent on unnecessary A lot of time is spent on unnecessary
training training
of big networksof big networks
7/16Slobodan Miletić sloba10@gmail.comNikola Jovanović nikolaj_ub@yahoo.co.uk
Existing solutionsExisting solutions
Constructive AlgorithmConstructive Algorithm Starts with a small neural networkStarts with a small neural network Adds nodes and connectionsAdds nodes and connections Uses input/error rate to form new nodesUses input/error rate to form new nodes Can miss optimal solutionCan miss optimal solution
8/16Slobodan Miletić sloba10@gmail.comNikola Jovanović nikolaj_ub@yahoo.co.uk
Proposed solutionProposed solution What’s better?What’s better?
General algorithm – fitness function change General algorithm – fitness function change
enables generation enables generation of different neural networks typesof different neural networks types
Created neural networks outperform Created neural networks outperform neural network models designed by handneural network models designed by hand
Besides the best network, it creates several Besides the best network, it creates several
suboptimal networks suboptimal networks that can that can also be used as a solutionsalso be used as a solutions
9/16Slobodan Miletić sloba10@gmail.comNikola Jovanović nikolaj_ub@yahoo.co.uk
Proposed solution Proposed solution
What’s new?What’s new?
No similar general algorithm on the No similar general algorithm on the
marketmarket
Original set of genetic algorithm Original set of genetic algorithm
mutationsmutations
10/16Slobodan Miletić sloba10@gmail.comNikola Jovanović nikolaj_ub@yahoo.co.uk
Proposed solution Proposed solution
What’s its future?What’s its future? Computer power and parallelism are Computer power and parallelism are
increasing increasing
which allows more complex neural which allows more complex neural networksnetworks
Automated neural network generation Automated neural network generation algorithmalgorithm
like this will allow creation like this will allow creation
of complex neural networksof complex neural networks
11/16Slobodan Miletić sloba10@gmail.comNikola Jovanović nikolaj_ub@yahoo.co.uk
f2
g5
p2
h7
f3
r2
No change
Add node
a4
b3
c7
d2
q5
g5 h
7
f3
a4
b3
d2
f2
c7
p2
r2
f2
p2
r2
g5 h
7
f3
c7a
4 b3
d2
f3
g5
h7
r2
a4 b
3
c7
f2
p2
d2
Drop node
Activation Function
h7
r2
a4 b
3
c7 f
2f5
p2
d2
f2
f2
f2
f2
f2
f2
f2
f2
g5
p2
h7
f3
r2
f2
a3
a3
f2
f2
g2
h2
f1
a3
b4
c5
c3
c4
c5
f2
d2
a3
q3
r2
h1
h1
h1
h1
f3
h2
f3
g3
g3
h2
a4 b
3
c7 d
2
Solution detailsSolution details
Number of Neurons
a4 b
1
c7 d
2
a4 b
3
c7 d
2
3
f2
g5
p2
h7
f3
r2
Drop Connection
a4 b
3
c7 d
2
Add Connection
f2
c7
p2
r2
12/16Slobodan Miletić sloba10@gmail.comNikola Jovanović nikolaj_ub@yahoo.co.uk
1 point Crossover2 point Crossover
a4
b3
c7
d2
q5
Solution detailsSolution detailsh7
r2
a4 b
3
c7 f
2f5
p2
d2
g5 h
7
f3
a4
b3
d2
a4 b
1
c7 d
2
f2
g5
p2
h7
f3
r2
a4 b
3
c7 d
2
f2
c7
p2
r2
a4
b3
d2
q5
Add node
No change
a4
b3
c7
d2
q5
Drop connection
a4
b3
c7
d2
q5
h7
f3
d2
a4
g5
a4
b3
b3
c7
d2
q5
a4
c7
d2
g5
h7
b3
f3
a4
d2
b3
q5
13/16Slobodan Miletić sloba10@gmail.comNikola Jovanović nikolaj_ub@yahoo.co.uk
1 point Crossover1 point Crossover2 point Crossover
f3
a4
d2
b3
q5
Solution detailsSolution details
a4
b3
c7
d2
q5
a4
b3
c7
d2
q5
a4
c7
d2
g5
h7
b3
f3
a4
d2
b3
q5
h7
f3
d2
a4
g5
a4
b3
b3
c7
d2
q5
a4
b3
d2
q5
g b
Activation function
Activation function
f15 h
7
f3
d2
a4
b3
c7
f3
a4
d2
a3
q5
5
Number of
Neurons
g
10
h7
f3
d2
a4
b3
c7 a
1 f3b
3 q5
h7
f3
d2
a4
b3
c7 a
4d2
g5
Drop node
f3b
3 q5
h7
f3
d2
a4
b3
c7
Add connectio
n
a4
d2
g5
Drop connection
g5 h
7
f3
d2
a4
b3
c7
Add node
f3
a4
d2
b3
q5
Add connection
14/16Slobodan Miletić sloba10@gmail.comNikola Jovanović nikolaj_ub@yahoo.co.uk
2 point Crossover
ConclusionConclusion
New way to create neural networksNew way to create neural networks
Results fully comparable Results fully comparable
with hand designed networkswith hand designed networks
Space for further improvementSpace for further improvement
More then one created network can More then one created network can
be used be used 15/16
Slobodan Miletić sloba10@gmail.comNikola Jovanović nikolaj_ub@yahoo.co.uk
QUESTIONS?QUESTIONS?
16/16Slobodan Miletić sloba10@gmail.comNikola Jovanović nikolaj_ub@yahoo.co.uk
Recommended