Soft computing

Lecture 7

Multi-Layer perceptrons

Why hidden layer is needed

Problem of XOR for simple perceptron

X2

X1

(0,1)

(0,0)(1,0)

(1,1)Class 1

Class 1

Class 2

Class 2

In this case it is not possible to draw descriminant line

Minimization of error

Main algorithm of training

Kinds of sigmoid used in perceptrons

Exponential

Rational

Hyperbolic tangent

Formulas for error back propagation algorithm

Modification of weights of synapsesof jth neuron connected with ith ones,xj – state of jth neuron (output)

For output layer

For hidden layersk – number of neuron in next layerconnected with jth neuron

(1)

(2)

(3)

(2), (1)

(1)

(3), (1)

(1)

Example of implementation TNN=Class(TObject) public State:integer; N,NR,NOut,NH:integer; a:real; Step:real; NL:integer; // ъюы-тю шЄхЁрЎшщ яЁш юсєўхэшш S1:array[1..10000] of integer; S2:array[1..200] of real; S3:array[1..5] of real; G3:array[1..5] of real; LX,LY:array[1..10000] of integer; W1:array[1..10000,1..200] of real; W2:array[1..200,1..5] of real; W1n:array[1..10000,1..200] of real; W2n:array[1..200,1..5] of real; SymOut:array[1..5] of string[32]; procedure FormStr; procedure Learn; procedure Work; procedure Neuron(i,j:integer); end;

Procedure of simulation of neuron; procedure TNN.Neuron(i,j:integer);var k:integer; Sum:real;begincase i of 1: begin if Form1.PaintBox1.Canvas.Pixels[LX[j],LY[j]]= clRed then S1[j]:=1 else S1[j]:=0; end; 2: begin Sum:=0.0; for k:=1 to NR do Sum:=Sum + S1[k]*W1[k,j]; if Sum> 0 then S2[j]:=Sum/(abs(Sum)+Net.a) else S2[j]:=0;end; 3: begin Sum:=0.0; for k:=1 to NH do Sum:=Sum + S2[k]*W2[k,j]; if Sum> 0 then S3[j]:=Sum/(abs(Sum)+Net.a) else S3[j]:=0; end; end;end;

Fragment of procedure of learning For i:=1 to NR do for j:=1 to NH do begin S:=0; for k:=1 to NOut do begin if (S3[k]>0) and (S3[k]<1) then D:=S3[k]*(1-S3[k]) else D:=1; W2n[j,k]:=W2[j,k]+Step*S2[j]*(G3[k]-S3[k])*D; S:=S+D*(G3[k]-S3[k])*W2[j,k] end; if (S2[j]>0) and (S2[j]<1) then D:=S2[j]*(1-S2[j]) else D:=1; S:=S*D; W1n[i,j]:=W1[i,j]+Step*S*S1[i]; end;end;

Generalization

Some of the test data are now misclassified. The problem is that the network, with two hidden units, now has too much freedom and has fitted a decision surface to the training data which follows its intricacies in pattern space without extracting the underlying trends.

Overfitting

Local minima

Two tasks solved by MLP

• Classification (recognition)– Usually binary outputs

• Regression (approximation)– Analog outputs

Theorem of Kolmogorov

“Any continuous function from input to output can be implemented in a three-layer net, given sufficient number of hidden units nH, proper nonlinearities, and weights.”

Advantages and disadvantages of MLP with back propagation

• Advantages:– Guarantee of possibility of solving of tasks

• Disadvantages:– Low speed of learning– Possibility of overfitting– Impossible to relearning– Selection of structure needed for solving of

concrete task is unknown

Increase of speed of learning

• Preliminary processing of features before getting to inputs of percepton

• Dynamical step of learning (in begin one is large, than one is decreasing)

• Using of second derivative in formulas for modification of weights

• Using hardware implementation

Fight against of overfitting

• Don’t select too small error for learning or too large number of iteration

Choice of structure

• Using of constructive learning algorithms– Deleting of nodes (neurons) and links

corresponding to one– Appending new neurons if it is needed

• Using of genetic algorithms for selection of suboptimal structure

Impossible to relearning

• Using of constructive learning algorithms– Deleting of nodes (neurons) and links

corresponding to one– Appending new neurons if it is needed