Aravind Baskar A0136344 ASSIGNMENT - 3 · Aravind Baskar – A0136344 ASSIGNMENT - 3 EE5904R/ME5404 Neural Networks Homework #3 Q1. Function Approximation with RBFN (20 Marks) Solution

Aravind Baskar – A0136344

ASSIGNMENT - 3

EE5904R/ME5404 Neural Networks

Homework #3

Q1. Function Approximation with RBFN (20 Marks)

Solution a): RBFN-Exact Interpolation Method:

Number of hidden units = number of data points

Several trainings are made because noise adding to the function is random, which

provide different solution based on randomly chosen values. The mean squared error for

the five trials and the average error are

Trial 1 Trial 2 Trial 3 Trial 4 Trial 5 Average

0.0866 0.2320 0.1501 0.1962 0.1136 0.1557

MATLAB CODE:

clear all;

x_train = -1:0.05:1;

x_test = -1:0.01:1;

N_train=length(x_train);

N_test=length(x_test);

for trial=1:5

noise = randn(N_train,1);

y_train = 1.2*sin(pi*x_train)-cos(2.4*pi*x_train)+0.3*noise';

y_test = 1.2*sin(pi*x_test)-cos(2.4*pi*x_test);

sigma = 0.1;

w_train= ones(N_train,N_train);

for j=1:N_train,for i=1:N_train

w_train(j,i)= exp(-((x_train(j)-x_train(i))^2)/(2*(sigma)^2));

end,end

w_train = inv(w_train)*y_train';

w_test = zeros(N_test,N_train);

for j=1:N_test,for i=1:N_train

w_test(j,i)= exp(-((x_test(j)-x_train(i))^2)/(2*(sigma)^2));

end,end

net_output = w_test*w_train;

error = y_test' - net_output;

disp(mse(error));

figure(trial);

plot(x_train,y_train,'b^');hold on;

plot(x_test,y_test,'go');hold on;

plot(x_test,net_output,'r');hold off;

legend('training data','test data','RBFN Output','Location','NorthWest');

title(['RBFN - Exact interpolation method; trial-',int2str(trial)]);

print(figure(trial),'-djpeg100',strcat('q1_a_',int2str(trial)))

end

(a) (b)

(c) (d)

(e)

Fig.1. RBFN-Exact Interpolation Method

This shows that RBFN – Exact Interpolation method is over fitting because we

included some noise to the exact solution, which provided inaccurate results with high

error values.

a) Follow the strategy of “Fixed Centers Selected at Random” (as described on

page 37 in the slides of lecture five), randomly select 15 centers among the

sampling points. Determine the weights of the RBFN. Evaluate the

approximation performance of the resulting RBFN using the test set. Compare it

to the result of part a).

Solution (b): Fixed Centers Selected at Random

In Fixed centers method, 15 center points are randomly assigned among the

sampling points i.e., 15 hidden layers.

(a) (b)

(c) (d)

(e)

Fig.2. Fixed Centers Selected at Random: No.of centers =15

The mean squared error for the five trials, when m=15 and the average error are


0.0643 0.7702 2.4387 0.3203 0.0557 1.8246

Comparison:

When comparing exact interpolation and fixed center method, the errors are small

in fixed center method in some trials but in other few trials the error was extremely large.

This is because the interpolation matrix is close to singular or badly scaled. The solution

depends on the number of centers we choose (i.e. the number of hidden units).

MATLAB CODE:

% Fixed Centers selected at Random

clear all; clc;

x_train = -1:0.05:1;

x_train = x_train(:);

x_test = -1:0.01:1;

x_test = x_test(:);



M = 15;

for trial=1:5


y_train = 1.2*sin(pi*x_train)-cos(2.4*pi*x_train)+0.3*noise;


% select M centers randomly from the training points

centerInd = randperm(N_train);

centers = x_train(centerInd(1:M));

dist = minmax(centers);

dmax = (dist(2)-dist(1))^2;

phimat = ones(N_train,M+1);

for i=1:N_train

for j=1:M

phimat(i,j+1)=exp(-M*(x_train(i)-centers(j))^2/dmax);

end

end

weights = inv(phimat'*phimat)*(phimat'*y_train);

phimattest = ones(N_test,M+1);

for i=1:N_test

for j=1:M

phimattest(i,j+1)=exp(-20*(x_test(i)-centers(j))^2/dmax);

end

end

net_output = phimattest*weights;

e = y_test - net_output;

E = mse(e);

disp(E);

figure(trial);




legend('training data','test data','RBFN Output','Location','NorthWest');

title(['RBFN- Fixed Centers Selected at Random; trial-',int2str(trial)]);

end

Solution (c): RBFN - Regularization Factor Method

The regularization factors are varied between 0.001 and 10 to see the effects on

the exact interpolation method. The mean squared errors are

Lamb

da

0.00

1

0.00

5

0.01 0.05 0.1 0.5 1 2 5 10

Error 0.056

6

0.049

8

0.047

2

0.042

8

0.041

2

0.039

7

0.043

1

0.054

9

0.105

3

0.201

2

When comparing RBFN and Regularization method, the error are considerably small

for lesser lambda values. But error shoot ups when lambda value is increased leads to

under fit the curve. And also increase in lambda increases the smoothness of the curve.

Fig.4. Regularization Factor Method

MATLAB CODE:

clear all;

x_train = -1:0.05:1;

x_test = -1:0.01:1;

N=length(x_train);


noise = randn(N,1);



sigma = 0.1;

W_train= ones(N,N);

for j=1:N,for i=1:N

W_train(j,i)= exp(-((x_train(j)-x_train(i))^2)/(2*(sigma)^2));

end,end

lambda = [0.001 0.005 0.01 0.05 0.1 0.5 1 2 5 10];

for trial=1:length(lambda)

w_train = inv(W_train'*W_train+lambda(trial)*eye(N,N))*W_train'*y_train';

W_test = zeros(N_test,N);

for j=1:N_test,for i=1:N

W_test(j,i)= exp(-((x_test(j)-x_train(i))^2)/(2*(sigma)^2));

end,end

net_output = W_test*w_train;


E(trial)=mse(error);

disp(mse(error));

figure(trial);




legend('training','test','RBFN Output','Location','NorthWest');

title(['Regularization Factor method; lambda =',num2str(lambda(trial))]);

end

loglam=log(lambda);

plot(loglam,E,'-r*');

xlabel('Log of Regularization Factor');ylabel('Mean Square Error');

Solution (d): RBFN Fixed center Vs Regularization Factor method

Similar to previous case, error will increase with increase in lambda in regularization

factor method.

Lamb

da

0.00

1

0.00

5

0.01 0.05 0.1 0.5 1 2 5 10

Error 0.027

9

0.037

3

0.048

6

0.088

2

0.114

1

0.22

8

0.292

2

0.370

3

0.469

0

0.539

7

The mean square error values are very less for lambda less than 0.5 when

comparing to Fixed center RBFN method. For lambda greater than 0.5 i.e. increase in

lambda value increases the curve smoothness but it also increase the mean square error

which is greater than Fixed center error.

Fig.5. Regularization Factor method Vs Fixed Center RBFN

MATLAB CODE:

clear all; clc;

x_train = -1:0.05:1;

x_test = -1:0.01:1;



M=15;




index = randperm(N_train);

mu = x_train(index(1:M));

minmaximum = minmax(mu);

dmax = minmaximum(2)-minmaximum(1);

dmax=dmax^2;

W_train= ones(N_train,M);

for j=1:N_train,for i=1:M

W_train(j,i)= exp(-15*((x_train(j)-mu(i))^2)/dmax);

end,end

lambda = [0.001 0.005 0.01 0.05 0.1 0.5 1 2 5 10];

for trial=1:length(lambda)

w_train = inv(W_train'*W_train+lambda(trial)*eye(M,M))*W_train'*y_train';

W_test = zeros(N_test,M);

for j=1:N_test,for i=1:M

W_test(j,i)= exp(-15*((x_test(j)-mu(i))^2)/dmax);

end,end

net_output = W_test*w_train;


E(trial)=mse(error);

disp(mse(error));

figure(trial);




legend('training','test','RBFN Output','Location','NorthWest');

title(['Regularization Factor method; lambda =',num2str(lambda(trial))]);

end

loglam=log(lambda);

plot(loglam,E,'-r*');

xlabel('Log of Regularization Factor');ylabel('Mean Square Error');

Q2. Handwritten alphabet classification using Self-organizing Map (20 Marks)

Solution:

The hand written pictures are loaded as binary image from SOM_database.mat

Learning rate = 0.001

No.of iterations = 1000

Sigma calculated as per size of the image = 7.07

'U' 'U' 'L' 'O' 'O' 'O' 'S' 'S' 'E' 'E'

'U' 'U' 'U' 'L' 'L' 'O' 'S' 'L' 'L' 'L'

'U' 'U' 'U' 'U' 'L' 'L' 'L' 'L' 'L' 'L'

'U' 'U' 'N' 'N' 'N' 'N' 'L' 'L' 'L' 'L'

'M' 'N' 'N' 'A' 'A' 'A' 'A' 'L' 'L' 'L'

'N' 'N' 'N' 'N' 'A' 'A' 'A' 'A' 'R' 'R'

'M' 'M' 'M' 'A' 'A' 'A' 'A' 'A' 'R' 'R'

'M' 'M' 'M' 'A' 'A' 'A' 'A' 'A' 'R' 'R'

Since SOM randomly initialized the values keeps on changes in every run.

1) For each neuron in the trained SOM, show the weights on a 10x10 map to visualize

the result of the training process and make comments about them, if any. A simple

way would be to reshape the weight of a neuron into a 20x16 matrix with double

precision (since that is the dimension of the training image) and display it as an

image. As seen, the weights look almost same as the label obtained in the previous

part. Since learning rate = 0.1.

2) Use the generated SOM to classify the test images (found in test_data). The

classification can be done in the following fashion. Apply a test sample to SOM, and

determine the winner neuron. Then label the test sample with the one

corresponding to the winner neuron (note that labels of all the neurons in the SOM

have already been determined in the first step). Compare the label resulting from

above classification scheme to the true label of the test sample, and calculate the

recognition rate for the whole test set.

The SOM is tested with the test images (found in test_data). The winner solution

is determined by applying the test sample to SOM then test sample is labelled with

corresponding winner neuron. Several trails are done because the initial weights are

chosen random. At each trial gives different performance results because the values are

assigned randomly. The average of five trials are taken and the performance in percentage

given below


72.22 74.44 75.556 70.665 71.33 73.293

'U' 'M' 'A' 'L' 'U' 'N' 'M' 'M' 'N' 'A' 'E' 'E'

'E' 'E' 'E' 'E' 'L' 'E' 'E' 'E' 'U' 'U' 'U' 'U'

'U' 'U' 'U' 'R' 'U' 'U' 'L' 'R' 'N' 'N' 'R' 'A'

'R' 'R' 'R' 'A' 'A' 'A' 'A' 'A' 'A' 'A' 'A' 'N'

'A' 'A' 'L' 'L' 'L' 'U' 'U' 'L' 'L' 'U' 'U' 'L'

'M' 'N' 'M' 'M' 'M' 'M' 'M' 'A' 'M' 'M' 'O' 'O'

'O' 'O' 'L' 'L' 'O' 'O' 'O' 'O' 'S' 'S' 'S' 'S'

'S' 'S' 'S' 'S' 'L' 'S'

When epochs = 1000; size=10x10; Learning rate=0.1; Recognition Rate = 72.22%

Testing of SOM for other design parameters

The parameters that can be varied and played around are

a) Learning rate

b) Size of the lattice

c) Sigma

d) Width

e) Number of iterations

The initial value of sigma depends on the size of the lattice as per the formula

given. So we concentrate on the size and no.of iteration.

Size of lattice

When we increase the size of lattice from 10x10 to 20x20, the accuracy rate also

increased but the time consumption for learning also increased when comparing to lower

size. Thus increasing in size also helps in improving the performance. But there is a trade-

off between time for computation and the performance.

Iterations:

Increase in no.of iteration increase the time conception. But the accuracy of

prediction is considerably increased. Obtained label values for 15000 iteration. The weights

here almost completely comply with the labels obtained. Also the accuracy obtained here

is 82.22 % which is higher compared to the previous part. Thus increasing the iterations

definitely helps in improving the performance. But there is a trade-off between time for

computation and the performance.

'N' 'A' 'L' 'R' 'R' 'E' 'E' 'E' 'S' 'S'

'N' 'A' 'R' 'R' 'E' 'E' 'E' 'S' 'S' 'S'

'N' 'N' 'A' 'R' 'E' 'E' 'E' 'O' 'O' 'S'

'M' 'M' 'M' 'M' 'R' 'R' 'R' 'O' 'O' 'O'

'M' 'M' 'N' 'N' 'M' 'R' 'O' 'O' 'O' 'O'

'N' 'N' 'M' 'M' 'U' 'L' 'L' 'L' 'O' 'O'

'N' 'M' 'M' 'M' 'U' 'L' 'L' 'L' 'U' 'U'

'M' 'M' 'M' 'M' 'U' 'L' 'L' 'L' 'U' 'U'

The corresponding weights can be visualized as

Documents

Aravind Baskar A0136344 ASSIGNMENT - 3 · Aravind Baskar – A0136344 ASSIGNMENT - 3 EE5904R/ME5404 Neural Networks Homework #3 Q1. Function Approximation with RBFN (20 Marks) Solution