EE-4253 RTSP PROJECT

BLIND SOURCE SEPARATION USING

ICA & SFA

By

Naveena Vemulapalli

Ali Aghdaei

I. INTRODUCTION

Blind Source Separation also called as the Blind Signal Separation is the method used to

separate the set of source signals from the mixed signals. Blind source separation has many

methods but in this project we mainly concentrate on two methods that is Independent

Component Analysis (ICA) and the Signal Fraction Analysis (SFA). In general when we have

a large amount of data for analysis it becomes very important to reduce this data, scientists in

many fields collect the measurements and observe different techniques to describe observe and

analyze the data that is collected. Selecting an appropriate technique is very important to

segregate this data

Some of the popular data analysis techniques for the data compression is Independent

Component Analysis (ICA), Principal Component Analysis (PCA) and the Signal Fraction

Analysis (SFA). Independent Component Analysis is a technique that is for the data sets that

are Non-Gaussian and are independent to each other. Principal Component Analysis is based

on the orthogonal transformation of the signals. It converts the correlated signals into

uncorrelated signal to the correlated signal.

II. INDEPENDENT COMPONENT ANALYSIS

Independent Component Analysis is mainly based on the assumption of having non Gaussian

signals which are independent of each other. These two conditions should always be true to

provide a good results. Although having some disadvantages this method is one of the best

method in the Blind Source Separation. This method is used to separate the signals which are

mixed together with the unknown mixing matrix. The algorithm for the Independent

Component Analysis is given as follows.

Let 𝑆𝑇 = [𝑠1, 𝑠2, 𝑠3, … … … … … . . 𝑠𝑛] be a source vector and 𝑋𝑇 = [𝑥1, 𝑥2, … … … . 𝑥𝑛] be the

source mixture random vector. The mixing matrix is considered to be A. using the matrix

notation we have equation (1)

𝑋 = 𝐴𝑆 (1)

In this above equation we know only X. we have no information about the A matrix and the S

matrix. We know that the elements of S are non-gaussian and statistically independent.

Therefore let us denote the linear combination of elements of X by y as shown in equation (2)

𝑦 = 𝑊𝑇𝑋 (2)

From equation 1 we can write the equation 3

𝑦 = 𝑊𝑇𝑋 = 𝑊𝑇𝐴𝑆 = 𝑍𝑇𝑆 (3)

The main matrix that is to be determined is the matrix W. From the equation (3) we know that

y is a linear combination of the original independent non-gaussian signals. According the

central limit theorem we know that the sum of all the non-gaussian independent signals leads

to the gaussian signal. Hence we should measure the amount of non-Gaussianity to apply on y

to find the separated independent sources. One of the popular method to measure the non-

Gaussianity is called the Kurtosis. In the Independent Component Analysis we need to have

several estimations to estimate the maximum likelihood which leads to the estimation of higher

order cumulants. This is called the kurtosis. The kurtosis of y is defined as shown in the

equation (4)

𝑘𝑢𝑟𝑡(𝑦) = 𝐸(𝑦4) − 3[𝐸(𝑦2)]2 (4)

Let us assume that y has a unity variance then the equation (4) becomes

𝑘𝑢𝑟𝑡(𝑦) = 𝐸(𝑦4) − 3 (5)

This kurtosis generally has three kinds of values which are shown below

Kurtosis>0 Sup-Gaussian

Kurtosis=0 Gaussian

Kurtosis >0 Sup-Gaussian

For the gaussian random variables the shape of the random variable is bell shape. For the sub-

Gaussian random variables the shape is flatter than the Gaussian. For the Sup-Gaussian the

shape is sharper and has a long tail. Applying fixed point algorithm to optimize the kurtosis

determines the matrix W through which we can determine the mixing matrix A.ICA also solves

the non-linear optimization problem which needs more complicated algorithms.

III. SIGNAL FRACTION ANALYSIS

In this section we intend to describe SFA algorithm which is one of Blind Source Separation

techniques. We assume that received signal is composed of two additive components. Let, X;

n > p be a multivariate data set, an addition of the desired data (signal) S, and undesired signal

(noise) N. Therefore, the observed data can be represented in equation (6)

𝑋 = 𝑆 + 𝑁 (6)

From the above equation, we conclude that:

𝑋𝑇𝑋 = 𝑆𝑇𝑆 + 𝑁𝑇𝑁 + 𝑆𝑇𝑁 + 𝑁𝑇𝑆 (7)

Then we define the signal to noise ratio (SNR), the ration of the variance of the signal to

variance of the noise is given in equation (8)

𝑆𝑁𝑅 = 𝑣𝑎𝑟 (𝑆)

𝑣𝑎𝑟 (𝑁) (8)

In addition, noise fraction is defined as the ratio of noise variance to the total variance. That is

given in equation (9),

𝑢 = 𝑣𝑎𝑟 (𝑁)

𝑣𝑎𝑟 (𝑋) (9)

The maximum signal fraction transformation is a linear transformation defined in equation (10)

𝑌 = 𝑋𝑎 = 𝑆𝑎 + 𝑁𝑎 (10)

Where a is a p by 1 vector. Here, the problem is to determine a in order to maximize the signal

to noise ratio for Y. The SNR for Y is represented by equation (11)

𝑆𝑁𝑅 = 𝑣𝑎𝑟 (𝑆𝑎)

𝑣𝑎𝑟 (𝑁𝑎) (12)

Thus,

𝑆𝑁𝑅 = 𝐸(𝑎𝑇𝑆𝑇𝑆𝑎) − 𝐸(𝑎𝑇𝑆𝑇)𝐸(𝑠𝑎)

𝐸(𝑎𝑇𝑁𝑇𝑁𝑎) − 𝐸(𝑎𝑇𝑁𝑇)𝐸(𝑁𝑎)

Considering this condition that all the data is zero mean, therefore:

𝑆𝑁𝑅 = 𝐸(𝑎𝑇𝑆𝑇𝑆𝑎)

𝐸(𝑎𝑇𝑁𝑇𝑁𝑎) (13)

From the equation (13) we get equation 14

𝑎𝑇𝑋𝑇𝑋𝑎 = 𝑎𝑇𝑆𝑇𝑆𝑎 + 𝑎𝑇𝑁𝑇𝑁𝑎 + 𝑎𝑇𝑆𝑇𝑁𝑎 + 𝑎𝑇𝑁𝑇𝑆𝑎 (14)

Therefore we will get equation 15

𝑆𝑁𝑅 = 𝐸(𝑎𝑇𝑋𝑇𝑋𝑎) − 𝐸(𝑎𝑇𝑆𝑇𝑁𝑎) − 𝐸(𝑎𝑇𝑁𝑇𝑆𝑎 − 𝐸(𝑎𝑇𝑁𝑇𝑁𝑎)

𝐸(𝑎𝑇𝑁𝑇𝑁𝑎) (15)

We assume that signal and noise are independent of each other, thus the second term in the

nominator of the above equation will be zero. The equation will be simplified to equation (16)

𝑆𝑁𝑅 = 𝐸(𝑎𝑇𝑋𝑇𝑋𝑎)

𝐸(𝑎𝑇𝑁𝑇𝑁𝑎) -1 (16)

Since we don't have access to the statistical information, we must use the estimation of the

covariance matrices to compute the SNR. Assuming n large, we can estimate the statistical

means via their sample mean. In this case, we have:

𝑆𝑁𝑅 = (𝑎𝑇𝑋𝑇𝑋𝑎)

(𝑎𝑇𝑁𝑇𝑁𝑎) -1

The SFA technique is based on maximizing the signal to noise ratio powers. Therefore, to

maximize the SNR we have to maximize the term

𝐷(𝑎) = (𝑎𝑇𝑋𝑇𝑋𝑎)

(𝑎𝑇𝑁𝑇𝑁𝑎) -1

This optimization problem is solved by taking the derivative of D(a)with respect to a and

setting it to zero

𝜕𝐷(𝑎)

𝜕𝑎=

2𝑋𝑇𝑋𝑎(𝑎𝑇𝑁𝑇𝑁𝑎) − 2𝑎𝑇𝑁𝑇𝑁𝑎(𝑎𝑇𝑋𝑇𝑋𝑎)

𝑎𝑇𝑁𝑇𝑁𝑎 (17)

XTXa(𝑎𝑇𝑁𝑇𝑁𝑎) = 𝑁𝑇𝑁𝑎(𝑎𝑇XTXa)

Here we define 𝛼 and 𝛽 as:

𝛼 = 𝑎𝑇𝑋𝑇𝑋𝑎

𝛽 = 𝑎𝑇𝑁𝑇𝑁𝑎

Thus we will get equation (18)

β𝑋𝑇𝑋𝑎 = α𝑁𝑇𝑁𝑎 (18) Now, defining

𝜆 =𝛽

𝛼 for 𝛽 ≠ 0

Thus:

𝜆 = 𝑆𝑁𝑅 + 1

And finally we have:

𝑋𝑇𝑋𝑎 = 𝜆𝑁𝑇𝑁𝑎

In order to solve above equation and calculate a we use eig( ) command in MATLAB while a

is the generalized singular vector and 𝜆 is the generalized singular value associated to a that

shows the connection between SFA and GSVD.

Therefore, given matrix X we can recover S by applying SFA under the following

Conditions:

1. n must be large enough (i.e. is a tall matrix) in order to estimate the statistical mean via

the sample mean.

2. E(S) = E (N) = E(X) = 0 (i.e., the columns of S, N and X are assumed to have zero mean).

3. Signals are independent from the noise, (i.e., E(STN) = E(NTS) = 0).

The generalized singular vectors (ai 's) are orthonormal with respect to matrix NTN,

and are orthogonal with respect to matrix XTX. FFA algorithm is shown in the below figure in

a nutshell:

Estimation of noise covariance matrix:

In SFA algorithm noise covariance matrix should be estimated. There are many methods in

order to calculate that, but we are going to introduce differencing method here.

As we explained before, the observed data can be represented by X = S +N. We denote the

difference matrix as dX in which each row i (i.e., dX(i; :)) is the difference between the (i + 1)-

th row of X (i.e., X(i + 1; :)) and the i-th row of X (i.e., X(i; :))).Suppose that we can estimate

the noise covariance using the difference matrix as: 𝑁𝐻𝑁 =𝑑𝑋𝐻𝑑𝑋

2.

Therefore:

Then,

Where 𝑋 ∈ 𝑅𝑛×𝑝 and𝑑𝑋 ∈ 𝑅(𝑛−1)×𝑝. Here we assumed we have the additive noise signal as:

𝑥𝑚(𝑡𝑖) = 𝑠𝑚(𝑡𝑖) + 𝑛𝑚(𝑡𝑖)

Therefore,

𝑥𝑚(𝑡𝑖) − 𝑥𝑚(𝑡𝑖−1) = 𝑠𝑚(𝑡𝑖) − 𝑠𝑚(𝑡𝑖−1) + 𝑛𝑚(𝑡𝑖) − 𝑛𝑚(𝑡𝑖−1)

To simplify, we calculate matrix dX including elements of 𝑥𝑚(𝑡𝑖) − 𝑥𝑚(𝑡𝑖−1) and compute

matrices dS and dN based on elements 𝑠𝑚(𝑡𝑖) − 𝑠𝑚(𝑡𝑖−1) and 𝑛𝑚(𝑡𝑖) − 𝑛𝑚(𝑡𝑖−1) respectively

when m = 1,...p and i = 1,…n.

𝑑𝑋 = 𝑑𝑆 + 𝑑𝑁

Which leads to:

𝑑𝑋𝑇𝑑𝑋 = 𝑑𝑆𝑇𝑑𝑆 + 𝑑𝑁𝑇𝑑𝑁

Here using 𝑑𝑁 = 𝑁𝑡 − 𝑁𝑡−1 , 𝑁𝑡−1𝑇 𝑁𝑡−1 = 𝑁𝑡

𝑇𝑁𝑡 and 𝑁𝑡−1𝑇 𝑁𝑡 = 𝑁𝑡

𝑇𝑁𝑡−1 = 0 , thus:

𝑑𝑋𝑇𝑑𝑋 = 𝑑𝑆𝑇𝑑𝑆 + 2𝑁𝑡𝑇𝑁𝑡

Assuming the sampling rate is high enough and therefore it leads to 𝑑𝑆 ≈ 0

𝑑𝑋𝑇𝑑𝑋 = 2𝑁𝑡𝑇𝑁𝑡

This means that we could use the differencing idea for the observed data and apply it to

estimate the noise covariance matrix.

IV PERFORMANCE EVALUATION:

The performance of the ICA and the SFA techniques is evaluated using the Normalized Mean

Square Error (NMSE). The equation for NMSE is shown in equation (19). NMSE is difference

between the observed value and the predicted value. The performance of the filter is better if

this value is as low as possible.

𝑚𝑠𝑒 =∑ |𝑠(𝑛) − 𝑠(𝑛)̂|

2𝑁𝑛=1

∑ |𝑠(𝑛)|2𝑁𝑛=1

(19)

The second performance metric that is used for the performance evaluation is the correlation.

Correlation is defined as the dependence of the signals on each other. The equation for

correlation is shown in equation (20). The measured signal changes as the original signal

changes.

𝑐𝑜𝑟𝑟 =1

𝑁∑ 𝑠(𝑛)̂ − 𝑠(𝑛) (20)𝑛

𝑛=1

V. SIMULATION RESULTS FOR ICA:

The simulation result for the ICA is shown below. The first signal is the voice signal and the

second signal is the music signal. These signals are selected because the voice and the music

is always independent of each other. For proper reconstruction of the signals the signals should

be independent of each other. The signals are first converted to a .wav file using the software

Audacity. The signals are sampled at the rate of 44100Hz. The length of both the signals is

made equal so as avoid any irregularities. The noise is adding to the signals and here we

consider 2 observed signals.

Figure1: The original Voice and Music Signal

Figure 2: The observed Voice and music signal

Figure 3: The reconstructed Voice and the music signal

SNR

NMSE

MUSIC VOICE S1 S2

52.488 9.1261 0.0040 0.0531

5.2488 0.9126 0.0028 0.0525

0.5248 0.09126 0.0030 0.0432

0.0525 0.00913 0.0030 0.0438

Table 1: Effect of the SNR on NMSE in ICA process

The NMSE for the independent Component Analysis decreases as the SNR increases but only

up to a certain value and then the NMSE increases. This performance is somewhat similar to

the SFA. But here in the ICA the breaking threshold is less than that of SFA. Therefore we can

say that the performance of SFA is better than that of the ICA

VI. SIMULATION RESULTS FOR SFA:

The simulation is done by taking two source signals. The first signal is the voice signal and the

second signal is the music signal. These signals are selected because the voice and the music

is always independent of each other. For proper reconstruction of the signals the signals should

be independent of each other. The signals are first converted to a .wav file using the software

Audacity. The signals are sampled at the rate of 44100Hz. The length of both the signals is

made equal so as avoid any irregularities. The noise is adding to the signals and here we

consider 2 observed signals.

Figure 4: The original Voice and the music signal

Figure 5: The observed voice and music signal

Figure 6: The separated voice and the music signal

EFFECT OF ALPHAS ON THE MSE & CORRELATION:

ALPHAS NMSE CORRELATION

𝜶𝟏 𝜶𝟐 𝜶𝟑 𝜶𝟒 S1(MUSIC) S2(VOICE) S1(MUSIC) S2(VOICE)

1 1 1 1 0.0713 0.0123 -0.0866 877.524

1 7 1 7 0.0713 0.0110 -0.5171 14399

7 1 7 1 0.0715 0.0139 0.0319 157.12

7 1 1 1 0.0715 0.0142 11.697 10.6281

7 1 1 7 0.0715 0.0142 6.812 6.9005

1 7 7 1 0.0715 0.0145 6.8271 -6.9113

1 1 1 7 0.0716 0.0141 12.3663 11.0153

Table 2: Effect of the alphas on NMSE in SFA process

From the table that is shown above we can make some interesting observations based on the

alpha values. The alphas that are shown here are nothing but the unknown mixing matrix. This

matrix is generally random and is unknown. As we need to evaluate the performance based on

these alphas we have taken some deterministic values. As the number of observed signals is

taken as 2 we have considered 4 different kinds of alphas. The Table (2) shows all the different

kinds of alphas and there performance on the mean square error and the correlation. From the

table we can observe that the correlation is very high when the values of

𝜶𝟏,𝜶𝟑 and𝜶𝟐,𝜶𝟒 are equal. When these values are equal the correlation between the signals

is high and they cannot be separated properly. When the reconstruction fails in the voice signal

we get the reconstructed signal as the complete noise signal and the music signal is the

observed signal.

EFFECT OF SNR ON THE NMSE:

SNR

VARIANCE

NMSE

CORRELATION

MUSIC VOICE S1 S2 S1 S2

0.00713 0.00137 10 0.0713 0.0137 -0.3991 1.6019

0.0713 0.0137 1 0.0713 0.0137 2.4484 16.1612

0.713 0.137 0.1 0.0711 0.0136 24.504 136.48

7.13 1.37 0.01 0.0710 0.0138 148.75 186.93

71.3 13.7 0.001 0.0715 0.0142 15.98 13.87

Table 3: Effect of the SNR on NMSE in SFA process

Again the noise is also random in nature. In order to evaluate the effect of noise on the

performance of the reconstruction we have considered some deterministic values of the noise.

Firstly, we know that as the SNR increases the MSE decreases. The Table (3) shows the values

of the NMSE for different values of the noise. From the table we can conclude that the up to

the variance the means square error decreases as the signal to noise ratio increases as the Signal

to Noise Ratio increases. If we decrease the variance beyond 0.1 MSE decreases which means

that we can consider the variance 0.1 as the breakpoint and after this the reconstruction of the

signals fail completely. We cannot separate the source signals at all.

VII. CONCLUSION:

Blind Source Separation method is mainly used to separate the source signals that are mixed

together by an unknown mixing matrix. We can conclude that the Signal Fraction analysis is a

better method than the Independent Component Analysis. They are many reasons to explain

this deduction. Firstly, the observed signal always contains the noise and moreover the noise

is also added during the transmission. As we know that in ICA the components should be non-

gaussian but in the presence of the noise ICA cannot distinguish properly from the noise.

Secondly, SFA algorithm is comparatively easy than the ICA algorithm. As ICA requires

several estimations like estimating the maximum likelihood and also kurtosis to measure the

amount of non-Gaussianity leads to the estimation of higher order cumulants. ICA also solves

the nonlinear optimization problem. SFA just needs to solve the GSVD problem.

Using the MATLAB SFA algorithm can be programmed easily and moreover we can evaluate

the effects of performance of different parameters like the alphas, noise variances on the

reconstructed signal is done above. We have clearly observed the cases where the

reconstruction fails and how the separated source signal is changing based on the SFA

algorithm.

VIII. REFERENCES:

[1]DISSERTATION-Signal Fraction Analysis For Subspace Processing Of High Dimensional Data by Fatemeh Emdad Department of Mathematics

IX. APPENDIX:

1) The code for the Independent Component Analysis is shown below

% SFA and ICA Computation and Comparison

close all; clear all;

% reading the Audio Signals

[inp fs] = wavread('H:\best.wav');

[inp2 fs2] = wavread('H:\ChillingMusic.wav');

inp = inp';

inp2 = inp2';

% Giving the Noise variances

Sigma = [0.001 0.01 0.1 1 0.001 0.01 0.1 1];

inp = inp(1,1:50000); VAR1 = var(inp);

inp2 = inp2(1,1:50000); VAR2 = var(inp2);

SNR = [VAR1 ./ Sigma(1,1:4) VAR2 ./ Sigma(1,5:8)]

vect = [inp;inp2];

NN=2;

std=0.1;

% Noisa Generation and adding it to the original signal;

for loop =1:8

for i=1:NN

for j=1:NN

ron(i,j)=1-0.1*abs(i-j);

end

end

R=chol(ron);

noi=randn(2,50000)'*std*R;

AA=rand(2,2)

Phi=[10 20];

for p=1:2;

teta(1:2,p)=(p-1)*pi*cos(Phi(p)*pi/180);

end

obs=AA*vect + Sigma(1,loop)*randn(size(vect))

% ICA Part

[E, D] = pcamat(obs);

[nv, wm, dwm] = whitenv(obs, E, D);

[A, W] = fpica(nv, wm, dwm);

x=W*obs;

figure

for i=1:2;

subplot(2,1,i);hold on

plot(vect(i,:));

title('The original signal');

end

% plotting the figures

figure

for i=1:2;

subplot(2,1,i);hold on

plot(obs(i,:));

title('The observed signal');

end

figure

for i=1:2;

subplot(2,1,i);hold on

plot(x(i,:));

title('The Seperated signals using ICA');

end

if loop <= 4

MSE_voice(loop) = ((sum((x(1,:) - vect(1,:)).^2)) / (sum(vect(1,:)).^2))/(50000)

else

MSE_music(loop) = ((sum((x(2,:) - vect(2,:)).^2)) / (sum(vect(2,:)).^2))/(50000)

end

end

2) The code for the Signal Fraction Analysis is shown below

% SFA and ICA Computation and Comparison

clear all

clc

close all

% Reading the audio signals

[inp fs] = wavread('H:\best.wav');

[inp2 fs2] = wavread('H:\ChillingMusic.wav');

inp = inp';

inp2 = inp2';

% Adding the noise to the signal

Sigma = [0.001 0.01 0.1 1 0.001 0.01 0.1 1];

inp = inp(1,1:50000); VAR1 = var(inp);

inp2 = inp2(1,1:50000); VAR2 = var(inp2);

SNR = [VAR1 ./ Sigma(1,1:4) VAR2 ./ Sigma(1,5:8)]

vect = [inp;inp2];

NN = 2;

Sigma = 0.01;

obs = zeros(2,50000);

AA = abs(rand(2,2));

% generating observed signal

for loop = 1:8

for j = 1:2

obs(j,1:end) = (AA(1,j) .* vect(1,:)) + (AA(2,j) .* vect(2,:)) +...

+ (Sigma .* randn(1,50000));

end

deltaX = zeros(2,50000);

% estimating covariance matrix using differencing method

deltaX(1,1:end) = abs(obs(1+1,1:end) - obs(1,1:end));

deltaX(2,1:length(obs)) = abs(obs(1,1:end) - obs(2,1:end));

noise_cov = deltaX * deltaX';

XTX = obs * obs';

% estimating filter coefficients

[a , lambda] = eig(XTX , (noise_cov/2));

% reconstructing the signal

for i =1:2

s_hat(i,:) = a(:,i)' * obs;

end

% plot original, observed and reconstructed signals

figure

for i=1:2;

subplot(2,1,i);hold on

plot(vect(i,:));

title('The original signal');

end

figure

for i=1:2;

subplot(2,1,i);hold on

plot(obs(i,:));

title('The observed signal');

end

figure

for i=1:2;

subplot(2,1,i);hold on

plot(s_hat(i,:));

title('The Seperated signals using SFA');

end

if loop <= 4

MSE_voice(loop) = ((sum((s_hat(1,:) - vect(1,:)).^2)) / (sum(vect(1,:)).^2))/(50000)

else

MSE_music(loop) = ((sum((s_hat(2,:) - vect(2,:)).^2)) / (sum(vect(2,:)).^2))/(50000)

end

end