Feature extraction using fuzzy complete linear discriminant analysis

The reporter ： Cui Yan

2012. 4. 26

The report outlines

1.The fuzzy K-nearest neighbor classifier (FKNN)

2.The fuzzy complete linear discriminant analysis

3.Expriments

The Fuzzy K-nearest neighbor classifier (FKNN)

Each sample should be classified

similarly to its surrounding samples,

therefore, a unknown sample could be

predicated by considering the

classification of its nearest neighbor

samples.

The K-nearest neighbor classifier (KNN)

KNN tries to classify an unknown sample based

on its k-known classification neighbors.

FKNN

Given a sample set , a fuzzy M

-class partition of these vectors specify the membership degrees of each sample corres-ponding to each class.

The membership degree of a training vector to o each of M classes is specified by ,

which is computed by the following steps:ijuijx

},,,{ 21 nxxxX

Step 1: Compute the distance matrix between

pairs of feature vectors in the training.

Step 2: Set diagonal elements of this matrix to

infinity (practically place large numeric

values there).

Step 3: Sort the distance matrix (treat each of

its column separately) in an ascending

order. Collect the class labels of the

patterns located in the closest neigh-

borhood of the pattern under consi-

deration (as we are concerned with k

neighbors, this returns a list of k

integers).

Step 4: Compute the membership grade to class i

for j-th pattern using the expression proposed

in [1].

[1] J.M. Keller, M.R. Gray, J.A. Givens, A fuzzy k-nearest neighbor algorithm, IEEE Trans. Syst.Man Cybernet. 1985, 15(4):580-585

pattern. theof label the if )(*0.49

pattern. theof label the if )(*49.051.0

j-thi

j-thiu

kn

kn

ijij

ij

A example for FKNN

No. Feature1 Feature2 class

1 0.2000 0.3000 1

2 0.3000 0.2000 1

3 0.4000 0.3000 1

4 0.5000 0.5000 2

5 0.6000 0.4000 2

6 0.5000 0.6000 2

7 0.7000 0.3000 3

8 0.8000 0.4000 3

9 0.7000 0.5000 3

1 2 3 4 5 6 7 8 9

0

0.1414

0.2000

0.3606

0.4123

0.4243

0.5000

0.5385

0.6083

0

0.1414

0.1414

0.3606

0.3606

0.4123

0.4472

0.5000

0.5385

0

0.1414

0.2000

0.2236

0.2236

0.3000

0.3162

0.3606

0.4123

0

0.1000

0.1414

0.2000

0.2236

0.2828

0.3162

0.3606

0.3606

0

0.1414

0.1414

0.1414

0.2000

0.2236

0.2236

0.3606

0.4123

0

0.1000

0.2236

0.2236

0.3162

0.3606

0.3606

0.4243

0.4472

0

0.1414

0.1414

0.2000

0.2828

0.3000

0.3606

0.4123

0.5000

0

0.1414

0.1414

0.2000

0.3162

0.3606

0.4123

0.5385

0.6083

0

0.1414

0.1414

0.2000

0.2000

0.2236

0.3606

0.5000

0.5385

No. 1 2 3 4 5 6 7 8 9

1 0 0.1414 0.2000 0.3606 0.4123 0.4243 0.5000 0.6083 0.5385

2 0.1414 0 0.1414 0.3606 0.3606 0.4472 0.4123 0.5385 0.5000

3 0.2000 0.1414 0 0.2236 0.2236 0.3162 0.3000 0.4123 0.3606

4 0.3606 0.3606 0.2236 0 0.1414 0.1000 0.2828 0.3162 0.2000

5 0.4123 0.3606 0.2236 0.1414 0 0.2236 0.1414 0.2000 0.1414

6 0.4243 0.4472 0.3162 0.1000 0.2236 0 0.3606 0.3606 0.2236

7 0.5000 0.4123 0.3000 0.2828 0.1414 0.3606 0 0.1414 0.2000

8 0.6083 0.5385 0.4123 0.3162 0.2000 0.3606 0.1414 0 0.1414

9 0.5385 0.5000 0.3606 0.2000 0.1414 0.2236 0.2000 0.1414 0

1 2 3 4 5 6 7 8 9

1 2 3 4 5 6 7 8 9

2 1 2 6 4 4 5 9 5

3 3 1 5 9 5 8 7 8

4 4 5 9 7 9 9 5 4

5 5 4 3 8 3 4 4 7

6 7 7 7 6 7 3 6 6

7 6 6 8 3

9 9 9 1 2

8

1

6

2

3

2

3

2

8 8 8 2 1 2 1 1 1

1 2 3 4 5 6 7 8 9

1 1 1 2 2 2 3 3 3

1 1 1 2 2 2 2 3 2

1 1 1 2 3 2 3 3 3

2 2 2 3 3 3 3 2 2

2 2 2 1 3 1 2 2 3

2 3 3 3 2 3 1 2 2

3 2 2 3 1

3 3 3 1 1

3

1

2

1

1

1

1

1

3 3 3 1 1 1 1 1 1

1 2 3 4 5 6 7 8 9

1 1 1 2 2 2 2 3 2

1 1 1 2 3 2 3 3 3

2 2 2 3 3 3 3 2 2

Set k=3

1 2 3 4 5 6 7 8 9

0.8367 0.8367 0.8367 0 0 0 0 0 0

0.1633 0.1633 0.1633 0.8367 0.6733 0.8367 0.1633 0.1633 0.3267

0 0 0 0.1633 0.3267 0.1633 0.8367 0.8367 0.6733

The fuzzy complete linear

discriminant analysis

For the training set , we define the i-th class mean by combining the fuzzy membership degree as

And the total mean as

},,,{ 21 nxxxX

.,,2,1 ,1

1 ciu

xum n

j ij

n

j jij

i

1

1

n

i in xm

(1)

(2)

Incorporating the fuzzy membership degree, the between-class, the within-class and the total class fuzzy scatter matrix of samples can be defined as

c

i Nj

TjjijtF

c

i Nj

TijijijwF

c

i

n

j

TiiijbF

i

i

mxmxuS

mxmxuS

mmmmuS

1

1

1 1

))((

))((

))((

(3)

step1: Calculate the membership degree matrix U

by the FKNN algorithm.

step 2: According toEqs.(1)-(3) work out the

between-class, within-class and total class

fuzzy scatter matrices.

step 3: Work out the orthogonal eigenvectors

p1, . . . , pl of the total class fuzzy scatter

matrix corresponding to positive

eigenvalues.

Algorithm of the fuzzy complete linear analysis

tFS

step 4: Let P = (p1, . . . , pl) and

, work out the orthogonal

eigenvectors g1, . . . , gr of correspending

the zero eigenvalues.

step 5: Let P1 = (g1, . . . , gr) and , work

out the orthogonal eigenvectors v1, . . . , vr of

, calculate the irregular discriminant

vectors by .

P, S P S wFT

wF ˆ

P SP S bFT

bF ˆ

1ˆ1~

PS PS bFT

bF

bFS~

vPPwir 1irw

wFS

step 6: Work out the orthogonal eigenvectors q1,…, qs

of correspending the non-zero eigenvalues.

step 7: Let P2 = (q1,…, qs) and

, work out the optimal

discriminant vectors vr+1, . . . , vr+s by the

Fisher LDA, calculate the regular discriminant

vectors by .

step 8: (Recognition): Project all samples into the

obtained optimal discriminant vectors and

classify.

wFS

,PS P S wFT

wF 2ˆ2

2ˆ2 PS P S bFT

bF

rw vPPwr 2

Experiments

• We compare Fuzzy-CLDA with CLDA, UWLDA, FLDA, Fuzzy Fisherface, FIFDA on 3 different data sets from the UCI data sources. The characteristics of the three datasets can be found from (http://archive.ics.uci.edu/ml/datasets).

• All data sets are randomly split to the train set and test set with the ratio 1:4. Experiments are repeated 25 times to obtain mean prediction error rate as a performance measure, NCC is adopted to classify the test samples by using L2 norm.

Thanks！

2012. 4. 26