Support Vector Machine basierte Klassifikation und Regression in der Geofernerkundung

Abteilung Geomatik

Geographisches Institut der Humboldt-Universität zu Berlin

Support Vector Machine basierte Klassifikation und Regression

in der Geofernerkundung

*[email protected]

Andreas Rabe*

Sebastian van der Linden

Benjamin Jakimow

Patrick Hostert

26. Oktober 2011

2


Bad news:• wrong usage leads to overfitting or underfitting• mostly used as a black box (complex mathematics)• works also well in one- or two-dimensional feature spaces, but nobody notice

Support Vector Machines (SVMs)What would be of interest for the audience?Support Vector Machines (SVMs)What would be of interest for the audience?

Good news:• a SVM is a state-of-the-art classifier (fits arbitrary class boundaries)• is widely used inside remote sensing applications• works well in high-dimensional feature spaces

Take-home-message:• you can always avoid overfitting or underfitting when using SVM• you can use SVM as a black box, ...• ... but you could gain a deeper understanding by looking at simple

one- or two-dimensional examples

3


Support Vector Machines (SVMs)What would be of interest for the audience?Support Vector Machines (SVMs)What would be of interest for the audience?

This talk...

• is not about the mathematics behind SVMs.• is not about specific remote sensing applications → colored maps are not helpful!

• is about understanding the concepts behind SVM and the influence of parameters.• is about learning from simple one- or two-dimensional examples, to be able to

generalize to high-dimensional, real world problems.

4


linear / separable

linear /non-separable

non-linear / separable

non-linear /non-separable

simple

kernel function

regularization

regularizationand

kernel function

Different settings for binary classification in 2DDifferent settings for binary classification in 2D

positive class negative class

To train a SVM we need to set appropriate parameter values g for the kernel function and C regularization.

5


SVM overviewSVM overview

A Support vector machine (SVM) ...

... is a universal learning machine for - pattern recognition (classification),- regression estimation and- distribution estimation.

... can be seen as an implementation of Vapnik's Structural Risk Minimisation principle inside the context of Statistical Learning Theory (Vapnik1998).

not today

6


The optimal separating hyperplane.

Suppose the training set:

(x1,y1),..., (xl,yl), xRn, y{+1,−1},

can be separated by a hyperplane

(w∙x)-b = 0.

The optimal separating hyperplane separates the vectors without error and maximizes the margin between the closest vectors to the hyperplane.

(Burges1998)

SVM classification overviewSVM classification overview

H1: (w∙x)-b = -1

H2: (w∙x)-b = +1

not today

7


To construct the optimal separating hyperplane one has to solve a quadratic optimization problem:

Formulated as lagrange functional:

Maximize the functional

under the constraints:

12

L w w w

1 if 1

1 if 1i i

i i

b , y

b , y

w x

w x

1 1

12

l l

i i j i j i ji i , j

W α αα y yα x x

1

0 and 0.l

i i ii

αy α

Minimize the functional




not today

8


Let α0 = (α10,..., αl

0) be a solution to this quadratic optimization problem.

The optimal hyperplane w0 is a linear combination of the vectors of the training set.

00 0 0

1

l

i i ii

f b α y bx w x x x

y sign fx x

00

1

l

i i ii

α yw x

The decision rule y(x) is based on the sign of the decision function f(x):



not today

9


1d

i iK ,x x x x

Kernel Function

When looking at the lagrange functional:

1 1

12

l l


W α αα y yα x x

it can be observed, that only dot products between vectors in the input space are calculated.

The idea is to replace the dot product in the input space by the dot product in a higher dimensional feature space, defined by a kernel function K(x,xi).

Polynomial kernel:

Gaussian RBF kernel: 2

expi iK , gx x x x


00

1

l

i i ii

f α y K , bx x x

This leads to a non-linear decision function:

not today

10


The concept of maximizing the margin between classes must be modified, to be able to handle non-separable classes.

We introduce so-called slack variables = (1,..., l), one for each vector in the training set.

1

12

l

ii

L Cw w w

1 if 1

1 if 1

0

i i i

i i i

i

b , y

b , y

i

w x

w x

Minimize the functional


(Burges1998)

Regularization


not today

11


1 1

12

l l


W α αα y y K ,α x x

1

0 and 0l

i i ii

αy α C

Formulated as lagrange functional:

Maximize the functional


Regularization


not today

12


simple separable examplesimple separable example

2D example - separable, linear (Burges1998)

1D example - separable, non-linear

2

1

expl

i i ii

f α y g bx x x class sign fx x

2D example - separable, non-linear (www.mblondel.org)

13


simple non-separable examplesimple non-separable example

1D example - non-separable, non-linear

2D example - non-separable, linear (Burges1998)

2

1

expl

i i ii

f α y g bx x x class sign fx x

2D example - non-separable, linear (www.mblondel.org)

14


influence of parametersinfluence of parameters

good fit

numerical

problems

overfitting

underfitting

ccgg

kernel parameter g and penalty parameter ckernel parameter g and penalty parameter c

15


influence of parametersinfluence of parameters

kernel parameter g and penalty parameter ckernel parameter g and penalty parameter c

good fit

numerical

problems

overfitting

underfitting

ccgg

16


imageSVM inside EnMAP-Box software (remote sensing software)imageSVM inside EnMAP-Box software (remote sensing software)

A SVM implementation for classification and regression imageSVM is freely available inside EnMAP-Box software (contact [email protected]).

Suitable parameters are estimated via grid search and cross-validation.

A SVM implementation for classification and regression imageSVM is freely available inside EnMAP-Box software (contact [email protected]).

Suitable parameters are estimated via grid search and cross-validation.

17


underfitting

outlook - SVM regressionoutlook - SVM regression

SVM regression - kernel parameter g and penalty parameter cSVM regression - kernel parameter g and penalty parameter c

cc gg

good fit

overfitting

18


SVM regression - epsilon-loss functionSVM regression - epsilon-loss function

outlook - SVM regressionoutlook - SVM regression

19


Thank you very much for your attention.

Any questions?

References

Burges, C. J. C. (1998). "A Tutorial on Support Vector Machines for Pattern Recognition." Data Mining and Knowledge Discovery 2(2): 121-167.

Chang, C.-C. and C.-J. Lin (2001). LIBSVM: a Library for Support Vector Machines. Software available at http://www.csie.ntu.edu.tw/~cjlin/libsvm.

Vapnik, V. (1999). The Nature of Statistical Learning Theory, Springer-Verlag.

Documents

Support Vector Machine basierte Klassifikation und Regression in der Geofernerkundung