Introduction to Gaussian Processes

Carlos Becker, Emtiyaz KhanDecember 2015

Pattern Recognition & Machine Learning Course, EPFL

Motivation2

Goals of this lecture

– Understand what a Gaussian Process (GP) is.

– Learn how GPs can be used for regression.

More specific to GPs, you will learn:

– What a covariance matrix means from a GP point of view.

– How a GP defines a prior over functions, and its relationship to its covariance matrix and correlation terms.

– What “conditioning on the measurements” means, in a probabilistic sense as well as mathematically.

Note: GPs for classification are outside the scope of this lecture. But if you understand regression GPs it won't be too difficult to learn how classification GPs work. Please see Rasmusen and William's “Gaussian Processes for Machine Learning” book.

Motivation: why Gaussian Processes?

Motivation4

Say we want to estimate a scalar function

from training data

x1 x2 x3

y1

y2y3

Motivation5

Say we want to estimate a scalar function

from training data

x1 x2 x3

f1

f2f3

x1 x2 x3

y1

y2y3

2nd Order Polynomial

Motivation6

Say we want to estimate a scalar function

from training data

x1 x2 x3

f1

f2f3

x1 x2 x3

y1

y2y3

3rd Order Polynomial

Motivation7

Say we want to estimate a scalar function

from training data

x1 x2 x3

f1

f2f3

x1 x2 x3

y1

y2y3

Radial Basis Function

Motivation8

Gaussian Processes let us place a prior on the 'shape' of

And this prior is formulated probabilistically

Let's get started!

Just before.10

Gaussian Processes11

Let's get back to finding , but now

– From a single observation

– We want to predict

x1

f1

Gaussian Processes12

x1

f1

Now, in a probabilistic manner.

Gaussian Processes13

x1

f1

Getting there.

Gaussian Processes14

What about multiple training points?

Gaussian Processes15

Gaussian Processes16

Gaussian Processes17

Gaussian Processes18

Gaussian Processes19

Gaussian Processes20

Gaussian Processes21

Gaussian Processes22

Gaussian Processes23

Gaussian Processes24

Exercise25

Time to get to the Real GP

Gaussian Processes27

Gaussian Processes28

1) A prior over functions

Gaussian Processes29

1) A prior over functions

Gaussian Processes30

1) A prior over functions

Gaussian Processes31

1) A prior over functions

Gaussian Processes32

1) A prior over functions

Gaussian Processes33

1) A prior over functions

Gaussian Processes34

2) Incorporating noise-free measurements

Gaussian Processes35

2) Incorporating noise-free measurements

Gaussian Processes36

3) Incorporating noisy measurements (as in real life)

Gaussian Processes37

3) Incorporating noisy measurements (as in real life)

Gaussian Processes38

Demo Time

Gaussian Processes39

A Real Example [Rasmussen, Williams, Gaussian Processes for Machine Learning]

?

Gaussian Processes40

A Real Example [Rasmussen, Williams, Gaussian Processes for Machine Learning]

Gaussian Processes41

A Real Example [Rasmussen, Williams, Gaussian Processes for Machine Learning]

Kernel Design:

11 parameters

Gaussian Processes42

What about Classification?

Gaussian Processes44

Summary