Ch. 4: Radial Basis Functions Stephen Marsland, Machine Learning: An

Algorithmic Perspective. CRC 2009based on slides from many Internet sources

Longin Jan LateckiTemple University

latecki@temple.edu

Perceptorn

In RBFN

architecture

Input layer Hidden layer Output layer

x1

x2

x3

xn

h1

h2

h3

hm

f(x)

W1

W2

W3

Wm

Three layers

• Input layer– Source nodes that connect to the network to its

environment

• Hidden layer– Hidden units provide a set of basis function– High dimensionality

• Output layer– Linear combination of hidden functions

architecture

Radial basis function

hj(x) = exp( -(x-cj)2 / rj2 )

f(x) = wjhj(x)j=1

m

Where cj is center of a region,

rj is width of the receptive field

architecture

7

Function Approximation with Radial Basis Functions

RBF Networks approximate functions using (radial) basis functions as the building blocks.

8

Exact Interpolation• RBFs have their origins in techniques for performing

exact function interpolation [Bishop, 1995]:– Find a function h(x) such that

h(xn) = tn n=1, ... N

• Radial Basis Function approach (Powel 1987):– Use a set of N basis functions of the form (||x-xn||),

one for each point,where (.) is some non-linear function.

– Output: h(x) = n wn (||x-xn||)

9

Exact Interpolation• Goal (exact interpolation):

– Find a function h(x) such that h(xn) = tn n=1, ... N

• Radial Basis Function approach (Powel 1987):– Use a set of N basis functions of the form (||x-xn||), one

for each point,where (.) is some non-linear function.– Output: h(x) = n wn (||x-xn||)

w1 (||x1-x1||) + w2 (||x1-x2||) + ... + wN (||x1-xN||) = t1

w1 (||x2-x1||) + w2 (||x2-x2||) + ... + wN (||x2-xN||) = t2 W = T...

w1 (||xN-x1||) + w2 (||xN-x2||) + ... + wN (||xN-xN||)= tN

10

Exact Interpolation

11

Exact Interpolation

12

• Due to noise that may be present in the data exact interpolation is rarely useful.

• By introducing a number of modifications, we arrive at RBF networks:

• Complexity rather than the size of the data is what is important

– Number of the basis functions need not be equal to N• Centers need not be constrained by the input• Each basis function can have its own adjustable width

parameter • Bias parameter may be included in the linear sum.

13

Illustrative Example - XOR Problem

2

1

14

Function Approximation via Basis Functions and RBF Networks

• Using nonlinear functions, we can convert a nonlinearly separable problem• into a linearly separable one.

• From a function approximation perspective, this is equivalent to implementing a complex function (corresponding to the nonlinearly separable decision boundary) using simple functions (corresponding to the linearly separable decision boundary)

• Implementing this procedure using a network architecture, yields the RBF• networks, if the nonlinear mapping functions are radial basis functions.

• Radial Basis Functions:– Radial: Symmetric around its center– Basis Functions: Also called kernels, a set of functions whose linear combination can

generate an arbitrary function in a given function space.

15

RBF Networks

16

RBF Networks

17

Network Parameters• What do these parameters represent?

• • : The radial basis function for the hidden layer. • This is a simple nonlinear mapping function (typically Gaussian)

that transforms the d- dimensional input patterns to a (typically higher) H-dimensional space. The complex decision boundary will be constructed from linear combinations (weighted sums) of these simple building blocks.

• • uJi: The weights joining the first to hidden layer. These weights constitute the center points of the radial basis functions. Also called prototypes of data.

• • : The spread constant(s). These values determine the spread (extend) of each radial basis function.

• • Wjk: The weights joining hidden and output layers. These are the weights which are used in obtaining the linear combination of the radial basis functions. They determine the relative amplitudes of the RBFs when they are combined to form the complex function.

• • ||x-uJ||: the Euclidean distance between the input x and the prototype vector uJ. Activation of the hidden unit is determined according to this distance through

18

19

Training RBF Networks

Approach 1: Exact RBFApproach 2: Fixed centers selected at randomApproach 3: Centers are obtained from clusteringApproach 4: Fully supervised training

20

Training RBF Networks

• Approach 1: Exact RBF • Guarantees correct classification of all training

data instances.

• Requires N hidden layer nodes, one for each training instance.

• No iterative training is involved: w are obtained by solving a set of linear equations

• Non-smooth, bad generalization

21

Exact Interpolation

22

Exact Interpolation

23

Too Many Receptive Fields?• In order to reduce the artificial complexity of the

RBF, we need to use fewer number of receptive fields.

•

• Approach 2: Fixed centers selected at random.• Use M < N data points as the receptive field centers.• Fast but may require excessive centers

• • Approach 3: Centers are obtained from

unsupervised learning (clustering). • Centers no longer has to coincide with data points• This is the most commonly used procedure, providing

good results.

24

Approach 2 Approach 3 Approach 3.b

25

Determining the Output Weights through learning (LMS)

26

RBFs for Classification

Homework

• Problem 4.1, p. 117• Problem 4.2, p. 117