Jim RamsayJim RamsayMcGill UniversityMcGill University

Basis Basics

OverviewOverview

What are basis functions?What are basis functions? What properties should they have?What properties should they have? How are they usually constructed?How are they usually constructed? Fourier bases.Fourier bases. B-spline bases: Lots of detail here.B-spline bases: Lots of detail here. Wavelet bases.Wavelet bases. Advice, wrap-up, and beyond.Advice, wrap-up, and beyond.

What are basis functions?What are basis functions?

We need flexible method for constructing a We need flexible method for constructing a function function f(t) f(t) that can track local curvature.that can track local curvature.

We pick a system ofWe pick a system of K K basisbasis functions functions φφkk(t), (t), and call this theand call this the basisbasis forfor f(t). f(t).

We express We express f(t) f(t) as a as a weighted sumweighted sum of these of these basis functions:basis functions:

f(t) =f(t) = aa11φφ11(t) + a(t) + a22φφ22(t) + … + a(t) + … + aKKφφKK(t)(t) The coefficients The coefficients aa11, … , a, … , aK K determine the determine the

shape of the function.shape of the function.

What do we want from basis What do we want from basis functions?functions?

Fast computationFast computation of individual basis functions. of individual basis functions. FlexibleFlexible: can exhibit the required curvature : can exhibit the required curvature

where needed, but also be nearly linear when where needed, but also be nearly linear when appropriate.appropriate.

Fast computationFast computation of coefficients of coefficients aakk: possible if : possible if matrices of values are diagonal, banded or matrices of values are diagonal, banded or sparse.sparse.

Differentiable Differentiable as required: We make lots of use as required: We make lots of use of derivatives in functional data analysis.of derivatives in functional data analysis.

ConstrainedConstrained as required, such as periodicity, as required, such as periodicity, positivity, monotonicity, asymptotes and etc.positivity, monotonicity, asymptotes and etc.

What are some commonly used What are some commonly used basis functions?basis functions?

PowersPowers: : 1, t, t1, t, t22, and so on. They are the basis , and so on. They are the basis functions for polynomials. These are not very functions for polynomials. These are not very flexible, and are used only for simple problems.flexible, and are used only for simple problems.

Fourier seriesFourier series: : 1, sin(1, sin(ωωt), cos(t), cos(ωωt), sin(2t), sin(2ωωt), t), cos(2cos(2ωωt),t), and so on for a fixed known frequency and so on for a fixed known frequency ωω. These are used for periodic functions.. These are used for periodic functions.

B-spline functionsB-spline functions: These have now more or : These have now more or less replaced polynomials for non-periodic less replaced polynomials for non-periodic problems. More explanation follows.problems. More explanation follows.

How do we construct basis How do we construct basis systems?systems?

We start with a prototype basis function We start with a prototype basis function φφ(t)(t). Let’s call it the . Let’s call it the mother functionmother function..

We apply three operations to it:We apply three operations to it: Lateral shiftLateral shift φφ(t+a) (t+a) to focus fitting power to focus fitting power

near a position near a position aa.. Scale changeScale change φφ(bt), (bt), to increase resolving to increase resolving

powerpower, , or the capacity for local curvatureor the capacity for local curvature.. SmoothingSmoothing, to increase differentiability and , to increase differentiability and

smoothness.smoothness.

Can I see an example?Can I see an example?

Consider the Consider the Fourier seriesFourier series:: Mother functionsMother functions: : φφ(t)(t) = = sin(sin(ωωt), t), and and φφ(t)(t) = 1 = 1 Lateral ShiftLateral Shift:: sin[ sin[ωω(t+(t+ππ/2)]/2)] = = cos(cos(ωωt)t) Scale changeScale change: sin(k: sin(kωωt), cos(kt), cos(kωωt), k=1,2,…t), k=1,2,… SmoothingSmoothing: None; these are already infinitely : None; these are already infinitely

differentiable functions.differentiable functions. StrengthsStrengths: Orthogonality makes computing : Orthogonality makes computing

coefficients very fastcoefficients very fast LimitationsLimitations: Fourier series is only natural if : Fourier series is only natural if f(t)f(t)

is periodic with period is periodic with period ωω..

Five Fourier Basis FunctionsFive Fourier Basis Functions

φφ11(t)(t) = = 11 A sine/cosine pair A sine/cosine pair

with period 1: with period 1: Captures low Captures low frequency variationfrequency variation

A sine/cosine pair A sine/cosine pair with period ½: with period ½: Captures higher Captures higher frequency variationfrequency variation

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

-1

-0.5

0

0.5

1

t

k(

t)

What about polynomials?What about polynomials? The The mother functionmother function is is φφ(t)(t) = = 11.. SmoothingSmoothing: Multiply by : Multiply by t t to increase both to increase both

differentiability and curvaturedifferentiability and curvature.. No No shift shift oror scaling scaling: Polynomials are invariant : Polynomials are invariant

with respect to shift and scaling.with respect to shift and scaling. StrengthsStrengths: Great for very simple curves such as : Great for very simple curves such as

linear or slight curvature.linear or slight curvature. LimitationsLimitations: Just not flexible enough. We can’t : Just not flexible enough. We can’t

focus fitting power on a specific location.focus fitting power on a specific location.

What basis should I use for non-What basis should I use for non-periodic functions?periodic functions?

Spline basesSpline bases have nearly everything: have nearly everything: Infinite and independent control over shift, Infinite and independent control over shift,

scale, and smoothness.scale, and smoothness. Potential to model sharp changes in Potential to model sharp changes in f(t)f(t) as as

well as its smooth variation.well as its smooth variation. As fast as polynomials to evaluate.As fast as polynomials to evaluate. Band-structured matrix of values.Band-structured matrix of values.

How do B-splines work?How do B-splines work? Start with a discrete Start with a discrete

mesh of values, called mesh of values, called knotsknots: : ξξ00, , ξξ11, …, , …, ξξLL. .

Space out the knots Space out the knots where you don’t need where you don’t need much resolution, make much resolution, make them dense where you them dense where you need a lot of need a lot of resolution.resolution.

-5 -4 -3 -2 -1 0 1 2 3 4 5-0.2

0

0.2

0.4

0.6

0.8

1

Mother functionMother function is the is the box function over box function over [[ξξ00, , ξξ11]: ]: φφ(t)(t) = = 1, 1, ξξ00 ≤ t ≤ t < < ξξ11, , andand 0 0 otherwise.otherwise.

ShiftShift: just shift to : just shift to another interval.another interval.

ScaleScale: determined by : determined by spacing between spacing between knots.knots.

0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8 20

0.2

0.4

0.6

0.8

1

What about smoothness?What about smoothness?

Smoothness control is the heart of spline Smoothness control is the heart of spline technology.technology.

It is defined by a It is defined by a recursion relationrecursion relation.. The recursion relation uses spline basis The recursion relation uses spline basis

functions at a given smoothness level to functions at a given smoothness level to make new ones at the next higher make new ones at the next higher smoothness level.smoothness level.

B-spline notationB-spline notation

The spline basis system we use is the The spline basis system we use is the B-splineB-spline system. system.

To keep notation simple, let the knots be To keep notation simple, let the knots be the natural numbers the natural numbers j = 0, 1, …j = 0, 1, …

We need two indices for a B-spline basis We need two indices for a B-spline basis function function BBjmjm(t):(t): jj to indicate the knot where to indicate the knot where it starts, and it starts, and mm to indicate its to indicate its orderorder or or degree of smoothness. degree of smoothness.

For our box functions, For our box functions, m = 1m = 1. .

B-splines of order 2 are B-splines of order 2 are tenttent functions, starting at a knot, functions, starting at a knot, rising linearly to 1 at the rising linearly to 1 at the next knot, and decaying next knot, and decaying linearly to 0 two knots over.linearly to 0 two knots over.

They are They are continuouscontinuous. Order . Order 2 implies a continuous 2 implies a continuous derivative of order 0.derivative of order 0.

Order 2 knots are piecewise Order 2 knots are piecewise linear.linear.

What about order 3 B-What about order 3 B-splines?splines?

0 0.5 1 1.5 2 2.5 30

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

The recursion for integer knotsThe recursion for integer knots

, 1 1, 1

( 1) ( )

( ) ( ) ( ) ( )jm

j m j m

m B t

t j B t m j t B t

For the B-spline function of order 2 beginning at 0, the recursion is

02 01 11( ) ( ) (2 ) ( )B t tB t t B t

02 01 11( ) ( ) (2 ) ( )B t tB t t B t

•In the first term, the mother box function B01(t) is multiplied by t, and is therefore t itself.•In the second, the mother box is shifted In the second, the mother box is shifted to the right and multiplied by to the right and multiplied by 2-t2-t. . •The second term amounts to a shift and The second term amounts to a shift and reflection of the first term.reflection of the first term.

Constructing a Tent Constructing a Tent BB0202(t)(t) from Two from Two Boxes Boxes BB0101(t)(t) and and BB1111(t)(t)

0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8 20

0.2

0.4

0.6

0.8

1 x

B 01(x

) and

(2-x

)B11

(x)

0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8 20

0.2

0.4

0.6

0.8

1

xB 01

(x) +

(2-x

)B11

(x)

Constructing Order 3 Constructing Order 3 BB0303(t) (t) from from Two Tent FunctionsTwo Tent Functions

0 0.5 1 1.5 2 2.5 30

0.5

1

0 0.5 1 1.5 2 2.5 30

0.5

1

0 0.5 1 1.5 2 2.5 30

0.5

1

Multiplying by Multiplying by (t-j)(t-j) increases the polynomial increases the polynomial degree of the B-spline in the first term by one.degree of the B-spline in the first term by one.

Multiplying Multiplying (m+j-t)(m+j-t) does the same thing for the does the same thing for the reflected and shifted B-spline in the second reflected and shifted B-spline in the second term.term.

The result is a new B-spline of one degree or The result is a new B-spline of one degree or order higher, spread over an additional interval, order higher, spread over an additional interval, and having an one more level of continuous and having an one more level of continuous derivatives.derivatives.

Order 3 B-splines are piece-wise quadratic, Order 3 B-splines are piece-wise quadratic, spread over 3 intervals, and have a continuous spread over 3 intervals, and have a continuous first derivative.first derivative.

How do we get the extra How do we get the extra smoothness?smoothness?

tBtB0101(t)(t) and and (2-t)B(2-t)B1111(t)(t) have the same have the same values at knot 1. The sum is continuous.values at knot 1. The sum is continuous.

tBtB0202(t)(t) and and (3-t)B(3-t)B1212(t)(t) have the same have the same derivative values at knots 1 and 2. The derivative values at knots 1 and 2. The sum has a continuous first derivative.sum has a continuous first derivative.

Each time we multiply by Each time we multiply by tt, and then shift , and then shift and reflect the result, we add another and reflect the result, we add another order of smooth derivative.order of smooth derivative.

Suppose I don’t want so much Suppose I don’t want so much smoothness at a fixed point?smoothness at a fixed point?

You can have multiple knots at a point.You can have multiple knots at a point. For each additional knot, the spline For each additional knot, the spline

function will have one less derivative at function will have one less derivative at that knot.that knot.

An order An order mm Bspline with Bspline with mm knots at a point knots at a point can be discontinuous at that point.can be discontinuous at that point.

Most spline software places Most spline software places mm knots at knots at each of the boundaries.each of the boundaries.

How many B-spline basis functions How many B-spline basis functions am I using?am I using?

When all interior knots are unique, the When all interior knots are unique, the total number of B-spline basis functions is total number of B-spline basis functions is equal to the number of equal to the number of interiorinterior knots plus knots plus the the orderorder..

When there are no interior knots, splines When there are no interior knots, splines become polynomials: The order equals become polynomials: The order equals the number of basis functions.the number of basis functions.

WaveletsWavelets

Wavelets take this shift/scale/smooth trio Wavelets take this shift/scale/smooth trio of operations even further. of operations even further.

They define basis functions organized into They define basis functions organized into orthogonal orthogonal levels of resolutionlevels of resolution. .

The resulting The resulting multi-resolutionmulti-resolution analysis is analysis is one of the most exciting developments in one of the most exciting developments in mathematics since Fourier’s series.mathematics since Fourier’s series.

Some adviceSome advice

If you have periodic data, you’ll probably If you have periodic data, you’ll probably use use Fourier seriesFourier series..

If you don’t, you’ll probably use If you don’t, you’ll probably use B-splinesB-splines. . Especially if you need to look at or work Especially if you need to look at or work

with derivatives. with derivatives. If you want a derivative of order If you want a derivative of order mm, fit the , fit the

data with a spline of order at least data with a spline of order at least m+2m+2, , although order although order 2m2m has some advantages. has some advantages.

Where have we been?Where have we been? Linear combinations of basis functions are a Linear combinations of basis functions are a

convenient way to model functions.convenient way to model functions. The more basis functions we use, the more The more basis functions we use, the more

complex the fitted function can be.complex the fitted function can be. Most basis function systems involve turning Most basis function systems involve turning

three knobs, marked three knobs, marked shift, scaleshift, scale, and , and smooth, smooth, to to modify a mother function. modify a mother function.

The best systems permit independent control of The best systems permit independent control of these features. these features.

Like splines and wavelets.Like splines and wavelets.

Where do we go from here?Where do we go from here?

Although we can control the resolution of a Although we can control the resolution of a spline fit by the number of spline basis spline fit by the number of spline basis functions and the spacing of knots, we functions and the spacing of knots, we often need easier more continuous control often need easier more continuous control over smoothness.over smoothness.

Roughness penaltiesRoughness penalties can give us the can give us the additional control that we need.additional control that we need.