Scientific Computing

Linear Least Squares

Interpolation vs Approximation

• Recall: Given a set of (x,y) data points, Interpolation is the process of finding a function (usually a polynomial) that passes through these data points.

Interpolation vs Approximation

• Given a set of (x,y) data points, Approximation is the process of finding a function (usually a line or a polynomial) that comes the “closest” to the data points.

• Data has “noise” – cannot find interpolating line.

General Least Squares Idea

• Given data and a class of

functions F, the goal is to find the “best” function f in F that approximates the data.

• Consider the data to be two vectors of length n + 1. That is,

x = [x0 x1 … xn]t and y = [y0 y1 … yn]t

General Least Squares Idea

• Definition: The error, or residual, of a given function with respect to this data is the vector r = y - f(x):

That is r = [r0 r1 … rn]t where ri = yi - f(xi)

• We want to find a function f such that the error is made as small as possible. How do we measure the size of the error? With vector norms.

Vector Norms

• A vector norm is a quantity that measures how large a vector is (the magnitude of the vector).

Our previous examples for vectors in Rn :• Manhattan• Euclidean• Chebyshev

• For Least Squares, the best norm to use will be the 2-norm.

Ordinary Least Squares

• Definition The least-squares best approximating function to a set of data, x, y from a class of functions, F, is the function f* in F that minimizes the 2-norm of the error. That is, if f * is the least squares best approximating function, then

• This is often called the ordinary least squares method of approximating data.

Linear Least Squares

• Linear Least Squares: We assume that the class of functions is the class of all possible lines.

• By the method in the last slide, we then want to find a linear function f(x) = ax + b that minimizes the 2-norm of the vector y-f(x), i.e. that minimizes:

Linear Least Squares

• Linear Least Squares: To minimize it is enough to minimize the term inside the square root. So, if f(x) = ax+b, we need to

minimize over all possible values of a and b. • From calculus, we know that the minimum will occur

where the partial derivatives with respect to a and b are zero.

Linear Least Squares

• Linear Least Squares:

• These equations can be written as:

• These last two equations are called the Normal Equations for the best line fit to the data.

Linear Least Squares

• Linear Least Squares:

• Note that these are two equations in the unknowns a and b. Let

• Then, the solution is

Matrix Formulation of Linear Least Squares

• Linear Least Squares: Want to minimize• Let

• Then, we want to find a vector c that minimizes the length squared of the error vector Ac-y (or y – Ac) That is, we want to minimize

• This is equivalent to minimizing the Euclidean distance from y to Ac.

b

ac

x

x

A

n

,

1

......

10

Matrix Formulation of Linear Least Squares

• Linear Least Squares: Find a vector c to minimize the Euclidian distance from y to Ac.

• Equivalently, minimize ||y–Ac||2 or (y–Ac)t (y–Ac)

• If we take all partials of this expression (with respect to c0 , c1 ) and set these equal to zero, we get

AcAcyAcyy

AcAcyAcAcyyyAcyAcyttttt

ttttttt

2

)()(

yAAcA

AcAyAtt

tt

022

Matrix Formulation of Linear Least Squares

• The equation At Ac = At y is also called the Normal Equation for the linear best fit. The equations one gets from At Ac = At y are exactly the same equations we had before:

• The solution c=[a b] for At Ac = At y gives the constants for the line ax+b.

Matlab Example % Example of how to find the linear least squares fit to noisy datax = 1:.1:6; % x valuesy = .1*x + 1; % linear y-valuesynoisy = y + .1*randn(size(x)); % add noise to y valuesplot(x, ynoisy,'.') % Plot the noisy datahold on % So we can plot more data later % Find d11, d12, d21, d22, e1, e2D=[sum(x.^2), sum(x); sum(x), length(x)];e1 = x*ynoisy';e2 = sum(ynoisy); % Solve for a and bdet = D(1,1)*D(2,2) - D(1,2)*D(2,1);a = (D(2,2)*e1 - D(1,2)*e2)/det;b = (D(1,1)*e2 - D(2,1)*e1)/det;% Create a vector of y-values for the linear best fitfit_y = a.*x + b; plot(x, fit_y,'-') % Plot the best fit line

Matlab Example At Ac = At y % Example of how to find the linear least squares fit to noisy datax = 1:.1:6; % x valuesy = .1*x + 1; % linear y-valuesynoisy = y + .1*randn(size(x)); % add noise to y valuesplot(x, ynoisy,'.') % Plot the noisy datahold on % So we can more data later % Create matrix AA = zeros(length(x), 2);A(:,1) = x;A(:,2) = ones(length(x),1);D = A'*A; e = A'*ynoisy'; % Solve for constants a and bc= D \ e;fit_y = c(1).*x + c(2);plot(x, fit_y,'O')