Lecturenotes123 (1)

8/3/2019 Lecturenotes123 (1)

1/32

Chapter 1

Numerical techniques for engineers(ENGT5140)

1.1 Introduction

This module presents the modern and advanced numerical techniques which are currently used in mechanicaland electronic engineering in a manner that is both interesting and understandable to the graduate studentsin these fields. The module contains the well recognised elements of advanced numerical simulation techniqueswhich universally underpin the formation of the professional engineer. It is comprised mainly of : numericalmethods for ODE and PDE, Finite element, finite volume and spectral methods. A detailed algorithm isgiven for each method presented based on the case studies, so that students can write computer programsimplementing the numerical technique in different application areas. Numerous engineering examples of theuse of numerical simulations are also included in the module. The principal aim of the module is to enhanceand develop the students understanding of and ability to use the numerical methods and algorithms forsolving engineering problems.

1.2 Module SyllabusIn this module we will cover the following contents:

MATLAB, Basic knowledge of numerical methodsTaylors theorem and linear system of equations

Ordinary differential equation (ODE) initial value problems

1. General feature of ODE

2. Eulers method

3. Runge-Kutta method

Partial differential equations (PDE)

1. General feature of partial differential equations

2. Parabolic PDE: Heat equationExplicit and implicit methods and Crank-Nicolson method

3. Hyperbolic PDEWave equation, Lax-Wendroff type methods

4. Elliptic PDEPoisson equation,Finite difference method

The finite element method for PDE

Other numerical methods of modern computing

Applications

1


2/32

2 CHAPTER 1. NUMERICAL TECHNIQUES FOR ENGINEERS (ENGT5140)

1.3 References and books

There are plenty references and books for this subject, we will base on the following books for this module:

Fausett L., Numerical Methods Algorithms and Applications, Pearson Education, 2003, ISBN 0-13-031400-5

Kiusalaas J., Numerical Methods in Engineering with Matlab, Cambridge, 2005.

Kreyszig E, Advanced Engineering Mathematics, Wiley International, 2006 ISBN 978-0-471-72897-9. Mathews J. and Fink K., Numerical Methods Using Matlab, Pearson Education, 2004, ISBN 0-13-

191178-3 Reddy J. And Garting D. The Finite Element Method in Heat Transfer and Fluid Dynamics,CRC Press 2001, ISBN 084932355X

Zienkiewicz, O. And Taylor R., The Finite Element Method, Vol 1 The Basis, 5th Ed. Oxford,Butterworth-Heineman, 2000, ISBN 0750650494

Zienkiewicz, O. And Taylor R., The Finite Element Method, Vol 2 Solid Mechanics, 5th Ed. Oxford,Butterworth-Heineman, 2000, ISBN 0750650559

There are also plenty webs for numerical methods, you can search the topic by using Google. The following

maths websits may be useful:

http://mathworld.wolfram.com(which has a wealth of material where you can look up definitions and formulas, etc.)

http://en.wikipedia.org/wiki/Catergory:Mathematics

http://helm.lboro.ac.uk/(a very good website to help engineers learn mathematics)

http://www.mathcentre.ac.uk/(which offers students quick reference guides, practice and revision materials, and also online practice

exercises on many branches of mathematics)There are, of course, hundreds of other websites, references and books to explore . . . .

Dr. Xin Kai Li Email: [email protected] Room Q3.14 (8695)


3/32

Chapter 2

Matlab

2.1 Introduction

This lecture note presents various MATLAB applications in mechanics and applied mathematics. Our ob-jective is to employ numerical methods in examples emphasizing the appeal of MATLAB as a programming

tool. The programs are intended for study as a primary component of the text. The numerical methodsused include interpolation, numerical integration, finite differences, linear algebra, Fourier analysis, roots ofnonlinear equations, linear differential equations, nonlinear differential equations, linear partial differentialequations, analytic functions, and optimization methods. The physical applications vary widely from solu-tion of linear and nonlinear differential equations in mechanical system dynamics to geometrical propertycalculations for areas and volumes.

MATLAB is a commercial software and a trademark of The MathWorks, Inc., USA. It is an integratedprogramming system, including graphical interfaces and a large number of specialized toolboxes. MATLABis getting increasingly popular in all fields of science and engineering. This chapter will provide some basicnotions needed for the understanding of the remainder of the book. A deeper study of MATLAB can beobtained from many MATLAB books and the very useful help of MATLAB.

2.2 MatricesMatrices are the fundamental object of MATLAB and are particularly important in this book. Matrices canbe created in MATLAB in many ways, the simplest one obtained by the commands

>> A=[1 2 3;4 5 6;7 8 9]

A =

1 2 3

4 5 6

7 8 9

Note the semi-colon at the end of each matrix line. We can also generate matrices by pre-defined functions,such as random matrices

>> rand(3)ans =

0.8147 0.9134 0.2785

0.9058 0.6324 0.5469

0.1270 0.0975 0.9575

Rectangular matrices can be obtained by specification of the number of rows and columns, as in

>> rand(2,3)

ans =

0.9649 0.9706 0.4854

0.1576 0.9572 0.8003

2.3 Operating with matricesWe can add, subtract, multiply, and transpose matrices. For example, we can obtain a matrix C = A + B,by the following commands

3


4/32

4 CHAPTER 2. MATLAB

>> a=rand(4)

a =

0.2769 0.6948 0.4387 0.1869

0.0462 0.3171 0.3816 0.4898

0.0971 0.9502 0.7655 0.4456

0.8235 0.0344 0.7952 0.6463

>> b=rand(4)

b =0.7094 0.6551 0.9597 0.7513

0.7547 0.1626 0.3404 0.2551

0.2760 0.1190 0.5853 0.5060

0.6797 0.4984 0.2238 0.6991

>> c=a+b

c =

0.9863 1.3499 1.3985 0.9381

0.8009 0.4797 0.7219 0.7449

0.3732 1.0692 1.3508 0.9515

1.5032 0.5328 1.0190 1.3454

The matrices can be multiplied, for example E = A

D, as shown in the following example

>> d=rand(4,1)

d =

0.8909

0.9593

0.5472

0.1386

>> e=a*d

e =

1.1792

0.6220

1.4787

1.2914The transpose of a matrix is given by the apostrophe, as AT

>> A=rand(3,2)

A =

0.1493 0.2543

0.2575 0.8143

0.8407 0.2435

>> A

ans =

0.1493 0.2575 0.8407

0.2543 0.8143 0.2435

If one wants to cancel the echo of the input, a semi-colon at the end of the statement suffices. Important tomention that MATLAB is case-sensitive, variables a and A being different objects. We can erase variablesfrom the workspace by using clear, or clear all. A given object can be erased, such as clear A.

Some examples of such functions are given in the following commands (here we build matrices by blocks)

>> [eye(3),diag(eye(3)),rand(3)]

ans =

1.0000 0 0 1.0000 0.9293 0.2511 0.3517

0 1.0000 0 1.0000 0.3500 0.6160 0.8308

0 0 1.0000 1.0000 0.1966 0.4733 0.5853

\

Another example of matrices built from blocks:

>> A=rand(3)

A =

0.5497 0.7572 0.5678


5/32

2.4. CONDITIONALS,IF AND LOOPS, FOR, WHILE 5

0.9172 0.7537 0.0759

0.2858 0.3804 0.0540

>> B = [A, zeros(3,2); zeros(2,3), ones(2)]

B =

0.5497 0.7572 0.5678 0 0

0.9172 0.7537 0.0759 0 0

0.2858 0.3804 0.0540 0 0

0 0 0 1.0000 1.00000 0 0 1.0000 1.0000

2.4 Conditionals,if and loops, for, while

Often a function needs to branch based on runtime conditions. MATLAB offers structures for this similarto those in most languages. Here is an example illustrating most of the features of if.

x=-1

if x==0

disp(Bad input!)

elseif max(x) > 0

y = x+1;else

y = x^2;

end

Many programs require iteration, or repetitive execution of a block of statements. Again, MATLAB issimilar to other languages here. This code for calculating the first 10 Fibonacci numbers illustrates the mostcommon type of for/end loop:

>> f=[1 2]

f =

1 2

>> for i=3:10;f(i)=f(i-1)+f(i-2);end;

>> ff =

1 2 3 5 8 1 3 2 1 3 4 5 5 8 9

It is sometimes necessary to repeat statements based on a condition rather than a fixed number of times.This is done with while.

>> x=10;while x > 1; x = x/2,end

x =

5

x =

2.5000

x =

1.2500x =

0.6250

Other examples of for/end loops:

>> x = []; for i = 1:4, x=[x,i^2], end+

x = 1

x = 1 4

x = 1 4 9

x = 1 4 9 16

and in inverse form

>> x = []; for i = 4:-1:1, x=[x,i^2], endx =

16

x =


6/32

6 CHAPTER 2. MATLAB

16 9

x =

1 6 9 4

x =

1 6 9 4 1

Note the initial values of x = [ ] and the possibility of decreasing cycles.

2.5 relations

Relations in MATLAB are shown below: Some relation operators:

< Less than

> Greater than

= Greater or equal than

== Equal to

~= Not equal

Note the difference between = and logical equal ==. Some logical operators:

& and

| or

~ not

The result if either 0 or 1, as in

>> 35,3==5+,

ans =

1

ans =

0

ans =0

The same is obtained for matrices, as in

>> a = rand(5), b = triu(a), a == b

a =

0.1419 0.6557 0.7577 0.7060 0.8235

0.4218 0.0357 0.7431 0.0318 0.6948

0.9157 0.8491 0.3922 0.2769 0.3171

0.7922 0.9340 0.6555 0.0462 0.9502

0.9595 0.6787 0.1712 0.0971 0.0344

b =

0.1419 0.6557 0.7577 0.7060 0.82350 0.0357 0.7431 0.0318 0.6948

0 0 0.3922 0.2769 0.3171

0 0 0 0.0462 0.9502

0 0 0 0 0.0344

ans =

1 1 1 1 1

0 1 1 1 1

0 0 1 1 1

0 0 0 1 1

0 0 0 0 1

2.6 Functions

Scalar functions:


7/32

2.6. FUNCTIONS 7

sin asin exp abs round

cos acos log sqrt floor

tan atan rem sign ceil

Vector functions:

max sum median any

min prod mean all

For example

a=rand(3,4)

a =

0.4387 0.7952 0.4456 0.7547

0.3816 0.1869 0.6463 0.2760

0.7655 0.4898 0.7094 0.6797

>> b=sin(a)

b =

0.4248 0.7140 0.4310 0.6851

0.3724 0.1858 0.6022 0.2725

0.6929 0.4704 0.6514 0.6286

>> c=sqrt(b)c =

0.6518 0.8450 0.6565 0.8277

0.6102 0.4310 0.7760 0.5220

0.8324 0.6859 0.8071 0.7928

Another example

>> x=1:10

x =

1 2 3 4 5 6 7 8 9 10

>> sum(x)

ans =

55>> mean(x)

ans =

5.5000

>> max(x)

ans =

10

Some important matrix functions are shown below

eig Eigenvalues and eigenvectors

chol Choleski factorization

inv Inverse

lu LU decompositionqr QR factorization

schur Schur decomposition

poly Characteristic polynomial

det Determinant

size Size of a matrix

norm 1-norm, 2-norm, F-norm, -norm

cond Conditioning number of 2-norm

rank Rank of a matrix

In some cases such functions may use more than one output argument, as in

>> A=rand(3)

A =0.8147 0.9134 0.2785

0.9058 0.6324 0.5469

0.1270 0.0975 0.9575


8/32

8 CHAPTER 2. MATLAB

>> y=eig(A)

y =

-0.1879

1.7527

0.8399

where we wish to obtain the eigenvalues only, or in

>> [V,D]=eig(A)

V =

0.6752 -0.7134 -0.5420

-0.7375 -0.6727 -0.2587

-0.0120 -0.1964 0.7996

D =

-0.1879 0 0

0 1.7527 0

0 0 0.8399

where we obtain the eigenvectors and the eigenvalues of matrix A.

2.7 Submatrix

In MATLAB it is possible to manipulate matrices in order to make code more compact or more efficient.For example, using the colon we can generate vectors, as in

>> x=1:8

x =

1 2 3 4 5 6 7 8

or using increments

>> x=1.2:0.5:3.7

x =

1.2000 1.7000 2.2000 2.7000 3.2000 3.7000

This sort of vectorization programming is quite efficient, no for/end cycles are used. This efficiency can beseen in the generation of a table of sines,

>> x=0:pi/2:2*pi

x =

0 1.5708 3.1416 4.7124 6.2832

>> b=sin(x)

b =

0 1.0000 0.0000 -1.0000 -0.0000

>> [x b]

ans =

0 01.5708 1.0000

3.1416 0.0000

4.7124 -1.0000

6.2832 -0.0000

The colon can also be used to access one or more elements from a matrix, where each dimension is given asingle index or vector of indices. A block is then extracted from the matrix, as illustrated next.

>> a=rand(3,4)

a =

0.6551 0.4984 0.5853 0.2551

0.1626 0.9597 0.2238 0.5060

0.1190 0.3404 0.7513 0.6991>> a(2,3)

ans =

0.2238


9/32

2.7. SUBMATRIX 9

1.12 Submatrix 11

>> a(1:2,2:3)

ans =

0.4984 0.5853

0.9597 0.2238

>> a(1,end)

ans =

0.2551>> a(1,:)

ans =

0.6551 0.4984 0.5853 0.2551

>> a(:,3)

ans =

0.5853

0.2238

0.7513

It is interesting to note that arrays are stored linearly in memory, from the first dimension, second, and soon. So we can in fact access vectors by a single index, as show below.

>> a=[1 2 3;4 5 6; 9 8 7]a =

1 2 3

4 5 6

9 8 7

>> a(3)

ans =

9

>> a(7)

ans =

3

>> a([1 2 3 4])

ans =

1 4 9 2

>> a(:)

ans =

1

4

9

2

5

8

3

6

7

Subscript referencing can also be used in both sides.

>> a

a =

1 2 3

4 5 6

9 8 7

>> b

b =

1 2 3

4 5 6

>> b(1,:)=a(1,:)

b =1 2 3

4 5 6

>> b(1,:)=a(2,:)


10/32

10 CHAPTER 2. MATLAB

b =

4 5 6

4 5 6

>> b(:,2)=[]

b =

4 6

4 6

>> a(3,:)=0a =

1 2 3

4 5 6

0 0 0

>> b(3,1)=20

b =

4 6

4 6

20 0

As you noted in the last example, we can insert one element in matrix B, and MATLAB automaticallyresizes the matrix.

2.8 M-files, scripts and functions

A M-file is a plain text file with MATLAB commands, saved with extension .m. The M-files can be scripts offunctions. By using the editor of MATLAB we can insert comments or statements and then save or compilethe m-file. Note that the percent sign % represents a comment. No statement after this sign will be executed.Comments are quite useful for documenting the file. M-files are useful when the number of statements islarge, or when you want to execute it at a later stage, or frequently, or even to run it in background. Asimple example of a script is given below.

% program 1

% programmer: Antonio ferreira

% date: 2008.05.30% purpose : show how M-files are built

% data: a - matrix of numbers; b: matrix with sines of a

a=rand(3,4);

b=sin(a);

Functions act like subroutines in fortran where a particular set of tasks is performed. A typical function isgiven below, where in the first line we should name the function and give the input parameters (m,n,p) inparenthesis and the output parameters (a,b,c) in square parenthesis.

function [a,b,c] = antonio(m,n,p)+

a = hilb(m);

b= magic(n);c= eye(m,p);

We then call this function as >> [a,b,c]=antonio(2,3,4) producing

>> [a,b,c]=antonio(2,3,4)

a =

1.0000 0.5000

0.5000 0.3333

b =

8 1 6

3 5 7

4 9 2

c =

1 0 0 00 1 0 0

It is possible to use only some output parameters.


11/32

2.9. LINEAR ALGEBRA 11

>> [a,b]=antonio(2,3,4)

a =

1.0000 0.5000

0.5000 0.3333

b =

8 1 6

3 5 7

4 9 2

2.9 Linear algebra

In our finite element calculations we typically need to solve systems of equations, or obtain the eigenvaluesof a matrix. MATLAB has a large number of functions for linear algebra. Only the most relevant for finiteelement analysis are here presented. Consider a linear system AX = B, where

>> a=rand(3)

a =

0.8909 0.1386 0.8407

0.9593 0.1493 0.25430.5472 0.2575 0.8143

>> b=rand(3,1)

b =

0.2435

0.9293

0.3500

The solution vector X can be easily evaluated by using the backslash command,

>> x=a\b

x =

0.7837

2.9335-1.0246

Consider two matrices (for example a stiffness matrix and a mass matrix), for which we wish to calculatethe generalized eigenproblem,

>> a=rand(4)

a =

0.1966 0.3517 0.9172 0.3804

0.2511 0.8308 0.2858 0.5678

0.6160 0.5853 0.7572 0.0759

0.4733 0.5497 0.7537 0.0540

>> b=rand(4)

b =0.5308 0.5688 0.1622 0.1656

0.7792 0.4694 0.7943 0.6020

0.9340 0.0119 0.3112 0.2630

0.1299 0.3371 0.5285 0.6541

>> [v,d]=eig(a,b)

v =

0.1886 -0.0955 1.0000 -0.9100

0.0180 1.0000 -0.5159 -0.4044

-1.0000 -0.2492 -0.2340 0.0394

0.9522 -0.8833 0.6731 -1.0000

d =

-4.8305 0 0 00 -0.6993 0 0

0 0 0.1822 0

0 0 0 0.7628


12/32


The MATLAB function eig can be applied to the generalized eigenproblem, producing matrix V, each columncontaining an eigenvector, and matrix D, containing the eigenvalues at its diagonal. If the matrices are thestiffness and the mass matrices then the eigenvectors will be the modes of vibration and the eigenvalues willbe the square roots of the natural frequencies of the system.

2.10 Matlab graphics

MATLAB can be easily used to plot the graphs. For example, to plot the function y = x2 .5x first enteran array of independent variables:

>> x = linspace(0,1,25)

> > y = x . ^ 2 - . 5 * x ;

>> plot(x,y)

The plot shows up in a new window. To plot in a different color, use

>> plot(x,y,r)

where the character string r means red. Use the helpwindow to see other options. To plot graphs on top ofeach other, use hold on.

>> hold on

>> z = exp(x);

>> plot(x,z)

>> plot(x,z,g)

hold off will stop simultaneous plotting. Alternatively, use

>> plot(x,y,r,x,z,g)

In the surface plots, x and y must give a rectangular array, and z is a matrix whose entries are the values ofthe function at the array points.

>> x =linspace(-1,1,40); y = x;

>> z = x * (y. ^ 2);>> surf(x,y,z)

Typing the command

>> rotate3d

will allow you to use the mouse interactively to rotate the graph to view it from other angles.

2.11 Computing errors

There are various kinds of errors that we encounter when using a computer for computation.

Truncation Error: Caused by adding up to a finite number of terms, while we should add infinitely

many terms to get the exact answer in theory.

Round-off Error: Caused by representing/storing numeric data in finite bits. Overflow/Underflow: Caused by too large or too small numbers to be represented/ stored properly

in finite bitsmore specifically, the numbers having absolute values larger/smaller than the maximum(fmax)/minimum (fmin) number that can be represented in MATLAB.

Negligible Addition: Caused by adding two numbers of magnitudes differing by over 52 bits, as can beseen in the last section.

Loss of Significance: Caused by a bad subtraction, which means a subtraction of a number from anotherone that is almost equal in value.

Error Magnification: Caused and magnified/propagated by multiplying/dividing a number containinga small error by a large/small number.

Errors depending on the numerical algorithms, step size, and so on


13/32

2.11. COMPUTING ERRORS 13

Although we cannot be free from these kinds of inevitable errors in some degree, it is not computers, butinstead human beings, who must be responsible for the computing errors. While our computer may insiston its innocence for an unintended lie, we programmers and users cannot escape from the responsibilityof taking measures against the errors and would have to pay for being careless enough to be deceived bya machine. We should, therefore, try to decrease the magnitudes of errors and to minimize their impacton the final results. In order to do so, we must know the sources of computing errors and also grasp thecomputational properties of numerical algorithms.

For instance, consider the following two formulas:

f1(x) =

x(

x + 1 x) f2(x) =

xx + 1 +

x

These are theoretically equivalent, hence we expect them to give exactly the same value. However, runningthe MATLAB program matlab1.m to compute the values of the two formulas, we see a surprising result

that, as x increases, the step of f1(x) incoherently moves hither and thither, while f2(x) approaches1

2at

a steady pace. We might feel betrayed by the computer and have a doubt about its reliability. Why doessuch a flustering thing happen with f1(x)? It is because the number of significant bits abruptly decreaseswhen the subtraction (

x + 1 x) is performed for large values of x, which is called loss of significance.

In order to take a close look at this phenomenon, let x = 1015. Then we have

x + 1 = 3.162277660168381 107 = 31622776.60168381

x = 3.162277660168379 107 = 31622776.60168379

These two numbers have 52 significant bits, or equivalently 16 significant digits (252 10523/10 1015) sothat their significant digits range from 108 to 108. Accordingly, the least significant digit of their sum anddifference is also the eighth digit after the decimal point (108).

x + 1 +

x = 63245553.20336761

x + 1 x = 0.00000001862645149230957 0.00000002

Note that the number of significant digits of the difference decreased to 1 from 16. Could you imagine that a

single subtraction may kill most of the significant digits? This is the very loss of significance, which is oftencalled catastrophic cancellation.

%matlab1

clear

f1 = inline(sqrt(x)*(sqrt(x + 1) - sqrt(x)),x);

f2 = inline(sqrt(x)./(sqrt(x + 1) + sqrt(x)),x);

x = 1 ;

format long e

for k = 1:15

fprintf(At x=%15.0f, f1(x)=%20.18f, f2(x) = %20.18f, x,f1(x),f2(x));

x = 10*x;

end

sx1 = sqrt(x+1); sx = sqrt(x); d = sx1 - sx; s = sx1 + sx;fprintf(sqrt(x+1) = %25.13f, sqrt(x) = %25.13f ,sx1,sx);

fprintf( diff = %25.23f, sum = %25.23f ,d,s);

The result should be

>> matlab1

At x= 1, f1(x)=0.414213562373095150, f2(x) = 0.414213562373095090

At x= 10, f1(x)=0.488088481701514750, f2(x) = 0.488088481701515480

At x= 100, f1(x)=0.498756211208899460, f2(x) = 0.498756211208902730

At x= 1000, f1(x)=0.499875062461021870, f2(x) = 0.499875062460964860

At x= 10000, f1(x)=0.499987500624854420, f2(x) = 0.499987500624960890

At x= 100000, f1(x)=0.499998750005928860, f2(x) = 0.499998750006249940

At x= 1000000, f1(x)=0.499999875046341910, f2(x) = 0.499999875000062490At x= 10000000, f1(x)=0.499999987401150920, f2(x) = 0.499999987500000580

At x= 100000000, f1(x)=0.500000005558831620, f2(x) = 0.499999998749999950

At x= 1000000000, f1(x)=0.500000077997506340, f2(x) = 0.499999999874999990


14/32


At x= 10000000000, f1(x)=0.499999441672116520, f2(x) = 0.499999999987500050

At x= 100000000000, f1(x)=0.500004449631168080, f2(x) = 0.499999999998750000

At x= 1000000000000, f1(x)=0.500003807246685030, f2(x) = 0.499999999999874990

At x= 10000000000000, f1(x)=0.499194546973835970, f2(x) = 0.499999999999987510

At x=100000000000000, f1(x)=0.502914190292358400, f2(x) = 0.499999999999998720

sqrt(x+1) = 31622776.6016838100000, sqrt(x) = 31622776.6016837920000

diff = 0.00000001862645149230957, sum = 63245553.20336760600000000000000

Tips for avoiding large errors are given below:First, in order to prevent loss of significance, it is important to avoid a bad subtraction (see the previousexample), that is, a subtraction of a number from another number having almost equal value.

Second, in order to decrease the magnitude of round-off errors and to lower the possibility of over-flow/underflow errors, make the intermediate result as close to 1 as possible in consecutive multiplica-

tion/division processes. According to this rule, when computingxy

z, we program the formula as

(xy)z

when x and y in the multiplication are very different in magnitude

xy

z

when y and z in the division are close in magnitude, and

xz

y when x and z in the division are close in magnitude

2.12 Toward good program

Among the various criteria about the quality of a general program, the most important one is how robust itsperformance is against the change of the problem properties and the initial values. A good program guidesthe program users who dont know much about the program and at least give them a warning message withoutruntime error for their minor mistake. There are many other features that need to be considered, such asuser friendliness, compactness and elegance, readability, and so on. But, as far as the numerical methodsare concerned, the accuracy of solution, execution speed (time efficiency), and memory utilization (spaceefficiency) are of utmost concern. Since some tips to achieve the accuracy or at least to avoid large errors

(including overflow/underflow) are given in the previous section, we will look over the issues of executionspeed and memory utilization.


15/32

Chapter 3

Taylors theorem

Recall from calculus the Taylors series for a function, f(x), expanded about some number, c, is written as

f(x) a0 + a1(x c) + a2(x c)2 + a3(x c)3 +

Here the symbol

is used to denote a formal series, meaning that convergence is not guaranteed in general.

The constants ai are related to the function f and its derivatives evaluated at x = c. When c = 0, this is aMacLaurin series.

For example we have the following Taylors series (with c = 0):

ex = 1 + x +x2

2!+

x3

3!+

sin(x) = x x3

3!+

x5

5!+

cos(x) = 1 x2

2!+

x4

4!

(3.1)

Theorem (Taylors Theorem)

If f(x) has derivatives of order n + 1 on the closed interval [a b], then for any x and c in this interval

f(x) =n

k=0

f(k)(c)(x c)kk!

+f(n+1)()(x c)n+1

(n + 1)!

where is some number between x and c; and fk(x) is the kth derivative of f at x.

We will use this theorem again and again in this class. The main usage is to approximate a function by thefirst few terms of its Taylors series expansion; the theorem then tells us the approximation is as good as thefinal term, also known as the error term. That is, we can make the following manipulation:

[f(x)

nk=0

f(k)(c)(x c)kk!

=

f(n+1)()(x c)n+1(n + 1)!

M|x c|n+1 (3.2)

Example 3.0.1 Find an approximation for f(x) = sin x expanded about c = 0, using n = 3.

Solution:

Solving for f(k) is fairly easy for this function. We find that

f(x) = sin x = sin(0) +cos(0)x

1!+

sin(0)x22!

+ cos(0)x3

3!+

sin()x4

4!

= x x3

6+

sin()x4

24

so

sin x x

x3

6 =

sin()x4

24

x4

24since | sin | 1.Example 3.0.2 Apply Taylors Theorem to expand f(x) = x3 21x2 + 17 around c = 1.

15


16/32

16 CHAPTER 3. TAYLORS THEOREM

Solution:

Simple calculus gives us

f(0)(x) = x3 21x2 + 17f(1)(x) = 3x2 42xf(2)(x) = 6x 42f(3)(x) = 6

f(k)(x) = 0

Evaluating these at c = 1 gives

f(x) = 3 + (39)(x 1) + (36)(x 1)2

2+

6(x 1)36

Note there is no error term, since the higher order derivatives are identically zero. By carrying out simplealgebra, you will find that the above expansion is, in fact, the function f(x).

Theorem (Taylors Theorem, Alternative Form)If f(x) has derivatives of order n + 1 on the closed interval [a b], then for any x in this interval and any hsuch that x + h is in this interval,

f(x + h) =n

k=0

f(k)(x)(h)k

k!+

f(n+1)()(h)n+1

(n + 1)!

where is some number between x and x + h.

We generally apply this form of the theorem with h 0. This leads to a discussion on the matter of Ordersof Convergence. We briefly describe it with the following example.Consider the Taylor expansion of ln x

ln(x + h) = ln x +(1/x)h

1+

(1/x2)h22

+(2/3)h3

6

Letting x = 1 we have

ln(1 + h) = h h22

+ 133

h3

Using the fact that is between 1 and 1 + h, as long as h is relatively small (say smaller than1

2), the term

1

33can be bounded by a constant, and thus

ln(1 + h)

h h2

2

= 133h3

Mh3

where M is a constant. Thus, we say that h h2

2is a O(h3) approximation to ln(1 + h), usually, we write

it as

ln(1 + h) = h h2

2+ O(h3)

It may be helpful for avoiding a bad subtraction, which we discussed in the previous section, to use theTaylor series expansion rather than using the exponential function directly for the computation of ex . Forexample, suppose we want to find

f(x) =ex 1

x, x = 0

We can use the Taylor series expansion up to just the fourth-order of ex about x = 0

ex 1 + x + x2

2+

1

6x3 +

1

24x4

to approximate the above function as

f(x) =ex 1

x 1 + x

2+

x2

6+

x3

24

The following MATLAB code shows the advantage by using Taylor series:


17/32

17

%matlab2: reduce the round-off error using Taylor series

f = inline((exp(x)-1)/x,x);

f1 = inline(((x/4+1)*x/3) + x/2+1,x);

x = 0; tmp = 1;

for k1 = 1:12

tmp = tmp * 0.1; x1 = x + tmp;

fprintf(At x = %14.12f, f(x) = %18.12e; f1(x) = %18.12e \n ,x1,f(x1),f1(x1));

end

The result is given below

At x = 0.100000000000, f(x) = 1.051709180756e+000; f1(x) = 1.084166666667e+000

At x = 0.010000000000, f(x) = 1.005016708417e+000; f1(x) = 1.008341666667e+000

At x = 0.001000000000, f(x) = 1.000500166708e+000; f1(x) = 1.000833416667e+000

At x = 0.000100000000, f(x) = 1.000050001667e+000; f1(x) = 1.000083334167e+000

At x = 0.000010000000, f(x) = 1.000005000007e+000; f1(x) = 1.000008333342e+000

At x = 0.000001000000, f(x) = 1.000000499962e+000; f1(x) = 1.000000833333e+000

At x = 0.000000100000, f(x) = 1.000000049434e+000; f1(x) = 1.000000083333e+000

At x = 0.000000010000, f(x) = 9.999999939225e-001; f1(x) = 1.000000008333e+000

At x = 0.000000001000, f(x) = 1.000000082740e+000; f1(x) = 1.000000000833e+000

At x = 0.000000000100, f(x) = 1.000000082740e+000; f1(x) = 1.000000000083e+000At x = 0.000000000010, f(x) = 1.000000082740e+000; f1(x) = 1.000000000008e+000

At x = 0.000000000001, f(x) = 1.000088900582e+000; f1(x) = 1.000000000001e+000


18/32

18 CHAPTER 3. TAYLORS THEOREM


19/32

Chapter 4

System of linear equations

4.1 Introduction

A number of problems in numerical analysis can be reduced to, or approximated by, a system of linear

equations. In this chapter, we deal with several numerical schemes for solving a system of equations

a11x1 + a12x2 + + a1nxn = b1a21x1 + a22x2 + + a2nxn = b2

......

......

an1x1 + an2x2 + + annxn = bn

(4.1)

which can be written in a compact form by using a matrixvector notation as

Ax = b

where A is a matrix, whose element in the ith row and jth column is aij ; and b is a column vector, whoseith entry is bi. This gives the easier way of writing this equation:

a11 a12 a13 a1na21 a22 a23 a2n

......

.... . .

...an1 an2 an3 ann

x1x2...

xn

=

b1b2...

bn

4.1.1 Elementary row operations

You may remember that one way to solve linear equations is by applying elementary row operations to agiven equation of the system. For example, if we are trying to solve the given system of equations, theyshould have the same solution as the following system:

a11 a12 a13 a1na21 a22 a23 a2n

......

.... . .

...kai1 kai2 kai3 kain

......

.... . .

...an1 an2 an3 ann

x1x2...

xi...

xn

=

b1b2...

kbi...

bn

where k is some given number which is not zero. It suffices to solve this system of linear equations, as it hasthe same solution(s) as our original system. Multiplying a row of the system by a nonzero constant is oneof the elementary row operations.

The second elementary row operation is to replace a row by the sum of that row and a constant times

19


20/32

20 CHAPTER 4. SYSTEM OF LINEAR EQUATIONS

another. Thus, for example, the following system of equations has the same solution as the original system:

a11 a12 a13 a1na21 a22 a23 a2n

......

.... . .

...ai1 + aj1 ai2 + aj2 ai3 + aj3 ain + ajn

.

..

.

..

.

..

.

. .

.

..an1 an2 an3 ann

x1x2...

xi.

..xn

=

b1b2...

bi + bj.

..bn

We have replaced the ith row by the ith row plus times the jth row.The third elementary row operation is to switch rows:

a21 a22 a23 a2na11 a12 a13 a1n

......

.... . .

...an1 an2 an3 ann

x2x1...

xn

=

b2b1...

bn

We have here switched the first and second rows.The idea of Gaussian Elimination is to use the elementary row operations to put a system into upper

triangular form then use back substitution. Well give an example here:

Example 4.1.1 Solve the set of linear equations

x1 + x2 x3 = 23x1 4x2 + 4x3 = 72x1 + x2 + x3 = 7

Solution:

We start by rewriting in the matrix form

1 1 1 | 2

3 4 4 | 72 1 1 | 7

We add 3 times the first row to the second, and 2 times the first row to the third to get

1 1 1 | 20 1 1 | 10 1 3 | 3

We now add 1 times the second row to the third row to get1 1 1 | 20 1 1 | 1

0 0 2 | 4

We now solve this by back substitution. Because the matrix is in upper triangular form, we can solve x3 bylooking only at the last equation; namely x3 = 2. However, once x3 is known, the second equation involvesonly one unknown, x2, and can be solved only by x2 = 3. Then the first equation has only one unknown,and is solved by x1 = 1.

All sorts of funny things can happen when you attempt Gaussian elimination: it may turn out that yoursystem has no solution, or has a single solution (as above), or an infinite number of solutions. We shouldexpect that an algorithm for automatic solution of systems of equations should detect these problems.

4.2 Algorithm problems

The algorithm we have outlined is far too right. Consider the following example.

Example 4.2.1 Solve the system of equations given by the matrix form0.0590 0.2372 | 0.35280.1080 0.4348 | 0.6452


21/32

4.3. PIVOTING STRATEGIES FOR GAUSSIAN ELIMINATION (GE) 21

Solution:

Note that the exact solution of this system is x1 = 10 and x2 = 1. Suppose, however, that the algorithmuses only 4 significant figures for its calculations. The algorithm pivots on the first equation. The multiplierfor the second row is

0.1080

0.0590 1.830508 which will be rounded to

1.831 by the algorithm.

The second entry in the matrix is replaced by

0.4348 (1.831)(0.2372) = 0.4348 + 0.4343 = 0.0005where the arithmetic is rounded to four significant figures each time. There is some serious subtractivecancellation going on here. We have lost three figures with this subtraction. The errors get worse from here.Similarly, the second matrix entry becomes

0.6452 (1.831)(0.3528) = 0.6452 0.6460 = 0.0008where, again, intermediate steps are rounded to four significant figures, and again there is subtractivecancelling. This puts the system in the form

0.0590 0.2372 | 0.35280 0.0005 | 0.0008

When the algorithm attempts back substitution, it gets the value

x2 =0.00080.0005 = 1.6

This is a bit off from the actual value of 1. The algorithm now finds

x1 = (0.3528 0.2372 1.6) = 12.41where each step has rounding to four significant figures. This is also a bit off.

4.3 Pivoting strategies for Gaussian elimination (GE)

Gaussian Elimination can fail when performed in the wrong order. If the algorithm selects a zero pivot, themultipliers are undefined, which is no good. We also saw that a pivot small in magnitude can cause failure,for example,

x1 + x2 = 1

x1 + x2 = 2

A simple method solves this as

x2 =2 11 1

= 1 1

x1 =

1

x2

=

1

1 If is very small, then

1

is enormous compared to both 1 and 2. With poor rounding, the algorithm solves

x2 = 1. Then it solves x1 = 0. This is nearly correct for x2, but is an awful approximation for x1. Note thatthis choice of x1, x2 satisfies the first equation, but not the second.

Now suppose the algorithm changed the order of the equations, then solved

x1 + x2 = 2

x1 + x2 = 1

The algorithm solves this as

x2 =1 21

x1 = 2 x2Theres no problem with rounding here.


22/32


4.4 Scaled partial pivoting

The simple GE algorithm uses the rows 1, 2, n 1 in order as pivot equations. As shown above, this cancause errors. Better is to pivot first row l1, then row l2, etc, until finally pivoting on row ln1, for somepermutation {li}ni=1 of the integers i = 1, 2,...,n. The strategy of scaled partial pivoting is to compute thispermutation so that GE works well.

In light of our example, we want to pivot on an element which is not small compared to other elements

in its row. So our algorithm first determines smallness by calculating a scale, row-wise:

si = max1jn

|ai,j|

The scales are only computed once. Then the first pivot, l1, is chosen to be the i such that

|ai1|si

is maximized. The algorithm pivots on row lk producing a bunch of zeros in the kth column.The slick way to implement this is to first set li = i, for i = 1, 2, , n. Then rearrange this vector in a

kind ofbubble sort: when you find the index that should be l1, swap them, i.e., find the j such that lj shouldbe the first pivot and switch the values of l1, lj .

Then at the kth step, search only those indices in the tail of this vector, i.e., only among lj for k

j

nand perform a swap.

We present an example of using scaled partial pivoting with GE. It is hard to come up with an examplewhere the numbers do not come out as ugly fractions. We look at the following example.

2 1 3 7 | 154 4 0 7 | 112 1 1 3 | 76 5 4 17 | 31

The scales are as follows: s1 = 7, s2 = 7, s3 = 3 and s4 = 17. We pick l1. It should be the index which

maximizes|ai1|

si. These values are:

2

7 ,

4

7 ,

2

3 ,

6

17We pick l1 = 3, and pivot

0 2 2 4 | 80 2 2 1 | 32 1 1 3 | 70 2 1 8 | 10

We pick l2. It should not be 3; and should be the index which maximizes|ai2|

si. These values are:

2

7,

2

7,

2

17

We have a tie. In this case we pick the second row, i.e., l2 = 2. We pivot

0 0 0 5 | 50 2 2 1 | 32 1 1 3 | 70 0 3 7 | 13

The matrix is in permuted upper triangular form. We could proceed, but would get a zero multiplier, andno changes would occur.

If we did proceed we would have l3 = 4. Then l4 = 1. Our row permutation is 3; 2; 4; 1. When we doback substitution, we work in this order reversed on the rows, solving x4; then x3; x2; x1. We get x4 = 1, so

x3 =1

3(13 7 1) = 2

x2 = 12 (3 1 1 + 2 2) = 0

x1 =1

2(7 3 1 1 2 1 0) = 1


23/32

4.5. LU FACTORIZATION 23

4.5 LU factorization

We examined GE to solve the system

Ax = b

where

a11 a12 a13 a1na21 a22 a23 a2na31 a32 a33 a3n

......

.... . .

...an1 an2 an3 ann

We want to show that GE actually factors A into lower and upper triangular parts (matrices), that is

A = LU

where

L =

1 0 0 . . . 0l21 1 0 0l31 l32 1

0

......

.... . . ...

ln1 ln2 ln3 1

, U =

u11 u12 u13 . . . u1n0 u22 u23 u2n0 0 u33

u3n

......

.... . . ...

0 0 0 unn

We call this a LU factorization of A.

Example 4.5.1 We consider solution of the following augmented form:

2 1 1 3 | 74 4 0 7 | 116 5 4 17 | 312 1 0 7 | 15

Solution:The GE reduces it to

2 1 1 3 | 70 2 2 1 | 30 0 3 7 | 130 0 0 12 | 18

We are going to run the GE gain, and see how it is a LU factorization. Since this is the GE version, we firstpivot on the first row. Our multipliers are 2; 3; 1. We pivot to get

2 1 1 3 | 70 2 2 1 | 30 2 1 8 | 100 2 1 4 | 8

Careful inspection shows that we have merely multiplied A and b by a lower triangular matrix M1

M1 =

1 0 0 02 1 0 03 0 1 01 0 0 1

This is like we are solving the system

M1Ax = M1b

We pivot on the second row to get 2 1 1 3 | 70 2 2 1 | 30 0 3 7 | 130 0 3 5 | 5


24/32


The multipliers are 1 and 1. We can view this pivot as a multiplication by M2 with

M2 =

1 0 0 00 1 0 00 1 1 00 1 0 1

We are solving M2M1Ax = M2M1b

We pivot on the third row, with a multiplier of 1. Thus we get

2 1 1 3 | 70 2 2 1 | 30 0 3 7 | 130 0 0 12 | 18

We have multiplied by M3

M3 =

1 0 0 00 1 0 00 0 1 00 0 1 1

Now we are solving

M3M2M1Ax = M3M2M1b

But we have an upper triangular form, that is, if we let

U =

2 1 1 30 2 2 10 0 3 70 0 0 12

Then we have

M3M2M1A = U

A = (M3M2M1)1U

A = M11 M12 M

13 U

A = LU

We are hoping that L is indeed lower triangular, and has ones on the diagonal. It turns out that the inverseof each Mi matrix has a nice form. We write them here (check it yourself)

L =

1 0 0 02 1 0 03 1 1 01

1

1 1

This is real interesting: the matrix L looks to be composed of ones on the diagonal and multipliers underthe diagonal.

Now we check to see if we made any mistakes.

LU =

1 0 0 02 1 0 03 1 1 01 1 1 1

2 1 1 30 2 2 10 0 3 70 0 0 12

=

2 1 1 34 4 0 76 5 4 172 1 0 7

= A

4.6 LU solution

We see that the GE algorithm can be used to actually calculate the LU factorization. We will look at this inmore detail in another example. We now examine how we can use the LU factorization to solve the equation

Ax = b


25/32

4.6. LU SOLUTION 25

Since we have A = LU, we first solve

Lz = b

then solve

U x = z

Since L is lower triangular, we can solve for z with a forward substitution. Similarly, since U is uppertriangular, we can solve for x with a back substitution. We drag out the previous example (which we nevergot around to solving):

2 1 1 3 | 74 4 0 7 | 116 5 4 17 | 312 1 0 7 | 15

We found that the LU factorization of A as

A = LU =

1 0 0 02 1 0 03 1 1 01 1 1 1

2 1 1 30 2 2 10 0 3 70 0 0 12

We solve

Lz =

1 0 0 02 1 0 03 1 1 01 1 1 1

z =

7113115

= b

We obtain

z =

731318

Now we solve

U x =

2 1 1 30 2 2 10 0 3 70 0 0 12

x =

731318

= z

We get this solution

x =

37

24

1712

5

6

3

2

T

Theorem If A is an n n matrix, and the Gaussian Elimination does not encounter a zero pivot, then thealgorithm generates a LU factorization of A, where L is the lower triangular part of the output matrix, andU is the upper triangular part

A = LU

This theorem relies on us using the fancy version of GE, which saves the multipliers in the spots where thereshould be zeros. If correctly implemented, then, L is the lower triangular part but with ones put on thediagonal.

Example 4.6.1 Find LU factorization for

A =

1 4 54 20 32

5 32 64

Solution:

We first write the desired product as

1 0 0l21 1 0

l31 l32 1

u11 u12 u130 u22 u23

0 0 u33

=

1 4 54 20 32

5 32 64


26/32


We begi by solving the first row of U and the first column of L:

(1)u11 = a11 = 1 (1)u12 = a12 = 4 (1)u13 = a13 = 5

l21u11 = a21 = 4 l31u11 = a31 = 5

Next, using these values, we find that the second row of U and the second column of L:

1 0 04 1 05 l32 1

1 4 50 u22 u230 0 u33

= 1 4 54 20 325 32 64

(4)(4) + u22 = 20 = u22 = 20 16 = 4(4)(5) + u23 = 32 = u23 = 12

(5)(4) + l32u22 = 32 = l32 = 3Finally, the only remaining unknown in matrix U is determined

1 0 04 1 0

5 3 1

1 4 50 4 12

0 0 u33

=

1 4 54 20 32

5 32 64

(5)(5) + (3)(12) + u33 = 64 = u33 = 3The factorization is

L =

1 0 04 1 0

5 3 1

, U =

1 4 50 4 12

0 0 3

4.7 Factorization of a general matrix

LU factorization of a nonsingular (square) matrix A means expressing the matrix as the multiplication ofa lower triangular matrix L and an upper triangular matrix U, where a lower/upper triangular matrix is amatrix having no nonzero elements above/below the diagonal. For the case where some row switching oper-

ation is needed like in the Gauss elimination, in this section we include a permutation matrix P representingthe necessary row switching operation(s) to write the LU decomposition as

P A = LU

The usage of a permutation matrix is exemplified by

P A =

0 0 11 0 0

0 1 0

a11 a12 a13a21 a22 a23

a31 a32 a33

=

a31 a32 a33a11 a12 a13

a21 a22 a23

which denotes switching the first and third rows followed by switching the second and third rows. Aninteresting and useful property of the permutation matrix is that its transpose agrees with its inverse

PTP = I PT = P1

To take a close look at the LU decomposition from the previous example, the process is that we begin byfinding u11 = a11, and then solving for the remaining elements in the first row of U and the first column ofL. At the second stage, we find u22 and then the remainder of te second row of U and the second columnof L. Continuing in this manner, we determine all of the elements of U and L, as shown in the previousexample. This process is described in the following algorithm

function [L,U,P] = ludcmp(A)

%This gives LU decomposition of A with the permutation matrix P

% denoting the row switch(exchange) during factorization

NA = size(A,1);

AP = [A eye(NA)]; %augment with the permutation matrix.for k = 1:NA - 1

%Partial Pivoting at AP(k,k)

[akx, kx] = max(abs(AP(k:NA,k)));


27/32

4.7. FACTORIZATION OF A GENERAL MATRIX 27

if akx < eps

error(Singular matrix and No LU decomposition)

end

mx = k+kx-1;

if kx > 1 % Row change if necessary

tmp_row = AP(k,:);

AP(k,:) = AP(mx,:);

AP(mx,:) = tmp_row;end

% LU decomposition

for m = k + 1: NA

AP(m,k) = AP(m,k)/AP(k,k); %Eq.(2.4.8.2)

AP(m,k+1:NA) = AP(m,k + 1:NA)-AP(m,k)*AP(k,k + 1:NA);

end

end

P = AP(1:NA, NA + 1:NA + NA); %Permutation matrix

for m = 1:NA

for n = 1:NA

if m == n, L(m,m) = 1.; U(m,m) = AP(m,m);

elseif m > n, L(m,n) = AP(m,n); U(m,n) = 0.;else L(m,n) = 0.; U(m,n) = AP(m,n);

end

end

end

if nargout == 0, disp(L*U = P*A with); L,U,P, end

%%%%%%%%%%%%%%%%%%%%%%%% the end of LU %%%%%%%%%%%%%%%%%

%

%You can check if P*L*U = A?

>> A = [1 2 5;0.2 1.6 7.4; 0.5 4 8.5]

>>[L,U,P] = lu_dcmp(A) %LU decomposition

L =

1.000000000000000e+000 0 0

5.000000000000000e-001 1.000000000000000e+000 02.000000000000000e-001 4.000000000000001e-001 1.000000000000000e+000

U =

1 2 5

0 3 6

0 0 4

P =

1 0 0

0 0 1

0 1 0

Once we have the LU decomposition of the coefficient matrix A = PTLU, it is more efficient to use the

lower/upper triangular matrices for solving equation Ax = b than to apply the Gauss elimination method.The procedure is as follows:

PTLU x = b, LU x = P b, U x = L1P b, x = U1L1P b

Note that the premultiplication ofL1 and U1 by a vector can be performed by the forward and backwardsubstitution, respectively. The following program applies the forward and backward substitutions:

function x = forsubst(L,B)

%forward substitution for a lower-triangular matrix equation Lx = B

N = size(L,1);

x(1,:) = B(1,:)/L(1,1);

for m = 2:N

x(m,:) = (B(m,:)-L(m,1:m - 1)*x(1:m-1,:))/L(m,m);end

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

function x = backsubst(U,B)


28/32


%backward substitution for a upper-triangular matrix equation Ux = B

N = size(U,2);

x(N,:) = B(N,:)/U(N,N);

for m = N-1: -1:1

x(m,:) = (B(m,:) - U(m,m + 1:N)*x(m + 1:N,:))/U(m,m);

end

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

4.8 Other factorization: Cholesky and SVD

There are several other matrix decompositions such as Cholesky decomposition, and singular value decom-position (SVD). Instead of looking into the details of these algorithms, we will simply survey the MATLABbuilt-in functions implementing these decompositions.

Cholesky decomposition factors a positive definite symmetric/Hermitian matrix into an upper triangularmatrix premultiplied by its transpose as

A = UTU (U : an upper triangular matrix)

and is implemented by the MATLAB built-in function chol().

>>A = [2 3 4;3 5 6;4 6 9]; %a positive definite symmetric matrix>> U=chol(A)

U=

1.414213562373095e+000 2.121320343559642e+000 2.828427124746190e+000

0 7.071067811865481e-001 1.256073966947019e-015

0 0 1.000000000000001e+000

>>U*U - A %to check if the result is right

SVD (singular value decomposition) is to express an M N matrix A in the following form

A = U SVT

where U is an orthogonal (unitary) M M matrix, V is an orthogonal (unitary) N N matrix, and S is areal diagonal M N matrix having the singular values of A (the square roots of the eigenvalues of ATA) in

decreasing order on its diagonal. This is implemented by the MATLAB built-in function svd().A = [1 2;2 3;3 5]; %a rectangular matrix

>> [U,S,V] = svd(A) %Singular Value Decomposition

U =

-3.092342188617865e-001 7.556724585107019e-001 -5.773502691896251e-001

-4.998144365796163e-001 -6.456409185090943e-001 -5.773502691896263e-001

-8.090486554414028e-001 1.100315400016061e-001 5.773502691896258e-001

S =

7.207096748493507e+000 0

0 2.403257328175770e-001

0 0

V =

-5.183792016010000e-001 -8.551508658403556e-001-8.551508658403556e-001 5.183792016010000e-001

err = U*S*V-A %to check if the result is right

err =

-1.110223024625157e-016 -2.220446049250313e-016

8.881784197001252e-016 4.440892098500626e-016

8.881784197001252e-016 0

4.9 Iterative methods to solve equations

We saw that Gaussian Elimination requires around 23

n3 operations just to find the LU factorization, then

about n2 operations to solve the system, when A is nn. Ifn is large, this may take too long to be practical.Thus we look for an alternative algorithm.


29/32

4.10. JACOBI METHOD 29

4.10 Jacobi method

Let us consider a simple case, find the solution of the equation

3x + 1 = 0

which can be cast into an iterative scheme as

2x = x 1, x = x + 1

2 , = xk+1 =1

2 xk 1

2Starting from some initial value x0 for k = 0, we can incrementally change k by 1 each time to proceed asfollows:

x1 = 21 21x0x2 = 21 21x1 = 21 + 22 + 22x0x3 = 21 21x2 = 21 + 22 23 23x0

Whatever the initial value x0 is, this process will converge to the sum of a geometric series with the ratio of

12

as

xk =a0

1 r = 1

21 12

= 1

3 k and what is better, the limit is the very true solution to the given equation. We are happy with this, butmight feel uneasy, because we are afraid that this convergence to the true solution is just a coincidence. Willit always converge, no matter how we modify the equation so that only x remains on the LHS?To answer this question, let us try another iterative scheme.

x = 2x 1, = xk+1 = 2xk 1x1 = 1 2x0x2 = 1 2x1 = 1 2(1 2x0) = 1 + 2 + 22x0x3 = 1 2x2 = 1 + 2 22 23x0

This iteration will diverge regardless of the initial value x0. But, we are never disappointed, since we knowthat no one can be always lucky. To understand the essential difference between these two cases, we shouldknow the fixed-point theorem, we omit it here. Apart from this, let us go into a system of equations.

3 21 2

x1x2

=

1

1

, = Ax = b

Dividing the first equation by 3 and transposing all term(s) other than x1 to the RHS and dividing thesecond equation by 2 and transposing all term(s) other than x2 to the RHS, we have

xk+11xk+12

=

0 2

3 12

0

xk1xk2

+

13 12

A matrix form would be

xk+1 = Axk + b (4.2)

Assuming that this scheme works well, we set the initial value to zero (x0 = 0) and proceed as

xk = [I + A + A2 + ]b = [I A]1b =

1 2/31/2 1

11/3

1/2

=1

1 1/3

1 2/31/2 1

1/3

1/2

=1

2/3

2/3

2/3

=

1

1

which will converge to the true solution x =

1

1

. This suggests another method of solving a system of

equations, which is called Jacobi iteration. It can be generalized for an N N matrixvector equation asfollows

am1x1 + am2x2 + + ammxm + + amNxN = bm

x(k+1)m = N

n=m

amnamm

xkn +bm

amm, m = 1, 2, , N


30/32


The matrix form isxk+1 = Axk + b

where

0 a12/a11 a1N/a11a21/a22 0 a2N/a22

......

......

aN1/aNN aN2/aNN 0

, b =

b1/a11b2/a22

...

bN/aNN

This scheme is implemented by the following MATLAB routine jacobi(). You can run it to solve the aboveequation.

function [x,J,c] = jacobi(A,b,n,z)

%

% x = jacobi(A,b,n,z)

%

% Jacobi iteration on system A*x = b with printing

% n -- number of iterations

% z -- initial vector (default 0)

%

% x -- final iterate% J -- Jacobi matrix

% c -- Jacobi vector

%

if nargin


31/32

4.12. THE CONVERGENCE OF JACOBI AND GAUSSSEIDEL ITERATIONS 31

%

% Gauss-Seidel iteration on system A*x = b with printing

% using matrix multiplication (not optimal)



%

% x -- final iterate

% G -- Gauss-Seidel matrix% c -- Gauss-Seidel vector

%

if nargin N

n=m

|amn|, for m = 1, 2, , N

This implies that the convergence of the iterative schemes is ensured if, in each row of coefficient matrixA, the absolute value of the diagonal element is greater than the sum of the absolute values of the otherelements. It should be noted, however, that this is a sufficient, not a necessary, condition. In other words,the iterative scheme may work even if the above condition is not strictly satisfied.

One thing to note is the relaxation technique, which may be helpful in accelerating the convergence ofGaussSeidel iteration. It is a slight modification of equation (4.3) as

x(k+1)m = (1 )x(k)m + bm

m1n=1 amnx

(k+1)n

Nn=m+1 amnx

(k)n

amm(4.4)

0 < < 2

and is called SOR (successive overrelaxation) for the relaxation factor 1 < < 2 and successive underrelax-ation for 0 < < 1. But regrettably, there is no general rule for selecting the optimal value of the relaxationfactor . The SOR method can be easily programmed in the following program sor().

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

function x = sor(A,b,n,w,z)

%

% x = sor(A,b,n,w,z)

%

% SOR iteration on system A*x = b with printing


% w -- SOR parameter


%% x -- final iterate

%

if nargin


32/32


[m,l] = size(A);

D = diag(diag(A));

J = D\(D - A);

c = D\b;

x=z;

for k = 1:n

for i=1:m

x(i,1) = (1-w)*x(i,1) + w*(J(i,:) * x + c(i,1));end

fprintf(1,%3d ,k)

fprintf(1,%5.4f ,x)

fprintf(1,\n)

end

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

Documents

Lecturenotes123 (1)