Lec04 Perturbation

8/9/2019 Lec04 Perturbation

1/39

ERROR AND SENSITIVTY ANALYSIS FOR SYSTEMS

OF LINEAR EQUATIONS

Read parts of sections 2.6 and 3.5.3

Conditioning of linear systems.

Estimating errors for solutions of linear systems

Backward error analysis

Relative element-wise error analysis

4-1 GvL 3.5; Heath 2.3; TB 12 pert


2/39

Perturbation analysis for linear systems (Ax=b)

Question addressed by perturbation analysis: determinethe variation of the solution x when the data, namely

A and b, undergoes small variations. Problem is Ill-

conditioned if small variations in data cause very largevariation in the solution.



3/39

Analysis I: Asymptotic First Order Analysis

Let E, be an n n matrix and eb be an n-vector.

Perturb A into A() =A +E and b into b+eb.

Note: A+ E is nonsingular for small enough.

Why?

The solution x() of the perturbed system is s.t.(A +E)x() =b+eb.



4/39

Let () =x() x. Then,

(A+ E)() = (b+eb) (A+E)x= (eb Ex)() = (A+ E)1(eb Ex).

x() is differentiable at = 0 and its derivative is

x(0) = lim0()

=A1 (eb Ex) .

A small variation[E, eb]will cause the solution to vary

by roughly x(0) =A1

(eb Ex).



5/39


6/39

The quantity (A) = A A1 is called thecondition

numberof the linear system with respect to the norm ..When using the p-norms we write:

p(A) = ApA1p

Note: 2(A) =max(A)/min(A)= ratio of largest to

smallest singular values ofA. Allows to define2(A)whenA is not square.

Determinant *is not* a good indication of sensitivity

Small eigenvalues *do not* always give a good indicationof poor conditioning.



7/39


8/39

Rigorous norm-based error bounds

Previous bound is valid only when perturbation is small

enough, where small is not precisely defined.

New bound valid within an explicitly given neighborhood.THEOREM 1: Assume that (A + E)y = b + eb and

Ax = b and that A1E < 1. Then A + E is

nonsingular andx y

x

A1 A

1 A1 E

E

A+

eb

b



9/39

To prove, first need to show that A+E is nonsingular

ifA is nonsingular and E is small. Begin with simple case:

LEMMA: IfE


10/39

d) (I E)1 = limk

ki=0 E

i. We write this as

(I E)1 =

i=0

Ei

e) Finally:

(I E)1 =

limk

k

i=0Ei

= limk

k

i=0Ei

limk

ki=0

Ei limk

ki=0

Ei

1

1 E



11/39

Can generalize result:

LEMMA: If A is nonsingular and A1

E < 1 thenA +E is non-singular and

(A +E)1 A1

1A1 E

Proof is based on relation A + E=A(I+ A1E) and useof previous lemma.



12/39

Now we can prove the theorem:

THEOREM 1: Assume that (A + E)y = b + eb andAx = b and that A1E < 1. Then A + E is

nonsingular and

x yx

A1

A1 A1 E

EA

+ ebb



13/39

Proof: From (A+E)y =b+eb and Ax=b we get

(A+ E)(y x) =eb Ex. Hence:

y x= (A +E)1(eb Ex)

Taking norms y x (A+E)1 [eb + Ex]

Dividing by x and using result of lemmay x

x (A +E)1 [eb/x + E]

A11 A1E

[eb/x + E]

A1A

1 A1E eb

Ax

+E

A

Result follows by using inequality Ax b.... QED



14/39


15/39

Another common form:

THEOREM 2: Let (A+ A)y = b+ b and Ax = bwhere A E, b eb, and assume that

A1E


16/39


17/39

Normwise backward error in just A or b

Suppose we model entire perturbation in RHS b.

Let r =b Ay be the residual.

Then y satisfies Ay =b+ b with b=r exactly.

The relative perturbation to the RHS is rb

.

Suppose we model entire perturbation in matrix A.

Then y satisfies

A ryT

yTy

y =b

The relative perturbation to the matrix is

ryT

yTy

2

/A2 = r2

Ay2



18/39


19/39

yis given (a computed solution). Eand ebto be selected

(most likely directions of perturbation for A and b).

Typical choice: E=A, eb =b

Explain why this is not unreasonable

Let r =b Ay. Then we have:

THEOREM 3: E,eb(y) = r

Ey+eb

Normwise backward error is for case E=A, eb =b:A,b(y) =

rAy+b



20/39

Show how this can be used in practice as a means to

stop some iterative method which computes a sequence of

approximate solutions to Ax=b.



21/39


22/39

Proof of Theorem 3

Let D Ey + eb and E,eb(y). The theoremstates that = r/D. Proof in 2 steps.

First: Any A,b pair satisfying (1) is such that

r/D. Indeed from (1) we have (recall that r =b Ay)

Ay+ Ay =b+ b r = Ay b

r Ay+b (Ey+eb) r

D

Second: We need to show an instance where the minimum

value ofr/D is reached. Take the pair A, b:

A=rzT; b=r with =Ey

D; =

eb

D

The vector z depends on the norm used - for the 2-norm:z =y/y2. Here: Proof only for 2-norm


23/39

a) We need to verify that first part of (1) is satisfied:

(A+ A)y = Ay+r yT

y2y =b r+r

= b (1 )r =b 1 Ey

Ey + eb r= b

eb

Dr =b+ r

(A+ A)y = b+ b The desired result

b) Finally: Must now verify that A = E and

b =eb. Exercise: Show that uvT2 = u2v2

A = ||y2

ryT = EyD

ryy2

=E

b = ||r =eb

Dr =eb QED



24/39


25/39

Show: a function which satisfies the first 2 requirements

of vector norms (1. (x) 0 (==0, iff x = 0) and 2.

(x) = ||(x)) satisfies the triangle inequality iff itsunit ball is convex.

(Continued) Use the above to construct a norm in R2

that is *not* absolute.



26/39

Define absolute *matrix* norms in same way. Which of

the norms A1, A, A2, and AFare absolute?

Recall that for any matrix f l(A) =A+E with |E|

u |A|. For an absolute matrix norm

EA

u

What does this imply?

Component-wise analysis requires that we use normsthat are *absolute*

We will restrict analysis to .

See sec. 2.6.5 of text.



27/39

Analogue of theorem 2 for case E= |A|, eb = |b|:

THEOREM 4 Let Ax = b and (A+ A)y = b+ bwhere |A| |A| and |b| |b|. Assume that

(A) =


28/39

Componentwise relative condition number :

C(A) |A1| |A|

Redo example seen after Theorem 3, (6 6 Vander-

monde system) using componentwise analysis.



29/39


30/39

Example of ill-conditioning: The Hilbert Matrix

Notorious example of ill conditioning.

Hn =

1 12

13

1n

1

2

1

3

1

4

1

n+1... ... ... ... ...1n

1n+1

12n1

i.e., hij = 1

i +j 1

Forn= 5 2(Hn) = 4.766.. 105.

Let bn =Hn(1, 1, . . . , 1)T.

Solution ofHnx=b is (1, 1, . . . , 1)T

. Let n= 5 and perturb h5,1 = 0.2 into 0.20001.

New solution: (0.9937, 1.1252, 0.4365, 1.865, 0.5618)T



31/39

Estimating condition numbers.

Avoid the expense of computing A1 explicitly.

Choose a random or carefully chosen vector v.

Solve Au=v using factorization already computed. Then A1 u / v is an guess-timate of A1.

Estimated condition number is (A) Au / v.

Typical choice for v: choose [ 1 ] with signs

chosen on the fly during back-substitution to maximize the

next entry in the solution, based on the upper triangular

factor from Gaussian Elimination.



32/39

Condition Number Measures How Close to Singularity

1/ relative distance to nearest singular matrix.

Let A, B be two n n matrices with A nonsingular and

B singular. Then

1

(A)

A B

A

Proof: B singular x = 0 such that Bx= 0.

x = A1Ax A1 Ax = A1(A B)x

A1

A BxDivide both sides by x (A) = xA A1

result. QED.



33/39

Example:

let A =

1 11 0.99

and B =

1 11 1

Then 1

1(A) 0.01

2 1(A)

20.01

= 200.

It can be shown that (Kahan)

1

(A) = minBA B

A | det(B) = 0



34/39

Estimating errors from residual norms

Let x an approximate solution to system Ax = b (e.g.,computed from an iterative process). We can compute

the residual norm:

r = b AxQuestion: How to estimate the error xx from r?

One option is to use the inequalityxx

x (A) r

b.

We must have an estimate of(A).



35/39

Proof of inequality.

First, note that A(x x) =b Ax=r. So:x x = A1r A1 r

Also note that from the relation b=Ax, we get

b = Ax A x x b

A

Therefore,

x x

x

A1 r

b/A= (A)

r

b

Show thatxx

x 1

(A)rb

.



36/39

S ll E l


37/39

Small Example

Solve Ax =b problem in 3-digit decimal arithmetic:0.641 0.242

0.321 0.121

x1x2

=

0.883

0.442

Solution by standard algorithm is y =

0.708

1.775

residual r =b Ay =

7.12

1.12

104.

(actually used 11 bit arithmetic, printed in decimal.)


E ti t f d i ll l


38/39

Estimate forward error in small example

Get estimated condition number from(1, 2) A= (0.001, 0.000)

Conclude A11 (1, 2)1

(0.001, 0.000)1

= 3000.

Combine with A1 = 0.962 1.

Get lower bound on Condition Number: 1(A) 3000.

So Theorem 1 simplified gives

x y

x A1 A

eb

b 3000

8.24 104

1.325 = 1.866,

predicting no accuracy!

Keep over 4 decimal digits to get any accuracy at all!



39/39

Documents

Lec04 Perturbation