Linear System of Equations -Conditioning

Numerical experimentsInput has uncertainties:

• Errors due to representation with finite precision• Error in the sampling

Onceyouselectyournumericalmethod,howmucherrorshouldyouexpecttoseeinyouroutput?

Isyourmethodsensitivetoerrors(perturbation)intheinput?

Demo “HilbertMatrix-ConditionNumber”

t⇒→

A x = b ✓A) random

① Defining A -B) Hilbert

(NXN)

② Start with a knowexact solution :

+true = [ 1 , I , - . - . , I ] (np.ones.CN2)

③ Compute be A@ xtrue

④ Solve Ab Is → Xsolve

⑤ Compute error H Xsolve - Xtrue H

Sensitivity of Solutions of Linear SystemsSuppose we start with a non-singular system of linear equations ! " = $.

We change the right-hand side vector $ (input) by a small amount Δ$.

How much the solution " (output) changes, i.e., how large is Δ"?

Output Relative errorInput Relative error = Δ1 / 1

Δ3 / 3 = Δ1 3Δ3 1

Fs -ra FT- --

At =b → exact

A- I = 5 → pertI = x tsx b = b t SbA- (xtsx) = btsb → /ADX=bb

Sensitivity of Solutions of Linear SystemsOutput Relative errorInput Relative error = Δ1 / 1

Δ3 / 3 = Δ1 3Δ3 1

Output Relative errorInput Relative error =

TAXISA-Ax- Ab

HA"H ⑧- A- Ab

" a.is. " ,if"iII=

in::i÷÷÷:

Δ"" ≤ $&' $ Δ%

%

Sensitivity of Solutions of Linear Systemsi÷s-

---

F To

Sensitivity of Solutions of Linear SystemsWe can also add a perturbation to the matrix ! (input) by a small amount (, such that

(! + () ," = $

and in a similar way obtain:

Δ"" ≤ !!" ! (

!

•O

-

T CornellA)

Condition number

Demo “HilbertMatrix-ConditionNumber”

The condition number is a measure of sensitivity of solving a linear system of equations to variations in the input.

The condition number of a matrix !:

./01 ! = !!" !

Recall that the induced matrix norm is given by

! = max# $" !"

And since the condition number is relative to a given norm, we should be precise and for example write:

./01% ! or ./01& !

--

p p

① -

Condition numberΔ"" ≤ ./01 ! Δ$

$

Small condition numbers mean not a lot of error amplification. Small condition numbers are good!

But how small?

A- → singloudCA) =D-

conedCA) = HAH HA' 'll

1) H X H > O - cordCA) so

2) ll X Y H f Il XII 11411 → HAA-' II f HAH HA

''ll

HAITTIA' ' Il > HI 11Hell = hfffx.gl/IHI--1/HAHHA-'H3aTf→

Condition numberΔ"" ≤ ./01 ! Δ$

$

Small condition numbers mean not a lot of error amplification. Small condition numbers are good!

Recall that 5 = max# $" 5 " = 1

Which provides with a lower bound for the condition number:

./01 ! = !!" ! ≥ !!"! = 5 = 1

If !!" does not exist, then ./01 ! = ∞ (by convention)

Recall Induced Matrix Norms

< 4 = max5 >674

8?65

< 9 = max6 >574

8?65

< : = max; @;

9' are the singular value of the matrix !

Maximum absolute column sum of the matrix !

Maximum absolute row sum of the matrix !=

A-'

e"Byo!

-

{ 100 , 13 , 0.5 } { too its ' o÷s }

11 A Hz = 100

HA' '

112 = 2

candCA) = 100 × 2 = 2001

Condition Number of Orthogonal Matrices

What is the 2-norm condition number of an orthogonal matrix A?

./01 ! = !!" ( ! (= !) ( ! ( = 1

That means orthogonal matrices have optimal conditioning.

They are very well-behaved in computation.

condra, {"

small → well - conditioned&

large → ill - conditionedN

About condition numbers1. For any matrix !, ./01 ! ≥1

2. For the identity matrix 5, ./01 5 = 1

3. For any matrix ! and a nonzero scalar ;, ./01 ;! = ./01 !

4. For any diagonal matrix <, ./01 < = *+# ,!*-. ,!

5. The condition number is a measure of how close a matrix is to being singular: a matrix with large condition number is nearly singular, whereas a matrix with a condition number close to 1 is far from being singular

6. The determinant of a matrix is NOT a good indicator is a matrix is near singularity detfA) = o → sing

Residual versus errorOur goal is to find the solution " to the linear system of equations ! " = $

Let us recall the solution of the perturbed problem

," = " + Δ"

which could be the solution of

! ," = $ + Δ$ , ! + ( ," = $, (! + () ," = $ + Δ$

And the error vector as = = Δ" = ," − "

We can write the residual vector as

? = $ − ! ,"

Demo “HilbertMatrix-ConditionNumber”

O-

te e

-- - -

- = -0

③ Its Eel ¥

Relative residual: !" #

(How well the solution satisfies the problem)

Relative error: $##(How close the approximated solution is

from the exact one)

→-

-

=

Residual versus error

It is possible to show that the residual satisfy the following inequality:

?! ," ≤ . @/

Where . is “large” constant when LU/Gaussian elimination is performed without pivoting and “small” with partial pivoting.

Therefore, Gaussian elimination with partial pivoting yields small relative residual regardless of conditioning of the system.

When solving a system of linear equations via LU with partial pivoting, the relative

residual is guaranteed to be small!

±

Residual versus errorLet us first obtain the norm of the error:

At = b ⇒ x = A-'

b

Il DX It = 11 I - x H = H Ajax - A' '

b ll = HA- 'fAb)Hr

HDX It = H A-'r ll H A

' ' II ll r ll

* Tix' Hittin-

YEI ' "IHA" ancordCA)

llsxH⇒

floridCA) HrdHxHHAH

Rule of thumb for conditioningSuppose we want to find the solution " to the linear system of equations ! " = $ using LU factorization with partial pivoting and backward/forward substitutions.

Suppose we compute the solution ,".

If the entries in ! and $ are accurate to S decimal digits,

and ./01 ! = BC0,

then the elements of the solution vector ," will be accurate to about

D −Edecimal digits

per -1%4%10's

-0-0gerehffyy-fcondcmttfbbuI-le.no?YoIs-w'Donee