Math for CS Lecture 2 1

• Solution of Linear Systems of Equations

• Consistency

• Rank

• Geometric Interpretation

• Gaussian Elimination

Lecture 2. Contents

Math for CS Lecture 2 2

Systems of linear equationsIn Linear equations the unknowns appear raised to the power

one.

If the variables in the equation are ordered as x1, … xn and the

missing variables

xi are written as … + 0·xi + … , then the linear equation can be

written in the

matrix notation, according to the matrix multiplication rules.

Solution of linear

equations was one of the reasons, for matrix notation invention.

Math for CS Lecture 2 3

Linear Systems in Matrix Form

nnnnnn

nn

nn

bxaxaxa

bxaxaxa

bxaxaxa

2211

22222121

11212111

nnnnnn

n

n

b

b

b

x

x

x

aaa

aaa

aaa

2

1

2

1

21

22221

11211

(1)

Math for CS Lecture 2 4

bxA Each side of the equation

bAxAA 11 Can be multiplied by A-1 :

Due to the definition of A-1: xxIxAA 1

Therefore the solution of (2) is:

(2)

bAx 1

Solution of Linear Systems

Math for CS Lecture 2 5

Consistency (Solvability)

• A-1 does not exist for every A

• The linear system of equations A·x=b has a solution, or said to be consistent IFF

Rank{A}=Rank{A|b}

• A system is inconsistent when

Rank{A}<Rank{A|b} Rank{A} is the maximum number of linearly independent columns or rows of A. Rank can be found by using ERO (Elementary Row Oparations) or ECO (Elementary column operations).

Math for CS Lecture 2 6

Elementary row and column operations• The following operations applied to the

augmented matrix [A|b], yield an equivalent linear system

– Interchanges: The order of two rows/columns can be changed

– Scaling: Multiplying a row/column by a nonzero constant

– Sum: The row can be replaced by the sum of that row and a nonzero multiple of any other row.

One can use ERO and ECO to find the Rank as follows:EROminimum # of rows with at least one nonzero entryorECOminimum # of columns with at least one nonzero entry

Math for CS Lecture 2 7

An inconsistent example: Geometric interpretation

5

4

42

21

2

1

x

x

00

21Rank{A}=1

Rank{A|b}=2 > Rank{A}

ERO:Multiply the first row with -2 and add to the second row

3

4

0

2

0

1

Math for CS Lecture 2 8

Uniqueness of solutions

• The system has a unique solution IFF

Rank{A}=Rank{A|b}=n

n is the order of the system

• Such systems are called full-rank systems

Math for CS Lecture 2 9

Full-rank systems

• If Rank{A}=n

Det{A} 0 A-1 exists Unique solution

2

4

11

21

2

1

x

x

Math for CS Lecture 2 10

Rank deficient matrices

• If Rank{A}=m<n

Det{A} = 0 A is singular so not invertible

infinite number of solutions (n-m free variables)

under-determined system

8

4

42

21

2

1

x

x

Consistent so solvable

Rank{A}=Rank{A|b}=1

Math for CS Lecture 2 11

Ill-conditioned system of equations

• A small deviation in the entries of A matrix, causes a large deviation in the solution.

47.1

3

99.048.0

21

2

1

x

x

1

1

2

1

x

x

47.1

3

99.049.0

21

2

1

x

x

0

3

2

1

x

x

Math for CS Lecture 2 12

Ill-conditioned continued.....

• A linear system

of equations is

said to be “ill-

conditioned” if

the coefficient

matrix tends to

be singular

Math for CS Lecture 2 13

Gaussian Elimination

– By using ERO, matrix A is transformed into an upper triangular matrix (all elements below diagonal 0)

– Back substitution is used to solve the upper-triangular system

n

i

n

i

nnnin

iniii

ni

b

b

b

x

x

x

aaa

aaa

aaa

11

1

1

1111

ERO

n

i

n

i

nn

inii

ni

b

b

b

x

x

x

a

aa

aaa

~

~

~00

~~0

111111

Back

sub

stit

uti

on

Math for CS Lecture 2 14

Pivotal Element

)1(

)1(3

)1(2

)1(1

3

2

1

)1()1(3

)1(2

)1(1

)1(3

)1(33

)1(32

)1(31

)1(2

)1(23

)1(22

)1(21

)1(1

)1(13

)1(12

)1(11

nnnnnnn

n

n

n

b

b

b

b

x

x

x

x

aaaa

aaaa

aaaa

aaaa

Pivotal element

The first coefficient of the first row (pivot) is used to zero out first coefficients of

Other rows.

Math for CS Lecture 2 15

First step of elimination

)2(

)2(3

)2(2

)1(1

3

2

1

)2()2(3

)2(2

)2(3

)2(33

)2(32

)2(2

)2(23

)2(22

)1(1

)1(13

)1(12

)1(11

)1(11

)1(11,

)1(11

)1(311,3

)1(11

)1(211,2

0

0

0

/

/

/

nnnnnn

n

n

n

nn b

b

b

b

x

x

x

x

aaa

aaa

aaa

aaaa

aam

aam

aam

First row, multiplied by appropriate factor is subtracted from other rows.

Math for CS Lecture 2 16

Second step of elimination

)2(

)2(3

)2(2

)1(1

3

2

1

)2()2(3

)2(2

)2(3

)2(33

)2(32

)2(2

)2(23

)2(22

)1(1

)1(13

)1(12

)1(11

0

0

0

nnnnnn

n

n

n

b

b

b

b

x

x

x

x

aaa

aaa

aaa

aaaa

Pivotal element

The second coefficient (pivot) of the second row is used to zero out second coefficients of

other rows.

Math for CS Lecture 2 17

Second step of elimination

)3(

)3(3

)2(2

)1(1

3

2

1

)3()3(3

)3(3

)3(33

)2(2

)2(23

)2(22

)1(1

)1(13

)1(12

)1(11

)2(22

)2(22,

)2(22

)2(322,3

00

00

0

/

/

nnnnn

n

n

n

nn b

b

b

b

x

x

x

x

aa

aa

aaa

aaaa

aam

aam

Second row, multiplied by appropriate factor

is subtracted from other rows (ERO).

Math for CS Lecture 2 18

Gaussion elimination algorithm

Define number of steps as p (pivotal row) For p=1,n-1

For r=p+1 to n

For c=p+1 to n

0

/)(

)()(,

prp

ppp

prppr

a

aam

)(,

)()1( ppcpr

prc

prc amaa

)(,

)()1( pppr

pr

pr bmbb

Math for CS Lecture 2 19

Back substitution algorithm

)(

)1(1

)3(3

)2(2

)1(1

1

3

2

1

)(

)(1

)(11

)3(3

)3(33

)2(2

)2(23

)2(22

)1(1

)1(13

)1(12

)1(11

0000

000

00

0

nn

nn

n

n

nnn

nnn

nnn

n

n

n

b

b

b

b

b

x

x

x

x

x

a

aa

aa

aaa

aaaa

Finally, the following system is obtained:

Math for CS Lecture 2 20

Back substitution algorithm

1,,2,11

1

1

)()()(

11

)1(1)1(

111

)(

)(

nnixaba

x

xaba

x

a

bx

n

ikk

iik

iii

iii

nn

nnnnn

nnn

nnn

nn

n

The answer is obtained as following: