Orthogonal DiagonalizationMATH 322, Linear Algebra I

J. Robert Buchanan

Department of Mathematics

Spring 2015

Objectives

In this lesson we will:I consider the problem of diagonalizing a symmetric matrix,I discover this problem is related to the problem of finding an

orthonormal basis for Rn consisting of eigenvectors of agiven n × n matrix,

I recognize that many matrices which arise in applicationsare symmetric.

Orthogonal Diagonalization Problem

There are three fundamental questions to be answered:1. Given an n × n matrix A, does there exist an orthonormal

basis for Rn consisting of eigenvectors of A?2. Given an n × n matrix A, does there exist an orthogonal

matrix P such that P−1AP = D = PT AP is a diagonalmatrix? If so, we say that A is orthogonallydiagonalizable and that P orthogonally diagonalizes A.

3. Which n × n matrices are orthogonally diagonalizable?

Orthogonally Similar Matrices

DefinitionIf A and B are square matrices, then we say that B isorthogonally similar to A if there is an orthogonal matrix Psuch that B = PT A P.

Remarks:I If B is orthogonally similar to A then A is orthogonally

similar to B.

B = PT A P ⇐⇒ P BPT = QT B Q = A

where Q = PT .

I When A is orthogonally similar to a diagonal matrix D thenA is orthogonally diagonalizable.

I Orthogonal matrix P such that D = PT A P orthogonallydiagonalizes A.

Orthogonally Similar Matrices

DefinitionIf A and B are square matrices, then we say that B isorthogonally similar to A if there is an orthogonal matrix Psuch that B = PT A P.

Remarks:I If B is orthogonally similar to A then A is orthogonally

similar to B.

B = PT A P ⇐⇒ P BPT = QT B Q = A

where Q = PT .I When A is orthogonally similar to a diagonal matrix D then

A is orthogonally diagonalizable.

I Orthogonal matrix P such that D = PT A P orthogonallydiagonalizes A.

Orthogonally Similar Matrices

DefinitionIf A and B are square matrices, then we say that B isorthogonally similar to A if there is an orthogonal matrix Psuch that B = PT A P.

Remarks:I If B is orthogonally similar to A then A is orthogonally

similar to B.

B = PT A P ⇐⇒ P BPT = QT B Q = A

where Q = PT .I When A is orthogonally similar to a diagonal matrix D then

A is orthogonally diagonalizable.I Orthogonal matrix P such that D = PT A P orthogonally

diagonalizes A.

Which Matrices are Orthogonally Diagonalizable?

Suppose A is orthogonally diagonalizable, then

D = PT AP = DT

which implies that A = PDPT and AT = PDPT ,

in otherwordsA is symmetric.

The converse of this statement is also true.

Which Matrices are Orthogonally Diagonalizable?

Suppose A is orthogonally diagonalizable, then

D = PT AP = DT

which implies that A = PDPT and AT = PDPT , in otherwordsA is symmetric.

The converse of this statement is also true.

Which Matrices are Orthogonally Diagonalizable?

Suppose A is orthogonally diagonalizable, then

D = PT AP = DT

which implies that A = PDPT and AT = PDPT , in otherwordsA is symmetric.

The converse of this statement is also true.

Equivalent Statements

TheoremIf A is an n × n matrix, then the following are equivalent.

1. A is orthogonally diagonalizable.2. A has an orthonormal set of n eigenvectors.3. A is symmetric.

Proof (1) =⇒ (2)

I Suppose A is orthogonally diagonalizable.I There exists an orthogonal matrix P such that PT A P is

diagonal.I The columns of P are eigenvectors of A.I Since P is orthogonal, the columns of P are orthonormal,

and thus the eigenvectors of A are an orthonormal set.

Proof (2) =⇒ (1)

I Suppose A has a orthonormal set of eigenvectors{p1,p2, . . . ,pn}.

I If the columns of matrix P are the eigenvectors of A, thenP diagonalizes A.

I Since the columns of P are orthonormal, P is anorthogonal matrix which orthogonally diagonalizes A.

Proof (1) =⇒ (3)

Let P be the orthogonal matrix which orthogonally diagonalizesA and let the diagonal matrix D:

D = PT A PP D PT = A

(P D PT )T = AT

P D PT = AT

A = AT

Thus A is symmetric.

Properties of Symmetric Matrices

TheoremIf A is a symmetric matrix, then:

1. the eigenvalues of A are all real numbers, and2. eigenvectors from different eigenspaces are orthogonal.

Proof of (2)

I Let v1 and v2 be eigenvectors corresponding to distincteigenvalues λ1 and λ2 respectively.

I Since A is symmetric.

λ1v1 · v2 = A v1 · v2 = v1 · AT v2 = v1 · Av2 = v1 · λ2v2

I Hence we may write (λ1 − λ2)(v1 · v2) = 0.I Since λ1 6= λ2 then v1 · v2 = 0 and thus v1 and v2 are

orthogonal.

Proof of (2)

I Let v1 and v2 be eigenvectors corresponding to distincteigenvalues λ1 and λ2 respectively.

I Since A is symmetric.

λ1v1 · v2 = A v1 · v2 = v1 · AT v2 = v1 · Av2 = v1 · λ2v2

I Hence we may write (λ1 − λ2)(v1 · v2) = 0.

I Since λ1 6= λ2 then v1 · v2 = 0 and thus v1 and v2 areorthogonal.

Proof of (2)

I Let v1 and v2 be eigenvectors corresponding to distincteigenvalues λ1 and λ2 respectively.

I Since A is symmetric.

λ1v1 · v2 = A v1 · v2 = v1 · AT v2 = v1 · Av2 = v1 · λ2v2

I Hence we may write (λ1 − λ2)(v1 · v2) = 0.I Since λ1 6= λ2 then v1 · v2 = 0 and thus v1 and v2 are

orthogonal.

Diagonalization of Symmetric Matrices

Steps:1. Find a basis for each eigenspace of A.2. Apply the Gram-Schmidt process to each of these bases to

obtain an orthonormal basis for each eigenspace.3. Form the matrix P from the basis vectors found in step (2).

Example

Find an orthogonal matrix P which diagonalizes

A =

[6 −2−2 3

].

Eigensystems:

Eigenvalue Eigenvector2 (1,2)7 (−2,1)

Example

Find an orthogonal matrix P which diagonalizes

A =

[6 −2−2 3

].

Eigensystems:

Eigenvalue Eigenvector2 (1,2)7 (−2,1)

Diagonalization

Eigensystems:

Eigenvalue Orthonormal Eigenvector2 (1/

√5,2/

√5)

7 (−2/√

5,1/√

5)

Let P =

[1/√

5 −2/√

52/√

5 1/√

5

]then

PT A P =

[1√5

2√5

− 2√5

1√5

] [6 −2−2 3

] [ 1√5− 2√

52√5

1√5

]=

[2 00 7

]

Example

Find an orthogonal matrix P which diagonalizes

A =

2 −1 −1−1 2 −1−1 −1 2

.

Eigensystems:

Eigenvalue Eigenvector3 (−1,0,1)3 (−1,1,0)0 (1,1,1)

Example

Find an orthogonal matrix P which diagonalizes

A =

2 −1 −1−1 2 −1−1 −1 2

.

Eigensystems:

Eigenvalue Eigenvector3 (−1,0,1)3 (−1,1,0)0 (1,1,1)

Diagonalization (1 of 2)

Using the Gram-Schmidt process we find that an orthonormalbasis for the eigenspace of A corresponding to λ1 = 3 is

p1 =(−1,0,1)‖(−1,0,1)‖

= (−1/√

2,0,1/√

2)

u2 = (−1,1,0)− 〈(−1,1,0),p1〉p1 = (−1/2,1,−1/2)

p2 =(−1/2,1,−1/2)‖(−1/2,1,−1/2)‖

= (−1/√

6,√

2/3,−1/√

6)

p3 =(1,1,1)‖(1,1,1)‖

= (1/√

3,1/√

3,1/√

3)

Diagonalization (2 of 2)

Let orthogonal matrix P =

− 1√

2− 1√

61√3

0√

23

1√3

1√2− 1√

61√3

then

PT A P

=

− 1√

20 1√

2

− 1√6

√23 − 1√

61√3

1√3

1√3

2 −1 −1−1 2 −1−1 −1 2

− 1√

2− 1√

61√3

0√

23

1√3

1√2− 1√

61√3

=

3 0 00 3 00 0 0

Example

Find an orthogonal matrix P which diagonalizes

A =

1 2 3 42 1 2 33 2 1 24 3 2 1

.

Eigensystems:

Eigenvalue Eigenvector4 +√

26 (1, 15(1 +

√26), 1

5(1 +√

26),1)4−√

26 (1,−15(1 +

√26),−1

5(1 +√

26),1)−2 +

√2 (−1,1 +

√2,−1−

√2,1)

−2−√

2 (−1,1−√

2,−1 +√

2,1)

Example

Find an orthogonal matrix P which diagonalizes

A =

1 2 3 42 1 2 33 2 1 24 3 2 1

.

Eigensystems:

Eigenvalue Eigenvector4 +√

26 (1, 15(1 +

√26), 1

5(1 +√

26),1)4−√

26 (1,−15(1 +

√26),−1

5(1 +√

26),1)−2 +

√2 (−1,1 +

√2,−1−

√2,1)

−2−√

2 (−1,1−√

2,−1 +√

2,1)

Diagonalization (1 of 3)

Since eigenvalues are all distinct we need only normalize theeigenvectors to find the columns of the orthogonal matrix P.

p1 =(1, 1

5(1 +√

26), 15(1 +

√26),1)

‖(1, 15(1 +

√26), 1

5(1 +√

26),1)‖

=

(5

2√

26 +√

26,

1 +√

26

2√

26 +√

26,

1 +√

26

2√

26 +√

26,

5

2√

26 +√

26

)

p2 =(1,−1

5(1 +√

26),−15(1 +

√26),1)

‖(1,−15(1 +

√26),−1

5(1 +√

26),1)‖

=

(5

2√

26 +√

26,−1−

√26

2√

26 +√

26,−1−

√26

2√

26 +√

26,

5

2√

26 +√

26

)

Diagonalization (2 of 3)

Since eigenvalues are all distinct we need only normalize theeigenvectors to find the columns of the orthogonal matrix P.

p3 =(−1,1 +

√2,−1−

√2,1)

‖(−1,1 +√

2,−1−√

2,1)‖

=

(−1

2√

2 +√

2,

1 +√

2

2√

2 +√

2,−1−

√2

2√

2 +√

2,

1

2√

2 +√

2

)

p4 =(−1,1−

√2,−1 +

√2,1)

‖(−1,1−√

2,−1 +√

2,1)‖

=

(−1

2√

2−√

2,

1−√

2

2√

2−√

2,−1 +

√2

2√

2−√

2,

1

2√

2−√

2

)

Diagonalization (3 of 3)

P =

52√

26+√

265

2√

26+√

26− 1

2√

2+√

2−1

2√

2−√

21+√

262√

26+√

26−1−

√26

2√

26+√

261+√

22√

2+√

21−√

22√

2−√

21+√

262√

26+√

26−1−

√26

2√

26+√

26−1−

√2

2√

2+√

2

√2−1

2√

2−√

25

2√

26+√

265

2√

26+√

261

2√

2+√

21

2√

2−√

2

Spectral Decomposition (1 of 2)If symmetric matrix A is orthogonally diagonalized by matrix

P =[

u1 u2 · · · un]

then D = PT A P and

A = P D PT

=[

u1 u2 · · · un]λ1 0 · · · 00 λ2 · · · 0...

......

0 0 · · · λn

uT1

uT2...

uTn

=[λ1u1 λ2u2 · · · λnun

]

uT1

uT2...

uTn

A = λ1u1uT

1 + λ2u2uT2 + · · ·+ λnunuT

n

Spectral Decomposition (2 of 2)

A = λ1u1uT1 + λ2u2uT

2 + · · ·+ λnunuTn

This formula is called the spectral decomposition of A.

Remarks:I The set of eigenvalues of A is sometimes called the

spectrum pf A.I Since ui is an n × 1 matrix, then λiuiuT

i is an n × n matrix.

Spectral Decomposition (2 of 2)

A = λ1u1uT1 + λ2u2uT

2 + · · ·+ λnunuTn

This formula is called the spectral decomposition of A.Remarks:

I The set of eigenvalues of A is sometimes called thespectrum pf A.

I Since ui is an n × 1 matrix, then λiuiuTi is an n × n matrix.

Interpretation of the Spectral Decomposition

A x = λ1u1uT1 x + λ2u2uT

2 x + · · ·+ λnunuTn x

Multiplication of x by a symmetric matrix is equivalent toprojecting x on the one-dimensional subspaces determined bythe eigenvectors of A, then scaling the projections by theeigenvalues, and then adding the scaled projections.

Example

Let A =

[6 −2−2 3

]and consider the multiplication of A by

vector x = (1,1).

The eigenvalues of A are λ1 = 2 and λ2 = 7 and the followingmatrix orthogonally diagonalizes A.

P =

[1√5− 2√

52√5

1√5

]=[

u1 u2]

Example

Let A =

[6 −2−2 3

]and consider the multiplication of A by

vector x = (1,1).

The eigenvalues of A are λ1 = 2 and λ2 = 7 and the followingmatrix orthogonally diagonalizes A.

P =

[1√5− 2√

52√5

1√5

]=[

u1 u2]

Illustration (1 of 3)

u1uT1 =

[1√5

2√5

] [1√5

2√5

]=

[ 15

25

25

45

]

u2uT2 =

[− 2√

51√5

] [− 2√

51√5

]=

[ 45 −2

5−2

515

]A = λ1u1uT

1 + λ2u2uT2

= 2[ 1

525

25

45

]+ 7

[ 45 −2

5−2

515

]=

[ 25

45

45

85

]+

[ 285 −14

5−14

575

]A =

[6 −2−2 3

]

Illustration (2 of 3)

A[

11

]= 2

[ 15

25

25

45

] [11

]+ 7

[ 45 −2

5−2

515

] [11

]= 2

[ 3565

]+ 7

[ 25−1

5

]=

[ 65

125

]+

[ 145−7

5

]=

[41

]

Illustration (3 of 3)

x Ax

λ=2

λ=7-2 -1 1 2 3 4 5

x

-2

-1

1

2

3

4y

Schur’s Theorem

TheoremIf A is an n × n matrix with real entries and real eigenvalues,then there is an orthogonal matrix P such that PT A P is anupper triangular matrix of the form

PT A P =

λ1 ∗ ∗ · · · ∗0 λ2 ∗ · · · ∗0 0 λ3 · · · ∗...

......

...0 0 0 · · · λn

= S

in which λ1, λ2, . . . , λn are the eigenvalues of A repeatedaccording to multiplicity.

Hessenberg’s Theorem

TheoremIf A is an n × n matrix with real entries, then there is anorthogonal matrix P such that PT A P is a matrix of the form

PT A P =

∗ ∗ · · · ∗ ∗ ∗∗ ∗ · · · ∗ ∗ ∗0 ∗ · · · ∗ ∗ ∗...

......

......

0 0 · · · ∗ ∗ ∗0 0 · · · 0 ∗ ∗

= H.

Homework

I Read Section 7.2I Exercises: 1, 3, 5, 7, 9, 15, 19