Lecture 15: QR decomposition and FourierSeries (Section 3.4)

Thang Huynh, UC San Diego2/14/2018

Gram-Schmidt

▶ Gram-Schmidt orthonormalization:

• Input: basis 𝑎𝑎𝑎1, … ,𝑎𝑎𝑎𝑛 for 𝑉• Output: orthonormal basis 𝑞𝑞𝑞1, … ,𝑞𝑞𝑞𝑛 for 𝑉 .

𝑏𝑏𝑏1 = 𝑎𝑎𝑎1, 𝑞𝑞𝑞1 = 𝑏𝑏𝑏1‖𝑏𝑏𝑏1‖

𝑏𝑏𝑏2 = 𝑎𝑎𝑎2 − ⟨𝑎𝑎𝑎2, 𝑞𝑞𝑞1⟩𝑞𝑞𝑞1, 𝑞𝑞𝑞2 = 𝑏𝑏𝑏2‖𝑏𝑏𝑏2‖

𝑏𝑏𝑏3 = 𝑎𝑎𝑎3 − ⟨𝑎𝑎𝑎3, 𝑞𝑞𝑞1⟩𝑞𝑞𝑞1 − ⟨𝑎𝑎𝑎3, 𝑞𝑞𝑞2⟩𝑞𝑞𝑞2, 𝑞𝑞𝑞3 = 𝑏𝑏𝑏3‖𝑏𝑏𝑏3‖

⋮

1

The QR decomposition

Let 𝐴 be an 𝑚 × 𝑛 matrix of rank 𝑛. Then we have the QRdecomposition 𝐴 = 𝑄𝑅,

• where 𝑄 is 𝑚 × 𝑛 and has orthonormal columns, and• 𝑅 is upper triangular, 𝑛 × 𝑛 and invertible.

▶ Example. Find the QR decomposition of 𝐴 =⎡⎢⎢⎣

1 2 40 0 50 3 6

⎤⎥⎥⎦

.

2

The QR decomposition

Let 𝐴 be an 𝑚 × 𝑛 matrix of rank 𝑛. Then we have the QRdecomposition 𝐴 = 𝑄𝑅,

• where 𝑄 is 𝑚 × 𝑛 and has orthonormal columns, and• 𝑅 is upper triangular, 𝑛 × 𝑛 and invertible.

▶ Example. Find the QR decomposition of 𝐴 =⎡⎢⎢⎣

1 2 40 0 50 3 6

⎤⎥⎥⎦

.

2

The QR decomposition▶ We apply Gram-Schmidt to the columns of 𝐴:

𝑏𝑏𝑏1 =⎡⎢⎢⎣

100

⎤⎥⎥⎦

= 𝑞𝑞𝑞1,

𝑏𝑏𝑏2 =⎡⎢⎢⎣

203

⎤⎥⎥⎦

− ⟨⎡⎢⎢⎣

203

⎤⎥⎥⎦

,𝑞𝑞𝑞1⟩𝑞𝑞𝑞1 = 𝑞𝑞𝑞1 ⟹ 𝑞𝑞𝑞2 =⎡⎢⎢⎣

001

⎤⎥⎥⎦

𝑏𝑏𝑏3 =⎡⎢⎢⎣

456

⎤⎥⎥⎦

− ⟨⎡⎢⎢⎣

456

⎤⎥⎥⎦

,𝑞𝑞𝑞1⟩𝑞𝑞𝑞1 − ⟨⎡⎢⎢⎣

456

⎤⎥⎥⎦

,𝑞𝑞𝑞2⟩𝑞𝑞𝑞2 ⟹ 𝑞𝑞𝑞3 =⎡⎢⎢⎣

010

⎤⎥⎥⎦

.

𝑄 = [ 𝑞𝑞𝑞1 𝑞𝑞𝑞2 𝑞𝑞𝑞3 ] =⎡⎢⎢⎣

1 0 00 0 10 1 0

⎤⎥⎥⎦

3

The QR decomposition▶ We apply Gram-Schmidt to the columns of 𝐴:

𝑏𝑏𝑏1 =⎡⎢⎢⎣

100

⎤⎥⎥⎦

= 𝑞𝑞𝑞1,

𝑏𝑏𝑏2 =⎡⎢⎢⎣

203

⎤⎥⎥⎦

− ⟨⎡⎢⎢⎣

203

⎤⎥⎥⎦

,𝑞𝑞𝑞1⟩𝑞𝑞𝑞1 = 𝑞𝑞𝑞1 ⟹ 𝑞𝑞𝑞2 =⎡⎢⎢⎣

001

⎤⎥⎥⎦

𝑏𝑏𝑏3 =⎡⎢⎢⎣

456

⎤⎥⎥⎦

− ⟨⎡⎢⎢⎣

456

⎤⎥⎥⎦

,𝑞𝑞𝑞1⟩𝑞𝑞𝑞1 − ⟨⎡⎢⎢⎣

456

⎤⎥⎥⎦

,𝑞𝑞𝑞2⟩𝑞𝑞𝑞2 ⟹ 𝑞𝑞𝑞3 =⎡⎢⎢⎣

010

⎤⎥⎥⎦

.

𝑄 = [ 𝑞𝑞𝑞1 𝑞𝑞𝑞2 𝑞𝑞𝑞3 ] =⎡⎢⎢⎣

1 0 00 0 10 1 0

⎤⎥⎥⎦ 3

The QR decompositionTo find 𝑅 in 𝐴 = 𝑄𝑅, note that Note that 𝑄𝑇𝐴 = 𝑄𝑇𝑄𝑅 = 𝑅.

𝑅 =⎡⎢⎢⎣

1 0 00 0 10 1 0

⎤⎥⎥⎦

⎡⎢⎢⎣

1 2 40 0 50 3 6

⎤⎥⎥⎦

=⎡⎢⎢⎣

1 2 40 3 60 0 5

⎤⎥⎥⎦

.

⟹⎡⎢⎢⎣

1 2 40 0 50 3 6

⎤⎥⎥⎦

=⎡⎢⎢⎣

1 0 00 0 10 1 0

⎤⎥⎥⎦

⎡⎢⎢⎣

1 2 40 3 60 0 5

⎤⎥⎥⎦

.

In general, to obtain 𝐴 = 𝑄𝑅.

• Gram-Schmidt on (columns of) 𝐴, to get 𝑄.• Then 𝑅 = 𝑄𝑇𝐴.

Why does this process ensure 𝑅 is an upper triangular matrix?

4

The QR decompositionTo find 𝑅 in 𝐴 = 𝑄𝑅, note that Note that 𝑄𝑇𝐴 = 𝑄𝑇𝑄𝑅 = 𝑅.

𝑅 =⎡⎢⎢⎣

1 0 00 0 10 1 0

⎤⎥⎥⎦

⎡⎢⎢⎣

1 2 40 0 50 3 6

⎤⎥⎥⎦

=⎡⎢⎢⎣

1 2 40 3 60 0 5

⎤⎥⎥⎦

.

⟹⎡⎢⎢⎣

1 2 40 0 50 3 6

⎤⎥⎥⎦

=⎡⎢⎢⎣

1 0 00 0 10 1 0

⎤⎥⎥⎦

⎡⎢⎢⎣

1 2 40 3 60 0 5

⎤⎥⎥⎦

.

In general, to obtain 𝐴 = 𝑄𝑅.

• Gram-Schmidt on (columns of) 𝐴, to get 𝑄.• Then 𝑅 = 𝑄𝑇𝐴.

Why does this process ensure 𝑅 is an upper triangular matrix?

4

The QR decompositionTo find 𝑅 in 𝐴 = 𝑄𝑅, note that Note that 𝑄𝑇𝐴 = 𝑄𝑇𝑄𝑅 = 𝑅.

𝑅 =⎡⎢⎢⎣

1 0 00 0 10 1 0

⎤⎥⎥⎦

⎡⎢⎢⎣

1 2 40 0 50 3 6

⎤⎥⎥⎦

=⎡⎢⎢⎣

1 2 40 3 60 0 5

⎤⎥⎥⎦

.

⟹⎡⎢⎢⎣

1 2 40 0 50 3 6

⎤⎥⎥⎦

=⎡⎢⎢⎣

1 0 00 0 10 1 0

⎤⎥⎥⎦

⎡⎢⎢⎣

1 2 40 3 60 0 5

⎤⎥⎥⎦

.

In general, to obtain 𝐴 = 𝑄𝑅.

• Gram-Schmidt on (columns of) 𝐴, to get 𝑄.• Then 𝑅 = 𝑄𝑇𝐴.

Why does this process ensure 𝑅 is an upper triangular matrix?

4

The QR decompositionTo find 𝑅 in 𝐴 = 𝑄𝑅, note that Note that 𝑄𝑇𝐴 = 𝑄𝑇𝑄𝑅 = 𝑅.

𝑅 =⎡⎢⎢⎣

1 0 00 0 10 1 0

⎤⎥⎥⎦

⎡⎢⎢⎣

1 2 40 0 50 3 6

⎤⎥⎥⎦

=⎡⎢⎢⎣

1 2 40 3 60 0 5

⎤⎥⎥⎦

.

⟹⎡⎢⎢⎣

1 2 40 0 50 3 6

⎤⎥⎥⎦

=⎡⎢⎢⎣

1 0 00 0 10 1 0

⎤⎥⎥⎦

⎡⎢⎢⎣

1 2 40 3 60 0 5

⎤⎥⎥⎦

.

In general, to obtain 𝐴 = 𝑄𝑅.

• Gram-Schmidt on (columns of) 𝐴, to get 𝑄.• Then 𝑅 = 𝑄𝑇𝐴.

Why does this process ensure 𝑅 is an upper triangular matrix?

4

The QR decompositionTo find 𝑅 in 𝐴 = 𝑄𝑅, note that Note that 𝑄𝑇𝐴 = 𝑄𝑇𝑄𝑅 = 𝑅.

𝑅 =⎡⎢⎢⎣

1 0 00 0 10 1 0

⎤⎥⎥⎦

⎡⎢⎢⎣

1 2 40 0 50 3 6

⎤⎥⎥⎦

=⎡⎢⎢⎣

1 2 40 3 60 0 5

⎤⎥⎥⎦

.

⟹⎡⎢⎢⎣

1 2 40 0 50 3 6

⎤⎥⎥⎦

=⎡⎢⎢⎣

1 0 00 0 10 1 0

⎤⎥⎥⎦

⎡⎢⎢⎣

1 2 40 3 60 0 5

⎤⎥⎥⎦

.

In general, to obtain 𝐴 = 𝑄𝑅.

• Gram-Schmidt on (columns of) 𝐴, to get 𝑄.• Then 𝑅 = 𝑄𝑇𝐴.

Why does this process ensure 𝑅 is an upper triangular matrix? 4

Practice Problems

▶ Example. Complete 13

⎡⎢⎢⎣

122

⎤⎥⎥⎦

, 13

⎡⎢⎢⎣

−2−12

⎤⎥⎥⎦

to an orthonormal basis of

ℝ3.

a) by using the FTLA to determine the orthogonalcomplement of the span you already have

b) by using Gram-Schmidt after throwing in an independent

vector such as⎡⎢⎢⎣

100

⎤⎥⎥⎦.

▶ Example. Find the 𝑄𝑅 decomposition of 𝐴 =⎡⎢⎢⎣

1 1 20 0 11 0 0

⎤⎥⎥⎦

.

5

The QR decomposition

▶ Example. The 𝑄𝑅 decomposition is very useful for solvingleast squares problems:

𝐴𝑇𝐴 𝑥𝑥𝑥 = 𝐴𝑇𝑏𝑏𝑏 ⟺ (𝑄𝑅)𝑇𝑄𝑅 𝑥𝑥𝑥 = (𝑄𝑅)𝑇𝑏𝑏𝑏⟺ 𝑅𝑇𝑅 𝑥𝑥𝑥 = 𝑅𝑇𝑄𝑇𝑏𝑏𝑏⟺ 𝑅 𝑥𝑥𝑥 = 𝑄𝑇𝑏𝑏𝑏

The last system is triangular and is solved by back substitution.

𝑥𝑥𝑥 is a least squares solution of 𝐴𝑥𝑥𝑥 = 𝑏𝑏𝑏 if and only if 𝑅 𝑥𝑥𝑥 = 𝑄𝑇𝑏𝑏𝑏,where 𝐴 = 𝑄𝑅.

6

The QR decomposition

▶ Example. The 𝑄𝑅 decomposition is very useful for solvingleast squares problems:

𝐴𝑇𝐴 𝑥𝑥𝑥 = 𝐴𝑇𝑏𝑏𝑏 ⟺ (𝑄𝑅)𝑇𝑄𝑅 𝑥𝑥𝑥 = (𝑄𝑅)𝑇𝑏𝑏𝑏⟺ 𝑅𝑇𝑅 𝑥𝑥𝑥 = 𝑅𝑇𝑄𝑇𝑏𝑏𝑏⟺ 𝑅 𝑥𝑥𝑥 = 𝑄𝑇𝑏𝑏𝑏

The last system is triangular and is solved by back substitution.

𝑥𝑥𝑥 is a least squares solution of 𝐴𝑥𝑥𝑥 = 𝑏𝑏𝑏 if and only if 𝑅 𝑥𝑥𝑥 = 𝑄𝑇𝑏𝑏𝑏,where 𝐴 = 𝑄𝑅.

6

The QR decomposition

▶ Example. The 𝑄𝑅 decomposition is very useful for solvingleast squares problems:

𝐴𝑇𝐴 𝑥𝑥𝑥 = 𝐴𝑇𝑏𝑏𝑏 ⟺ (𝑄𝑅)𝑇𝑄𝑅 𝑥𝑥𝑥 = (𝑄𝑅)𝑇𝑏𝑏𝑏⟺ 𝑅𝑇𝑅 𝑥𝑥𝑥 = 𝑅𝑇𝑄𝑇𝑏𝑏𝑏⟺ 𝑅 𝑥𝑥𝑥 = 𝑄𝑇𝑏𝑏𝑏

The last system is triangular and is solved by back substitution.

𝑥𝑥𝑥 is a least squares solution of 𝐴𝑥𝑥𝑥 = 𝑏𝑏𝑏 if and only if 𝑅 𝑥𝑥𝑥 = 𝑄𝑇𝑏𝑏𝑏,where 𝐴 = 𝑄𝑅.

6

Application: Fourier series

Given an orthogonal basis 𝑣𝑣𝑣1, 𝑣𝑣𝑣2, … , we express a vector 𝑥𝑥𝑥 as

𝑥𝑥𝑥 = 𝑐1𝑣𝑣𝑣1 + 𝑐2𝑣𝑣𝑣2 + … , where 𝑐𝑖 = ⟨𝑥𝑥𝑥, 𝑣𝑣𝑣𝑖⟩⟨𝑣𝑣𝑣𝑖, 𝑣𝑣𝑣𝑖⟩

.

A Fourier series of a function 𝑓 (𝑥) is an infinite expansion:

𝑓 (𝑥) = 𝑎0 + 𝑎1 cos(𝑥) + 𝑏1 sin(𝑥) + 𝑎2 cos(2𝑥) + 𝑏2 sin(2𝑥) + …

7

Application: Fourier series

Given an orthogonal basis 𝑣𝑣𝑣1, 𝑣𝑣𝑣2, … , we express a vector 𝑥𝑥𝑥 as

𝑥𝑥𝑥 = 𝑐1𝑣𝑣𝑣1 + 𝑐2𝑣𝑣𝑣2 + … , where 𝑐𝑖 = ⟨𝑥𝑥𝑥, 𝑣𝑣𝑣𝑖⟩⟨𝑣𝑣𝑣𝑖, 𝑣𝑣𝑣𝑖⟩

.

A Fourier series of a function 𝑓 (𝑥) is an infinite expansion:

𝑓 (𝑥) = 𝑎0 + 𝑎1 cos(𝑥) + 𝑏1 sin(𝑥) + 𝑎2 cos(2𝑥) + 𝑏2 sin(2𝑥) + …

7

Application: Fourier series▶ Example.

8

Application: Fourier series

▶ Example.

9

Application: Fourier series

▶ We are working in the vector space of functions ℝ → ℝ.

• More precisely, "nice" (e.g., piecewise continuous)functions that have period 2𝜋.

• There are infinite dimensional vector spaces.

▶ The functions

1, cos(𝑥), sin(𝑥), cos(2𝑥), sin(2𝑥), …

are a basis of this space. In fact, an orthogonal basis!

10

Application: Fourier series

▶ We are working in the vector space of functions ℝ → ℝ.

• More precisely, "nice" (e.g., piecewise continuous)functions that have period 2𝜋.

• There are infinite dimensional vector spaces.

▶ The functions

1, cos(𝑥), sin(𝑥), cos(2𝑥), sin(2𝑥), …

are a basis of this space.

In fact, an orthogonal basis!

10

Application: Fourier series

▶ We are working in the vector space of functions ℝ → ℝ.

• More precisely, "nice" (e.g., piecewise continuous)functions that have period 2𝜋.

• There are infinite dimensional vector spaces.

▶ The functions

1, cos(𝑥), sin(𝑥), cos(2𝑥), sin(2𝑥), …

are a basis of this space. In fact, an orthogonal basis!

10

Application: Fourier series

▶ We are working in the vector space of functions ℝ → ℝ.

• More precisely, "nice" (e.g., piecewise continuous)functions that have period 2𝜋.

• There are infinite dimensional vector spaces.

▶ The functions

1, cos(𝑥), sin(𝑥), cos(2𝑥), sin(2𝑥), …

are a basis of this space. In fact, an orthogonal basis!

10

Application: Fourier series

But what is the inner product on the space of functions?

▶ Recall that for vectors in ℝ𝑛 ∶ ⟨𝑣𝑣𝑣,𝑤𝑤𝑤⟩ = 𝑣1𝑤1 + … + 𝑣𝑛𝑤𝑛.▶ Functions:

⟨𝑓 , 𝑔⟩ = ∫2𝜋0 𝑓 (𝑥)𝑔(𝑥)𝑑𝑥.

▶ Example. Show that cos(𝑥) and sin(𝑥) are orthogonal.▶ Example. What is the norm of cos(𝑥)?

11

Application: Fourier series

But what is the inner product on the space of functions?▶ Recall that for vectors in ℝ𝑛 ∶ ⟨𝑣𝑣𝑣,𝑤𝑤𝑤⟩ = 𝑣1𝑤1 + … + 𝑣𝑛𝑤𝑛.

▶ Functions:⟨𝑓 , 𝑔⟩ = ∫2𝜋

0 𝑓 (𝑥)𝑔(𝑥)𝑑𝑥.

▶ Example. Show that cos(𝑥) and sin(𝑥) are orthogonal.▶ Example. What is the norm of cos(𝑥)?

11

Application: Fourier series

But what is the inner product on the space of functions?▶ Recall that for vectors in ℝ𝑛 ∶ ⟨𝑣𝑣𝑣,𝑤𝑤𝑤⟩ = 𝑣1𝑤1 + … + 𝑣𝑛𝑤𝑛.▶ Functions:

⟨𝑓 , 𝑔⟩ = ∫2𝜋0 𝑓 (𝑥)𝑔(𝑥)𝑑𝑥.

▶ Example. Show that cos(𝑥) and sin(𝑥) are orthogonal.▶ Example. What is the norm of cos(𝑥)?

11

Application: Fourier series

But what is the inner product on the space of functions?▶ Recall that for vectors in ℝ𝑛 ∶ ⟨𝑣𝑣𝑣,𝑤𝑤𝑤⟩ = 𝑣1𝑤1 + … + 𝑣𝑛𝑤𝑛.▶ Functions:

⟨𝑓 , 𝑔⟩ = ∫2𝜋0 𝑓 (𝑥)𝑔(𝑥)𝑑𝑥.

▶ Example. Show that cos(𝑥) and sin(𝑥) are orthogonal.

▶ Example. What is the norm of cos(𝑥)?

11

Application: Fourier series

But what is the inner product on the space of functions?▶ Recall that for vectors in ℝ𝑛 ∶ ⟨𝑣𝑣𝑣,𝑤𝑤𝑤⟩ = 𝑣1𝑤1 + … + 𝑣𝑛𝑤𝑛.▶ Functions:

⟨𝑓 , 𝑔⟩ = ∫2𝜋0 𝑓 (𝑥)𝑔(𝑥)𝑑𝑥.

▶ Example. Show that cos(𝑥) and sin(𝑥) are orthogonal.▶ Example. What is the norm of cos(𝑥)?

11

Application: Fourier seriesFourier series of 𝑓 (𝑥):

𝑓 (𝑥) = 𝑎0 + 𝑎1 cos(𝑥) + 𝑏1 sin(𝑥) + 𝑎2 cos(2𝑥) + 𝑏2 sin(2𝑥) + …

How can we find 𝑎𝑘 and 𝑏𝑘?

𝑎0 = ⟨𝑓 (𝑥), 1⟩⟨1, 1⟩ = 1

2𝜋 ∫2𝜋0 𝑓 (𝑥)𝑑𝑥,

𝑎𝑘 = ⟨𝑓 (𝑥), cos(𝑘𝑥)⟩⟨cos(𝑘𝑥), cos(𝑘𝑥)⟩ = 1

𝜋 ∫2𝜋0 𝑓 (𝑥) cos(𝑘𝑥) 𝑑𝑥,

𝑏𝑘 = ⟨𝑓 (𝑥), sin(𝑘𝑥)⟩⟨sin(𝑘𝑥), sin(𝑘𝑥)⟩ = 1

𝜋 ∫2𝜋0 𝑓 (𝑥) sin(𝑘𝑥) 𝑑𝑥,

▶ Example. Find the Fourier series of the 2𝜋-periodic function𝑓 (𝑥) defined by

𝑓 (𝑥) =⎧{⎨{⎩

1, for 𝑥 ∈ (0, 𝜋),−1, for 𝑥 ∈ (𝜋, 2𝜋).

12

Application: Fourier seriesFourier series of 𝑓 (𝑥):

𝑓 (𝑥) = 𝑎0 + 𝑎1 cos(𝑥) + 𝑏1 sin(𝑥) + 𝑎2 cos(2𝑥) + 𝑏2 sin(2𝑥) + …How can we find 𝑎𝑘 and 𝑏𝑘?

𝑎0 = ⟨𝑓 (𝑥), 1⟩⟨1, 1⟩ = 1

2𝜋 ∫2𝜋0 𝑓 (𝑥)𝑑𝑥,

𝑎𝑘 = ⟨𝑓 (𝑥), cos(𝑘𝑥)⟩⟨cos(𝑘𝑥), cos(𝑘𝑥)⟩ = 1

𝜋 ∫2𝜋0 𝑓 (𝑥) cos(𝑘𝑥) 𝑑𝑥,

𝑏𝑘 = ⟨𝑓 (𝑥), sin(𝑘𝑥)⟩⟨sin(𝑘𝑥), sin(𝑘𝑥)⟩ = 1

𝜋 ∫2𝜋0 𝑓 (𝑥) sin(𝑘𝑥) 𝑑𝑥,

▶ Example. Find the Fourier series of the 2𝜋-periodic function𝑓 (𝑥) defined by

𝑓 (𝑥) =⎧{⎨{⎩

1, for 𝑥 ∈ (0, 𝜋),−1, for 𝑥 ∈ (𝜋, 2𝜋).

12

Application: Fourier seriesFourier series of 𝑓 (𝑥):

𝑓 (𝑥) = 𝑎0 + 𝑎1 cos(𝑥) + 𝑏1 sin(𝑥) + 𝑎2 cos(2𝑥) + 𝑏2 sin(2𝑥) + …How can we find 𝑎𝑘 and 𝑏𝑘?

𝑎0 = ⟨𝑓 (𝑥), 1⟩⟨1, 1⟩ = 1

2𝜋 ∫2𝜋0 𝑓 (𝑥)𝑑𝑥,

𝑎𝑘 = ⟨𝑓 (𝑥), cos(𝑘𝑥)⟩⟨cos(𝑘𝑥), cos(𝑘𝑥)⟩ = 1

𝜋 ∫2𝜋0 𝑓 (𝑥) cos(𝑘𝑥) 𝑑𝑥,

𝑏𝑘 = ⟨𝑓 (𝑥), sin(𝑘𝑥)⟩⟨sin(𝑘𝑥), sin(𝑘𝑥)⟩ = 1

𝜋 ∫2𝜋0 𝑓 (𝑥) sin(𝑘𝑥) 𝑑𝑥,

▶ Example. Find the Fourier series of the 2𝜋-periodic function𝑓 (𝑥) defined by

𝑓 (𝑥) =⎧{⎨{⎩

1, for 𝑥 ∈ (0, 𝜋),−1, for 𝑥 ∈ (𝜋, 2𝜋).

12