Embed Size (px)
Citation preview
Linear Algebra Basics
Machine Learning CS 4641
Nakul Gopalan
Georgia Tech
These slides are based on slides from Le Song and Andres Mendez-Vazquez, Chao Zhang, Mahdi Roozbahani, Rodrigo Valente
Some logistics
β’ Creating team.
β’ Office hours are started from next week.
β’ First quiz out this Thursday.
β’ First assignment out this Thursday (early release).
Outline
β’ Linear Algebra Basics
β’ Norms
β’ Multiplications
β’ Matrix Inversion
β’ Trace and Determinant
β’ Eigen Values and Eigen Vectors
β’ Singular Value Decomposition
β’ Matrix Calculus
Why Linear Algebra?
π Γ ππ π
ππ
π
π
1
Example
Linear Algebra Basics
π Γ π π Γ π
π Γ π
Outline
β’ Linear Algebra Basics
β’ Norms
β’ Multiplications
β’ Matrix Inversion
β’ Trace and Determinant
β’ Eigen Values and Eigen Vectors
β’ Singular Value Decomposition
Norms
π
π
π
π
π
Norms
10
π
ππ
Vector Norm Examples
Special Matrices
π Γ π
π
π Γ π
π
Outline
β’ Linear Algebra Basics
β’ Norms
β’ Multiplications
β’ Matrix Inversion
β’ Trace and Determinant
β’ Eigen Values and Eigen Vectors
β’ Singular Value Decomposition
Multiplications
π Γ π π Γ π
π Γ π π
π
π
π π
π₯π¦π
π₯ππ¦: π₯ β π¦
π₯π¦π
Multiplications
Tπ₯ β π¦ =
Inner Product Properties
Inner Product Properties
Inner Product Properties
Example
Matrix multiplication geometric meaning
Outline
β’ Linear Algebra Basics
β’ Norms
β’ Multiplications
β’ Matrix Inversion
β’ Trace and Determinant
β’ Eigen Values and Eigen Vectors
β’ Matrix Decomposition
Linear Independence and Matrix Rank
ππ
π
π
π Γ π
Matrix Rank: Examples
What are the ranks for the following matrices?
Geometric meaning
Matrix Inverse
π Γ π
Outline
β’ Linear Algebra Basics
β’ Norms
β’ Multiplications
β’ Matrix Inversion
β’ Trace and Determinant
β’ Eigen Values and Eigen Vectors
β’ Singular Value Decomposition
Matrix Trace
π Γ π
π Γ π
π Γ π
π Γ π
π
Matrix Determinant
π
Properties of Matrix Determinant
Outline
β’ Linear Algebra Basics
β’ Norms
β’ Multiplications
β’ Matrix Inversion
β’ Trace and Determinant
β’ Eigen Values and Eigen Vectors
β’ Singular Value Decomposition
Eigenvalues and Eigenvectors
π Γ ππ
Computing Eigenvalues and Eigenvectors
π.
π π
(π΄ β ππΌ)
(π΄ β ππΌ) (π΄ β ππΌ)
(π΄ β ππΌ)
Eigenvalue Example
Slide credit: Shubham Kumbhar
Matrix Eigen Decomposition
columns
π
π
Outline
β’ Linear Algebra Basics
β’ Norms
β’ Multiplications
β’ Matrix Inversion
β’ Trace and Determinant
β’ Eigen Values and Eigen Vectors
β’ Singular Value Decomposition
Covariance matrix
Covariance matrix
Correlation matrix
Correlation matrix
Singular Value Decomposition
43
ππΓπn: instancesd: dimensionsX is a centered matrix
π = πΞ£ππ
ππΓπ β π’πππ‘πππ¦ πππ‘πππ₯ β π Γ ππ = πΌ
Ξ£πΓπ β ππππππππ πππ‘πππ₯
VπΓπ β π’πππ‘πππ¦ πππ‘πππ₯ β π Γ ππ = πΌ
=
ddd
d
dd
nn
n
vv
vv
uu
uu
X
.........
...
...
......
.........
000
000
00
00
00
.........
...
...
......
.........
1
11111
11
111
π Ξ£ ππ
π < π
πΆπΓπ =πππ
π
π = πΞ£ππ
πΆ =πππ
π
πΆ =πΞ£ππππΞ£ππ
π=πΞ£2ππ
π
Covariance matrix:
πΆ =πΞ£2ππ
π= π
Ξ£2
πππ
So, we can directly calculate eigenvalue of a covariance matrix by having the singular value of matrix X directly
ππ =Ξ£π2
πβ The eigenvalues of covariance matrix
According to Eigen-decomposition definition βπΆπ = VΞ
πΆπ = πΞ£2
ππππ = π
Ξ£2
π
ππ: Eigenvalue of πΆ or covariance matrix
Ξ£π: Singular value of π matrix
Geometric Meaning of SVD
Image Credit: Kevin Binz
Summary
β’ Linear Algebra Basics
β’ Norms
β’ Multiplications
β’ Matrix Inversion
β’ Trace and Determinant
β’ Eigen Values and Eigen Vectors
β’ Singular Value Decomposition
