Upload
others
View
2
Download
0
Embed Size (px)
Citation preview
Lecture 2: MLSC - Prof. Sethu Vijayakumar 1
Lecture 2: Linear Algebra Revisited
Overview • Vector spaces, Hilbert & Banach Spaces, Metrics & Norms
• Matrices, Eigenvalues, Orthogonal Transformations, Singular Values
• Operators, Operator Norms, Function Spaces
Note: We will need many of these concepts as basic tools to quantify and evaluate the
performance of machine learning algorithms and also to come up with more efficient
and effective solutions …
Lecture 2: MLSC - Prof. Sethu Vijayakumar 2
Vectors
Multiplication by scalar (ax).
Addition of vectors (x+y) – x and y have to be of same dimensions.
Linear combination.
u = ax+by (x and y have to be of same dimensions).
Angle between vectors.
Linear independence.
When one vector cannot be written as a linear combination of other, then the vectors are said to be linearly independent.
Usually denoted by lower case, bold letters, e.g. x, y
Operations :
wv
wv, cos
Lecture 2: MLSC - Prof. Sethu Vijayakumar 3
Metric
Definition 1 (Metric/ Distance)
Example 2 (Manhattan Distance)
Example 1 (Trivial Metric)
Lecture 2: MLSC - Prof. Sethu Vijayakumar 4
Vector Spaces
Definition 2 (Vector Spaces)
Definition 4 (Completeness)
Definition 3 (Cauchy Series)
Lecture 2: MLSC - Prof. Sethu Vijayakumar 5
Examples of Vector Spaces
Rational Numbers, Real Numbers, Polynomials are all Vector Spaces
Lecture 2: MLSC - Prof. Sethu Vijayakumar 6
Norm & Banach Spaces
Definition 5 (Norm / Length)
Definition 6 (Banach Space)
Lecture 2: MLSC - Prof. Sethu Vijayakumar 7
Examples of Banach Spaces
Lecture 2: MLSC - Prof. Sethu Vijayakumar 8
Dot Products & Hilbert Spaces
Definition 7 (Dot Product/ Inner Product)
Definition 8 (Hilbert Space)
543; 222
1
vv,v
4
3v
Example :
Lecture 2: MLSC - Prof. Sethu Vijayakumar 9
Examples of Hilbert Spaces
Lecture 2: MLSC - Prof. Sethu Vijayakumar 10
Matrices
m n
M
mv
v
1
v
nu
u
1
u
Review:
• Addition of Matrices
• Multiplication of matrices by scalars, vectors and matrices.
• Domain and Range of a Matrix
Range
Domain
u =Mv
Lecture 2: MLSC - Prof. Sethu Vijayakumar 11
Special matrices
Square and Diagonal Matrix A square matrix has equal number of rows and columns. A diagonal matrix has all off-diagonal elements zero.
Symmetric Matrix
Anti-symmetric Matrix
Orthogonal Matrix
(Often denoted as O(m) )
Lecture 2: MLSC - Prof. Sethu Vijayakumar 12
Matrix Concepts
Rank of a Matrix
Range, Domain and Null Space
Range of M, denoted by R(M), is the space of all vectors that can be obtained by the
operation of M on the vectors in the domain of M denoted by D(M).
Null Space of M, denoted by N(M), is a subspace of all the vectors in the domain of M
D(M) that map to the zero (null) vector in R(M) when operated upon by the matrix M.
vutsDuiffRv MMM ..)()(
0MM viffNv )(
Lecture 2: MLSC - Prof. Sethu Vijayakumar 13
Matrix Invariants
Trace
Properties of Trace :
)()(
)()()(
)()(
BAAB
BABA
AA
trtr
trtrtr
traatr
Determinant
Determinant can be written as the product of the eigenvalues :
Note: Trace and Determinant are invariant under orthogonal transformation
Lecture 2: MLSC - Prof. Sethu Vijayakumar 14
Matrix Norms
Frobeius Norm
Operator Norm
Lecture 2: MLSC - Prof. Sethu Vijayakumar 15
Eigensystems
Intuitive Explanation. A square matrix M is a mapping from n-dim to n-dim space.
Most vectors change both direction and length when undergoing this mapping transformation.
Those vectors which only change length (i.e., multiplying them by a matrix is similar to multiplying by a scalar) are called eigenvectors and the eigenvalue indicates how much they are shortened or lengthened.
Definition 9 (Eigenvalues/ Eigenvectors)
IMP: Defined only for square Matrices
Lecture 2: MLSC - Prof. Sethu Vijayakumar 16
Eigensystems II
o All eigenvalues of symmetric matrices are real
o Symmetric matrices are fully diagonalizable, i.e. we can find m eigenvectors
o All eigenvectors of symmetric matrices M with different eigenvalues are mutually orthogonal (Prove !!)
Eigenvectors/Eigenvalues of Symmetric Matrices
Decomposition of Symmetric Matrices
Example
Lecture 2: MLSC - Prof. Sethu Vijayakumar 17
Positive Matrices
Definition 10 (Positive Definite Matrices)
Induced Norm and Metrics
Lecture 2: MLSC - Prof. Sethu Vijayakumar 18
Mahalanobis Distance
10
01IM
Myxyx,yx,MM
Td )(
10
02IM
14.1
4.12M
= Euclidean Distance
Lecture 2: MLSC - Prof. Sethu Vijayakumar 19
Singular Value Decomposition
Note: Eigenvalue/Eigenvector decompositions are valid only for square matrices
Do we have some decomposition for rectangular matrices ??
Singular value Decomposition (SVD)
Lecture 2: MLSC - Prof. Sethu Vijayakumar 20
Matrix Inverse
m n
mv
v
1
v
nu
u
1
uRange
Domain M
1M
IMMIMM11 ;
Note: A regular inverse exists only for square matrices with linearly independent column vectors
Interpretation: We need a one-to-one mapping to uniquely go from one element of a space to another and back. Square matrices and linearly independent columns ensure this !!
Lecture 2: MLSC - Prof. Sethu Vijayakumar 21
Pseudoinverse
11 )()( TTTT or MMMMMMM
For rectangular matrices and square matrices with linearly dependent
columns, there exists the pseudoinverse or generalized inverse
which performs the inverse mapping. In general, these inverses are not
unique.
The above generalized inverse is called the Moore-Penrose
Psuedoinverse and is unique. Among the multiple inverse solutions, it
chooses the one with the minimum norm.
The Moore-Penrose Pseudoinverse (M+)
Lecture 2: MLSC - Prof. Sethu Vijayakumar 22
Operators
Linear Operators
Generalization of matrix – a mapping from one Banach space to another. Norms and
eigenvalues/eigenvectors are defined as for matrices; so are Range & Null Spaces.
Notation
A : FG denotes a linear operator A mapping from space F to space G.
Matrix Transpose Adjoint Operator
Symmetric Matrix Self Adjoint Operator
Orthogonal Matrix Isometry
A Matrix-Operator Correspondence
.,,, * GgFfallforgfgf AA
.,,, * GgFfallforgfgf AA
.,,, GgFfallforgfgf AA
TA
TAA
TAA 1
Lecture 2: MLSC - Prof. Sethu Vijayakumar 23
Linear Operators - Examples
Input transformation
Sampling
Sampling from a function to yield scalar outputs.
( We will see later why this is so !!! )
1x 2x 3x
f
Fourier Transform
Lecture 2: MLSC - Prof. Sethu Vijayakumar 24
Range and Null Space of Operators
R(A) R(A*)
A N(A*)
N(A)
A*
Recollect: Definition of Range, Domain and Null space of a matrix