Upload
lecong
View
226
Download
0
Embed Size (px)
Citation preview
K236: Basis of Data ScienceLecture 4. Review of linear algebra
Lecturer: Tu Bao Ho and Hieu Chi DamTA: Moharasan Gandhimathi
and Nuttapong Sanglerdsinlapachai
2
Schedule of K236
1. Introduction to data science ���*"�3 6/9
2. Introduction to data science ���*"�3 6/13
3. Data and databases ��������� 6/16
4. Review of univariate statistics !2+. 6/20
5. Review of linear algebra ,#�$ 6/23
6. Data mining software ��������� ��� 6/27
7. Data preprocessing �����' 6/30
8. Classification and prediction (1) �5��& (1) 7/4
9. Knowledge evaluation )1/� 7/7
10. Classification and prediction (2) �5��& (2) 7/11
11. Classification and prediction (3) �5��& (3) 7/14
12. Mining association rules (1) (4����-% 7/18
13. Mining association rules (2) (4����-% 7/21
14. Cluster analysis ����-% 7/25
15. Review and Examination �����06 (the data is not fixed) 7/27
Outline
1. Vector and matrix2. Operations on vectors and matrices3. Linear transformation4. Other topics
3
Linear)algebra)is)the)math)of)vectors)and)matrices.)
Application of matrices• To represent linear transformations
The rotation of vectors in 3D space is a linear transformation, which can be represented by a rotation matrix !: if " is a column vector describing the position of a point in space, the product !" is a column vector describing the position of that point after a rotation.
• To represent the composition of two linear transformations by the product of two transformation matrices.
• To solve systems of linear equations.
• Insight into the geometry of a linear transformation is obtainable (along with other information) from the matrix's eigenvalues and eigenvectors.
4https://en.wikipedia.org/wiki/Matrix_(mathematics)
Vectors and matrices
• Let # be a positive integer and let ℝ denote the set of real numbers, then ℝ% is the set of all #-tuples of real numbers.
• A vector &'⃗ ∈ ℝ% is an #-tuple of real numbers.
• A matrix * ∈ ℝ+×% is a rectangular array of real numbers with -rows and # columns. For example, a 3 × 2 matrix looks like this:
• Which math operations can perform on vectors and matrices?
5
Vectors and matrices
6
., 0 Dentry:)123#&567: &-
#&96:;-#: &#
-1<5.=&>.?@: &-&=&#
.-th row vector: 52 = 12B, 12C, … , 12% , 0-column vector: 93 =
1B31C3
⋮1+3
If * = 123 %×%Then Trace of A, F5 * = 1BB + 1CC + ⋯+ 1%%&&
Properties: F5 * ± J = F5 * ± F5 J , F5 *J = F5(J*)
* = 123 =
1BB1CB⋮
12B⋮
1+B
1BC1CC⋮
12C⋮
1+C
⋯
1B31C3
⋮
123
⋮
1+3
⋯
1B%1C%⋮
12%⋮
1+%+×%
Vector operations
• Operations can perform on vectors ; &= ;B, ;C, ;M &and '⃗ = 'B, 'C, 'M
are:
• ;. '⃗ = ; & '⃗ 96>O, O is the angle between two vectors.• ; and '⃗ are orthogonal if O = 90R. In this case ;. '⃗ = 0.
7
Addition
Subtraction
Scaling
norm (length))
dot)product
cross)product
Linear independence
8
Matrix operations
• Matrix addition:
ST&* = & 123 +×%, J = & U23 +×%
&&&&&&Fℎ@#&* + J = 123 +×%+& U23 +×%
= 123 + U23 +×%
• Matrix subtraction:
* − J = * + −1 J
• Scalar multiplication:
ST&* = & 123 +×%&1#Y&9&.>&1&>91:15,&
Fℎ@#&&9* = 9123 +×%
9
Matrix operations
• Matrix-vector product
• Matrix multiplication
• Matrix inverse (denoted *ZB)
• Determinant (denoted det * 65& * &)
• Matrix transpose (denoted *^)
10
Property:) *J ^ = J^*^
Matrix multiplication
11
pnijnmij bBaA !! == ][ ,][ If
pmijpnijnmij cbaAB !!! == ][][][Then
njin
n
kjijikjikij babababac +++==!
=
!1
2211where
Properties: (1) A+B = B+A, (2) BAAB !
Size of AB
!!!!
"
#
$$$$
%
&
=
!!!!!
"
#
$$$$$
%
&
!!!!!!
"
#
$$$$$$
%
&
inijii
nnnjn
nj
nj
nnnn
inii
n
ccccbbb
bbbbbb
aaa
aaa
aaa
!!!!
""""!"!!
"!
""!
"""!
21
1
2221
1111
21
21
11211
Matrix-vector product
• Two interpretations: ! In column picture (C), the multiplication of the matrix * by the vector =⃗
produces a linear combination of the columns of the matrix: _⃗ = *=⃗ =
=B*[:,B] + =C*[:,C], where *[:,B]&and *[:,C]&are the first and second columns of the matrix *.
! In the row picture, (R), multiplication of the matrix A by the vector =⃗produces a column vector with coefficients equal to the dot products of rows of the matrix with the vector =⃗.
12
Linear transformations
• A mapping Fb:ℝ% ⟶ ℝ+ is called linear transformation if for any two vectors ;, ' ∈ ℝ% and any scalar 9, we have Fb ; + ' = Fb ; + Fb(')
and Fb 9; = 9Fb(;).
• Given a matrix * ∈ ℝ+×%&and&= ∈ ℝ%, the function Fb = = *= describes the linear transformation Fb that takes #-vectors as inputs and produces --vectors as outputs:
Fb:ℝ% ⟶ ℝ+.&
• We simply write *=⃗ = U.
• Applying the linear transformation Fb to the vector =⃗ corresponds to the product of the matrix * and the column vector =⃗. We say Fb is represented by the matrix *.
13
Linear transformations
14
!!"
!!#
$
=+++
=+++=+++
mnmnmm
nn
nn
bxaxaxa
bxaxaxabxaxaxa
!"!!
2211
22222121
11212111
= = =
A x b
equationslinear �m
equationmatrix Single
bx A = 1 !! nnm 1!m!
!!!
"
#
$$$$
%
&
=
!!!!
"
#
$$$$
%
&
!!!!
"
#
$$$$
%
&
mnmnmm
n
n
b
bb
x
xx
aaa
aaaaaa
!!"
!!!!""
2
1
2
1
21
22221
11211
Matrix inverse
• Consider matrix *%×%. If matrix J%×% such that *J = J* = S%
Then (1) * is invertible (or nonsingular), (2) J is the inverse of *.
• The inverse matrix *ZB&undoes the effects of the matrix *
*ZB* = S% ≡
1 0
⋱
0 1
• To solve for h in h* = J, multiple both sides by *ZB&from the right,
we have h = J*ZB.
• To solve for h in *Jihj = k, multiple both sides by jZB&on the right,
and *ZB, JZB, iZB from the left, we have
h = iZBJZB*ZBkjZB.
15
Systems of equations as matrix equations
• Consider the matrix equation had the form *=⃗ = U, where * is
a 2 × 2 matrix, =⃗&is the vector of unknowns, and U is a vector of
constants.
• We can solve for =⃗ by multiplying both sides of the equation by the
matrix inverse *ZB:
• But how did we know what the inverse matrix *ZB is?
16
Find the inverse of a matrix by
Gauss-Jordan elimination method
• l1;>> − m65Y1#&@:.-.#1<.6#:& *&|&S → S&|&*B
17
!"
#$%
&''
=3141
ASolution *h = S
!"
#$%
&=!
"
#$%
&!"
#$%
&'' 10
013141
2221
1211
xxxx
!"
#$%
&=!
"
#$%
&''''++
1001
3344
22122111
22122111
xxxxxxxx
(2) 1304
(1) 0314
2212
2212
2111
2111
=!!=+=!!=+"
xxxxxxxx
18
Find the inverse of a matrix by Gauss-Jordan elimination method
!"#
$%& '((( )(!"
#$%&
''*'
110301
031141)1(
)4(21
)1(12 ,
!!
!! rr
!"#
$%& '((( )(!"
#$%&
''*'
110401
131041)2(
)4(21
)1(12 ,
!!
!! rr
1 ,3 2111 =!=" xx
1 ,4 2212 =!=" xx
)( 1143 1-
2221
12111 AAIAXxxxx
AX ==!"
#$%
& ''=!
"
#$%
&== '
Thus
11104301
10310141
1
inationJordanElimGauss
, )4(21
)1(12
!
!
"#
$%&
' !!((( )("
#
$%&
'!!
!
AIIA
rr !!
!!
If * can’t be row reduced to S, then * is singular.
Fundamental vector spaces
• A vector space consists of a set of vectors and all linear combinations of
these vectors. For example the vector space p = >q1#{'⃗B, '⃗C} consists of all
vectors of the form '⃗ = t'⃗B + u'⃗C, where t, u ∈ ℝ.
• The column space v * &of a matrix * is the set of vectors that can be
produced as linear combinations of the columns of the matrix *:
v * ≡ & _⃗ ∈ ℝ+|&_⃗ = *=⃗&&for&some&x ∈ ℝ%
• The null space }(*)&of * ∈ ℝ+×%&is&} * ≡ x ∈ ℝ%&|&*=⃗ = 0 .
• The row space ℛ * &of a matrix *&is the set of linear combinations of the
rows of *. The row space ℛ * &is the orthogonal complement of the null
space } * , .. @. ,&for all vectors '⃗ ∈ ℛ * and all vectors 7 ∈ } * ,&we
have '⃗. 7 = 0&and&ℝ% = } * ⊕ ℛ * where ⊕ stands for orthogonal
direct sum.
19
Basis
• A basis is any set of vectors that can be used as a coordinate system
for a vector space.
• The standard basis for the =_-plane, ' described by coordinate pair
'Å, 'Ç with respect to these axes, or equivalently '⃗ = 'ÅÉ̂ + 'ÇÖ̂,&&
where É̂ = 1,0 &1#Y&Ö̂ = 0,1 are unit vectors that point along the
&=-axis and _-axis respectively.
• Definition (Basis). A basis for a #-dimensional vector space Ü is any
set of # linearly independent vectors that are part of Ü.
• Any set of two linearly independent vectors eáB, eáC can serve as a
basis for ℝC. We can write any vector ' ∈ ℝC as a linear combination
of these basis vectors '⃗ = 'B@̂B + 'C@̂C.
20
Basis
• Note the same vector '⃗&&corresponds to different coordinate pairs
depending on the basis used: '⃗ = 'Å, 'Ç in the standard basis
Jà ≡ É,̂ Ö̂ , and '⃗ = 'B, 'C in the basis Jâ ≡ @̂B, @̂C
• Itis important to keep in mind the basis with respect to which the
coefficients are taken, and if necessary specify the basis as a
subscript, e.g., 'Å, 'Ç äã&or 'B, 'C äå
.
• Converting a coordinate vector from the basis Jâ to the basis Jà is
essence of data transformation, and performed as a multiplication by
a change of basis matrix:
• The change of basis from the Bs-basis to the Be-basis is accomplished
using the inverse matrix
21
Dimension and bases for vector spaces
• The dimension of a vector space is defined as the number of
vectors in a basis for that vector space.
• There is a general procedure for finding a basis for a vector space.
• Suppose you are given a description of a vector space in terms of
- vectors ç = >q1# '⃗B, '⃗C, … , '⃗+ and you are asked to find a basis
for ç and the dimension of ç.
• To find a basis for ç, you must find a set of linearly independent
vectors that span ç.&We can use the Gauss–Jordan elimination
procedure to accomplish this task.
22
Determinants
• The determinant of a matrix, denoted det&(*)&or |*|, is a special
way to combine the entries of a matrix that serves to check if a
matrix is invertible or not. The determinant formulas for 2 × 2 and
3 × 3 matrices are
23
Rank of a matrix
• Rank of a matrix&*+×%, denoted by 51#è(*), &is the maximum number
of its linearly independent rows (it is proven that number is equal
the maximum number of its linearly independent columns).
• We have 0 ≤ 51#è(*) ≤ min&(-, #)
24
Determinant)|ë|• Real+number• Defined+to+square+matrix+only• * ≠ 0& ⟹&∃*ZB
• * = 0,+we+only+know+that+all+the+columns+(or+rows)+together+are+linearly+dependent,+but+don’t+know+which+subset+of+columns++(or+rows)+which+are+linearly+independent.
!ïñó(ë)
• Integer• Defined+to+any+rectangular+matrix
• When+A+is+a&&#!#&square+matrix,+51#è(*) = 0& ⟹&∃*ZB
Eigenvalues and eigenvectors
• The vector " is an eigenvector of matrix A and ò is an eigenvalue of
A if:
*' = ò' (assume non-zero ')
• Interpretation: the linear transformation implied by A cannot
change the direction of the eigenvectors ', only their magnitude.
• To find the eigenvalues ò of a matrix *, find the roots of the
characteristic polynomial det * − òS = 0.
• Example:
23
Eigenvalues and Eigenvectors
• Eigenvalues and eigenvectors are only defined for square matrices
(i.e., -& = &#)
• Eigenvectors are not unique (e.g., if ' is an eigenvector, so is è')
• Suppose ò1, ò2, … , ò#&are the eigenvalues of *, then:
26
Homework for K236-L4
Based on the key issues mentioned in the class, choose
and use your suitable documents to study or recover what
you have learnt about linear algebra.
Recommended site
http://www.statisticshowto.com/matrices-and-matrix-
algebra/
Note: You don’t have to submit the report of this
homework.
27