Embed Size (px)
Citation preview
Organisation
Lectures:
I Monday 24/09: 10.00 - 12.00 and 13.00 - 15.00 (HoF1.28/Shanghai)
I Tuesday 25/09: 10.00 - 12.00 and 13.00 - 15.00 (HoFE.20/DZ Bank)
I Wednesday 26/09: 10.00 - 12.00 and 13.00 - 15.00 (HoFE.01/Deutsche Bank)
I Thursday 27/09: 10.00 - 12.00 and 13.00 - 15.00 (HoFE.01/Deutsche Bank)
I Friday 28/09: 10.00 - 12.00 and 13.00 - 15.00 (HoFE.01/Deutsche Bank)
Tutorials:
I Daily problem sets
I Need not be handed in
Outline
Linear Algebra Overview
MATLAB Introduction
MATLAB Vectors and Matrices
MATLAB Plotting
MATLAB Root �nding and Optimization,Integration and Interpolation,Random numbers,Ordinary di�erential equations
Linear Algebra
Linear equations
Matrix notationSystem of linear equations:
a11x1 + ...+ a1nxn = b1
... = ...
am1x1 + ...+ amnxn = bm
Can be represented by: a11 . a1n. . .
am1 . amn
x1.xn
=
b1.bm
Fundamentals
FieldA Field is a set of not fewer than two numbers, which is closed withrespect to the four rational operations of addition, subtraction,multiplication, and division by any non-zero number.
→ e.g. real numbers, complex numbers, rational numbers
VectorLet F be a �eld. A vector x of order n > 0 over F is an ordered set(x1, x2, ..., xn) of n numbers which belong to F .
Multiplication by a scalar and addition
Multiplication of a vector by a scalar
Let x = (x1, x2, ..., xn) be de�ned on �eld F . Then for α ∈ F wehave
αx = (αx1, αx2, ..., αxn)
Vector additionVector addition is de�ned for vectors of the same order by thefollowing formula
(x1, x2, ..., xn) + (y1, y2, ..., yn) = (x1 + y1, x2 + y2, ..., xn + yn)
Relations
Commutative and Associative relationFor x , y and z being vectors the following relations hold:
x + y = y + x
x + (y + z) = (x + y) + z
α(x + y) = αx + αy
(α+ β)x = αx + βx
Linear independence
Linear independence
The vectors x1, x2, ..., xk are linearly independent if the linearcombination
α1x1 + α2x2 + ...+ αkxk = 0
implies thatα1 = α2 = ... = αk = 0.
Examples
1. Are (1, 2) and (2, 4) linearly independent?
2. Are (2, 1) and (2, 4) linearly independent?
Linear independence cont.
Example 1
α11+ α22 = 0
α12+ α24 = 0
⇒ α1 = −2α2
⇒ Linear dependence of (1, 2) and (2, 4)
Example 2
α12+ α21 = 0
α12+ α24 = 0
⇒ α1 = α2 = 0⇒ Linear independence of (2, 1) and (2, 4)
Dot product
Dot product
The dot product of two vectors with the same length (x1, x2, ..., xn)and (y1, y2, ..., yn) is de�ned as:
x · y =n∑
i=1
xiyi = x1y1 + x2y2 + ...+ xnyn
Example
Dot product of x = (1, 2,−5) and y = (2,−1,−2):
x · y = x1y1 + x2y2 + x3y3 = 1 · 2− 2 · 1+ 5 · 2 = 10
Vector space and Inner product space
Vector space
A vector space of order n over F is a set B of vectors of order nover F with the property that whenever x , y ∈ B , α ∈ F , we have
I x + y ∈ B
I αx ∈ B
I 0 = (0, 0, ..., 0) ∈ B
I −x = (−x1,−x2, ...,−xn) ∈ B
Vectorspace and Inner product space cont.
Inner product space
An inner product space is a vector space B over F together with amap 〈·, ·〉 : B × B → F which satis�es the following for x , y , z ∈ Band α ∈ F :
I 〈x , y〉 = 〈y , x〉I 〈αx , y〉 = α 〈x , y〉I 〈x + y , z〉 = 〈x , z〉+ 〈y , z〉I 〈x , x〉 ≥ 0
→ In Euclidean spaces (Rn) inner product is equal to dot product
Vector space and Inner product space cont.
Example
u = (u1, u2, ..., un), v = (v1, v2, ..., vn) ∈ Rn and w1,w2, ...,wn arepositive real numbers.Then the weighted Euclidean inner product is given by
〈u, v〉 = w1u1v1 + w2u2v2 + ...+ wnunvn
Check that ful�lls axioms of inner product space:
1.axiom: 〈u, v〉 = 〈v , u〉
〈u, v〉 = w1u1v1 + w2u2v2 + ...+ wnunvn
= w1v1u1 + w2v2u2 + ...+ wnvnun = 〈v , u〉
Vector space and Inner product space cont.
Example cont.
2. axiom: 〈cu, v〉 = c〈u, v〉 with c being a scalar.
〈cu, v〉 = w1cu1v1 + w2cu2v2 + ...+ wncunvn
= c(w1u1v1 + w2u2v2 + ...+ wnunvn) = c〈u, v〉
3.axiom: 〈u + v , a〉 = 〈u, a〉+ 〈v , a〉 with a = (a1, a2, ..., an) ∈ Rn
〈u + v , a〉 = w1(u1 + v1)a1 + w2(u2 + v2)a2 + ...+ wn(un + vn)an
= (w1u1a1 + w2u2a2 + ...+ wnunan)
+ (w1v1a1 + w2v2a2 + ...+ wnvnan)
= 〈u, a〉+ 〈v , a〉
4.axiom: 〈u, u〉 ≥ 0
〈u, u〉 = w1u21 + w2u
22 + ...+ wnu
2n ≥ 0
Vector space and Inner product space cont.
NormSuppose that V is an inner product space. The norm of a vector uin V is given by
||u|| = 〈u, u〉1/2.
DistanceSuppose that V is an inner product space and that u and v are twovectors in V . The distance between u and v ,denoted by d(u, v) isgiven by
d(u, v) = ||u − v ||.
Vector space and Inner product space cont.
Cauchy-Schwarz Inequality
Suppose u and v are two vectors in an inner product space, then
|〈u, v〉| ≤ ||u|| · ||v ||.
Triangle Inequality
Suppose u and v are two vectors in an inner product space , then
||u + v || ≤ ||u||+ ||v ||.
Orthogonality
Orthogonality
Two vectors of the same length x = (x1, x2, ..., xn) andy = (y1, y2, ..., yn) in an inner product space B are orthogonal iftheir dot product x · y = 0.
Example
Are x = (1, 5,−2) and y = (3,−1,−1) orthogonal?
x · y = x1y1 + x2y2 + x3y3 = 1 · 3− 5 · 1+ 2 · 1 = 0
⇒ x and y are orthogonal
Matrices
De�nitionA m × n matrix A over a �eld F is a rectangular array of numbersin F consisting of m rows and n columns.
A =
a11 a12 . a1na21 a22 . a2n. . . .
am1 am2 . amn
Special matrices
Square matrix
A square matrix of order n is a n× n matrix. The elements a11, a22,... , ann are called the diagonal elements.
Zero matrixA zero matrix is a m × n matrix where all elements are 0.
Identity matrix
A identity matrix is a n × n matrix where the diagonal elements are1 and the rest 0 (denoted by In).
Special matrices cont.
Diagonal matrix
A diagonal matrix is a n× n matrix where all non-diagonal elementsare 0.
Triangular matrix
I An upper triangular matrix is a n × n matrix where allelements below the diagonal are 0.
I A lower triangular matrix is a n × n matrix where all elementsabove the diagonal are 0.
Idempotent matrix
The matrix A is idempotent if A = A2.
Matrix operations
SummationIf both A and B are m × n matrices, then (A+ B)ij = Aij + Bij .
Example
A =
[1 34 1
], B =
[3 −1−2 1
][1 34 1
]+
[3 −1−2 1
]=
[1+ 3 3− 14− 2 1+ 1
]=
[4 22 2
]
Matrix operations cont.
Multiplication
Let A be a k ×m and B a m × n matrix. Then (AB)ij =m∑l=1
AilBlj .
⇒ A · B 6= B · AFor a square matrix A (n × n) we have A0 = In and Ar = Ar−1A.
Example
A =
[1 34 1
], B =
[3 −1−2 1
][1 34 1
]·[
3 −1−2 1
]=
[1 · 3+ 3 · (−2) 1 · (−1) + 3 · 14 · 3+ 1 · (−2) 4 · (−1) + 1 · 1
]=
[−3 210 −3
]
Matrix operations cont.
Transpose
The transpose A′ of a matrix A is a matrix obtained from A byinterchanging rows and columns.
Example
A =
[1 3 24 −1 2
]⇒
A′ =
1 43 −12 2
Matrix operations cont.
Epsilon sign
For x = (x1, x2, ..., xn) we have ε(x) = sgn∏
1≤r<s≤n(xs − xr ).
Determinant of a matrixThe determinant of a n × n matrix A = [aij ] is the number∑
(λ1,λ2,...,λn)
ε(λ1, λ2, ..., λn)a1λ1a2λ2 ...anλn
where the summation is over all the n! arrangements(λ1, λ2, ..., λn) of (1, 2, ..., n). This determinant is denoted by
|A| =
∣∣∣∣∣∣∣∣a11 a12 . a1na21 a22 . a2n. . . .
an1 an2 . ann
∣∣∣∣∣∣∣∣or det(A) or |aij |n.
Matrix operations cont.
Example
A =
2 4 51 2 32 1 3
, x = (1, 2, 3)
Compute ε(x) for all arrangements:
x = (1, 2, 3) : ε(x) = sgn((2− 1)(3− 1)(3− 2)) = 1
x = (1, 3, 2) : ε(x) = sgn((3− 1)(2− 1)(2− 3)) = −1x = (2, 1, 3) : ε(x) = sgn((1− 2)(3− 1)(3− 2)) = −1x = (2, 3, 1) : ε(x) = sgn((3− 2)(1− 3)(1− 2)) = 1
x = (3, 1, 2) : ε(x) = sgn((2− 1)(1− 3)(2− 3)) = 1
x = (3, 2, 1) : ε(x) = sgn((2− 3)(1− 3)(1− 2)) = −1
Matrix operations cont.
Example cont.
det(A) = 1 · 2 · 2 · 3+ (−1) · 2 · 3 · 1+ (−1) · 4 · 1 · 3+ 1 · 4 · 3 · 2+ 1 · 5 · 1 · 1+ (−1) · 5 · 2 · 2 = 3
Matrix operations cont.
Determinant of a matrix cont.Further, if (µ1, µ2, ..., µn) is any �xed arrangement of (1, 2, ..., n),then the determinant can be expressed as∑
(λ1,λ2,...,λn)
ε
(λ1, λ2, ..., λnµ1, µ2, ..., µn
)aλ1µ1aλ2µ2 ...aλnµn
with ε
(λ1, λ2, ..., λnµ1, µ2, ..., µn
)= ε(λ1, λ2, ..., λn) · ε(µ1, µ2, ..., µn).
Matrix operations cont.
Determinant - Special cases
2× 2 matrix - Example
|A| =∣∣∣∣ 4 2−3 1
∣∣∣∣ = 4 · 1− 2 · (−3) = 10
3× 3 matrix - Example
|A| =
∣∣∣∣∣∣4 2 −12 3 15 −2 −1
∣∣∣∣∣∣= 4 · 3 · (−1) + 2 · 1 · 5+ 2 · (−2) · (−1)− 5 · 3 · (−1)− 2 · 2 · (−1)− 4 · (−2) · 1= 29
Matrix operations cont.
Properties of the determinant
I det(In) = 1
I det(AT ) = det(A)
I det(A−1) = 1
det(A)
I det(AB) = det(A) det(B)
I det(cA) = cn det(A) for n × n matrix A
I If A is a n × n triangular matrix then
det(A) = a1,1 · a2,2 · ... · an,n =n∏
i=1
ai ,i
Matrix operations cont.
MinorFor a n × n matrix A, the minor of it's entry aij is denoted by Mij
and is de�ned to be the determinant of the submatrix obtained byremoving from A it's i th row and j th column.
Example
A =
4 2 −12 3 15 −2 −1
M23 =
∣∣∣∣ 4 25 −2
∣∣∣∣ = 4 · (−2)− 2 · 5 = −18
Matrix operations cont.
CofactorFor a n × n matrix A with Mij being the minor of it's entry aij , Cij
is called the cofactor of aij and is de�ned as
Cij = (−1)i+jMij
Example
A =
4 2 −12 3 15 −2 −1
C23 = (−1)(2+3)M23 = (−1)5
∣∣∣∣ 4 25 −2
∣∣∣∣ = (−1)(4·(−2)−2·5) = 18
Matrix operations cont.
Cofactors and DeterminantFor a n× n matrix A with Cij being the cofactor of it's entry aij thefollowing holds:
|A| =n∑
i=1
aijCij for any j
=n∑
j=1
aijCij for any i
Matrix operations cont.
Cofactors and Determinant
Example
A =
4 2 −12 3 15 −2 −1
C13 = (−1)(1+3)M13 = (−1)4∣∣∣∣ 2 35 −2
∣∣∣∣ = 2 · (−2)− 3 · 5 = −19
C23 = (−1)(2+3)M23 = (−1)5∣∣∣∣ 4 25 −2
∣∣∣∣ = (−1)(4 · (−2)− 2 · 5) = 18
C33 = (−1)(3+3)M33 = (−1)6∣∣∣∣ 4 22 3
∣∣∣∣ = (4 · 3− 2 · 2) = 8
|A| =3∑
i=1
ai3Ci3 = (−1) · (−19) + 1 · 18+ (−1) · 8 = 29
Matrix operations cont.
Singularity
A n × n matrix A is said to be singular if |A| = 0.→ Matrix is singular if linearly dependent vectors.
InverseThe inverse of a n × n matrix A is a matrix A−1 such that
A · A−1 = I ,
where I is the identity matrix.
Adjugate
The adjugate matrix A∗ of the n × n matrix A is the transpose ofthe matrix of cofactors of the elements of A.
Matrix operations cont.
Example adjugate
A =
4 2 −12 3 15 −2 −1
C11 = M11 = −1,C12 = (−1)M12 = 7,C13 = M13 = −19C21 = (−1)M21 = 4,C22 = M22 = 1,C23 = (−1)M23 = 18
C31 = M31 = 5,C32 = (−1)M32 = −6,C33 = M33 = 8
A∗ =
−1 7 −194 1 185 −6 8
′ = −1 4 5
7 1 −6−19 18 8
Methods to compute Inverse
Search inverse of
A =
4 2 −12 3 15 −2 −1
Gauss-Jordan-Algorithm 4 2 −1 1 0 0
2 3 1 0 1 05 −2 −1 0 0 1
1 0 0 − 1
29
4
29
5
29
0 1 0 7
29
1
29− 6
29
0 0 1 −19
29
18
29
8
29
Methods to compute Inverse cont.
With the help of the Adjugate
A−1 =1
|A|A∗
A−1 =1
29
−1 4 57 1 −6−19 18 8
=
− 1
29
4
29
5
297
29
1
29− 6
29
−19
29
18
29
8
29
Special case 2× 2 matrix
A =
[4 2−3 1
]A−1 =
1
|A|A∗ =
1
10
[1 −23 4
]=
[1
10− 2
103
10
4
10
]
Matrix operations cont.
Vec operator
Let A be a m × n matrix. Then B = vec(A) is a mn × 1 matrixobtained from stacking the columns of A.
Tracetr(A) is the trace of the n × n matrix A and is the sum of thediagonal elements of A.
Example
A =
4 2 −12 3 15 −2 −1
tr(A) = 4+ 3− 1 = 6
Matrix operations cont.
Minor of order kLet A be a m × n matrix. If k ≤ m and l ≤ n, then any k rows andl columns of A determine a k × l submatrix of A. The determinantof a k × k submatrix of A is called a k-rowed minor of A, or aminor of order k .
RankThe rank R(A) or rank(A) of a non-zero matrix A is the maximumvalue of r for which there exists a non-vanishing r -rowed minor ofA. Therefore the rank of A is smaller or equal to the minimum ofthe numbers of rows and columns.
→ Rank (A) is the maximum number of linearly independent rowor column vectors.
Matrix operations cont.
Quadratic formA quadratic form on Rn is a real valued functionQ(x1, x2, ..., xn) =
∑i≤j
aijxixj which can be represented using
matrices as Q(x) = x ′Ax with
A =
a11
1
2a12 . 1
2a1n
1
2a12 a22 . 1
2a2n
. . . .1
2a1n
1
2a2n . ann
→ Example for matrix A: Variance-Covariance-Matrix
Matrix operations cont.
Positive / negative (semi-)de�nite
For any nonzero (n × 1) vector x a n × n matrix A is said to be
I positive de�nite if x ′Ax > 0.
I positive semi-de�nite if x ′Ax ≥ 0.
I negative de�nite if x ′Ax < 0.
I negative semi-de�nite if x ′Ax ≤ 0.
Matrix operations cont.
Example
A =
4 2 −12 3 15 −2 −1
, x =
121
x ′Ax = [ 1 2 1 ]
4 2 −12 3 15 −2 −1
121
= [ 13 5 0 ]
121
= 23
⇒ A is positive (semi-)de�nite.
Matrix operations cont.
Eigenvalue and Eigenvector
For the n × n matrix A a non-zero n × 1 vector v is an eigenvectorof A if there exists a scalar α such that
Av = αv .
Computation
The eigenvalues of A are precisely the solutions α to the equation
det(A− αI ) = 0.
Matrix operations cont.
Example
Compute eigenvalues and eigenvectors for A =
[4 2−3 1
].
Eigenvalues:
det(A− αI ) =∣∣∣∣ 4− α 2
3 1− α
∣∣∣∣ = 4− 5α+ α2 − 6 = 0
α1/2 =5
2±√
25
4+ 2
α1 = 5.3723, α2 = −0.3723
Matrix operations cont.
Example cont.
Eigenvectors:
α = 5.3723:[4 23 1
] [v11
v12
]= 5.3723
[v11
v12
]⇒ v1 =
[0.82460.5658
]α = −0.3723:[
4 23 1
] [v21
v22
]= −0.3723
[v21
v22
]⇒ v2 =
[−0.41600.9094
]
Linear equations
Matrix of Coe�cients and augmented matrix
In the system of linear equations
a11x1 + ...+ a1nxn = b1
... = ...
am1x1 + ...+ amnxn = bm
the matrix A =
a11 . a1n. . .
am1 . amn
is called the matrix of coe�cients
and B =
a11 . a1n b1. . . .
am1 . amn bm
is called the augmented matrix.
Linear equations cont.
Theorem 1: Consistency
A necessary and su�cient condition for a system of linear equationsto be consistent is that the matrix of coe�cients has the same rankas the augmented matrix.
Theorem 2: SolutionsLet A be the matrix of coe�cients and B the augmented matrix.Then the associated system of linear equations possesses an in�nitenumber of solutions if and only if rank(A) = rank(B) < n. Itpossesses a unique solution if and only if rank(A) = rank(B) = n.It possesses no solution if and only if rank(A) < rank(B).
Linear equations cont.
Theorem 3: Nontrivial solution for homogeneous system
The homogeneous system Ax = 0 in n unknowns possesses anontrivial solution if rank(A) < n.
Theorem 4The solution to the homogeneous system Ax = 0 in n unknownsconstitutes a vector space of dimensionality n − rank(A).
Functions
Vector derivativesThe derivative of a m × 1 vector y with respect to a n × 1 vector xis the n ×m matrix
∂y
∂x=
∂y1∂x1
∂y2∂x1
. ∂ym∂x1
∂y1∂x2
∂y2∂x2
. ∂ym∂x2
. . . .∂y1∂xn
∂y2∂xn
. ∂ym∂xn
Functions
Gradient and Hessian matrixLet f : Rn → R then the gradient (denoted by ∇ and is also called'del' or 'nabla') of f at x = (x1, ..., xn)
′
∇f (x) = ∂f (x)
∂x=
∂f (x)∂x1.
∂f (x)∂xn
and its Hessian is
H(x) =∂2f (x)
∂x ′∂x=
(∂
∂x1∇f (x), ..., ∂
∂xn∇f (x)
)→ The Hessian is a n × n matrix and symmetric.
