E 600 Chapter 1: Introduction to Vector Spaces · PreviewIntroduction The Algebraic Structure of V.S. Normed V.S. and Continuity Convex sets and the separating hyperplane theorem

Preview Introduction The Algebraic Structure of V.S. Normed V.S. and Continuity Convex sets and the separating hyperplane theorem

E 600

Chapter 1: Introduction to Vector Spaces

Simona Helmsmueller

August 12, 2018


Goals of this lecture:

• Understand formal mathematical thinking and notation,

including the difference between properties and definitions

• Know the concepts of span, linear independence, basis and

dimension

• Gain an intuition for open and closed sets, continuity and

convergence

• Be able to graphically illustrate convex sets, a convex hull and

the separating hyperplane theorem

Following lectures (both in this class and other

courses) will assume these goals have been reached!


Contents

Introduction

The Algebraic Structure of V.S.

Definition

Subspaces and Linear Dependence

Normed V.S. and Continuity

Norms

Open sets, Closed sets, Compact sets

Continuity

Convex sets and the separating hyperplane theorem

Convex sets

Planes, halfspaces and the Separating Hyperplane Theorem


Introduction

Main objective of the theory of vector spaces:

Geometrical insights at hand with 2-or 3-dimensional real vectors

are really helpful. Can we, in some way, generalize these insights to

other mathematical objects, for which a geometric picture is not

available?

IN THIS CHAPTER, BY VECTOR WE NEED NOT MEAN A

N-TUPLE OF REAL NUMBERS, BUT MAY REFER TO MANY

MORE OBJECTS (FUNCTIONS, SEQUENCES, MATRICES,...)!


Introduction

Main objective of the theory of vector spaces:

Geometrical insights at hand with 2-or 3-dimensional real vectors

are really helpful. Can we, in some way, generalize these insights to

other mathematical objects, for which a geometric picture is not

available?

IN THIS CHAPTER, BY VECTOR WE NEED NOT MEAN A

N-TUPLE OF REAL NUMBERS, BUT MAY REFER TO MANY

MORE OBJECTS (FUNCTIONS, SEQUENCES, MATRICES,...)!


Contents

Introduction


Definition



Norms


Continuity


Convex sets



Definition(Real Vector Space)

Let X := (X,+, ·) be a set of vectors with two operations: the

vector addition + : X× X 7→ X and the scalar multiplication

· : X× R 7→ X. X is called a vector space if

(i) Vector addition and scalar multiplication are closed operations:

∀x, y ∈ X, λ ∈ R x + y ∈ X and λ · x ∈ X

(ii) Vector addition is commutative: ∀x, y ∈ X x + y = y + x

(iii) Vector addition is associative: ∀x, y, z ∈ X

x + (y + z) = (x + y) + z

(iv) There exists a null element 0 in X such that: ∀x ∈ X

x + 0 = x


Definition

(v) Scalar multiplication is associative: ∀λ, µ ∈ R ∀x ∈ X

λ(µx) = (λµ)x

(vi) Scalar multiplication is distributive over vector and scalar

additions:

∀λ ∈ R ∀x, y ∈ X λ(x + y) = λx + λy

∀ λ, µ ∈ R ∀ x ∈ X (λ+ µ)x = λx + µx

(vii) If 1 denotes the scalar multiplicative identity and 0 the scalar

zero, then:

∀x ∈ X 1x = x and 0x = 0n


Exercise:

Does the following define a vector space?

for [a, b] ⊂ R, define V := {f : [a, b]→ [a, b]},

∀f ∈ V , a ∈ R : af := f : [a, b]→ [a, b] with f (x) := af (x)

and

∀f , g ∈ V : f + g := h : [a, b]→ [a, b] with h(x) := f (g(x))


Definition(Cartesian product)

Let X := (X,+, ·) and Y := (Y,+, ·) be two real vector spaces.

We define the cartesian product of X and Y, denoted X×Y as the

collection of ordered pairs (x , y) with x element of X and y

element of Y together with two operations: addition and scalar

multiplication, defined respectively as

(x1, y1) + (x2, y2) = (x1 + x2, y1 + y2) and λ(x , y) = (λx , λy).


Definition(Dot product)

Let x = (x1, ..., xn), y = (y1, ..., yn) ∈ Rn. Then the dot product of

these two n-dimensional vectors is a real number:

x • y = x1 · y1 + ...+ xn · yn.

Example:

If x = (1, 2, 3) and y = (5, 6, 7) then

x • y = 1 · 5 + 2 · 6 + 3 · 7 = 38.


Exercise:

Pick three vectors u, v ,w ∈ R4 and a scalar λ. Verify that the

following properties hold and discuss with your neighbor why this is

so.

Theorem(Properties of the dot product)

(a) u • v = v • u(b) u • (v + w) = u • v + u • w(c) u • (λv) = λ(u • v) = (λu) • v(d) u • u ≥ 0

(e) u • u = 0→ u = 0

(f) (u + v) • (u + v)) = u • u + 2(u • v) + v • v


Definition(Closure Under an Operation)

Let X := (X,+, ·) be a real vector space. We say that Y ⊆ X is

closed under the addition if and only if, for any two elements y1and y2 in Y, we have that y1 + y2 belongs to Y. Similarly, we can

define closure under scalar multiplication.

Definition(Vector Subspace)

Let X := (X,+, ·) be a real vector space and Y a non empty subset

of X. We say that Y is a subspace of X if and only if Y is closed

under vector addition and scalar multiplication.


Definition(Closure Under an Operation)

Let X := (X,+, ·) be a real vector space. We say that Y ⊆ X is

closed under the addition if and only if, for any two elements y1and y2 in Y, we have that y1 + y2 belongs to Y. Similarly, we can

define closure under scalar multiplication.

Definition(Vector Subspace)

Let X := (X,+, ·) be a real vector space and Y a non empty subset

of X. We say that Y is a subspace of X if and only if Y is closed

under vector addition and scalar multiplication.


Theorem( Intersection and Addition of Subspaces)

Let M and N be subspaces of a real vector space X. Then their

intersection, M ∩ N, is a subspace of X.

What about the union?


Theorem( Intersection and Addition of Subspaces)

Let M and N be subspaces of a real vector space X. Then their

intersection, M ∩ N, is a subspace of X.

What about the union?


Theorem(Generated Subspace (a.k.a Span))

Let Y be a subset of a real vector space X. Then, the set

Span(Y), which consists of all vectors in X that can be expressed

as linear combinations of vectors in Y, is a subspace of X. It is

called the subspace generated by Y or span of Y and it is the

smallest subspace which contains Y.


Examples:

Let Y1 = {(1, 0), (0, 1)}. What is Span(Y1)?

Let Y2 = {(1, 0), (0, 2), (0, 0.5)}. What is Span(Y2)?

Let Y3 = {f (x) = x + 1, g(x) = x2 + 2}. What is Span(Y3)?


Example:

Let u, v ∈ Rn. Then, Span(u, v) = {λu + µv : λ, µ ∈ R}. If u is a

multiple of v , then Span(u, v) is simply the line spanned by u, and

Span(u, v) = Span(u) = Span(v). However, if u is not a multiple

of v , then Span(u, v) is a two-dimensional plane, which contain

the lines Span(u) and Span(v).


Definition(Linear Dependence, Linear Independence)

Let x be an element of a real vector space X. x is said to be

linearly dependent upon a set S of vectors of X if it can be

expressed as a linear combination of vectors from S. Equivalently,

x is linearly dependent upon S if and only if x ∈ Span(S). If that

is not the case, the vector x is said to be linearly independent of

the set S. Finally, a set of vectors is said to be a linearly

independent set if each vector of the set is linearly independent of

the remainder of the set.


Theorem(Testing Linear Independence)

A necessary and sufficient condition for the set of vectors

x1, x2, ..., xn to be linearly independent is that:

Ifn∑

k=1

λkxk = 0, then ∀k = 1, 2, ..., n λk = 0.


Example (from Simon & Blume (1994)):

The vectors

e1 =

1

0...

0

, ..., en =

0...

0

1

∈ Rn

are linearly independent, because if c1, ..., cn ∈ R such that

c1e1 + ...+ cnen = 0,

c1

1

0...

0

+ c2

0

1...

0

+ ...+ cn

0

0...

1

=

c1

c2...

cn

=

0

0...

0

.The last vector equation implies that c1 = c2 = ... = cn = 0.


Definition(Basis and Space Dimension)

A finite set S of linearly independent vectors is said to be a basis

for the space X if S generates X. The number of elements in the

basis of vector space X is called its dimension.


Theorem(Uniqueness of the Dimension)

Any two bases for a finite dimensional vector space contain the

same number of elements.


Exercise:

What is the dimension of a plane in Rn?

What is the dimension of the set of all real-valued functions

defined on [a, b]?


Contents

Introduction


Definition



Norms


Continuity


Convex sets



Definition(Normed Space)

Let X be a vector space. If we can define a real-valued function ‖.‖which maps each element x in X into a real number ‖x‖, and if

that function is such that:

(i) ∀x ∈ X, ‖x‖≥ 0, ‖x‖= 0 if and only if x = 0,

(non-negativity)

(ii) ∀x , y ∈ X, ‖x + y‖≤ ‖x‖+‖y‖, (triangle

inequality)

(iii) ∀x ∈ X ∀λ ∈ R, ‖λx‖= |λ|‖x‖. (absolute

homogeneity1)

Then ‖.‖ is called a norm for X and (X, ‖.‖) a normed space.


An important example for us is the Euclidean norm in Rn:

∀x = (x1, ..., xn) ∈ Rn ‖x‖:=

(n∑

i=1

x2i

)1/2

= (x ′x)1/2

The Euclidean norm is important (e.g. OLS) and intuitive

(geometric interpretation). All following results and definitions

make use of the Euclidean norm. However, you should bear in

mind that there are more general concepts of norms and the

following definitions can be generalized to suit these. Read the

lecture notes for the general version.


An important example for us is the Euclidean norm in Rn:

∀x = (x1, ..., xn) ∈ Rn ‖x‖:=

(n∑

i=1

x2i

)1/2

= (x ′x)1/2

The Euclidean norm is important (e.g. OLS) and intuitive

(geometric interpretation). All following results and definitions

make use of the Euclidean norm. However, you should bear in

mind that there are more general concepts of norms and the

following definitions can be generalized to suit these. Read the

lecture notes for the general version.


Definition(ε-Open Ball)

Let (X, ||·||) be a Euclidean space, x0 be an element of X, and ε be

a strictly positive real number. The ε-open ball Bε(x0) centered at

x0 is the set of points whose distance from x0 is strictly smaller

than ε, that is:

Bε(x0) = {x |x ∈ X, ||x − x0||) < ε}.


Definition(ε-Closed Ball)

Let (X, ||·||) be a Euclidean space, x0 be an element of X, and ε be

a strictly positive real number. The ε-closed ball Bε[x0] centered

on x0 is the set of points whose distance from x0 is smaller than or

equal to ε, that is:

Bε[x0] = {x |x ∈ X, ||x − x0||≤ ε}.


Definition(Interior Point, Interior)

Let A be a subset of a metric space X. The point a in A is said to

be an interior point of A if and only if there exists ε > 0 such that

the ε-open ball centered at a lies entirely inside A. The collection

of all interior points of A is called the interior of A, denoted Int(A)

or A.


Definition(Open Set)

Let A be a subset of a metric space X. A is said to be an open set

if and only if A =Int(A).


Definition(Closure Point, Closure)

Let A be a subset of a metric space X. The point x in X is said to

be a closure point of A if and only if, for every ε > 0, the ε-open

ball centered at x contains at least one point a that belongs to A.

The collection of all closure points of A is called the closure of A,

denoted A.


Definition(Closed Set)

Let A be a subset of a metric space X. A is said to be a closed set

if an only if A = A.


Definition(Boundary Point, Boundary)

Let A be a subset of a metric space X. The point x in X is said to

be a boundary point of A if and only if, for every ε > 0, the ε-open

ball centered on x contains at least one point a that belongs to A

and at least one point ac that belongs to the complement of A,

AC. The collection of all boundary points of A is called the

boundary of A and denoted ∂A.


Theorem(Properties of Open Sets)

Let (X, d(., .)) be a metric space. Then

(i) ∅ and X are open in X.

(ii) A set A is open if and only if its complement is closed.

(ii) The union of an arbitrary (possibly infinite) collection of open

sets is open.

(iii) The intersection of a finite collection of open sets is open.


Theorem(Properties of Closed Sets)

Let (X, d(., .)) be a metric space. Then

(i) ∅ and X are closed in X.

(ii) A set A is closed if and only if its complement is open.

(iii) The union of a finite collection of closed sets is closed.

(iv) The intersection of an arbitrary (possibly infinite) collection of

closed sets is closed.


Definition(Continuous Function)

A function mapping from space X to Y is continuous at x0 ∈ X if

and only if, for every ε > 0, there is a δ > 0 such that if

||x − x0||< δ, then ||f (x)− f (x0)||< ε. A function that is

continuous at every point of its domain is said to be continuous.


Definition(Convergence)

An infinite sequence of vectors {xn}n∈N in X is said to converge to

a vector x ∈ X iff the sequence {||xn − x ||}n∈N of real numbers

converges to 0. That is,

∀ε > 0 ∃N ∈ N ∀n > N ||xn − x ||< ε.

In this case, we write xn →n→∞ x .


Exercise: Show that the limit of a converging sequence is unique

in a metric space.


Theorem(Characterization of Continuity)

A function mapping from X to Y is continuous at x0 ∈ X if and

only if xn → x0 implies f (xn)→ f (x0).


Contents

Introduction


Definition



Norms


Continuity


Convex sets



Definition(Convex Combination)

A convex combination of the vectors x1, x2,...,xn is a linear

combination of the vectors, i.e., a sumn∑

i=1

λixi , λi ∈ R, such that

the following additional requirements hold:

n∑i=1

λi = 1 and ∀i λi ∈ [0, 1]


Exercise:

Let n = 2 and x , y ∈ R with x < y . Graphically display the convex

combination of these x and y .


Definition(Characterization of Convex Sets)

Let Y be a subset of a vector space X. Y is convex if and only if

the convex combination between any two of its elements is

contained in Y.


Definition(Convex Hull)

Let Y be the subset of a vector space X. The convex hull, denoted

Co(Y) is the smallest convex set containing Y.

The convex hull of Y may also be expressed as the set of all

possible convex combinations of the elements of Y:

Co(Y) =

{x ∈ X : ∃y1, y2, · · · , yn ∈ Y and λ ∈ [0, 1]n

s.t.n∑

i=1

λi = 1 and x =n∑

i=1

λiyi

}


TheoremLet u, v be vectors in Rn. In the plane spanned by the two vectors,

let θ be the angle between them (see picture below). Then

u • v = ||u||·||v ||cosθ.


Definition(Hyperplane)

Let X be a subspace of Rn. Then, a hyperplane of X is a set of the

form:

Hba := {x ∈ X | a • x = b}

where a is an element of Rn that is different from 0 and b is an

element of R.


Example: Lines


Example: Planes


Definition(Halfspace)

Let X be a subspace of Rn. Then, a halfspace of X is a set of the

form:

Hb−a := {x ∈ X | a′x ≤ b}

or

Hb+a := {x ∈ X | a′x ≥ b}

where a is an element of Rn that is different from 0 and b is an

element of R.


Theorem(Separating Hyperplane Theorem)

Let C and D be two convex sets in a metric space X. Further,

assume C∩D = ∅. Then, there exists a 6= 0 in Rn and b in R such

that for all x in C a′x ≤ b and for all x in D a′x ≥ b. The

hyperplane {x ∈ X | a′x = b} is called a separating hyperplane for

the sets C and D.

Documents

E 600 Chapter 1: Introduction to Vector Spaces · PreviewIntroduction The Algebraic Structure of V.S. Normed V.S. and Continuity Convex sets and the separating hyperplane theorem