IE 521 Convex Optimization

Lecture 2: Convex Geometry

Niao He

4th February 2019

IE 521 ConvexOptimization

Niao He

Warm-up

Quick Review

Questions

Convex Geometry

Radon’s Theorem

Helley’s Theorem

Separation Theorem

Outline

Warm-upQuick ReviewQuestions

Convex GeometryRadon’s TheoremHelley’s TheoremSeparation Theorem

1 / 25

IE 521 ConvexOptimization

Niao He

Warm-up

Quick Review

Questions

Convex Geometry

Radon’s Theorem

Helley’s Theorem

Separation Theorem

Quick Review

I Convex setI X is convex iff λx + (1− λ)y ∈ X ,∀x , y ∈ X , λ ∈ [0, 1]

I Convex hullI Conv(X ) ={∑k

i=1 λixi : k ∈ N, λi ≥ 0,∑k

i=1 λi = 1, xi ∈ X ,∀i}

I Convexity-preserving operatorsI Taking intersection, Cartesian product, summationI Taking affine mapping, inverse affine mapping

I Topological propertiesI For convex sets, rint(X ) is dense in cl(X )

I Representation theoremI Any point in the convex hull of set X with dimension d

can be written as the convex combination of at mostd + 1 points in X .

2 / 25

IE 521 ConvexOptimization

Niao He

Warm-up

Quick Review

Questions

Convex Geometry

Radon’s Theorem

Helley’s Theorem

Separation Theorem

Question 1

Can you find a partition of the sets whose convex hullsintersect?

Figure: Four sets

3 / 25

IE 521 ConvexOptimization

Niao He

Warm-up

Quick Review

Questions

Convex Geometry

Radon’s Theorem

Helley’s Theorem

Separation Theorem

Question 2

If I1, I2 and I3 are intervals on the real line such that any twohave a point in common, do all three have a point incommon?

4 / 25

IE 521 ConvexOptimization

Niao He

Warm-up

Quick Review

Questions

Convex Geometry

Radon’s Theorem

Helley’s Theorem

Separation Theorem

Question 3

Which group is different from others?

Figure: Four groups of disjoint sets

5 / 25

IE 521 ConvexOptimization

Niao He

Warm-up

Quick Review

Questions

Convex Geometry

Radon’s Theorem

Helley’s Theorem

Separation Theorem

The Mathematicians

Figure: JohannRadon (1887–1956)

Figure: EduardHelley (1884–1943)

Figure: HermannMinkowski(1864–1909)

6 / 25

IE 521 ConvexOptimization

Niao He

Warm-up

Quick Review

Questions

Convex Geometry

Radon’s Theorem

Helley’s Theorem

Separation Theorem

Radon’s Theorem (J. Radon, 1921)

Theorem. Let S be a collection of N points in Rd withN ≥ d + 2. Then we can write S = S1 ∪ S2 s.t.

S1 ∩ S2 = ∅, and Conv(S1) ∩ Conv(S2) 6= ∅.

Remark.I Any set of d + 2 points in Rd can be partitioned into

two disjoint sets whose convex hulls intersect.I Can be used to show the VC-dimension of the class of

halfspaces (linear separators) in d-dimensions is d + 1.

Figure: 3 points separable vs 4 points nonseparable

7 / 25

IE 521 ConvexOptimization

Niao He

Warm-up

Quick Review

Questions

Convex Geometry

Radon’s Theorem

Helley’s Theorem

Separation Theorem

Proof of Radon’s Theorem

I Let S = {x1, ..., xN} with N ≥ d + 2.

I Consider the linear system{∑Ni=1 γixi = 0∑Ni=1 γi = 0

⇒ (d + 1) equationsN ≥ (d + 2) unknowns

So there exists a non-zero solution γ1, ..., γN .

I Let I = {i : γi ≥ 0}, J = {j : γj < 0} anda =

∑i∈I γi = −

∑j∈J γj , then∑

i∈Iγixi =

∑j∈J

(−γj)xj ⇒∑i∈I

γiaxi =

∑j∈J

−γja

xj

I The partition S1 = {xi , i ∈ I} and S2 = {xj : j ∈ J}gives the desired result.

8 / 25

IE 521 ConvexOptimization

Niao He

Warm-up

Quick Review

Questions

Convex Geometry

Radon’s Theorem

Helley’s Theorem

Separation Theorem

Back to the Question

If I1, I2 and I3 are intervals on the real line such that any twohave a point in common, do all three have a point incommon?

9 / 25

IE 521 ConvexOptimization

Niao He

Warm-up

Quick Review

Questions

Convex Geometry

Radon’s Theorem

Helley’s Theorem

Separation Theorem

Helley’s Theorem (E. Helly, 1923)

Theorem. Let S1, ...,SN be a collection of convex sets in Rd

with N > d . Assume every (d + 1) sets of them have apoint in common, then all the sets have a point in common.

Figure: Four convex sets in R2

Q. Does the theorem still hold if we relax N =∞?Q. Does the theorem still hold if we relax (d + 1) sets to dsets?

10 / 25

IE 521 ConvexOptimization

Niao He

Warm-up

Quick Review

Questions

Convex Geometry

Radon’s Theorem

Helley’s Theorem

Separation Theorem

Helley’s Theorem (E. Helly, 1923)

Theorem. Let S1, ...,SN be a collection of convex sets in Rd

with N > d . Assume every (d + 1) sets of them have apoint in common, then all the sets have a point in common.

Remark.I Not true for infinite collection:

I E.g. Si = [i ,∞),∩+∞i=1 Si = ∅

I Not true if reduce (d + 1) sets to d sets.

Corollary. Let {Sα} be any collection of compact convex setsin Rd . If every (d + 1) sets have a point in common, then allsets have a points in common.

11 / 25

IE 521 ConvexOptimization

Niao He

Warm-up

Quick Review

Questions

Convex Geometry

Radon’s Theorem

Helley’s Theorem

Separation Theorem

Proof of Helley’s Theorem

Figure: Illustration of N = 4, d = 2

12 / 25

IE 521 ConvexOptimization

Niao He

Warm-up

Quick Review

Questions

Convex Geometry

Radon’s Theorem

Helley’s Theorem

Separation Theorem

Proof of Helley’s Theorem

By induction on N.

I Base case: N = d + 1, obviously true.

I Induction step: Assume the collection of N(≥ d + 1)sets have common point if every (d + 1) of them havecommon point. Show this holds for N + 1 sets.

I From the assumption, ∃ {x1, x2, ..., xN+1} such thatxi ∈ S1 ∩ ... ∩ Si−1 ∩ Si+1 ∩ ... ∩ SN+1 6= ∅.

I By Radon’s theorem, we can split it into two disjointsets, {x1, . . . , xk} and {xk+1, . . . , xN}, and

Conv({x1, ..., xk}) ∩ Conv({xk+1, ..., xN+1}) 6= ∅.

I Let z ∈ Conv({x1, ..., xk}) ∩ Conv({xk+1, ..., xN+1}). Itcan be shown that z ∈ S1 ∩ ... ∩ SN+1 (why?).

13 / 25

IE 521 ConvexOptimization

Niao He

Warm-up

Quick Review

Questions

Convex Geometry

Radon’s Theorem

Helley’s Theorem

Separation Theorem

Application of Helley’s Theorem

Baby Theorem Let X contain a finite set of points in theplane, such that every three of them are contained in a diskof radius 1. Then X is contained in a disk of radius 1.

Jung’s Theorem. Let X contain a finite set of points in theplane, such that any two of them has distance no greaterthan 1. Then X is contained in a disk of radius 1/

√3.

Jung’s Theorem. Let X ⊂ Rn be a compact set such thatany two of them has Euclidean distance no greater than 1.

Then X is contained in a ball with radius√

n2(n+1) .

14 / 25

IE 521 ConvexOptimization

Niao He

Warm-up

Quick Review

Questions

Convex Geometry

Radon’s Theorem

Helley’s Theorem

Separation Theorem

Application of Helley’s Theorem

Question. Consider the optimization problem

p∗ = minx∈R10

g0(x), s.t. gi (x) ≤ 0, i = 1, ..., 521.

I Suppose ∀t ∈ R, X0 ={x ∈ R10 : g0(x) ≤ t

}is convex,

Xi ={x ∈ R10 : gi (x) ≤ 0

}is convex.

I How many constraints can you drop without affectingthe optimal value?

15 / 25

IE 521 ConvexOptimization

Niao He

Warm-up

Quick Review

Questions

Convex Geometry

Radon’s Theorem

Helley’s Theorem

Separation Theorem

Other Applications of Helley’s Theorem

Helleys theorem is a very fundamental result in convexgeometry and can be applied to show many results.

I The centerpoint theorem

I Farkas Lemma

I Sion-Kakutani Theorem

I Chebyshev approximation

16 / 25

IE 521 ConvexOptimization

Niao He

Warm-up

Quick Review

Questions

Convex Geometry

Radon’s Theorem

Helley’s Theorem

Separation Theorem

Back to the Question

When can we separate two sets?

Figure: Four groups of disjoint sets

17 / 25

IE 521 ConvexOptimization

Niao He

Warm-up

Quick Review

Questions

Convex Geometry

Radon’s Theorem

Helley’s Theorem

Separation Theorem

Separation of Sets

Definition. Let S and T be two nonempty convex sets in Rn.A hyperplane H =

{x ∈ Rn : aT x = b

}with a 6= 0 is said to

separate S and T if S ∪ T 6⊂ H and

S ⊂ H− ={x ∈ Rn : aT x ≤ b

}T ⊂ H+ =

{x ∈ Rn : aT x ≥ b

}

Figure: Separation of two sets

18 / 25

IE 521 ConvexOptimization

Niao He

Warm-up

Quick Review

Questions

Convex Geometry

Radon’s Theorem

Helley’s Theorem

Separation Theorem

Strict Separation of Sets

Definition. Let S and T be two nonempty convex sets in Rn.A hyperplane H =

{x ∈ Rn : aT x = b

}with a 6= 0 is said to

strictly separate S and T if

S ⊂ H−− ={x ∈ Rn : aT x < b

}T ⊂ H++ =

{x ∈ Rn : aT x > b

}

Figure: Strict Separation of two sets

19 / 25

IE 521 ConvexOptimization

Niao He

Warm-up

Quick Review

Questions

Convex Geometry

Radon’s Theorem

Helley’s Theorem

Separation Theorem

Strong Separation of Sets

Definition. Let S and T be two nonempty convex sets in Rn.A hyperplane H =

{x ∈ Rn : aT x = b

}with a 6= 0 is said to

strongly separate S and T if there exits b′ < b < b′′ suchthat

S ⊂{x ∈ Rn : aT x ≤ b′

}T ⊂

{x ∈ Rn : aT x ≤ b′′

}Remark.

I Strict separation does not necessarily imply strongseparation.

I Strong separation is equivalent to say

supx∈S

aT x < infx∈T

aT x .

20 / 25

IE 521 ConvexOptimization

Niao He

Warm-up

Quick Review

Questions

Convex Geometry

Radon’s Theorem

Helley’s Theorem

Separation Theorem

Separation Hyperplane Theorem

Theorem. Let S and T be two nonempty convex sets. ThenS and T can be separated if and only if

rint(S) ∩ rint(T ) = ∅.

21 / 25

IE 521 ConvexOptimization

Niao He

Warm-up

Quick Review

Questions

Convex Geometry

Radon’s Theorem

Helley’s Theorem

Separation Theorem

Supporting Hyperplane Theorem

Theorem. Let S be a nonempty convex set and x0 ∈ ∂S .Then there exists a hyperplane H =

{x : aT x = aT x0

}with

a 6= 0 such that

S ⊂{x : aT x ≤ aT x0

}, and x0 ∈ H.

Figure: Supporting hyperplane

I This follows directly from the previous theorem.

I Such a hyperplane is called a supporting hyperplane.

22 / 25

IE 521 ConvexOptimization

Niao He

Warm-up

Quick Review

Questions

Convex Geometry

Radon’s Theorem

Helley’s Theorem

Separation Theorem

Strict Separation Hyperplane Theorem I

Theorem. Let S be closed and convex and x0 6∈ S , Thenthere exists a hyperplane that strictly separates x0 and S .

Figure: Strict separation

I Closedness of the set is crucial here.

I Separating hyperplane can be constructed based on theprojection.

23 / 25

IE 521 ConvexOptimization

Niao He

Warm-up

Quick Review

Questions

Convex Geometry

Radon’s Theorem

Helley’s Theorem

Separation Theorem

Strict Separation Hyperplane Theorem II

Theorem. Let S and T be two nonempty convex sets andS ∩ T = ∅. If S − T is closed, then S and T can be strictlyseparated.

Remark.

I Even if both S and T are closed convex, S − T mightnot be closed, and they might not be strictly separated.

I When both S and T are closed convex, S ∩ T = ∅ andat least one of them is bounded, then S − T is closed,and S and T can be strictly separated

24 / 25

IE 521 ConvexOptimization

Niao He

Warm-up

Quick Review

Questions

Convex Geometry

Radon’s Theorem

Helley’s Theorem

Separation Theorem

References

I Boyd & Vandenberghe, Chapter 2.5

I Ben-Tal & Nemirovski, Chapter 1.2.2-1.2.6

25 / 25