Convexity

S ⊆ Rn is called convex if for all x , y ∈ S and all 0 ⩽ α ⩽ 1

f : S → R is called convex on S ⊆ Rn if for \ x , y ∈ S and all 0 ⩽ α ⩽ 1

Q: Give an example of a convex, but not strictly convex function.

179

Convexity: Consequences

If f is convex, . . .

If f is strictly convex, . . .

180

Optimality ConditionsIf we have found a candidate x

∗ for a minimum, how do we know it

actually is one? Assume f is smooth, i.e. has all needed derivatives.

181

Optimization: Observations

Q: Come up with a hypothetical approach for finding minima.

Q: Is the Hessian symmetric?

Q: How can we practically test for positive definiteness?

182

If ex O V Y

Yes CSchwartz th m Ey t Fax

Cholesky A L E

In-Class Activity: Optimization Theory

In-class activity: Optimization Theory

183

Sensitivity and Conditioning (1D)How does optimization react to a slight perturbation of the minimum?

184

I f xx fix a tot X is the true min

fifth Ighth't Hot

loft f ext h't stol

EX Ihle FIEF8 half as many

16

digits Fff

Sensitivity and Conditioning (nD)

How does optimization react to a slight perturbation of the minimum?

185

f x't th s fix it hxfex s

direction É s HAI S the11511 1

IN E Eminentconditioning depends on Hf

Unimodality

Would like a method like bisection, but for optimization.

In general: No invariant that can be preserved.

Need extra assumption.

186

Golden Section Search

Suppose we have an interval with f unimodal:

Would like to maintain unimodality.

187

Golden Section Search: Efficiency

Where to put x1, x2?

Convergence rate?

Demo: Golden Section Proportions [cleared]

188

Newton’s Method

Reuse the Taylor approximation idea, but for optimization.

Demo: Newton’s Method in 1D [cleared]

189

fix th a fix t f ath tf ex Ich

f approx f with f at ticmin I ch to get X.tl

I ch o f Xia f Xia h

h e fifty solving f ex o with Newton

quadratic

In-Class Activity: Optimization Methods

In-class activity: Optimization Methods

190

Steepest DescentGiven a scalar function f : R

n→ R at a point x , which way is down?

Demo: Steepest Descent [cleared]191

Direction of steepest descent ofline search e.g

Golden section

1 Xo init guess

2 Sk Pf Xk3 min f Xia task4 X reel Xk t ASK

Steepest Descent: ConvergenceConsider quadratic model problem:

f (x) =1

2xTAx + c

Tx

where A is SPD. (A good model of f near a minimum.)

192

If A Xtc

en Xk Xt then

Hertilla Fitted ÉiIleLinear convergence

I