An Introduction to Numerical Optimizationdesi.iteso.mx/.../CIR_OPTIMIZATION/optimization_1.pdf · Introduction to Numerical Optimization(Part 1) Dr. José Ernesto Rayas-Sánchez March

Introduction to Numerical Optimization (Part 1)Dr. José Ernesto Rayas-Sánchez

March 28, 2008

1

1

An Introduction to Numerical Optimization

(Part 1)

Dr. José Ernesto Rayas Sánchez

2Dr. J. E. Rayas Sánchez

Outline

Mathematical formulations

Types of optimization problems

Minimizers

Recognizing a minimizer

Types of optimization methods

Line search methods

Trust region methods


March 28, 2008

2


Unconstrained Optimization Problem

Given u: ℜn→ℜ, find

nuu ℜ∈≤ xxx allfor )()( *

)(minarg* xx x u=

x* is a global minimizer ⇔

x* is a local minimizer ⇔

Replacing ≤ by < we have a strict global minimizer and a strict local minimizer, respectively

ℵ∈≤ xxx allfor )()( * uu:

2

* ε<−ℜ∈=ℵ xxx n

where u is the objective function (or cost function), and xcontains the optimization variables


Example 1 – Quadratic

22

21 )12()1()( −+−= xxu x ⎥⎦

⎤⎢⎣

⎡=

2

1

xx

xwhere

-2-1

01

23

-2

0

2

40

10

20

30

40

x1

Quadratic Function

x2

-2 -1 0 1 2 3-2

-1

0

1

2

3

x1

x 2

Quadratic Function


March 28, 2008

3


Example 2 – Rosenbrock

-2 -1.5 -1 -0.5 0 0.5 1 1.5 2-2

-1.5

-1

-0.5

0

0.5

1

1.5

2

x1

x 2

Rosenbrock Function

21

2212 )1()(100)( xxxu −+−=x ⎥⎦

⎤⎢⎣

⎡=

2

1

xx

xwhere

-2-1

01

2

-2-1

0

120

1000

2000

3000

4000

x1

Rosenbrock Function

x2


Example 3 – Sinc

rru )sin()( =x ⎥⎦

⎤⎢⎣

⎡=

2

1

xx

xwhere

-10-5

05

10

-10-5

0

510

-0.5

0

0.5

1

x1

Sinc Function

x2

-10 -5 0 5 10-10

-5

0

5

10

x1

x 2

Sinc Function

22

21 xxr +=


March 28, 2008

4


Example 4 – Styblinski and Tang

)5165160.5()( 222

421

21

41 xxxxxxu +−++−=x ⎥⎦

⎤⎢⎣

⎡=

2

1

xx

xwhere

-4 -3 -2 -1 0 1 2 3 4-4

-3

-2

-1

0

1

2

3

4

x1x 2

Styblinski and Tang (1990) Function

-4-2

02

4

-4-2

0

24

-80

-60

-40

-20

0

20

x1

Styblinski and Tang (1990) Function

x2


Example 5 – Venkataraman

dcqxdpxcbxaxu ++−−+= )cos()cos()( 2122

21x

⎥⎦

⎤⎢⎣

⎡=

2

1

xx

xwhere ππ 4,3,4.0,3.0,2,1 ====== qpdcba

-1-0.5

00.5

1

-1-0.5

0

0.510

2

4

6

8

10

x1

Venkataraman (2002) Function

x2

-1 -0.5 0 0.5 1-1

-0.5

0

0.5

1

x1

x 2

Venkataraman (2002) Function


March 28, 2008

5


Constrained Optimization Problem


Ω∈≤ xxx ),()( * uu

)(minarg* xxx

uΩ∈

=

x* is a global minimizer ⇔

x* is a local minimizer ⇔

u is the objective function (or cost function), x contains the optimization variables, Ω is the feasible set or region

ℵ∈≤ xxx allfor )()( * uu:

2

* εΩ <−∈=ℵ xxx

if u(x) is convex, a local minimizer is also the global minimizer


Constrained Optimization Problem (cont)


⎩⎨⎧

∈≥∈=

=Ij

Ei

jcic

uΩΩ

0)(0)(

subject to )(minarg*

xx

xx x

ci are equality constrains, cj are inequality constrains

If u(x) and all the constraints are linear functions of x, the problem is a linear programming problem

If the objective function or at least one of the constraints is nonlinear, then the problem is a nonlinear programming problem


March 28, 2008

6


Recognizing a Local Minimizer

xs is a stationary point if ∇u(xs) = 0

First order necessary condition: If x* is a local minimizer then ∇u(x*) = 0

Second order sufficient conditions: If ∇u(x*) = 0 and H(u(x*)) is positive definite then x* is a strict local minimizer

If a point is a stationary point and not a local minimizer or maximizer, the point is called a “saddle point”


Optimization Methods: A Broad Classification

Non-descent methods

Descent methods, u(xi+1) < u(xi) for every i, or u(xi+1) < u(xi) for i > N, where N is a number of initial steps

Two fundamental strategies:

• Line search methods

• Trust region methods


March 28, 2008

7


Line Search Methods

At the i-th iteration, the algorithm chooses a direction diand searches along this direction from the current iterate xi for a new iterate xi+1 with a lower function value

The search direction and the step size can be selected in several manners


Trust Region Methods

At the i-th iteration, a model mi of the objective function u is created. The algorithm restricts the search for a minimizer of mi to some region around xi. If the minimizer of mi does not produce a sufficient decrease in u, the trust region is shrunk and the model is again minimized

There are several ways to construct the model mi, and to define the trust region (ball, elliptical, box-shaped, etc.)


March 28, 2008

8


A Generic Line Search Algorithm

begini = 0, xi = x0repeat until stopping_criterion

di = SearchDirection(u, xi)αi = LineSearch(u, xi, di)xi+1 = xi + αιdii = i + 1

end


Stopping Criteria

A maximum number of iterations has been reached

The objective function is practically not decreasing

The absolute change in the optimization variables is small enough

11)()( ε<− +ii uu xx

221 |||| ε<−+ ii xx

maxii >


March 28, 2008

9


Stopping Criteria (cont)

The relative change in the optimization variables is small enough

The gradient is small enough

The Hessian is positive definite (near the solution, xi ≈ x*)

)||(|||||| 42321 εε +<−+ iii xxx

52||)(|| ε<∇ iu x

611 )))((()(21 ε<−− ++ iii

Tii u xxxHxx


Search Directions for Line Search Methods

Steepest descent direction)( ii u xd −∇=


March 28, 2008

10


Search Directions for Line Search Meth. (cont)

The steepest descent direction with exact line searches moves in orthogonal steps



The steepest descent direction can be extremely slow (zigzagging)


March 28, 2008

11



Newton direction

in this case the most used step length is 1

di is a linear transformation of −∇u(xi)

Quasi-Newton directions

Bi is an approximation of H(∇u(xi)) which is updated after each iteration to take into account the additional knowledge gained during the step

)())(( 1iii uu xxHd ∇−= −

)(1iii u xBd ∇−= −



The two most popular updating formulas for Bi in Quasi-Newton directions:

SR1 formula (Symmetric-Rank-One)

BFGS formula (Broyden, Fletcher, Goldfarb and Shanno)

where

iT

iii

Tiiiiii

ii ssBysBysByBB

)())((

1 −−−+=+

iTi

Tii

iiTi

iTiii

ii syyy

sBsBssBBB +−=+1

iii xxs −= +1 )()( 1 iii uu xxy ∇−∇= +


March 28, 2008

12



Conjugate Gradient direction

where βi is a scalar that ensures that di and di−1 are conjugate

Any two vectors a and b are conjugate with respect to a symmetric positive definite matrix A if aTAb = 0

1)( −+−∇= iiii u dxd β


Finding the Step Size in Line Search Methods

Exact line search

The exact line search stops at a point where the local gradient is orthogonal to the search direction

Soft line search (See: P.E. Frandsen, K. Jonasson, H.B. Nielsen and O. Tingleff, Unconstrained Optimization. Lyngby, Denmark: Department of Mathematical Modeling, Technical University of Denmark, 1999, pp. 26-30)

)(minarg)(minarg00

ααααα

vu iii >>=+= dx


March 28, 2008

13


A Generic Trust Region Algorithm

begini = 0, xi = x0, r = r0 > 0repeat until stopping_criterion

mi(s) = BuildModel(u, xi)

if a > 0.75 then r = 2rif a < 0.25 then r = r/3if a > 0 then xi+1 = xi + sii = i + 1

end

)]()(/[)]()([ iiiiii mmuua ssxx −+−= 0

rmii ≤==∈

sssss

: where)(minarg ΩΩ

Documents

An Introduction to Numerical Optimizationdesi.iteso.mx/.../CIR_OPTIMIZATION/optimization_1.pdf · Introduction to Numerical Optimization(Part 1) Dr. José Ernesto Rayas-Sánchez March