General Nonlinear Programming (NLP) Software CAS 737 / CES 735 Kristin Davies Hamid Ghaffari Alberto Olvera-Salazar Voicu Chis January 12, 2006

General Nonlinear Programming (NLP) Software

CAS 737 / CES 735

Kristin DaviesHamid Ghaffari

Alberto Olvera-Salazar Voicu Chis

January 12, 2006

Outline Intro to NLP Examination of:

IPOPT PENNON CONOPT LOQO KNITRO

Comparison of Computational Results Conclusions

Intro to NLP

The general problem:

.

,,,

,...,1,0)(

,...,1,0)(..

)(min

Condefinedfunctionsare

ghfandsetcertainaisCxwhere

Cx

mJjxg

pIixhts

xf

jinn

j

i

Either the objective function or some of the constraints may be nonlinear

(NLP)

Intro to NLP (cont’d…) Recall:

The feasible region of any LP is a convex set

if the LP has an optimal solution, there is an extreme point of the feasible set that is optimal

However: even if the feasible region of an NLP is

a convex set, the optimal solution might not be an extreme point of the feasible region

Intro to NLP (cont’d…) Some Major approaches for NLP

Interior Point Methods Use a log-barrier function

Penalty and Augmented Lagrange Methods Use the idea of penalty to transform a constrained

problem into a sequence of unconstrained problems.

Generalized reduced gradient (GRG) Use a basic Descent algorithm.

Successive quadratic programming (SQP) Solves a quadratic approximation at every iteration.

Summary of NLP Solvers

PEN N ON

A u gm en tedLag ran g ian M eth ods

KN I TR O ( T R )I PO PT, LO QO

( lin e search)

I n terio r P o in tM eth ods

CO N O PT

R ed u ced G rad ientM eth ods

NLP

IPOPT SOLVER (Interior Point OPTimizer)

Creators Andreas Wachter and L.T. Biegler at CMU

(~2002)

Aims Solver for Large-Scale Nonlinear

Optimization problems Applications

General Nonlinear optimization Process Engineering, DAE/PDE Systems, Process

Design and Operations, Nonlinear Model Predictive control, Design Under Uncertainty


Input Format Can be linked to Fortran and C code MATLAB

and AMPL. Language / OS

Fortran 77, C++ (Recent Version IPOPT 3.x) Linux/UNIX platforms and Windows

Commercial/Free Released as open source code under the

Common Public License (CPL). It is available from the COIN-OR repository


Key Claims Global Convergence by using a Line

Search. Find a KKT point Point that Minimizes Infeasibility (locally)

Exploits Exact Second Derivatives AMPL (automatic differentiation) If not Available use QN approx (BFGS) Sparsity of the KKT matrix.

IPOPT has a version to solve problems with MPEC Constraints. (IPOPT-C)


Algorithm Interior Point method with a novel line

search filter. Optimization Problem

0

0)(..

)(min

x

xcts

xfnx

0)(..

)log()()(min )(

xcts

xxfxi

il

xln

The bounds are replaced by a logarithmic Barrier term. The method solves a sequence of barrier problems for decreasing values of l


Algorithm (For a fixed value of )

Solve the Barrier Problem Search Direction (Primal-Dual IP)

Use a Newton method to solve the primal dual equations.

Hessian Approximation (BFGS update) Line Search (Filter Method) Feasibility Restoration Phase

l


Optimization Problem

0

0)(..

)(min

x

xcts

xfnx

0)(..

)log()()(min )(

xcts

xxfxi

il

xln

l

Outer LoopOuter Loop

The bounds are replaced by a logarithmic Barrier term.

The method solves a sequence of barrier problems for decreasing values of


Algorithm (For a fixed value of ) Solve the Barrier Problem

Search Direction (Primal-Dual IP) Use a Newton method to solve the

primal dual equations Hessian Approximation (BFGS update)

l


InnerLoopInnerLoop

0)(

0

0)()(

xc

eXVe

vxcxf

Optimality conditionsOptimality conditions

eXv

VariablesDual1

0)(..

)log()()(min )(

xcts

xxfxi

il

xln

Barrier Barrier NLPNLP

eeVX

xc

vxcxf

d

d

d

VX

xc

IxcH

kk

k

kkT

kk

vk

k

xk

kk

Tk

kk

)(

)()(

0

00)(

)(

),(2kkxxk xLH

At a Newton's iteration At a Newton's iteration ((xxkk,,kk,v,vkk))

)(

)()(

)(

)(

k

kkT

kk

k

xk

cT

k

kHkk

xc

vxcx

d

d

Ixc

xcIH

At a Newton's iteration At a Newton's iteration ((xxkk,,kk,v,vkk))

Algorithm Core: Solution of this Linear systemAlgorithm Core: Solution of this Linear system


Algorithm (For a fixed value of ) Line Search (Filter Method)

A trial point is accepted if improves feasibility or if improves the barrier function

Assumes Newton directions are “Good” especially when using Exact 2nd Derivatives

l

vkkkk

xkkkk

dvv

dxx

1

1

kkk

kkk

xx

or

xcxc

)(

)(If


Line Search - Feasibility Restoration Phase

When a new trial point does not provides sufficient improvement.

0..

)(min2

2

xts

xcnx

Restore FeasibilityMinimize constraint

violation

0

0)(..

min2

2

x

xcts

xx kx n

Force Unique SolutionFind closest feasible point.

Add Penalty function

The complexity of the problem increases when complementarity conditions are introduced from:

miyw

ywx

ywxc

st

ywxf

ii

ywx mmn

1,0

0,,

0),,(

.

),,(min

)()(

,,

miyw

ywxc

st

ywx

ywxf

ii

m

i

im

i

in

i

i

ywx mmn

1,0

0),,(

.

lnlnln

),,(min

)()(

1

)(

1

)(

1

)(

,,

)()( ii yw

•The interior Point method for NLPs has been extended to handle complementarity problems. (Raghunathan et al. 2003).

0)()( ii yw is relaxed as 0)(

)()()(

i

iii

s

syw



Additional IPOPT 3x. Is now programmed in C++. Is the primary NLP Solver in an undergoing

project for MINLP with IBM. References

Ipopt homepage:http://www.coin-or.org/Ipopt/ipopt-fortran.html

A. Wächter and L. T. Biegler, On the Implementation of a Primal-Dual Interior Point Filter Line Search Algorithm for Large-Scale Nonlinear Programming, Research Report, IBM T. J. Watson Research Center, Yorktown, USA, (March 2004 - accepted for publication in Mathematical Programming)

PENNON (PENalty method for NONlinear & semidefinite programming)

Creators Michal Kocvara & Michael Stingl (~2001)

Aims NLP, Semidefinite Programming (SDP),

Linear & Bilinear Matrix Inequalities (LMI & BMI), Second Order Conic Programming (SOCP)

Applications General purpose nonlinear optimization,

systems of equations, control theory, economics & finance, structural optimization, engineering

SDP (SemiDefinite Programming)

Minimization of a linear function subject to the constraint that an affine combination of symmetric matrices is positive semidefinite

m

iii

T

FxFxFwhere

xFts

xc

10)(

0)(..

min

),...,(matricessymmetric1 0 mFFm

Linear Matrix Inequality (LMI)

defines a convex constraint on x

SDP (SemiDefinite Programming)

-always an optimal point on the boundary

-boundary consists of piecewise algebraic surfaces

SOCP (Second-Order Conic Programming)

Minimization of a linear function subject to a second-order cone constraint

iTiii

T

dxcbxAts

xc

..

min

tuRtRu

t

uC k

k ,,1

Called a second-order cone constraint since the unit second-order cone of dimension k is defined as:

Which is called thequadratic, ice-cream,or Lorentz cone


Input Format MATLAB function, routine called from C

or Fortran, stand-alone program with AMPL

Language Fortran 77

Commercial/Free Variety of licenses ranging from

Academic – single user ($460 CDN) to Commercial – company ($40,500 CDN)


Key Claims 1st available code for combo NLP, LMI, &

BMI constraints Aimed at (very) large-scale problems Efficient treatment of different sparsity

patterns in problem data Robust with respect to feasibility of

initial guess Particularly efficient for large convex

problems


Algorithm Generalized version of the Augmented

Langrangian method (originally by Ben-Tal & Zibulevsky)

0/)(..

)(min

iigi pxgpts

xf

Augmented Problem

Augmented Lagrangian

gm

iiigii pxgpuxfpuxF

1

/)()(),,(

multiplierLagrange

functionpenalty

parameterpenalty0

sconstraintinequalityof#

,...,1

i

g

i

g

g

u

p

m

mi


The Algorithm Consider only inequality constraints from

(NLP)

Based on choice of a penalty function, φg, that penalizes the inequality constraints

Penalty function must satisfy multiple properties such that the original (NLP) has the same solution as the following “augmented” problem:

0

,...1,0/)(..

,)(min

i

giigi

pwith

mipxgpts

xxf

(NLPφ)

[3] Kocvara & Stingl


The Algorithm (Cont’d…) The Lagrangian of (NLPφ) can be viewed as a

(generalized) augmented Lagrangian of (NLP):

gm

iiigii pxgpuxfpuxF

1

/)()(),,(

Lagrange multiplier

Penalty parameter Penalty

function

Inequality constraint



The Algorithm STEPS

.,...,1,0.)0( 111gii mipLetgivenbeuandxLet


.

,...2,1

satisfiedis

criteriumstoppingauntilrepeatkFor

gki

ki

gki

kig

ki

ki

kkkx

k

mippiii

mipxguuii

KpuxFthatsuchxFindi

,...,1,)(

,...,1,/)(

,,)(

1

1'1

11


The Algorithm STEPS

.,...,1,0.)0( 111gii mipLetgivenbeuandxLet

Initialization Can start with an arbitrary primal variable ,

therefore, choose Calculate initial multiplier values Initial p= , typically between 10 - 10000

1iu

01 xx


0


The Algorithm STEPS

KpuxFthatsuchxFindi kkkx

k ,,)( 11

(Approximate) Unconstrained Minimization Performed either by Newton with Line Search, or

by Trust Region Stopping criteria:


11,,,,

/,,

1.0,,

1

2

1'

2

1

2

1

H

kkkxH

kkkx

ki

kig

ki

ki

kkkx

kkkx

puxFpuxF

orpxguupuxF

orpuxF

1)'(' ),,(minarg1 kk

x

k puxFx


The Algorithm STEPS

5.0,1 typically

gki

kig

ki

ki mipxguuii ,...,1,/)( 1'1

Update of Multipliers Restricted in order to satisfy:

with a positive

If left-side violated, let If right side violate, let

11

ki

ki

u

u


newiu

/1newiu


The Algorithm STEPS

gki

ki mippiii ,...,1,)( 1

Update of Penalty Parameter No update during first 3 iterations Afterwards, updated by a constant

factor dependent on initial penalty parameter

Penalty update is stopped if peps (10-6) is reached [3] Kocvara & Stingl


The Algorithm

Choice of Penalty Function Most efficient penalty function for

convex NLP is the quadratic-logarithmic function:

[4] Ben-Tal & Zibulevsky

holdpropertiesthatsocrtcctc

andrwherertctctct

i

g

6,...,1)log(

)1,1()(

654

322

21

1


The Algorithm

Overall Stopping Criteria


7

1

10

)(1

)()(

)(1

),,()(

where

xf

xfxfor

xf

puxFxfk

kk

k

kkk


Assumptions / Warnings More tuning for nonconvex problems is

still required Slower at solving linear SDP problems

since algorithm is generalized


References Kocvara, Michal & Michael Stingl. PENNON: A Code for

Convex and Semidefinite Programming. Optimization Methods and Software, 8(3):317-333, 2003.

Kocvara, Michal & Michael Stingl. PENNON-AMPL User’s Guide. www.penopt.com . August 2003.

Ben-Tal, Aharon & Michael Zibulevsky. Penalty/Barrier Multiplier Methods for Convex Programming Problems. Siam J. Optim., 7(2):347-366, 1997.

Pennon Homepage. www.penopt.com/pennon.html Available online January 2007.

Documents

General Nonlinear Programming (NLP) Software CAS 737 / CES 735 Kristin Davies Hamid Ghaffari Alberto Olvera-Salazar Voicu Chis January 12, 2006