Lecture 8 – Nonlinear Programming Models

Topics

• General formulations

• Local vs. global solutions

• Solution characteristics

• Convexity and convex programming

• Examples

• In LP ... the objective function & constraints are linear and the problems are “easy” to solve.

• Many real-world engineering and business problems have nonlinear elements and are hard to solve.

Nonlinear Optimization

Minimize f(x)

s.t. gi(x) (, , =) bi, i = 1,…,m

x = (x1,…,xn) is the n-dimensional vector of decision variables

f (x) is the objective function

gi(x) are the constraint functions

bi are fixed known constants

General NLP

Example 1 Max f (x) =3x1 + 2x2

4

s.t. x1 + x2 1, x1 0, x2 unrestricted

2

Examples 2 and 3 can be reformulated as LPs

Example 2 Max f (x) = ec1x1 ec2x

2 … ecn xn

s.t. Ax = b, x 0 n

Example 3 Min j =1

fj (xj )

s.t. Ax = b, x 0

where each fj(xj ) is of the form

Problems with“decreasing efficiencies”

fj(xj)

xj

Examples of NLPs

Max f(x1, x2) = x1x2

s.t. 4x1 + x2 8

x1 0, x20

2

8

f(x) = 2

f(x) = 1

x2

Optimal solution will lie on the line g(x) = 4x1 + x2 – 8 = 0.

x1

NLP Graphical Solution Method

• Solution is not a vertex of feasible region. For this particular problem the solution is on the boundary of the feasible region. This is not always the case.

• In a more general case, f (x1, x2) = g (x1,

x2) with 0. (In this case, = 1.)

Gradient of f (x) = f (x1, x2) (f/x1, f/x2)T

This gives f/x1 = x2, f/x2 = x1

and g/x1 = 4, g/x2 = 1

At optimality we have f (x1, x2) = g (x1, x2)

or x1* = 1 and x2

* = 4

Solution Characteristics

f(x)

x

localmin

globalmax stationary

point

localmin

localmax

Let S n be the set of feasible solutions to an NLP.

Definition: A global minimum is any x0 S such than

f (x0) f (x)

for all feasible x not equal to x0.

Nonconvex Function

At (1, 0), f(x1, x2) = 1g1(x1, x2) + 2g1(x1, x2)

or (0, 6) = 1(1, 0) + 2(0, 1), 1 0, 2 0

so 1 = 0 and 2 = 6

If g1 = x1 0 and g2 = x2 0, what is the optimum ?

Function with Unique Global Minimum at x = (1, –3)

Min { f (x)= sin(x) : 0 x 5}

Function with Multiple Maxima and Minima

Constrained Function with Unique Global Maximum and Unique Global Minimum

Convex function: If you draw a straight line between any two points on f (x) the line will be above or on f (x).

Concave function: If f (x) is convex than – f (x) is concave.

Linear functions are both convex and concave.

Convexity

d2 f (x)dx2

≥ 0 for all x

Convexity condition for univariate f :

x1 x2

f (x)

Definition of Convexity

Let x1 and x2 be two points (vectors) in S n. A function f (x) is convex if and only if

f (x1 + (1–)x2) ≤ f (x1) + (1–)f (x2)

for all 0 < < 1. It is strictly convex if the inequality sign ≤ is replaced with the sign <.

1-dimensional example

f(x)

x1 x2x1+(1−)x2

f(x1+(1−)x2)

f(x1)+(1−)f(x2).

.

..

. . .

f(x)

x

Nonconvex -- Nonconave Function

x1 x2

d2f

dx12

d2fdx1dx2

. . . d2f

dx1dxn

d2fdx2dx1

d2fdxndx1

. . . d2f

dxn2

Hessian of f at x : 2f (x) =

.

.

.

. . .

.

.

.

A positively weighted sum of convex functions is convex:

If fk(x) is convex for k =1,…,m and 1,…,m 0,

then f (x) = k fk(x) is convex.

m

k =1

Theoretical Result for Convex Functions

Used to determine convexity.

Determining Convexity

One-Dimensional Functions:

A function f (x) C 1 is convex if and only if it is

underestimated by linear extrapolation; i.e.,

f (x2) ≥ f (x1) + (df (x1)/dx)(x2 – x1) for all x1 and x2.

A function f (x) C 2 is convex if and only if its second

derivative is nonnegative.

d2f (x)/dx2 ≥ 0 for all x

If the inequality is strict (>), then f (x) is strictly convex.

x1 x2

f(x)

Multiple Dimensional Functions

Definition: The Hessian matrix H(x) associated with

f (x) is the n n symmetric matrix of second partial

derivatives of f (x) with respect to the components of x.

Example: f (x) = 3(x1)2 + 4(x2)3 – 5x1x2 + 4x1

⎥⎦

⎤⎢⎣

⎡−

−=⎟⎟

⎠

⎞⎜⎜⎝

⎛−

+−=∇

2122

21

245

56)( and

512

456)(

xxx

xxf xHx

When f (x) is quadratic, H(x) has only constant terms;

when f (x) is linear, H(x) does not exist.

f (x) is convex if only if f (x2) ≥ f (x1) + Tf (x1)(x2 – x1) for all x1 and x2.

Properties of the Hessian

•H(x) is positive definite if and only if xTHx > 0 for all x 0.

•H(x) is positive semi-definite if and only if xTHx ≥ 0 for all x and there exists and x 0 such that xTHx = 0.

•H(x) is indefinite if and only if xTHx > 0 for some x, and xTHx < 0 for some other x.

How can we use Hessian to determine whether or not f(x) is convex?

Multiple Dimensional Functions and Convexity

• f (x) is strictly convex (or just convex) if its associated Hessian matrix H(x) is positive definite (semi-definite) for all x.

• f (x) is neither convex nor concave if its associated Hessian matrix H(x) is indefinite

The terms negative definite and negative-semi- definite are also appropriate for the Hessian and provide symmetric results for concave functions. Recall that a function f (x) is concave if –f (x) is convex.

Testing for Definiteness

Definition: The ith leading principal submatrix of H is

the matrix formed taking the intersection of its first i

rows and i columns. Let Hi be the value of the

corresponding determinant:

obtained. is untilon so and , ,2221

12112111 nH

hh

hhHhH ==

Let Hessian, H =

⎥⎥⎥⎥⎥⎥⎥⎥

⎦

⎤

⎢⎢⎢⎢⎢⎢⎢⎢

⎣

⎡

nnnn

n

n

hhh

hhh

hhh

...

..

..

..

...

...

21

22221

11211

, where hij = 2f

(x)/xixj

Rules for Definiteness

• H is positive definite if and only if the determinants of all the leading principal submatrices are positive; i.e., Hi > 0 for i = 1,…,n.

• H is negative definite if and only if H1 < 0 and the remaining leading principal determinants alternate in sign:

H2 > 0, H3 < 0, H4 > 0, . . .

Positive-semidefinite and negative semi-definiteness require that all principal submatrices satisfy the above conditions for the particular case.

Quadratic Functions

Example 1: f (x) = 3x1x2 + x12 +

3x22

⎥⎦

⎤⎢⎣

⎡=⎟⎟

⎠

⎞⎜⎜⎝

⎛++

=∇63

32)( and

63

23)(

21

12 xHxxx

xxf

so H1 = 2 and H2 = 12 – 9 = 3

Conclusion f (x) is convex

because H(x) is positive definite.

Quadratic Functions (cont’d)

Example 2: f (x) = 24x1x2 + 9x12 +

16x22

⎥⎦

⎤⎢⎣

⎡=⎟⎟

⎠

⎞⎜⎜⎝

⎛++

=∇3224

2418)( and

3224

1824)(

21

12 xHxxx

xxf

so H1 = 18 and H2 = 576 – 576 = 0

• Thus H is positive semi-definite (determinants of

all submatrices are nonnegative) so f (x) is

convex.• Note, xTHx = 2(3x1 + 4x2)2 ≥ 0. For x1 = 4, x2 =

3, we get xTHx = 0.

Nonquadratic Functions

Example 3: f (x) = (x2 – x12)2 + (1 – x1)2

⎥⎦

⎤⎢⎣

⎡

−−++−

=24

42124)(

1

1212

x

xxxxH

Thus the Hessian depends on the point under consideration:

At x = (1, 1), which is positive definite.

At x = (0, 1), which is indefinite.

Thus f(x) is not convex although it is strictly convex near (1, 1).

⎥⎦

⎤⎢⎣

⎡−

−=

24

410)1,1(H

2 0(0 1)

0 2,

−⎡ ⎤=⎢ ⎥

⎣ ⎦H

What You Should Know About Nonlinear Programming

• How to develop models with nonlinear functions.

• The definition of convexity.

• Rules for positive and negative definiteness

• How to identify a convex function.

• The difference between a local and global solution.