04 Lagrange Multipliers SV

04 – METHOD OF LAGRANGE MULTIPLIERS

CIVL 4440, Fall 2015

Introduction

* Method of Lagrange multipliers is based on calculus.

* It uses calculus to identify stationary points of an objective function.

* These points are then compared to determine which of them is the global optimum.

* The method was originally developed for constrained problems with equality constraints.

* If the problem contains inequality constraints, they will have to be converted to equality constraints first.

Stationary Points

maximum

minimum

inflection

saddle point

http

://w

ww

.mat

h.ok

stat

e.ed

u/~

yqw

ang/

teac

hing

/mat

h455

3_sp

ring

09/d

emo/

min

max

1.pn

g

Problem with Equality Constraints

Max Z = F(X 1, X 2, X 3, …, X n)

s.t.

g1(X 1, X 2, X 3, …, X n) = b 1

g2(X 1, X 2, X 3, …, X n) = b 2

g3(X 1, X 2, X 3, …, X n) = b 3

:

:

gm(X 1, X 2, X 3, …, X n) = b m

To apply the method of Lagrange multipliers, the objective function, F and constraints, g1 to gm should be continuous and differentiable.

The problem should also be bounded, i.e., the optimal value of Z should not be infinity or negative infinity.

By definition, b1 to bm are constants.

Why are these conditions

necessary?

The LagrangianThe first step is to convert the problem from a constrained one to an unconstrained one.

This is done by multiplying each constraint with a Lagrange multiplier, λi and subtracting the result of that from the objective function.

The modified objective function is called the Lagrangian. For a maximization problem:

L = F – λ1(g1 – b1) – λ2(g2 – b2) – … – λm(gm – bm)

L = F – ∑ λ����� (gi – bi) The multiplier for constraint i, λi can be

thought of as the unit penalty incurred when that constraint is violated.

Identifying Stationary Points…Stationary points of the Lagrangian, L can now be determined by setting the following n + mpartial derivatives to zero.

dL/dX 1 = 0

dL/dX 2 = 0

:

:

dL/dX n = 0

dL/d λ1 = 0

dL/d λ2 = 0

:

:

dL/d λm = 0

X1, X 2, …, X n are decision variables. λ1, λ2, …, λn are new decisions variables that are introduced with the construction of the Lagrangian.

Identifying Stationary Points…Solving the partial derivatives as a series of simultaneous equations gives a set of stationary points.

Each stationary point refers to a unique combination X1, X2, …, X n and λ1, λ2, …, λn values.

One or more of the stationary points may be infeasible, i.e., they are in violation of at least one constraint.

Once determined, those that are feasible are compared and the one giving the maximum objective function value is the global optimum.

Identifying Stationary Points…

For minimization problems, the procedure is the same except that the Lagrangian, L is defined as:

L = F + λ1(g1 – b1) + λ2(g2 – b2) + … + λm(gm – bm)

L = F + ∑ λ����� (gi – bi)

And the global optimum is the stationary point that gives the least (instead of the maximum) objective function value.

Does it really matter whether we define L with negatives

or positives in front of the λi

?

It is now a “+” instead of a “–”.

Example of a Problem with Equality Constraints

MAX Z = X3 – 4X + Y2 + 5

s.t.

X + Y = 4

X = 0

To apply the method of Lagrange multipliers, modify the objective function to the following to convert the problem from a constrained problem to an unconstrained problem.

Consider the following constrained optimization problem:

L = X3 – 4X + Y2 + 5 – λ1 (X + Y – 4) – λ2 X

Example of a Problem with Equality Constraints

To find the stationary points of L:

dL/dX = 3X2 – 4 – λ1 - λ2 = 0 … (1)

dL/dY = 2Y – λ1 = 0 … (2)

dL/dλ1 = –X – Y + 4 = 0 … (3)

dL/dλ2 = X = 0 … (4)

Substituting (4) into (3) gives:

X = 0, Y = 4

Substituting that into (2) gives:

λ1 = 8

And that with X and Y into (1) gives:

λ2 = -12

Because of the equality constraints, it so happens that this particular problem has just one stationary point. Therefore, there is no need for a comparison of stationary points to be done, and the stationary point identified is the optimum.

Example of a Problem with Equality Constraints

Example of a Problem with Equality Constraints

Principle 1:

When the Lagrange multiplier associated with a constraint is non-zero, it means the constraint is binding. And when the multiplier is zero, it means the constraint is non-binding. Both λ1 and λ2 are non-zero. Therefore, both constraints are binding.

Principle 2:

The Lagrange multiplier represents the improvement in the objective function, Z with respect to a change in the RHS of the corresponding constraint.

λ1 = 8 means that if the RHS of the first constraint were to be increased by 1 unit, the objective function value at optimum will increase (since this is a maximization problem) by 8 units.

λ2 = -12 means that if the RHS of the second constraint were to be increased by a 1 unit, the objective function value at optimum will decrease by 12 units.

Interpretation of Lagrange multipliers, λλλλ1 = 8 and λλλλ2 = -12

For a maximization problem, an improvement means an increase, and for a minimization problem, a decrease.

Example of a Problem with Equality Constraints

Let us test out Principle 2 by solving and resolving the problem for different values of the RHS of the constraints. To save time, we will use Solver in Excel.

Interpretation of Lagrange multipliers, λλλλ1 = 8 and λλλλ2 = -12

RHS of Constraint Change in RHS Objective, Z Change in Z ∆∆∆∆ Z / ∆∆∆∆ RHS

Constraint 1, λ1 = 8

RHS of Constraint Change in RHS Objective, Z Change in Z ∆∆∆∆ Z / ∆∆∆∆ RHS

Constraint 2, λ2 = -12

Does it make sense now why it is important how the signs in front of the Lagrangian multipliers in the Lagrangian, L are defined depending on whether the problem is a maximization or minimization problem?

Example of a Problem with Equality Constraints

Example of a Problem with Inequality Constraints

Consider the following constrained optimization problem:

MIN Z = X 3 – 4X + Y 2 + 5s.t.

X + Y = 4X >= 0Y <= 10

To apply the method of Lagrange multipliers, the constraints are first converted to equality constraints. To do that, slack and surplus variables are added at appropriate places.

X + Y = 4X >= 0 ---> X – S 1

2 = 0Y <= 10 ---> Y + S 2

2 = 10

The objective function can now be modified to give Lagrangian:

L = X 3 – 4X + Y 2 + 5 + λ1(X + Y – 4) + λ2(X – S 12) + λ3(Y + S 2

2 – 10)

Example of a Problem with Inequality Constraints

To find the stationary points of the Lagrangian:

dL / dX = 3X2 – 4 + λ1 + λ2 = 0 … (5)

dL / dY = 2Y + λ1 + λ3 = 0 … (6)

dL / dλ1 = X + Y – 4 = 0 … (7)

dL / dλ2 = X – S12 = 0 … (8)

dL / dλ3 = Y + S22 – 10 = 0 … (9)

dL / dS1 = –2 λ2 S1 = 0 ---> λ2 = 0 or S1 = 0

dL / dS2 = –2 λ3 S2 = 0 ---> λ3 = 0 or S2 = 0

Eqn (6): λ1 = –2Y

Eqn (5): 3X2 – 4 – 2Y = 0 .… (10)

Eqn (10) + 2 × Eqn (7):

---> 3X2 + 2X – 12 = 0---> X = 1.694 or X = –2.361

This gives the first stationary point:

---> X = 1.694, Y = 2.306---> λ1 = –4.612, λ2 = 0, λ3 = 0---> S1

2 = 1.694, S22 = 7.694

---> Z = 8.403

Example of a Problem with Inequality Constraints

Possible solution 1: λ2 = 0 and λ3 = 0

Eqn (9): Y = 10Eqn (7): X = –6Eqn (8): S1

2 = –6

This gives the second stationary point, but this point is infeasible since S1

2 is negative.

Example of a Problem with Inequality Constraints

Possible solution 2: λ2 = 0 and S2 = 0

Eqn (8): X = 0

Eqn (7): Y = 4

This gives the third stationary point:---> X = 0, Y = 4---> λ1 = –8, λ2 = 12, λ3 = 0---> S1

2 = 0, S22 = 6

---> Z = 21

Example of a Problem with Inequality Constraints

Possible solution 3: S1 = 0 and λ3 = 0

Eqn (8): X = 0

Eqn (9): Y = 10

This gives the fourth stationary point, but this point is infeasible since it violates Eqn (7).

Example of a Problem with Inequality Constraints

Possible solution 4: S1 = 0 and S2 = 0

By comparing the four stationary points, which is the optimum? Which constraints are binding or non-binding? Graphically? From the Lagrange multipliers? From the slack and surplus variables?

There are four stationary points. Before going further, let us have a graphical look at where those stationary points lie. What pattern do you observe?

Example of a Problem with Inequality Constraints

-6 -4 -2 0 2 4 6 0

2

4

6

8

10

X

X >= 0

Y <= 1 0

Z = 15

Z = 8.4

Z = 20

Y

2 4

3

1

Example of a Problem with Inequality Constraints

Interpretation of Lagrange multipliers, λ1 = –4.612, λ2 = 0, λ3 = 0

Constraint 1, λ1 = –4.612

Constraint 2, λ2 = 0

Constraint 3, λ3 = 0

As before, let us test the Lagrange multipliers by solving the problem multiple times for different values of the RHS of the constraints.

RHS of Constraint Change in RHS Objective, Z Change in Z ∆∆∆∆ Z / ∆∆∆∆ RHS

In-Class ExerciseThere are two polluters, the first discharging R 1 kg of waste into the atmosphere everyday and the second R 2 kg of waste. The profits of the polluters increase with R 1

and R 2:

Profit of polluter 1 (million $) = 10 R 12

Profit of polluter 2 (million $) = 5 R 22

Use the method of Lagrange multipliers to find the values of R 1 and R 2 that give the maximum total profit. Note that regulations restrict the sum of R 1 and R 2 to be no greater than 5 kg.

Based on your results, estimate the increase in profit you would expect if the restriction on the sum of R 1 and R2 were to be made less strict and increased by 0.1 kg.