Introduction to Optimization - Indian Institute of Technology … · Ant colony Newspaper hawker Forensic artist Optimization in real life. R.K. Bhattacharjya/CE/IITG Books •K

R.K. Bhattacharjya/CE/IITG

Engineering Optimization

Rajib Kumar BhattacharjyaProfessor

Department of Civil EngineeringIIT Guwahati

Email: [email protected]

24 April 2015 1


Are you using optimization?

The word optimization may be very familiar or may be quite new to you.

. but whether you know about optimization or not, you are usingoptimization in many occasions of your day to day life .

.Examples

4/24/2015 2

R.K. Bhattacharjya/CE/IITG4/24/2015 3

Cooking

Ant colony

Newspaper

hawker

Forensic

artist

Optimization in real life


Books

K. Deb., Optimization for Engineering Design: Algorithms and Examples, PHI Pvt Ltd., 1998.

J. S. Arora, Introduction to Optimum Design, McGraw Hill International Edition, 1989.

S.S. Rao, Engineering optimization: Theory and Practice, New age international (P) Ltd. 2001

D. E. Goldberg, Genetic Algorithms in search and optimization, Pearson publication, 1990.

K. Deb, Multi-Objective Optimization Using Evolutionary Algorithms, Chichester, UK : Wiley

4/24/2015 4


Example

A farmer has 2400 m of fencing and wants to fence off a rectangular field that borders

a straight river. He needs no fence along the river. What are the dimensions of the

field that has the largest area?

100100

2200

1000

700700

400

1600

400

x

yy

RIVER

Maximize =

Subject to , = + 2 = 2400


A manufacturer needs to make a cylindrical can that will hold 1.5 liters of liquid.

Determine the dimensions of the can that will minimize the amount of material used in

its construction.

Constraint: r2h = 1500

Minimize: A = 2r2 + 2rh

Example

Dimension is in cm


Objectives

Topology: Optimal connectivity of the

structure

Minimum cost of material: optimal cross

section of all the members

We will consider the second objective only

The design variables are the cross sectional area

of the members, i.e. A1 to A7

Using symmetry of the structure

A7=A1, A6=A2, A5=A3

You have only four design variables, i.e., A1 to A4

Example


Optimization formulation

Objective

One essential constraint is non-negativity of design variables, i.e.

A1, A2, A3, A4 >= 0

Is it complete now?

What are the constraints?


First set of constraints Another constraint may be the

minimization of deflection at CAnother constraint is buckling

of compression members



What is Optimization?

Optimization is the act of obtaining the best result under a givencircumstances.

Optimization is the mathematical discipline which is concerned withfinding the maxima and minima of functions, possibly subject toconstraints.

4/24/2015 11

Introduction to optimization

4/24/2015 12

0

5

10

15

20

25

30

0 2 4 6 8 10

Minimum = 5 2 Equation of the line

How to find out the minimum of the function

= 2 5 = 0

= 5 Optimal solution

0

5

10

15

20

25

30

-5 -3 -1 1 3 5

Maximum

= 25 + 2 Equation of the line

= 2 = 0

= 0 Optimal solution

4/24/2015 13


, = 2 + 2 + 4

Equation of the surface

In this case, we can obtain the optimal

solution by taking derivatives with respect

to variable and and equating them to zero

= 2 = 0 = 0

= 2 = 0 = 0Optimal solution is 0,0

Single variable optimization

Objective function is defined as

Minimization/Maximization f(x)

4/24/2015 14


Single variable optimization

Stationary points

For a continuous and differentiable function f(x), a stationary point x* is apoint at which the slope of the function is zero, i.e. f (x) = 0 at x = x*,

4/24/2015 15

Minima Maxima Inflection point


Global minimum and maximum

4/24/2015 16

A function is said to have a global or absolute

maximum at x = x* if f (x* ) f(x) for all x in the

domain over which f(x) is defined.

A function is said to have a global or absolute

minimum at x = x* if f (x* ) f(x) for all x in the

domain over which f(x) is defined.

4/24/2015 17

Global optimaLocal optima

Local optima

Local optima

Local optima

f

X



Necessary and sufficient conditions for optimality

4/24/2015 18

Necessary condition

If a function () is defined in the interval and has a relative minimum at = , Where and if / exists as a finite number at = , then / = 0

Proof

/ = lim0

+ ()

Since is a relative minimum () +

For all values of sufficiently close to zero, hence

+ ()

0

+ ()

0

if 0

if 0


Thus

/ 0 If tends to zero through +ve value

/ 0 If tends to zero through -ve value

Thus only way to satisfy both the conditions is to have

/ = Note:

This theorem can be proved if is a relative maximum Derivative must exist at

The theorem does not say what happens if a minimum or maximum occurs at an end point of

the interval of the function

It may be an inflection point also.

Necessary and sufficient conditions for optimality


Sufficient conditions for optimality

Sufficient condition

Suppose at point x* , the first derivative is zero and first nonzero higher derivative is denoted by n, then

1. If n is odd, x* is an inflection point

2. If n is even, x* is a local optimum1. If the derivative is positive, x* is a local minimum

2. If the derivative is negative, x* is a local maximum

4/24/2015 20



Proof Apply Taylors series

+ = + +2

2! ++

1

1 !1 +

!

Since = = = 1 = 0

+ =

!

When is even

! 0

Thus if is positive + is positive Hence it is local minimum

Thus if negative + is negative Hence it is local maximum

When is odd

!changes sign with the change in the sign of h.

Hence it is an inflection point


= 3 10 22 10


Take an example

= 32 10 4 = 0

Solving for = 2.61 1.28 These two points are stationary points

Apply necessary condition

Apply sufficient condition = 6 4

2.61 = 11.66 positive and n is odd

= 2.61 is a minimum point

1.28 = 11.68 negative and n is odd

= 1.28 is a maximum point


Multivariable optimization without constraints

Minimize () Where =

12

Necessary condition for optimality

If () has an extreme point (maximum or minimum) at = and if the first partial Derivatives of () exists at , then

1=

2= =

= 0


Sufficient condition for optimality

The sufficient condition for a stationary point to be an extreme point is that the matrix of second partial derivatives of () evaluated at is

(1) positive definite when is a relative minimum(2) negative definite when is a relative maximum(3) neither positive nor negative definite when is neither a minimum nor a maximum

Proof Taylor series of two variable function

+ , + = , +

+

+1

2!2

2

2+ 2

2

+ 2

2

2+

+ , + = , +

+1

2!

2

22

2

2

2

+



+ = + +1

2! +

Since is a stationary point, the necessary condition gives that = 0

Thus

+ =1

2! +

Now, will be a minima, if is positive

will be a maxima, if is negative

will be positive if H is a positive definite matrix

will be negative if H is a negative definite matrix

A matrix H will be positive definite if all the eigenvalues are positive, i.e. all the values are positive which satisfies the following equation

= 0



a b

This is the narrow region

where optima exists

Line search techniques


0

5

10

15

20

25

30

-5 -3 -1 1 3 5

-30

-25

-20

-15

-10

-5

0

-5 -3 -1 1 3 5

Unimodal and duality principle

Minimization = Maximization

Optimal solution = 0

Optimal solution = 0


a bx1 x2

Quiz

For the unimodal function

Optima is not

a. Between , 1b. Between 1, 2c. Between 2, d. Between ,

Ans: a


a bx1 x2

Quiz


Optima is not


Ans: c


a bX1 X2

Quiz


Optima is not


Ans: a, c


a bX1 X2Xm

Interval halving method


a bx1 X2Xm



a bx1 X2Xm

In this case



a bx1 X2

Xm

CONTINUE

In this case



Golden ratio

4/24/2015 36

Golden ratio 0.618=1/1.618


a bX

1

Golden Section Search Method

X

2

Apply region

elimination rules

Suppose

L

1

1

1 > 2


a bX

1

X

2

Apply region

elimination rules

Suppose

1

1

L

X

2



a bX

1

X

2

Continue

iteration

1



a bcd

a cd

e

= + (1)

= (2)

If < ()

= + (3)

Putting (1) in (3), we have

= + +

= + 2 (4)

Equating (4) and (2), we have

= + 2

2 + 1 = 0 Solving =0.618, -1.618 0.618 is the golden



Newton-Raphson method

+1

= +1

+1

+1 =

Rearranging and putting

+1 =0

+2

Continue iteration


F

+1

+1 =

+2

F

Continue

iteration

Incase optimization problem, = 0

Considering =

+1 =

Newton-Raphson method


1. If is an unimodal convex function in the interval , , then is a) Positive

b) Negative

c) It may be negative or may be positive

d) None of the above

2. For the same function, take any point between , . If is less than 0, then minima is not within the range

a) , b) , c) , d) None of the above

2. For the same function, take any point between , . If is greater than 0, then minima not within the range

a) , b) , c) , d) None of the above

QUIZ

Ans. b

Ans. a

Ans. b


Take a point

Bisection method

= +

2

< 0 then area between , will be eliminated

> 0 then area between , will be eliminated

Disadvantage

Magnitude of the derivatives

is not considered


1 2

Considering similar triangle

2

1

1

2

22

= 2

12 1

= 2 2

2 1

2 1

In this case > 0

then area between , 2 will be eliminated

Apply region elimination technique

Bisection method


Multivariable problem

OPTIMAL POINT


A multivariable problem can be converted to a single variable problem using the

following equation


+1 = +

Take an example , = 2 2 + 4

0 =11

=10

1 = 0 +

1 =11

+ 10

=1 + 1

Putting in equation (1)

= 1 + 2 12 + 4

Taking first derivative

= 2 2 = 0

= 1

2 = 1 +

2 =01

+ 01

=0

1 +

Putting in equation (1)

= 0 + 1 + 2 + 4

Taking first derivative

= 2 2 = 0

= 1

1 =1 + 1

=01

2 =0

1 + =

00

OPTIMAL

SOLUTION


X

Y

0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 50

0.5

1

1.5

2

2.5

3

3.5

4

4.5

5

XY

0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 50

0.5

1

1.5

2

2.5

3

3.5

4

4.5

5



X

Y

0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 50

0.5

1

1.5

2

2.5

3

3.5

4

4.5

5

A search direction is a descent direction at point if the condition . < 0 is satisfied in the vicinity of the point .

+1 = +

= + .

The +1 <

When . < 0Or, . < 0


Descent direction

Steepest descent direction


Newtons method for multi-variable problem

+ = + +1

2! +Taylor series

+1 = + +1 +

1

2!+1

+1 +

By setting partial derivative of the equation to zero for minimization of , we have

= 0 + + +1 = 0

+ =

Since higher order derivative terms have been neglected, the above equation can be

iteratively used to find the value of the optimal solution

= +

In this

case

= 2

=

=

=

Transformation methodRajib Bhattacharjya

Department of Civil Engineering

IIT Guwahati


-15

-10

-5

0

5

10

15

0 1 2 3 4 5

f(x)

x

= 3 10 22 + 10

Minimize

Subject to = 3

Or, = 3 0

Infeasible

regionFeasible

region

The problem can be written

as

F , = + 2

Where, = 0 if 3

= otherwise

The bracket operator

can be implemented using

, 0 function


= 3 10 22 + 10

Minimize

Subject to = 3

Or, = 3 0

Infeasible

regionFeasible

region

The problem can be written

as

F , = 3 10 22 + 10 + 3 2

F , = 3 10 22 + 10 + 3,02

-15

-10

-5

0

5

10

15

0 1 2 3 4 5

f(x)

x


F , = 3 10 22 + 10 + 3,02

Minimize

By changing R value, it is

possible to avoid the

infeasible solution

The minimization of the

transformed function will

provide the optimal

solution which is in the

feasible region only


= 3 10 22 + 10

Minimize

Subject to = 3

Or, = 3 0

Infeasible

regionFeasible

region

The problem can also be

converted as

F , = 3 10 22 + 10 + 1

F , = 3 10 22 + 10 + 1

3

-15

-10

-5

0

5

10

15

0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5

f(x)

x

This term is

added in

feasible side

only


F , = 3 10 22 + 10 + 1

3Minimize

By changing R value, it is

possible to avoid the

infeasible solution

The minimization of the

transformed function will

provide the optimal

solution which is in the

feasible region only


Exterior penalty method Interior penalty method


F , = + ,

The transformation function can be written

as

This term is called Penalty

term

is called penalty parameter


Penalty terms

Parabolic penalty

=

0

5

10

15

20

25

-5 0 5

h(x)


Penalty terms

Log penalty

=

-2

-1

0

1

2

3

-1 1 3 5f(

x)

g(x)


Penalty terms

Inverse penalty

= 1

-10

-5

0

5

10

-5 0 5

f(x)

g(x)


Penalty terms

Bracket operator

=

0

5

10

15

20

25

-5 0 5

f(x)

g(x)


Take an example

Minimize = 1 42 + 2 4

2

Subject to = 1 + 2 5

The transform function can be written as

Minimize F = 1 42 + 2 4

2 + 1 + 2 52


X

Y

0 1 2 3 4 5 6 7 80

1

2

3

4

5

6

7

8

02

46

8

0

2

4

6

8

0

10

20

30

40

XY

X

Y

0 1 2 3 4 5 6 7 80

1

2

3

4

5

6

7

8

Minimize = 1 42 + 2 4

2

Subject to = 1 + 2 5


Minimize F = 1 42 + 2 4

2 + 1 + 2 52

= 0.5

Optimal solution is

3.250 3.250

X

Y

0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 50

0.5

1

1.5

2

2.5

3

3.5

4

4.5

5

X

Y

0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 50

0.5

1

1.5

2

2.5

3

3.5

4

4.5

5

= 1

Optimal solution is

3.000 3.000


R x1 x2 f(x) h(x) F

0 4.000 4.000 0.000 3.000 0.000

0.5 3.250 3.250 1.125 1.500 2.250

1 3.000 3.000 2.000 1.000 3.000

5 2.636 2.636 3.719 0.273 4.091

10 2.571 2.571 4.082 0.143 4.286

20 2.537 2.537 4.283 0.073 4.390

30 2.525 2.525 4.354 0.049 4.426

50 2.515 2.515 4.411 0.030 4.455

100 2.507 2.507 4.455 0.015 4.478

200 2.504 2.504 4.478 0.007 4.489

500 2.501 2.501 4.491 0.003 4.496

1000 2.501 2.501 4.496 0.001 4.498

10000 2.500 2.500 4.500 0.000 4.500


X

Y

0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 50

0.5

1

1.5

2

2.5

3

3.5

4

4.5

5

= 1000

2.501 2.501Optimal solution is

0

1

2

3

4

5

0

1

2

3

4

5

0

1

2

3

x 104

XY

Quadratic approximation

Rajib Kumar BhattacharjyaDepartment of Civil Engineering

IIT Guwahati


-1 -0.5 0 0.5 1 1.5 2 2.5-10

0

10

20

30

40

50

60

70

X

f(X

)

= 24 3 + 52 12 + 1

/ = 83 32 + 10 12 = 0

Solving for

= 0.8831 and = 5.1702


-1 -0.5 0 0.5 1 1.5 2 2.5-10

0

10

20

30

40

50

60

70

X

f(X

)

= 24 3 + 52 12 + 1

Solution is

= 1.2 and = 3.7808and / = 9.5040

Quadratic approximation of the

function at can be written as

= + / +0.5*

// 2

Now we can minimize the

functionMinimize / +0.5*

// 2

Approximate function for = 0

This is the solution of the approximate

function: First trial


-1 -0.5 0 0.5 1 1.5 2 2.5-10

0

10

20

30

40

50

60

70

X

f(X

)

= 24 3 + 52 12 + 1

Solution is

= 0.9456, = 5.1229 and / = 1.5377



= + / +0.5*

// 2



// 2

Approximate function for = 1.2


function: Second trial


-1 -0.5 0 0.5 1 1.5 2 2.5-10

0

10

20

30

40

50

60

70

X

f(X

)

= 24 3 + 52 12 + 1

Solution is

= 0.8864 and = 5.1701and / = 0.0785



= + / +0.5*

// 2



// 2



function: Third trial


-1 -0.5 0 0.5 1 1.5 2 2.5-10

0

10

20

30

40

50

60

70

X

f(X

)

= 24 3 + 52 12 + 1

Solution is

= 0.8831 and = 5.1702and / = 0.00099



= + / +0.5*

// 2



// 2



function: Fourth trial Gradient is negligible

STOP ITERATION


Minimize 1, 2 = 12 + 2 11

2+ 1 + 2

2 72

Now take an example of multivariable problem

-5

0

5

-5

0

50

200

400

600

800

1000

XY

X

Y

-5 -4 -3 -2 -1 0 1 2 3 4 5-5

-4

-3

-2

-1

0

1

2

3

4

5


Minimize 1, 2 = 12 + 2 11

2+ 1 + 2

2 72

= 2 2

The quadratic approximation of the function at = 2 2 can be

written as = +

+

For first approximation

= +

Minimize

= 1 2 2 24218

+ 1 2 2 214 1616 30

1 22 2

Or,

-5

0

5

-5

0

50

200

400

600

800

1000

-5

0

5

-5

0

5

0

1000

2000

3000

4000

YX


Solution

Trial

7.9268 -0.5610 -42 -18

5.7945 -4.4555 1628 99

4.4415 -3.1670 457 -296

113 -833.7952 -2.3927

3.6086 -1.9928 20 -22

1.5811 -4.56373.5858 -1.8623

0.0457 -0.41063.5844 -1.8483

-0.0042 -0.00533.5844 -1.8481

-0.0028 0.00063.5844 -1.8481

X value Gradient

1

2

3

4

5

6

7

8

9

Optimal solution

X

Y

-5 -4 -3 -2 -1 0 1 2 3 4 5-5

-4

-3

-2

-1

0

1

2

3

4

5

Gradient is almost negligible


THANKS

24 April 2015 78

Documents

Introduction to Optimization - Indian Institute of Technology … · Ant colony Newspaper hawker Forensic artist Optimization in real life. R.K. Bhattacharjya/CE/IITG Books •K