MULTI VARIABLE OPTIMIZATION
Min f(x1, x2, x3,----- xn)
UNIDIRECTIONAL SEARCH
- CONSIDER A DIRECTION S
sxx rr αα +=)(
- REDUCE TO
)(αfMin- SOLVE AS
A SINGLE VARIABLE PROBLEM
srMin Point
Uni directional search (example)
Min f(x1, x2) = (x1-10)2 + (x2-10)2
S = (2, 5) (search direction)
X = (2, 1) (Initial guess)
DIRECT SERACH METHODS
- SEARCH THROUGH MANY DIRECTIONS- FOR N VARIABLES 2N DIRECTIONS-Obtained by altering each of the n values and taking all combinations
EVOLUTIONARY OPTIMIZATION METHOD
- COMPARE ALL 2N+1 POINTS & CHOOSE THE BEST.- CONTINUE TILL THERE IS AN IMPROVE MENT - ELSE DECREASE INCREAMENT
STEP 1: x0 = INITIAL POINT
= STEP REDUCTION PARAMETER FOR EACH VARIABLE
= TERMINATION PARA METER
i∆
∈
2&
2;2/:4)12(:3
2/2
:2
0
0
GOTOxxELSE
GOTOxxIFSTEPMinxSTEP
xxPOINTSCREATEELSE
STOPIFSTEP
ii
N
iii
N
=
∆=∆−=
+=
∆±=
<∈∆
Hooke Jeeves pattern search
• Pattern Search ---– Create a set of search directions iteratively– Should be linearly independent
• A combination of exploratory and pattern moves– Exploratory – find the best point in the vicinity of the
current point– Pattern – Jump in the direction of change, if better then
continue, else reduce size of exploratory move and continue
Exploratory move
• Current solution is xc; set i = 1; x = xc
• S1: f = f(x), f+ = f(xi + ∆i), f- = f(xi - ∆i)• S2: fmin = min (f, f+, f-); set x corresponding
to fmin
• S3: If i = N, go to 4; else i = i + 1, go to 1• S4: If x ≠ xc, success, else failure
Pattern Move
• S1: Choose x(0), ∆I, for I = 1, 2, …N, ε, and set k = 0
• S2: Perform exploratory move with xk as base point; – If success, xk+1 = x, go to 4 else goto 3
• S3: If |∆| < ε, terminate– Else set ∆i = ∆i / α --- ∀i, go to 2
Pattern Move (contd)
• S4: k = k+1; xpk+1 = xk + (xk – xk-1)
• S5: Perform another exploratory move with xp
k+1 as the base point; Result = xk+1
• S6: If f(xk+1) < f(xk), goto S4– Else goto S3
Example :• Consider the Himmelblau function:
• SolutionStep 1 Selection of initial conditions1. Initial Point : 2. Increment vector : 3. Reduction factor : 4. Termination parameter :5. Iteration counter : k = 0
2221
22
2121 )7()11(),( −++−+= xxxxxxf
Tx )0,0()0( =T)5.0,5.0(=∆
2=α310 −=ε
• Step 2Perform an Iteration of the exploratory move with base point as
Thus we set
The exploratory move will be performed with the following steps
1)0,0()0( === iandxx T
)0(xx =
Steps for the Exploratory moveStep 1 : Explore the vicinity of the variable x1
Calculate the function values at three points
Step 2 : Take the Minimum of above function and corresponding point
Step 3 : As : all variables are not explored Increment counter i=2 and explore second variable.First iteration completed
TTxx )5.0,5.0()( )0(1
)0( =∆+
Tx )0,0()0( =
TTxx )5.0,5.0()( )0(1
)0( −=∆−
81.157))5.0,5.0(( ==+ Tff
170))0,0(( == Tff
81.171))5.0,5.0(( =−=− Tff
1≠i
Step1: At this point the base point is explore the variable x2 and calculate the function values.
Step 2 : fmin = 144.12 and point , x=(0.5,0.5)
Step 3 : As i=2 move to the step 4 of the exploratory moveStep 4 : ( of the Exploratory move )Since the move is success and we set
Tx )0,5.0(=
12.144))5.0,5.0(( ==+ Tff
81.157))0,5.0(( == Tff
62.165))5.0,5.0(( =−=− Tff
cxx ≠Tx )5.0,5.0(=
• As the move is success, set moveto step 4
STEP 4 : We set k=1 and perform Pattern move
Step 5 : Perform another exploratory move as before and with as the base point.
The new point is Set the new point Step 6 : is smaller than Proceed to next step to perform another pattern move
Txx )5.0,5.0()1( ==
TTTTp xxxx )1,1()0,0()5.0,5.0(2))(( )0()1()1()2( =−=−+=
)2(
pxTx )5.1,5.1(=
Txx )5.1,5.1()2( ==12.63)( )2( =xf 12.144)( )1( =xf
STEP 4 : Set k=2 and create a new point
Note: as is better than , a jump along the direction is made, this will take search closer to
true minimum
STEP 5 : Perform another exploratory move to find any better point around the new point.
Performing the move on both variables we have New point
This point is the true minimum point
Tp xxx )5.2,5.2()2( )1()2()3( =−=
)2(x )1(x)( )1()2( xx −
Tx )0.2,0.3()3( =
In the example the minimum of the Hookes-Jeevesalgorithm happen in two iterations : this may not be the case always
Even though the minimum point is reached there is no way of finding whether the optimum is reachedor notThe algorithm proceeds until the norm if the increment vector is small.
STEP 6 : function value at new point
Thus move on to step 412.63)(0)( 23 =<= xfxf
STEP 4 : The iteration counter k = 3 and the new point
STEP 5 : With the new point as base the search is success and x=(4.0,2.0) and thus we set
STEP 6 : The function value is 50, which is larger that the earlier i.e. 0 . Thus we move to step 3
Step 3 : Since we reduce the increment vector
and proceed to Step 2 to perform the iterations
Tp xxx )5.2,5.4()2( )2()3()4( =−=
Tx )0.2,0.4()4( =
ε⊄=∆ 5.0TT )25.0,25.0(2/)5.0,5.0( =−=∆
Step 2 : Perform an exploratory move with the following as current pointThe exploratory move on both the variables is failure and we obtain thus we proceed to Step 3
Step 3 : Since is not small reduce the increment vector and move to Step 2.
The new increment vector is The algorithm now continues with step 2 and step 3
until is smaller than the termination factor. The final solution is with the function
value 0
Tx )0.2,0.3()3( =
Tx )0.2,0.3()3( =
∆
T)125.0,125.0(=∆
Tx )0.2,0.3(* =
∆
POWELL’S CONJUGATE DIRECTION METHOD
For a quadratic function IN 2 VARIABLES- TAKE 2 POINTS x1 & x2 AND- A DIRECTION ‘d’
IF y1 IS A SOLUTION OF MIN &)( 1 dxf λ+y2 IS A SOLUTION OF MIN )( 2 dxf λ+
THEN (y2- y1) IS CONJUGATE TO d
OPTIMUM LIES ALONG (y2- y1)
x2
x1
y1
y2
FOR N VARIABLES
STEP 1: TAKE x0 & N LINEARLY INDEPENDENT DIRECTIONS s1, s2, s3, s4,------- sN si = ei
STEP 2: - MINIMIZE ALONG N UNI-DIRECTIONAL SEARCHDIRECTIONS,USING PREVIOUS BEST EVERY TIME
- PERFORM ANOTHER SEARCH ALONG s1
STEP 3: FROM THE CONJUGATE DIRECTION “d”
STEP 4: IF IS SMALL TERMINATE
ELSE
d
2/
&1
1
GOTOdds
ss jjj
=
∀= −
GRADIENT BASED METHODS
DESCENT DIRECTION
IF
⎥⎦
⎤⎢⎣
⎡∂∂
−−−−−∂∂
∂∂
∂∂
=∇
∇−∇
−∇=
≤∇
nxxxx
xfdxf
xfd
dxf
321
)().(
)(
0).(
IF
IS MAXIMALLY NEGATIVE
IS STEEPEST DESCENT IDRECTION
WHERE
CAUCHY’S METHOD (STEEPEST DESCENT)
STEP 1: CHOOSE M (MAX. NO. OF ITERATIONS)
STEP 2: CALCULATE
STEP 3: IF TERMINATE
IF TERMINATE
STEP 4: UNIDIRECTIONAL SEARCH USING 2
1
021
)(
)(
0,,,
∈
≥
≤∈∇
∇
=∈∈
Mk
xf
xf
kx
k
k
1
))(()(
1
1
1
+=
≤∈−
∇−=
+
+
kk
x
xx
xfxfxf
k
kk
kkk αMIN.
STEP 5: IF
ELSE GOTO STEP 2
- METHOD WORKS WELL WHEN xk IS FAR FROM x* (OPTIMUM)- IF POINT IS CLOSE TO x* THEN CHANGE IN GRADIENT VECTOR IS
VERY SMALL.- OTHER METHODS USE VARIATION
-SECOND DERIVATIVES(NEWTON’S)-COMBIMATION-CONJUGATE GRADIENT METHOD
SOLVING SIMULTANEOUS EQUATIONS
X+Y=5X-Y=2
MIN. [(X+Y-5)2+(X-Y-2)2]
PRACTICESOLVE PROBLEMS 3-1 TO 3-13 USING TECHNIQUES DONEIN CLASS.
Penalty function approach
• Transformation method- convert to a sequence of unconstrained
problems.• Give penalty to (violated) constraints.• Add to objective function.• Solve.• Use result as starting point for next iteration.• Alter penalties ands repeat.
• MinimiseSubjected toPenalty =
2221
22
21 )7()11( −++−+ xxxx
026)5( 22
21 ≥−+− xx
222
21 026)5(1.0 >≥−+−< xx
1x
2x
Minimum point
Infeasible regionFeasible region
Process 1. Choose2. Form modified objective function
3. Start with . Find so as to minimize P.(use )
4. If Terminate.
5. Else k=k+1; go to step 2.
.,,, 21 ΩRεε
))(),(,()(),( kkkkkk xhxgRxfRxP Ω+=
kx 1+kx
1ε
211 ),(),( ε<− −+ kkkk RxPRxP
,1 cRRk =+
• At any stage minimize P(x,R) = f(x)+Ω(R,g(x),h(x))R = set of penalty parametersΩ= penalty function
Types of penalty function• Parabolic penalty
- for equality constraints- only for infeasible points
• Interior penalty functions- penalize feasible points
• Exterior penalty functions- penalize infeasible points
• Mixed penalty functions- combination of both
2Rh(x)=Ω
• Infinite barrier penalty
- inequality constraints.- R is very large.- Exterior.
• Log penaltyΩ=-R ln[g(x)] -inequality constraints- for feasible points- interior.-initially large R.- larger penalty close to border.
∑=Ω )(xgR j
• Inverse penalty
- interior.- larger penalty close to border.- initially large R
• Bracket order penalty
- <A>=A if A<0.- exterior.- initially small R.
⎥⎦
⎤⎢⎣
⎡=Ω)(
1xg
R
2)( >< xgR
Direct search
• Variable elimination- for equality constraints- Express one variable as a function of others
and- Eliminate one variable - Remove all equality constraints