Upload
others
View
33
Download
1
Embed Size (px)
Citation preview
Numerical Optimization in MATLAB
Andrii Parkhomenko
Universitat Autonoma de Barcelona and Barcelona GSE
Spring 2017
Andrii Parkhomenko (UAB & Barcelona GSE) Numerical Optimization in MATLAB 0 / 24
Optimization
Typical problem in economics: find an optimum
Case 1:
maxx
f (x) = 0.5x − 0.02x2
Easy: just solve for x that satisfies f ′(x) = 0
Case 2:
maxx
f (x) = 0.5 ln x − 0.02x2 +1
(1 + x)53
Can’t solve analytically ⇒ need to use numerical methods
Andrii Parkhomenko (UAB & Barcelona GSE) Numerical Optimization in MATLAB 1 / 24
Numerical Optimization
Continuous differentiable objective function: derivative-based methods
Arbitrary objective function: derivative-free methods
Andrii Parkhomenko (UAB & Barcelona GSE) Numerical Optimization in MATLAB 2 / 24
Numerical Optimization in MATLAB
These slides cover three functions contained in MATLAB’s Optimization
Toolbox:
fminunc: unconstrained optimization, derivative-based
fmincon: constrained optimization, derivative-based
fminsearch: unconstrained optimization, derivative-free
Andrii Parkhomenko (UAB & Barcelona GSE) Numerical Optimization in MATLAB 3 / 24
fminunc
Finds parameters that minimize a given function
Syntax:x = fminunc(objfun,x0)
I x0: starting guessI objfun: objective function specified in a different scriptI x: solution
Andrii Parkhomenko (UAB & Barcelona GSE) Numerical Optimization in MATLAB 4 / 24
fminunc
Example 1:
maxx
f (x) = 0.5 ln x − 0.02x2 +1
(1 + x)53
0 1 2 3 4 5−1.4
−1.2
−1
−0.8
−0.6
−0.4
−0.2
0
0.2
0.4
0.6
x
f(x)
Andrii Parkhomenko (UAB & Barcelona GSE) Numerical Optimization in MATLAB 5 / 24
fminunc
Example 1:
maxx
f (x) = 0.5 ln x − 0.02x2 +1
(1 + x)53
Step 1. Main code
x0 = 1;
x = fminunc(@objfun,x0)
Step 2. Function
function y = objfun(x)
y = 0.5*log(x) - 0.02*x^2 + (1+x)^(-5/3);
y = -y;
end
Andrii Parkhomenko (UAB & Barcelona GSE) Numerical Optimization in MATLAB 6 / 24
fminunc
fminunc uses the BFGS Quasi-Newton method:
xk+1 = xk + αkpk
1 Compute direction pk
pk = −B−1k ∇fk
Bk+1 = Bk −Bksks
′kBk
s ′kBksk+
yky′k
y ′ksk
where sk ≡ xk+1 − xk and yk ≡ fk+1 − fk
2 Compute stepsize αk . Use line search to approximately solve
minαk
f (xk + αkpk)
Andrii Parkhomenko (UAB & Barcelona GSE) Numerical Optimization in MATLAB 7 / 24
fminunc
Introduce options:
options = optimset(’Opt1’,Opt1Val,’Opt2’,Opt2Val,...);
x = fminunc(@objfun,x0, options)
Important algorithm options:
Display:
optimset(’Display’, ’off’)
Tolerance level:
optimset(’TolFun’, 1e-6)
optimset(’TolX’, 1e-6)
Maximum number of iterations or evaluations
optimset(’MaxIter’, 400)
optimset(’MaxFunEvals’, 100)
Andrii Parkhomenko (UAB & Barcelona GSE) Numerical Optimization in MATLAB 8 / 24
fminunc
Caution:
Objective function must be continuous
Local minima and flat regions are big issues
Andrii Parkhomenko (UAB & Barcelona GSE) Numerical Optimization in MATLAB 9 / 24
fminunc
Example 2. Discontinuous objective function:
maxx
f (x) =
{0.5 ln x − 0.02x2 + 1
(1+x)53, 0 < x ≤ 4
0.5 ln x − 0.02x2 + 1
(1+x)53
+ 1, 4 < x
0 1 2 3 4 5−1.5
−1
−0.5
0
0.5
1
1.5
x
f(x)
Andrii Parkhomenko (UAB & Barcelona GSE) Numerical Optimization in MATLAB 10 / 24
fminunc
Example 3. Objective function with flat regions:
maxx
f (x) =
{0.5 ln x − 0.02x2 + 1
(1+x)53, 0 < x ≤ 4
0.441546, 4 < x
0 1 2 3 4 5−1.4
−1.2
−1
−0.8
−0.6
−0.4
−0.2
0
0.2
0.4
0.6
x
f(x)
Andrii Parkhomenko (UAB & Barcelona GSE) Numerical Optimization in MATLAB 11 / 24
fminunc
Example 4. Objective function with local maxima:
maxx
f (x) = sin(3x) + x − 0.25x2
0 1 2 3 4 5−1
−0.5
0
0.5
1
1.5
2
x
f(x)
Andrii Parkhomenko (UAB & Barcelona GSE) Numerical Optimization in MATLAB 12 / 24
fminunc
To obtain more details on the workings of fminunc:
Read documentation: Help → Type ’fminunc’
Read the code: Type ’open fminunc’
Andrii Parkhomenko (UAB & Barcelona GSE) Numerical Optimization in MATLAB 13 / 24
fmincon
Finds parameters that minimize a given function, subject to a
constraint
Same algorithm as fminunc
Syntax:x = fmincon(objfun,x0,A,b,Aeq,beq,lb,ub,nlcon,options)
I x0: starting guessI objfun: objective function specified in a different scriptI A, b: linear inequality constraint: Ax ≤ bI Aeq, beq: linear equality constraint: Ax = bI lb, ub: lower and upper bounds: x ∈ [lb, ub]I nlcon: nonlinear constraintI x: solution
Andrii Parkhomenko (UAB & Barcelona GSE) Numerical Optimization in MATLAB 14 / 24
fmincon
Example 1:
maxx
f (x) = 0.5 ln x − 0.02x2 +1
(1 + x)53
s.t. x ≤ a
0 1 2 3 4 5−1.4
−1.2
−1
−0.8
−0.6
−0.4
−0.2
0
0.2
0.4
0.6
x
f(x)
Andrii Parkhomenko (UAB & Barcelona GSE) Numerical Optimization in MATLAB 15 / 24
fmincon
Sometimes it’s better to reformulate a constrained problem as anunconstrained
I Can use a formulation with Lagrange multipliers
Andrii Parkhomenko (UAB & Barcelona GSE) Numerical Optimization in MATLAB 16 / 24
fminsearch
Finds parameters that minimize a given function
Derivative-free algorithm (Nelder-Mead)
Syntax:x = fminsearch(objfun,x0,options)
I x0: starting guessI objfun: objective function specified in a different scriptI x: solution
Andrii Parkhomenko (UAB & Barcelona GSE) Numerical Optimization in MATLAB 17 / 24
fminsearch
Nelder-Mead algorithm:
1 select 2 points: x1 and x2
2 order s.t. x = x1 and x = x2 if f (x) ≥ f (x); and x = x2 and x = x1
otherwise
3 reflection: xR = x + (x − x)
4 if f (xR) > f (x) and f (xR) > f (x), expansion: xE = x + α(xR − x), α > 1if f (xR) ≤ f (x) and f (xR) > f (x), update x = xR and go to step 2if f (xR) ≤ f (x) and f (xR) ≤ f (x), contraction: xC = x + β(x − x), β < 1
2
5 if f (xE ) > f (xR), update x = xE and go to step 2if f (xE ) < f (xR), update x = xR and go to step 2
6 if f (xC ) > f (x), update x = xC and go to step 2if f (xC ) < f (x), shrink: replace x = x + γ(x − x) and go to step 2, γ < 1
Iterate until convergence: x ≈ x and/or f (x) ≈ f (x)
Andrii Parkhomenko (UAB & Barcelona GSE) Numerical Optimization in MATLAB 18 / 24
fminsearch
Example 1:
maxx
f (x) = 0.5 ln x − 0.02x2 +1
(1 + x)53
0 1 2 3 4 5−1.4
−1.2
−1
−0.8
−0.6
−0.4
−0.2
0
0.2
0.4
0.6
x
f(x)
Andrii Parkhomenko (UAB & Barcelona GSE) Numerical Optimization in MATLAB 19 / 24
fminsearch
Example 2. Discontinuous objective function:
maxx
f (x) =
{0.5 ln x − 0.02x2 + 1
(1+x)53, 0 < x ≤ 4
0.5 ln x − 0.02x2 + 1
(1+x)53
+ 1, 4 < x
0 1 2 3 4 5−1.5
−1
−0.5
0
0.5
1
1.5
x
f(x)
Andrii Parkhomenko (UAB & Barcelona GSE) Numerical Optimization in MATLAB 20 / 24
fminsearch
Example 3. Objective function with flat regions:
maxx
f (x) =
{0.5 ln x − 0.02x2 + 1
(1+x)53, 0 < x ≤ 4
0.441546, 4 < x
0 1 2 3 4 5−1.4
−1.2
−1
−0.8
−0.6
−0.4
−0.2
0
0.2
0.4
0.6
x
f(x)
Andrii Parkhomenko (UAB & Barcelona GSE) Numerical Optimization in MATLAB 21 / 24
fminsearch
Example 4. Objective function with local maxima:
maxx
f (x) = sin(3x) + x − 0.25x2
0 1 2 3 4 5−1
−0.5
0
0.5
1
1.5
2
x
f(x)
Andrii Parkhomenko (UAB & Barcelona GSE) Numerical Optimization in MATLAB 22 / 24
fminsearch
Example 5. Objective function with many discontinuities:
maxx
f (x) = bxc+ 0.2x − 0.04x2
0 5 10 15 200
1
2
3
4
5
6
7
8
9
f(x)
x
Andrii Parkhomenko (UAB & Barcelona GSE) Numerical Optimization in MATLAB 23 / 24
fminunc vs fminsearch
Rule of thumb: use fminunc for continuous differentiable functions,
fminsearch otherwise
No option guarantees that you find a (global) optimum
For complicated cases:I understand the problem wellI try different algorithmsI try different starting points
Andrii Parkhomenko (UAB & Barcelona GSE) Numerical Optimization in MATLAB 24 / 24