Upload
dangngoc
View
243
Download
5
Embed Size (px)
Citation preview
1
• Generic descent algorithm
• Generalization to multiple dimensions
• Problems of descent methods, possible improvements
• Fixes
• Local minima
Lecture 10: descent methods
Gradient descent (reminder)
f
x
f(x)
f(m)
m
guess
Minimum of a function is found by following the slope of the function
2
Gradient descent (illustration)
f
x
f(x)
f(m)
m
guess
next step
Gradient descent (illustration)
f
x
f(x)
f(m)
m
guess
next step
new gradient
3
Gradient descent (illustration)
f
x
f(x)
f(m)
m
guess
next step
Gradient descent (illustration)
f
x
f(x)
f(m)
m
guess
stop
4
Gradient descent: algorithm
Start with a point (guess)
Repeat
Determine a descent direction
Choose a step
Update
Until stopping criterion is satisfied
guess
Gradient descent: algorithm
Start with a point (guess)
Repeat
Determine a descent direction
Choose a step
Update
Until stopping criterion is satisfied
Direction: downhill
5
Gradient descent: algorithm
Start with a point (guess)
Repeat
Determine a descent direction
Choose a step
Update
Until stopping criterion is satisfied
step
Gradient descent: algorithm
Start with a point (guess)
Repeat
Determine a descent direction
Choose a step
Update
Until stopping criterion is satisfied
Now you are here
6
Gradient descent: algorithm
Start with a point (guess)
Repeat
Determine a descent direction
Choose a step
Update
Until stopping criterion is satisfied
Stop when “close”
from minimum
Gradient descent: algorithm
Start with a point (guess) guess = x
Repeat
Determine a descent direction direction = -f’(x)
Choose a step step = h > 0
Update x:=x–hf’(x)
Until stopping criterion is satisfied f’(x)~0
7
Example of 2D gradient: pic of the MATLAB demo
Illustration of the gradient in 2D
Example of 2D gradient: pic of the MATLAB demo
Illustration of the gradient in 2D
8
Example of 2D gradient: pic of the MATLAB demo
Illustration of the gradient in 2D
Example of 2D gradient: pic of the MATLAB demo
Definition of the gradient in 2D
This is just a genaralization of the derivative in two dimensions.
This can be generalized to any dimension.
9
Example of 2D gradient: pic of the MATLAB demo
Illustration of the gradient in 2D
Example of 2D gradient: pic of the MATLAB demo
Gradient descent works in 2D
10
Generalization to multiple dimensions
Start with a point (guess)
Repeat
Determine a descent direction
Choose a step
Update
Until stopping criterion is satisfied
guess
Generalization to multiple dimensions
Start with a point (guess)
Repeat
Determine a descent direction
Choose a step
Update
Until stopping criterion is satisfied
Direction: downhill
11
Generalization to multiple dimensions
Start with a point (guess)
Repeat
Determine a descent direction
Choose a step
Update
Until stopping criterion is satisfied
step
Generalization to multiple dimensions
Start with a point (guess)
Repeat
Determine a descent direction
Choose a step
Update
Until stopping criterion is satisfied
Now you are here
12
Generalization to multiple dimensions
Start with a point (guess)
Repeat
Determine a descent direction
Choose a step
Update
Until stopping criterion is satisfied
Stop when “close”
from minimum
Generalization to multiple dimensions
Start with a point (guess) guess = x
Repeat
Determine a descent direction direction = -f(x)
Choose a step step = h > 0
Update x:=x–h Vf’(x)
Until stopping criterion is satisfied Vf’(x)~0
13
Multiple dimensions
Everything that you have seen with derivatives can be generalized
with the gradient.
For the descent method, f’(x) can be replaced by
In two dimensions, and by
in N dimensions.
Example of 2D gradient: MATLAB demo
The cost to buy a portfolio is:
If you want to minimize the price to buy your portfolio, you
need to compute the gradient of its price:
Sto
ck 1
Sto
ck 2
Sto
ck i
Sto
ck N
14
Problem 1: choice of the step
When updating the current computation:
- small steps: inefficient
- large steps: potentially bad results
f
x
f(x)
f(m)
m
guess
stop
Too many steps:
takes too long to converge
Problem 1: choice of the step
When updating the current computation:
- small steps: inefficient
- large steps: potentially bad results
f
x
f(x)
f(m)
m
Current point
Next point (went too far)
15
Problem 2: « ping pong effect »
[S. Boyd, L. Vandenberghe, Convex Convex Optimization lect. Notes, Stanford Univ. 2004 ]
Problem 2: « ping pong effect »
[S. Boyd, L. Vandenberghe, Convex Convex Optimization lect. Notes, Stanford Univ. 2004 ]
16
Problem 2: (other norm dependent issues)
[S. Boyd, L. Vandenberghe, Convex Convex Optimization lect. Notes, Stanford Univ. 2004 ]
Problem 3: stopping criterion
Intuitive criterion:
In multiple dimensions:
Or equivalently
Rarely used in practice.
More about this in EE227A (convex optimization, Prof. L. El Ghaoui).
17
Fixes
Several methods exist to address this problem
- Line search methods, in particular
- Backtracking line search
- Exact line search
- Normalized steepest descent
- Newton steps
Fundamental problem of the method: local minima
Local minima: pic of the MATLAB demo
The iterations of the algorithm
converge to a local minimum
18
Local minima: pic of the MATLAB demo
View of the algorithm is « myopic »