View
7
Download
0
Category
Preview:
Citation preview
Introduction Nonsmooth Optimization Standard Bundle Method The Goal of Research
Nonsmooth Optimization:
Bundle Methods
Kaisa Joki
kjjoki@utu.fi
University of Turku
5.11.2013
1/ 26 Kaisa Joki Bundle Methods
Introduction Nonsmooth Optimization Standard Bundle Method The Goal of Research
Outline
1 Introduction
2 Nonsmooth OptimizationConvex Nonsmooth AnalysisOptimality Condition
3 Standard Bundle MethodTheoretical BackgroundAlgorithm
4 The Goal of Research
2/ 26 Kaisa Joki Bundle Methods
Introduction Nonsmooth Optimization Standard Bundle Method The Goal of Research
Nonsmooth Optimization and Application Areas
In nonsmooth optimization (NSO) functions don't need to bedi�erentiable
The general problem is that we are minimizing functions thatare typically not di�erentiable at their minimizers
This type of problems arise in many �elds of applications
� Economics� Mechanics� Engineering� Computational chemistry and biology� Optimal control� Data mining
3/ 26 Kaisa Joki Bundle Methods
Introduction Nonsmooth Optimization Standard Bundle Method The Goal of Research
Cause of Nonsmoothness
Inherent: Original phenomenon contains variousdiscontinuities and irregularities.
Technological: Caused by some extra technologicalconstraints which may cause a nonsmooth dependencebetween variables and functions.
Methodological: Some algorithms for constrainedoptimization may lead to a nonsmooth problem (for example,the exact penalty function method).
Numerical: So called "sti� problems" which are analyticallysmooth but numerically unstable and behave like nonsmoothproblems.
4/ 26 Kaisa Joki Bundle Methods
Introduction Nonsmooth Optimization Standard Bundle Method The Goal of Research
Di�culties Caused by Nonsmoothness
The gradient does not exist at every point so we
can't utilize the classical theory of optimization because itrequires certain di�erentiability and strong regularityassumptions
can't use smooth (gradient based) methods because they maylead failure in convergence, in optimality test or in gradientapproximation
have di�culties de�ning a descent direction
5/ 26 Kaisa Joki Bundle Methods
Introduction Nonsmooth Optimization Standard Bundle Method The Goal of Research
Nonsmooth Optimization Problem
General problem
Lets consider a nonsmooth optimization problem of the form{min f(x)
s. t. x ∈ X,
where
Set X ⊆ Rn is a set of feasible solutions
Objective function f : Rn → R is
� not required to have continuous derivatives� supposed to be locally Lipschitz continuous on the set X
In the following the objective function f is assumed to be convex.
6/ 26 Kaisa Joki Bundle Methods
Introduction Nonsmooth Optimization Standard Bundle Method The Goal of Research
Convex Analysis
De�nition 1
Let function f : Rn → R be convex. The subdi�erential of f atx ∈ Rn is a set
∂f(x) ={ξ ∈ Rn | f(y) ≥ f(x) + ξT (y − x) for all y ∈ Rn
}.
Each vector ξ ∈ ∂f(x) is called a subgradient of f at point x.
The subdi�erential ∂f(x) is
a nonempty, convex and compact set
a generalization of a classical derivative because if f : Rn → R isconvex and di�erentiable at x ∈ Rn, then ∂f(x) = {∇f(x)}
7/ 26 Kaisa Joki Bundle Methods
Introduction Nonsmooth Optimization Standard Bundle Method The Goal of Research
Convex Analysis
Subdi�erential
8/ 26 Kaisa Joki Bundle Methods
Introduction Nonsmooth Optimization Standard Bundle Method The Goal of Research
Convex Analysis
Theorem 2
If f : Rn → R is convex then for all y ∈ Rn
f(y) = max{f(x) + ξT (y − x) |x ∈ Rn, ξ ∈ ∂f(x)
}.
Theorem 3
The direction d ∈ Rn is a descent direction for a convex function
f : Rn → R at x ∈ Rn if
ξTd < 0 for all ξ ∈ ∂f(x).
9/ 26 Kaisa Joki Bundle Methods
Introduction Nonsmooth Optimization Standard Bundle Method The Goal of Research
Optimality Condition
For convex functions we have the following necessary and su�cientoptimality condition:
Theorem 4
A convex function f : Rn → R attains its global minimum at point
x, if and only if
0 ∈ ∂f(x).
10/ 26 Kaisa Joki Bundle Methods
Introduction Nonsmooth Optimization Standard Bundle Method The Goal of Research
Methods for Nonsmooth Optimization
The main problem: We usually don't know the whole subdi�erential ofthe function but only one arbitrary subgradient at each point.
Di�erent methods to solve a nonsmooth optimization problem
Bundle Methods
Derivative Free Methods
Subgradient Methods
Gradient Sampling Methods
Hybrid Methods
Special Methods
11/ 26 Kaisa Joki Bundle Methods
Introduction Nonsmooth Optimization Standard Bundle Method The Goal of Research
About Standard Bundle Method
We consider an unconstrained convex nonsmooth problem{min f(x)
s. t. x ∈ Rn
Assumption: At every point x ∈ Rn we can evaluate the valuef(x) and one arbitrary ξ ∈ ∂f(x)Converges to the global minimum of f (if it exists)
12/ 26 Kaisa Joki Bundle Methods
Introduction Nonsmooth Optimization Standard Bundle Method The Goal of Research
The main idea:
Approximate the subdi�erential of the objective function witha bundle
Bundle consists of subgradients from previous iterations
Subgradient information is used to construct a piecewise linearapproximation to the objective function
This approximation is used to determine a descent direction
If approximation is not adequate then we add moreinformation to the bundle
13/ 26 Kaisa Joki Bundle Methods
Introduction Nonsmooth Optimization Standard Bundle Method The Goal of Research
Bundle
At iteration k in the current iteration point xk our bundle is
Bk ={(yj , f(yj), ξj) | j ∈ Jk
}.
where
� yj ∈ Rn is a trial point� ξj ∈ ∂f(yj) is a subgradient� Jk is a nonempty subset of {1, 2, . . . , k}
By using the bundle we can construct a cutting plane model
which is a piecewise linear approximation of function f
14/ 26 Kaisa Joki Bundle Methods
Introduction Nonsmooth Optimization Standard Bundle Method The Goal of Research
Cutting Plane Model
The cutting plane model is
f̂k(x) = maxj∈Jk
{f(yj) + ξ
Tj (x− yj)
}for all x ∈ Rn.
It is a convex function and f̂k(x) ≤ f(x).This approximation can be written in equivalent form
f̂k(x) = maxj∈Jk
{f(xk) + ξ
Tj (x− xk)− αk
j
}with the linearization error
αkj = f(xk)− f(yj)− ξ
Tj (xk − yj) ≥ 0 for all j ∈ Jk.
15/ 26 Kaisa Joki Bundle Methods
Introduction Nonsmooth Optimization Standard Bundle Method The Goal of Research
Cutting Plane Model
Linearization error Cutting plane model
16/ 26 Kaisa Joki Bundle Methods
Introduction Nonsmooth Optimization Standard Bundle Method The Goal of Research
Algorithm: Direction Finding Problem
To determine the search direction dk we need to solve
mind∈Rn
{f̂k(xk + d) +
1
2dTMkd
}where
Mk is a positive de�nite and symmetric n× n matrix.12d
TMkd is a stabilizing term which
� guarantees existence of the unique solution dk� keeps approximation local enough
The search direction dk is also a descent direction to the originalobjective
17/ 26 Kaisa Joki Bundle Methods
Introduction Nonsmooth Optimization Standard Bundle Method The Goal of Research
Algorithm: Quadratic Direction Finding Problem
mind∈Rn
{maxj∈Jk
{f(xk) + ξ
Tj d− αk
j
}+
1
2dTMkd
}(1)
The problem (1) can be rewritten as a smooth quadratic
direction �nding problemmin v + 1
2dTMkd
s. t. ξTj d− αkj ≤ v ∀j ∈ Jk,
v ∈ R, d ∈ Rn
(2)
18/ 26 Kaisa Joki Bundle Methods
Introduction Nonsmooth Optimization Standard Bundle Method The Goal of Research
Algorithm: Search Direction and Stopping Condition
The solution (vk,dk) of (2) can also be calculated from the dualproblem
The next iteration candidate is yk+1 = xk + dk
Valuevk = f̂k(yk+1)− f(xk)
is the predicted descent of f at yk+1
If vk = 0 then the current point xk is the global minimum
It is convenient to stop algorithm when −vk ≤ ε where ε > 0 is a�nal accuracy tolerance
19/ 26 Kaisa Joki Bundle Methods
Introduction Nonsmooth Optimization Standard Bundle Method The Goal of Research
Algorithm: Serious and Null Step
Now we perform either a serious step or a null step
A serious step
xk+1 = yk+1
is performed iff(yk+1)− f(xk) ≤ mvk
where m ∈ (0, 1/2) is a line search parameter.
Otherwise we make a null step
xk+1 = xk
20/ 26 Kaisa Joki Bundle Methods
Introduction Nonsmooth Optimization Standard Bundle Method The Goal of Research
Algorithm: Updating the Bundle
In both steps we improve the approximation by adding
(yk+1, f(yk+1), ξk+1)
into the bundle where ξk+1 ∈ ∂f(yk+1)
Easiest way to do this is to set Jk+1 = Jk ∪ {k + 1}� stores all subgradients� causes di�culties with storage and computation
By using the subgradient aggregation strategy we can keep thesize of the bundle bounded
21/ 26 Kaisa Joki Bundle Methods
Introduction Nonsmooth Optimization Standard Bundle Method The Goal of Research
Nonconvex Bundle Methods
Objective function is only supposed to be locally Lipschitzcontinuous
Cannot guarantee even local optimality of a solution withoutsome convexity assumption
Not as e�cient methods as convex bundle methods
Best nonconvex bundle methods are generalizations of convexbundle methods with suitable modi�cations
Not developed from the nonconvex perspective
22/ 26 Kaisa Joki Bundle Methods
Introduction Nonsmooth Optimization Standard Bundle Method The Goal of Research
Di�erence of two Convex functions
De�nition 5
A function f : Rn → R is called a DC function if it can be written in theform
f(x) = f1(x)− f2(x)
where f1 and f2 are convex functions on Rn.
If a DC function f is nonsmooth then at least one of the functionsf1 and f2 is nonsmooth
For DC functions it is still possible to utilize convex analysis andconvex optimization theory to some extent
Many problems of nonconvex optimization can be described byusing DC functions
23/ 26 Kaisa Joki Bundle Methods
Introduction Nonsmooth Optimization Standard Bundle Method The Goal of Research
Future Work
Only a few e�cient nonconvex nonsmooth optimization methods exist somy goal is
Develop a local bundle method for unconstrained nonconvexnonsmooth problems where the objective is a DC function
Add good features of gradient sampling methods to possibly get abetter method
Extend the method to constrained case
Modify the local method so that we get a global solution both inunconstrained and constrained case
24/ 26 Kaisa Joki Bundle Methods
Introduction Nonsmooth Optimization Standard Bundle Method The Goal of Research
References
[1] Bagirov, A.M. and Ugon, J.: Codi�erential method for
minimizing nonsmooth DC functions. Journal of GlobalOptimization, Vol. 50(1), 2011, pages 3�22.
[2] Haarala, M.: Large-Scale Nonsmooth Optimization: Variable
metric bundle method with limited memory. Doctoral Thesis,University of Jyväskylä, 2004.
[3] Mäkelä, M.M.: Survey of bundle methods for nonsmooth
optimization. Optimization Methods and Software, Vol. 17(1),2002, pages 1�29.
[4] Mäkelä, M.M. and Neittaanmäki, P.: Nonsmooth Optimization:
Analysis and Algorithms with Applications to Optimal Control.
World Scienti�c Publishing Co., Singapore, 1992.
25/ 26 Kaisa Joki Bundle Methods
Introduction Nonsmooth Optimization Standard Bundle Method The Goal of Research
Thank you for your attention!
26/ 26 Kaisa Joki Bundle Methods
Recommended