63
A Secant Method for Nonsmooth Optimization Asef Nazari CSIRO Melbourne CARMA Workshop on Optimization, Nonlinear Analysis, Randomness and Risk Newcastle, Australia 12 July, 2014 Asef Nazari A Secant Method for Nonsmooth Optimization

A Secant Method for Nonsmooth Optimization · 2014. 7. 12. · A Secant Method for Nonsmooth Optimization Asef Nazari CSIRO Melbourne CARMA Workshop on Optimization, Nonlinear Analysis,

  • Upload
    others

  • View
    6

  • Download
    0

Embed Size (px)

Citation preview

Page 1: A Secant Method for Nonsmooth Optimization · 2014. 7. 12. · A Secant Method for Nonsmooth Optimization Asef Nazari CSIRO Melbourne CARMA Workshop on Optimization, Nonlinear Analysis,

A Secant Method for Nonsmooth Optimization

Asef Nazari

CSIRO Melbourne

CARMA Workshop on Optimization, Nonlinear Analysis, Randomness and RiskNewcastle, Australia

12 July, 2014

Asef Nazari A Secant Method for Nonsmooth Optimization

Page 2: A Secant Method for Nonsmooth Optimization · 2014. 7. 12. · A Secant Method for Nonsmooth Optimization Asef Nazari CSIRO Melbourne CARMA Workshop on Optimization, Nonlinear Analysis,

OutlineBackground

Secant Method

1 BackgroundThe ProblemOptimization Algorithms and componentsSubgradient and Subdifferential

2 Secant MethodDefinitionsOptimality Condition and Descent DirectionThe Secant AlgorithmNumerical Results

Asef Nazari Secant Method

Page 3: A Secant Method for Nonsmooth Optimization · 2014. 7. 12. · A Secant Method for Nonsmooth Optimization Asef Nazari CSIRO Melbourne CARMA Workshop on Optimization, Nonlinear Analysis,

OutlineBackground

Secant Method

The ProblemOptimization Algorithms and componentsSubgradient and Subdifferential

The problem

Unconstrained Optimization Problem

minx∈Rn

f (x)

f : Rn → Rn

Locally Lipshitz O

Why just unconstrained?

Asef Nazari Secant Method

Page 4: A Secant Method for Nonsmooth Optimization · 2014. 7. 12. · A Secant Method for Nonsmooth Optimization Asef Nazari CSIRO Melbourne CARMA Workshop on Optimization, Nonlinear Analysis,

OutlineBackground

Secant Method

The ProblemOptimization Algorithms and componentsSubgradient and Subdifferential

The problem

Unconstrained Optimization Problem

minx∈Rn

f (x)

f : Rn → Rn

Locally Lipshitz O

Why just unconstrained?

Asef Nazari Secant Method

Page 5: A Secant Method for Nonsmooth Optimization · 2014. 7. 12. · A Secant Method for Nonsmooth Optimization Asef Nazari CSIRO Melbourne CARMA Workshop on Optimization, Nonlinear Analysis,

OutlineBackground

Secant Method

The ProblemOptimization Algorithms and componentsSubgradient and Subdifferential

Transforming Constrained into Unconstrained

Constrained Optimization Problem

minx∈Y

f (x)

where Y ⊂ Rn.

Distance function

dist(x ,Y ) = miny∈Y‖y − x‖

Theory of penalty function (under some conditions)

minx∈Rn

f (x) + σdist(x , y)

f (x) + σdist(x , y) is a nonsmooth function

Asef Nazari Secant Method

Page 6: A Secant Method for Nonsmooth Optimization · 2014. 7. 12. · A Secant Method for Nonsmooth Optimization Asef Nazari CSIRO Melbourne CARMA Workshop on Optimization, Nonlinear Analysis,

OutlineBackground

Secant Method

The ProblemOptimization Algorithms and componentsSubgradient and Subdifferential

Transforming Constrained into Unconstrained

Constrained Optimization Problem

minx∈Y

f (x)

where Y ⊂ Rn.

Distance function

dist(x ,Y ) = miny∈Y‖y − x‖

Theory of penalty function (under some conditions)

minx∈Rn

f (x) + σdist(x , y)

f (x) + σdist(x , y) is a nonsmooth function

Asef Nazari Secant Method

Page 7: A Secant Method for Nonsmooth Optimization · 2014. 7. 12. · A Secant Method for Nonsmooth Optimization Asef Nazari CSIRO Melbourne CARMA Workshop on Optimization, Nonlinear Analysis,

OutlineBackground

Secant Method

The ProblemOptimization Algorithms and componentsSubgradient and Subdifferential

Transforming Constrained into Unconstrained

Constrained Optimization Problem

minx∈Y

f (x)

where Y ⊂ Rn.

Distance function

dist(x ,Y ) = miny∈Y‖y − x‖

Theory of penalty function (under some conditions)

minx∈Rn

f (x) + σdist(x , y)

f (x) + σdist(x , y) is a nonsmooth function

Asef Nazari Secant Method

Page 8: A Secant Method for Nonsmooth Optimization · 2014. 7. 12. · A Secant Method for Nonsmooth Optimization Asef Nazari CSIRO Melbourne CARMA Workshop on Optimization, Nonlinear Analysis,

OutlineBackground

Secant Method

The ProblemOptimization Algorithms and componentsSubgradient and Subdifferential

Sources of Nonsmooth Problems

Minimax Problemminx∈Rn

f (x)

f (x) = max1≤i≤m

fi (x)

System of Nonlinear Equationsfi (x) = 0, i = 1, . . . ,m,

we often dominx∈Rn

‖f (x)‖

where f (x) = (f1(x), . . . , fm(x))

Asef Nazari Secant Method

Page 9: A Secant Method for Nonsmooth Optimization · 2014. 7. 12. · A Secant Method for Nonsmooth Optimization Asef Nazari CSIRO Melbourne CARMA Workshop on Optimization, Nonlinear Analysis,

OutlineBackground

Secant Method

The ProblemOptimization Algorithms and componentsSubgradient and Subdifferential

Sources of Nonsmooth Problems

Minimax Problemminx∈Rn

f (x)

f (x) = max1≤i≤m

fi (x)

System of Nonlinear Equationsfi (x) = 0, i = 1, . . . ,m,

we often dominx∈Rn

‖f (x)‖

where f (x) = (f1(x), . . . , fm(x))

Asef Nazari Secant Method

Page 10: A Secant Method for Nonsmooth Optimization · 2014. 7. 12. · A Secant Method for Nonsmooth Optimization Asef Nazari CSIRO Melbourne CARMA Workshop on Optimization, Nonlinear Analysis,

OutlineBackground

Secant Method

The ProblemOptimization Algorithms and componentsSubgradient and Subdifferential

Sources of Nonsmooth Problems

Minimax Problemminx∈Rn

f (x)

f (x) = max1≤i≤m

fi (x)

System of Nonlinear Equationsfi (x) = 0, i = 1, . . . ,m,

we often dominx∈Rn

‖f (x)‖

where f (x) = (f1(x), . . . , fm(x))

Asef Nazari Secant Method

Page 11: A Secant Method for Nonsmooth Optimization · 2014. 7. 12. · A Secant Method for Nonsmooth Optimization Asef Nazari CSIRO Melbourne CARMA Workshop on Optimization, Nonlinear Analysis,

OutlineBackground

Secant Method

The ProblemOptimization Algorithms and componentsSubgradient and Subdifferential

Structure of Optimization Algorithms

Step 1 Initial Step x0 ∈ Rn

Step 2 Termination Criteria

Step 3 Finding descent direction dk at xk

Step 4 Finding step size f (xk + αkdk) < f (xk)

Step 5 Loop xk+1 = xk + αkdk and go to step 2.

Asef Nazari Secant Method

Page 12: A Secant Method for Nonsmooth Optimization · 2014. 7. 12. · A Secant Method for Nonsmooth Optimization Asef Nazari CSIRO Melbourne CARMA Workshop on Optimization, Nonlinear Analysis,

OutlineBackground

Secant Method

The ProblemOptimization Algorithms and componentsSubgradient and Subdifferential

Classification of Algorithms Based on dk and αk

Directionsdk = −∇f (xk) Steepest Descent Methoddk = −H−1(xk)∇f (xk) Newton Methoddk = −B−1∇f (xk) Quasi-Newton Method

Step sizes

h(α) = f (xk + αdk)1 exactly solve h′(α) = 0 exact line search2 loosly solve it, inexact line serach

Trust region methods

Asef Nazari Secant Method

Page 13: A Secant Method for Nonsmooth Optimization · 2014. 7. 12. · A Secant Method for Nonsmooth Optimization Asef Nazari CSIRO Melbourne CARMA Workshop on Optimization, Nonlinear Analysis,

OutlineBackground

Secant Method

The ProblemOptimization Algorithms and componentsSubgradient and Subdifferential

Classification of Algorithms Based on dk and αk

Directionsdk = −∇f (xk) Steepest Descent Methoddk = −H−1(xk)∇f (xk) Newton Methoddk = −B−1∇f (xk) Quasi-Newton Method

Step sizesh(α) = f (xk + αdk)

1 exactly solve h′(α) = 0 exact line search2 loosly solve it, inexact line serach

Trust region methods

Asef Nazari Secant Method

Page 14: A Secant Method for Nonsmooth Optimization · 2014. 7. 12. · A Secant Method for Nonsmooth Optimization Asef Nazari CSIRO Melbourne CARMA Workshop on Optimization, Nonlinear Analysis,

OutlineBackground

Secant Method

The ProblemOptimization Algorithms and componentsSubgradient and Subdifferential

When the objective function is smooth

Descent direction f ′(xk , d) < 0

f ′(xk , d) = 〈∇f (xk), d〉 ≤ 0 descent direction

−∇f (xk) steepest descent direction

‖∇f (xk)‖ ≤ ε good stopping criteria (First Order NecessaryConditions)

Asef Nazari Secant Method

Page 15: A Secant Method for Nonsmooth Optimization · 2014. 7. 12. · A Secant Method for Nonsmooth Optimization Asef Nazari CSIRO Melbourne CARMA Workshop on Optimization, Nonlinear Analysis,

OutlineBackground

Secant Method

The ProblemOptimization Algorithms and componentsSubgradient and Subdifferential

Subgradient

Basic Inequality for convex differentiable function:

f (x) ≥ f (y) +∇f (y)T .(x − y) ∀x ∈ Dom(f )

Asef Nazari Secant Method

Page 16: A Secant Method for Nonsmooth Optimization · 2014. 7. 12. · A Secant Method for Nonsmooth Optimization Asef Nazari CSIRO Melbourne CARMA Workshop on Optimization, Nonlinear Analysis,

OutlineBackground

Secant Method

The ProblemOptimization Algorithms and componentsSubgradient and Subdifferential

Subgradient

Basic Inequality for convex differentiable function:

f (x) ≥ f (y) +∇f (y)T .(x − y) ∀x ∈ Dom(f )

Asef Nazari Secant Method

Page 17: A Secant Method for Nonsmooth Optimization · 2014. 7. 12. · A Secant Method for Nonsmooth Optimization Asef Nazari CSIRO Melbourne CARMA Workshop on Optimization, Nonlinear Analysis,

OutlineBackground

Secant Method

The ProblemOptimization Algorithms and componentsSubgradient and Subdifferential

Subgradient

g ∈ Rn is a subgradient of a convex function f at y if

f (x) ≥ f (y) + gT .(x − y) ∀x ∈ Dom(f )

if x < 0, g = −1

if x > 0, g = 1

if x = 0, g ∈ [−1, 1]

Asef Nazari Secant Method

Page 18: A Secant Method for Nonsmooth Optimization · 2014. 7. 12. · A Secant Method for Nonsmooth Optimization Asef Nazari CSIRO Melbourne CARMA Workshop on Optimization, Nonlinear Analysis,

OutlineBackground

Secant Method

The ProblemOptimization Algorithms and componentsSubgradient and Subdifferential

Subgradient

g ∈ Rn is a subgradient of a convex function f at y if

f (x) ≥ f (y) + gT .(x − y) ∀x ∈ Dom(f )

if x < 0, g = −1

if x > 0, g = 1

if x = 0, g ∈ [−1, 1]

Asef Nazari Secant Method

Page 19: A Secant Method for Nonsmooth Optimization · 2014. 7. 12. · A Secant Method for Nonsmooth Optimization Asef Nazari CSIRO Melbourne CARMA Workshop on Optimization, Nonlinear Analysis,

OutlineBackground

Secant Method

The ProblemOptimization Algorithms and componentsSubgradient and Subdifferential

Subgradient

g ∈ Rn is a subgradient of a convex function f at y if

f (x) ≥ f (y) + gT .(x − y) ∀x ∈ Dom(f )

if x < 0, g = −1

if x > 0, g = 1

if x = 0, g ∈ [−1, 1]

Asef Nazari Secant Method

Page 20: A Secant Method for Nonsmooth Optimization · 2014. 7. 12. · A Secant Method for Nonsmooth Optimization Asef Nazari CSIRO Melbourne CARMA Workshop on Optimization, Nonlinear Analysis,

OutlineBackground

Secant Method

The ProblemOptimization Algorithms and componentsSubgradient and Subdifferential

Subgradient

g ∈ Rn is a subgradient of a convex function f at y if

f (x) ≥ f (y) + gT .(x − y) ∀x ∈ Dom(f )

if x < 0, g = −1

if x > 0, g = 1

if x = 0, g ∈ [−1, 1]

Asef Nazari Secant Method

Page 21: A Secant Method for Nonsmooth Optimization · 2014. 7. 12. · A Secant Method for Nonsmooth Optimization Asef Nazari CSIRO Melbourne CARMA Workshop on Optimization, Nonlinear Analysis,

OutlineBackground

Secant Method

The ProblemOptimization Algorithms and componentsSubgradient and Subdifferential

Subgradient

g ∈ Rn is a subgradient of a convex function f at y if

f (x) ≥ f (y) + gT .(x − y) ∀x ∈ Dom(f )

if x < 0, g = −1

if x > 0, g = 1

if x = 0, g ∈ [−1, 1]

Asef Nazari Secant Method

Page 22: A Secant Method for Nonsmooth Optimization · 2014. 7. 12. · A Secant Method for Nonsmooth Optimization Asef Nazari CSIRO Melbourne CARMA Workshop on Optimization, Nonlinear Analysis,

OutlineBackground

Secant Method

The ProblemOptimization Algorithms and componentsSubgradient and Subdifferential

Subdifferential

Set of all subgradients of f at x , ∂f (x), is calledsubdifferential.

for f (x) = |x |, the subdifferential at x = 0 is ∂f (0) = [−1, 1]For x > 0 ∂f (x) = {1}For x < 0 ∂f (x) = {−1}f is differentiable at x ⇐⇒ ∂f (x) = {∇f (x)}for Lipschitz functions (Clarke)

∂f (x0) = conv{lim∇f (xi ) : xi −→ x0 ∇f (xi )exists}

Asef Nazari Secant Method

Page 23: A Secant Method for Nonsmooth Optimization · 2014. 7. 12. · A Secant Method for Nonsmooth Optimization Asef Nazari CSIRO Melbourne CARMA Workshop on Optimization, Nonlinear Analysis,

OutlineBackground

Secant Method

The ProblemOptimization Algorithms and componentsSubgradient and Subdifferential

Subdifferential

Set of all subgradients of f at x , ∂f (x), is calledsubdifferential.

for f (x) = |x |, the subdifferential at x = 0 is ∂f (0) = [−1, 1]For x > 0 ∂f (x) = {1}For x < 0 ∂f (x) = {−1}

f is differentiable at x ⇐⇒ ∂f (x) = {∇f (x)}for Lipschitz functions (Clarke)

∂f (x0) = conv{lim∇f (xi ) : xi −→ x0 ∇f (xi )exists}

Asef Nazari Secant Method

Page 24: A Secant Method for Nonsmooth Optimization · 2014. 7. 12. · A Secant Method for Nonsmooth Optimization Asef Nazari CSIRO Melbourne CARMA Workshop on Optimization, Nonlinear Analysis,

OutlineBackground

Secant Method

The ProblemOptimization Algorithms and componentsSubgradient and Subdifferential

Subdifferential

Set of all subgradients of f at x , ∂f (x), is calledsubdifferential.

for f (x) = |x |, the subdifferential at x = 0 is ∂f (0) = [−1, 1]For x > 0 ∂f (x) = {1}For x < 0 ∂f (x) = {−1}f is differentiable at x ⇐⇒ ∂f (x) = {∇f (x)}

for Lipschitz functions (Clarke)

∂f (x0) = conv{lim∇f (xi ) : xi −→ x0 ∇f (xi )exists}

Asef Nazari Secant Method

Page 25: A Secant Method for Nonsmooth Optimization · 2014. 7. 12. · A Secant Method for Nonsmooth Optimization Asef Nazari CSIRO Melbourne CARMA Workshop on Optimization, Nonlinear Analysis,

OutlineBackground

Secant Method

The ProblemOptimization Algorithms and componentsSubgradient and Subdifferential

Subdifferential

Set of all subgradients of f at x , ∂f (x), is calledsubdifferential.

for f (x) = |x |, the subdifferential at x = 0 is ∂f (0) = [−1, 1]For x > 0 ∂f (x) = {1}For x < 0 ∂f (x) = {−1}f is differentiable at x ⇐⇒ ∂f (x) = {∇f (x)}for Lipschitz functions (Clarke)

∂f (x0) = conv{lim∇f (xi ) : xi −→ x0 ∇f (xi )exists}

Asef Nazari Secant Method

Page 26: A Secant Method for Nonsmooth Optimization · 2014. 7. 12. · A Secant Method for Nonsmooth Optimization Asef Nazari CSIRO Melbourne CARMA Workshop on Optimization, Nonlinear Analysis,

OutlineBackground

Secant Method

The ProblemOptimization Algorithms and componentsSubgradient and Subdifferential

Optimality Condition

For a smooth convex function

f (x∗) = infx∈Dom(f )

f (x)⇐⇒ 0 = ∇f (x∗)

For a nonsmooth convex function

f (x∗) = infx∈Dom(f )

f (x)⇐⇒ 0 ∈ ∂f (x∗)

Asef Nazari Secant Method

Page 27: A Secant Method for Nonsmooth Optimization · 2014. 7. 12. · A Secant Method for Nonsmooth Optimization Asef Nazari CSIRO Melbourne CARMA Workshop on Optimization, Nonlinear Analysis,

OutlineBackground

Secant Method

The ProblemOptimization Algorithms and componentsSubgradient and Subdifferential

Optimality Condition

For a smooth convex function

f (x∗) = infx∈Dom(f )

f (x)⇐⇒ 0 = ∇f (x∗)

For a nonsmooth convex function

f (x∗) = infx∈Dom(f )

f (x)⇐⇒ 0 ∈ ∂f (x∗)

Asef Nazari Secant Method

Page 28: A Secant Method for Nonsmooth Optimization · 2014. 7. 12. · A Secant Method for Nonsmooth Optimization Asef Nazari CSIRO Melbourne CARMA Workshop on Optimization, Nonlinear Analysis,

OutlineBackground

Secant Method

The ProblemOptimization Algorithms and componentsSubgradient and Subdifferential

Directional derivative

In Smooth Case

f ′(xk , d) = limλ→0+

f (xk + λd)− f (xk)

λ= 〈∇f (xk), d〉

In nonsmooth case

f ′(xk , d) = limλ→0+

f (xk + λd)− f (xk)

λ= sup

g∈∂f (xk )〈gT , d〉

Asef Nazari Secant Method

Page 29: A Secant Method for Nonsmooth Optimization · 2014. 7. 12. · A Secant Method for Nonsmooth Optimization Asef Nazari CSIRO Melbourne CARMA Workshop on Optimization, Nonlinear Analysis,

OutlineBackground

Secant Method

The ProblemOptimization Algorithms and componentsSubgradient and Subdifferential

Directional derivative

In Smooth Case

f ′(xk , d) = limλ→0+

f (xk + λd)− f (xk)

λ= 〈∇f (xk), d〉

In nonsmooth case

f ′(xk , d) = limλ→0+

f (xk + λd)− f (xk)

λ= sup

g∈∂f (xk )〈gT , d〉

Asef Nazari Secant Method

Page 30: A Secant Method for Nonsmooth Optimization · 2014. 7. 12. · A Secant Method for Nonsmooth Optimization Asef Nazari CSIRO Melbourne CARMA Workshop on Optimization, Nonlinear Analysis,

OutlineBackground

Secant Method

The ProblemOptimization Algorithms and componentsSubgradient and Subdifferential

Descent Direction

In Smooth Case, if f ′(xk , d) = 〈∇f (xk), d〉 < 0. (−∇f (xk)steepest descent direction)

In nonsmooth case, if f ′(xk , d) < 0. It is proved that

d = −argming∈∂f (xk )

‖g‖

Asef Nazari Secant Method

Page 31: A Secant Method for Nonsmooth Optimization · 2014. 7. 12. · A Secant Method for Nonsmooth Optimization Asef Nazari CSIRO Melbourne CARMA Workshop on Optimization, Nonlinear Analysis,

OutlineBackground

Secant Method

The ProblemOptimization Algorithms and componentsSubgradient and Subdifferential

Descent Direction

In Smooth Case, if f ′(xk , d) = 〈∇f (xk), d〉 < 0. (−∇f (xk)steepest descent direction)

In nonsmooth case, if f ′(xk , d) < 0. It is proved that

d = −argming∈∂f (xk )

‖g‖

Asef Nazari Secant Method

Page 32: A Secant Method for Nonsmooth Optimization · 2014. 7. 12. · A Secant Method for Nonsmooth Optimization Asef Nazari CSIRO Melbourne CARMA Workshop on Optimization, Nonlinear Analysis,

OutlineBackground

Secant Method

The ProblemOptimization Algorithms and componentsSubgradient and Subdifferential

In Summary

Optimality condition

f (x∗) = infx∈Dom(f )

f (x)⇐⇒ 0 ∈ ∂f (x∗)

Directional Derivative

f ′(xk , d) = limλ→0+

f (xk + λd)− f (xk)

λ= sup

g∈∂f (xk )〈gT , d〉

Steepest Descent Direction

d = −argming∈∂f (xk )

‖g‖

Asef Nazari Secant Method

Page 33: A Secant Method for Nonsmooth Optimization · 2014. 7. 12. · A Secant Method for Nonsmooth Optimization Asef Nazari CSIRO Melbourne CARMA Workshop on Optimization, Nonlinear Analysis,

OutlineBackground

Secant Method

The ProblemOptimization Algorithms and componentsSubgradient and Subdifferential

Methods for nonsmooth problems

The way we treat ∂f (x) leads to different types of algorithms innonsmooth optimization

Subgradient method (one subgradient at each iteration)

Bundle method (a bundle of subgradients in each iteration)

Gradient Sampling method

smoothing technique

Asef Nazari Secant Method

Page 34: A Secant Method for Nonsmooth Optimization · 2014. 7. 12. · A Secant Method for Nonsmooth Optimization Asef Nazari CSIRO Melbourne CARMA Workshop on Optimization, Nonlinear Analysis,

OutlineBackground

Secant Method

DefinitionsOptimality Condition and Descent DirectionThe Secant AlgorithmNumerical Results

Definition of r -secant

S1 = {g ∈ Rn : ‖g‖ = 1} the unit sphere in Rn

g ∈ S1 we define

gmax = max {|gi |, i = 1, . . . , n} .

g ∈ S1 and gj = gmax , v ∈ ∂f (x + rg)

s = s(x , g , r) ∈ Rn where

s = (s1, . . . , sn) : si = vi , i = 1, . . . , n, i 6= j

and

sj =f (x + rg)− f (x)− r

∑ni=1,i 6=j sigi

rgj

is called an r -secant of the function f at a point x in thedirection g .

Asef Nazari Secant Method

Page 35: A Secant Method for Nonsmooth Optimization · 2014. 7. 12. · A Secant Method for Nonsmooth Optimization Asef Nazari CSIRO Melbourne CARMA Workshop on Optimization, Nonlinear Analysis,

OutlineBackground

Secant Method

DefinitionsOptimality Condition and Descent DirectionThe Secant AlgorithmNumerical Results

Definition of r -secant

S1 = {g ∈ Rn : ‖g‖ = 1} the unit sphere in Rn

g ∈ S1 we define

gmax = max {|gi |, i = 1, . . . , n} .

g ∈ S1 and gj = gmax , v ∈ ∂f (x + rg)

s = s(x , g , r) ∈ Rn where

s = (s1, . . . , sn) : si = vi , i = 1, . . . , n, i 6= j

and

sj =f (x + rg)− f (x)− r

∑ni=1,i 6=j sigi

rgj

is called an r -secant of the function f at a point x in thedirection g .

Asef Nazari Secant Method

Page 36: A Secant Method for Nonsmooth Optimization · 2014. 7. 12. · A Secant Method for Nonsmooth Optimization Asef Nazari CSIRO Melbourne CARMA Workshop on Optimization, Nonlinear Analysis,

OutlineBackground

Secant Method

DefinitionsOptimality Condition and Descent DirectionThe Secant AlgorithmNumerical Results

Definition of r -secant

S1 = {g ∈ Rn : ‖g‖ = 1} the unit sphere in Rn

g ∈ S1 we define

gmax = max {|gi |, i = 1, . . . , n} .

g ∈ S1 and gj = gmax , v ∈ ∂f (x + rg)

s = s(x , g , r) ∈ Rn where

s = (s1, . . . , sn) : si = vi , i = 1, . . . , n, i 6= j

and

sj =f (x + rg)− f (x)− r

∑ni=1,i 6=j sigi

rgj

is called an r -secant of the function f at a point x in thedirection g .

Asef Nazari Secant Method

Page 37: A Secant Method for Nonsmooth Optimization · 2014. 7. 12. · A Secant Method for Nonsmooth Optimization Asef Nazari CSIRO Melbourne CARMA Workshop on Optimization, Nonlinear Analysis,

OutlineBackground

Secant Method

DefinitionsOptimality Condition and Descent DirectionThe Secant AlgorithmNumerical Results

Definition of r -secant

S1 = {g ∈ Rn : ‖g‖ = 1} the unit sphere in Rn

g ∈ S1 we define

gmax = max {|gi |, i = 1, . . . , n} .

g ∈ S1 and gj = gmax , v ∈ ∂f (x + rg)

s = s(x , g , r) ∈ Rn where

s = (s1, . . . , sn) : si = vi , i = 1, . . . , n, i 6= j

and

sj =f (x + rg)− f (x)− r

∑ni=1,i 6=j sigi

rgj

is called an r -secant of the function f at a point x in thedirection g .

Asef Nazari Secant Method

Page 38: A Secant Method for Nonsmooth Optimization · 2014. 7. 12. · A Secant Method for Nonsmooth Optimization Asef Nazari CSIRO Melbourne CARMA Workshop on Optimization, Nonlinear Analysis,

OutlineBackground

Secant Method

DefinitionsOptimality Condition and Descent DirectionThe Secant AlgorithmNumerical Results

Facts about r -secant

If n = 1,

s =f (x + rg)− f (x)

rg

Mean Value Theorem for r -secants

f (x + rg)− f (x) = r〈s(x , g , r), g〉

Set of all possible r -secants of the function f at the point x

Sr f (x) = {s ∈ Rn : ∃g ∈ S1 : s = s(x , g , r)}

1 compact for r > 02 (x , r) 7→ Sr f (x), r > 0 is closed and upper semi-continuous

Asef Nazari Secant Method

Page 39: A Secant Method for Nonsmooth Optimization · 2014. 7. 12. · A Secant Method for Nonsmooth Optimization Asef Nazari CSIRO Melbourne CARMA Workshop on Optimization, Nonlinear Analysis,

OutlineBackground

Secant Method

DefinitionsOptimality Condition and Descent DirectionThe Secant AlgorithmNumerical Results

Facts about r -secant

If n = 1,

s =f (x + rg)− f (x)

rg

Mean Value Theorem for r -secants

f (x + rg)− f (x) = r〈s(x , g , r), g〉

Set of all possible r -secants of the function f at the point x

Sr f (x) = {s ∈ Rn : ∃g ∈ S1 : s = s(x , g , r)}

1 compact for r > 02 (x , r) 7→ Sr f (x), r > 0 is closed and upper semi-continuous

Asef Nazari Secant Method

Page 40: A Secant Method for Nonsmooth Optimization · 2014. 7. 12. · A Secant Method for Nonsmooth Optimization Asef Nazari CSIRO Melbourne CARMA Workshop on Optimization, Nonlinear Analysis,

OutlineBackground

Secant Method

DefinitionsOptimality Condition and Descent DirectionThe Secant AlgorithmNumerical Results

Facts about r -secant

If n = 1,

s =f (x + rg)− f (x)

rg

Mean Value Theorem for r -secants

f (x + rg)− f (x) = r〈s(x , g , r), g〉

Set of all possible r -secants of the function f at the point x

Sr f (x) = {s ∈ Rn : ∃g ∈ S1 : s = s(x , g , r)}

1 compact for r > 02 (x , r) 7→ Sr f (x), r > 0 is closed and upper semi-continuous

Asef Nazari Secant Method

Page 41: A Secant Method for Nonsmooth Optimization · 2014. 7. 12. · A Secant Method for Nonsmooth Optimization Asef Nazari CSIRO Melbourne CARMA Workshop on Optimization, Nonlinear Analysis,

OutlineBackground

Secant Method

DefinitionsOptimality Condition and Descent DirectionThe Secant AlgorithmNumerical Results

Facts about r -secant

If n = 1,

s =f (x + rg)− f (x)

rg

Mean Value Theorem for r -secants

f (x + rg)− f (x) = r〈s(x , g , r), g〉

Set of all possible r -secants of the function f at the point x

Sr f (x) = {s ∈ Rn : ∃g ∈ S1 : s = s(x , g , r)}

1 compact for r > 0

2 (x , r) 7→ Sr f (x), r > 0 is closed and upper semi-continuous

Asef Nazari Secant Method

Page 42: A Secant Method for Nonsmooth Optimization · 2014. 7. 12. · A Secant Method for Nonsmooth Optimization Asef Nazari CSIRO Melbourne CARMA Workshop on Optimization, Nonlinear Analysis,

OutlineBackground

Secant Method

DefinitionsOptimality Condition and Descent DirectionThe Secant AlgorithmNumerical Results

Facts about r -secant

If n = 1,

s =f (x + rg)− f (x)

rg

Mean Value Theorem for r -secants

f (x + rg)− f (x) = r〈s(x , g , r), g〉

Set of all possible r -secants of the function f at the point x

Sr f (x) = {s ∈ Rn : ∃g ∈ S1 : s = s(x , g , r)}

1 compact for r > 02 (x , r) 7→ Sr f (x), r > 0 is closed and upper semi-continuous

Asef Nazari Secant Method

Page 43: A Secant Method for Nonsmooth Optimization · 2014. 7. 12. · A Secant Method for Nonsmooth Optimization Asef Nazari CSIRO Melbourne CARMA Workshop on Optimization, Nonlinear Analysis,

OutlineBackground

Secant Method

DefinitionsOptimality Condition and Descent DirectionThe Secant AlgorithmNumerical Results

A new set

Sc0 f (x) = conv{v ∈ Rn : ∃(g ∈ S1, rk → +0, k → +∞) :

v = limk→+∞

s(x , g , rk)},

For regular and semismooth function f at a point x ∈ Rn:

∂f (x) = Sc0 f (x).

Asef Nazari Secant Method

Page 44: A Secant Method for Nonsmooth Optimization · 2014. 7. 12. · A Secant Method for Nonsmooth Optimization Asef Nazari CSIRO Melbourne CARMA Workshop on Optimization, Nonlinear Analysis,

OutlineBackground

Secant Method

DefinitionsOptimality Condition and Descent DirectionThe Secant AlgorithmNumerical Results

A new set

Sc0 f (x) = conv{v ∈ Rn : ∃(g ∈ S1, rk → +0, k → +∞) :

v = limk→+∞

s(x , g , rk)},

For regular and semismooth function f at a point x ∈ Rn:

∂f (x) = Sc0 f (x).

Asef Nazari Secant Method

Page 45: A Secant Method for Nonsmooth Optimization · 2014. 7. 12. · A Secant Method for Nonsmooth Optimization Asef Nazari CSIRO Melbourne CARMA Workshop on Optimization, Nonlinear Analysis,

OutlineBackground

Secant Method

DefinitionsOptimality Condition and Descent DirectionThe Secant AlgorithmNumerical Results

Optimality condition

Let x ∈ Rn be a local minimizer of the function f and it isdirectionally differentiable at x . Then

0 ∈ Sc0 f (x).

x ∈ Rn is an r -stationary point for a function f on Rn if0 ∈ Sc

r f (x).

x ∈ Rn is an (r , δ)-stationary point for a function f on Rn if0 ∈ Sc

r f (x) + Bδ where

Bδ = {v ∈ Rn : ‖v‖ ≤ δ}.

Asef Nazari Secant Method

Page 46: A Secant Method for Nonsmooth Optimization · 2014. 7. 12. · A Secant Method for Nonsmooth Optimization Asef Nazari CSIRO Melbourne CARMA Workshop on Optimization, Nonlinear Analysis,

OutlineBackground

Secant Method

DefinitionsOptimality Condition and Descent DirectionThe Secant AlgorithmNumerical Results

Descent Direction

If x ∈ Rn is not an r -stationary point of a function f on Rn,

0 6∈ Scr f (x).

we can compute a descent direction using the set Scr f (x)

Let x ∈ Rn and for given r > 0

min{‖v‖ : v ∈ Scr f (x)} = ‖v0‖ > 0.

Then for g0 = −‖v0‖−1v0

f (x + rg0)− f (x) ≤ −r‖v0‖.

minimize ‖v‖2

subjectto v ∈ Scr f (x).

Asef Nazari Secant Method

Page 47: A Secant Method for Nonsmooth Optimization · 2014. 7. 12. · A Secant Method for Nonsmooth Optimization Asef Nazari CSIRO Melbourne CARMA Workshop on Optimization, Nonlinear Analysis,

OutlineBackground

Secant Method

DefinitionsOptimality Condition and Descent DirectionThe Secant AlgorithmNumerical Results

Descent Direction

If x ∈ Rn is not an r -stationary point of a function f on Rn,

0 6∈ Scr f (x).

we can compute a descent direction using the set Scr f (x)

Let x ∈ Rn and for given r > 0

min{‖v‖ : v ∈ Scr f (x)} = ‖v0‖ > 0.

Then for g0 = −‖v0‖−1v0

f (x + rg0)− f (x) ≤ −r‖v0‖.

minimize ‖v‖2

subjectto v ∈ Scr f (x).

Asef Nazari Secant Method

Page 48: A Secant Method for Nonsmooth Optimization · 2014. 7. 12. · A Secant Method for Nonsmooth Optimization Asef Nazari CSIRO Melbourne CARMA Workshop on Optimization, Nonlinear Analysis,

OutlineBackground

Secant Method

DefinitionsOptimality Condition and Descent DirectionThe Secant AlgorithmNumerical Results

Descent Direction

If x ∈ Rn is not an r -stationary point of a function f on Rn,

0 6∈ Scr f (x).

we can compute a descent direction using the set Scr f (x)

Let x ∈ Rn and for given r > 0

min{‖v‖ : v ∈ Scr f (x)} = ‖v0‖ > 0.

Then for g0 = −‖v0‖−1v0

f (x + rg0)− f (x) ≤ −r‖v0‖.

minimize ‖v‖2

subjectto v ∈ Scr f (x).

Asef Nazari Secant Method

Page 49: A Secant Method for Nonsmooth Optimization · 2014. 7. 12. · A Secant Method for Nonsmooth Optimization Asef Nazari CSIRO Melbourne CARMA Workshop on Optimization, Nonlinear Analysis,

OutlineBackground

Secant Method

DefinitionsOptimality Condition and Descent DirectionThe Secant AlgorithmNumerical Results

An algorithm for descent direction (Alg1)

step 1. compute an r -secant s1 = s(x , g1, r) . Set W 1(x) = {s1} andk = 1.

step 2. ‖wk‖2 = min{‖w‖2 : w ∈ coW k(x)}. If

‖wk‖ ≤ δ,

then stop. Otherwise go to Step 3.

step 3. gk+1 = −‖wk‖−1wk .

step 4. f (x + rgk+1)− f (x) ≤ −cr‖wk‖, then stop. Otherwise go toStep 5.

step 5. sk+1 = s(x , gk+1, r) with respect to the direction gk+1,construct the set W k+1(x) = co

{W k(x)

⋃{sk+1}

}, set

k = k + 1 and go to Step 2.

Asef Nazari Secant Method

Page 50: A Secant Method for Nonsmooth Optimization · 2014. 7. 12. · A Secant Method for Nonsmooth Optimization Asef Nazari CSIRO Melbourne CARMA Workshop on Optimization, Nonlinear Analysis,

OutlineBackground

Secant Method

DefinitionsOptimality Condition and Descent DirectionThe Secant AlgorithmNumerical Results

An algorithm for descent direction (Alg1)

step 1. compute an r -secant s1 = s(x , g1, r) . Set W 1(x) = {s1} andk = 1.

step 2. ‖wk‖2 = min{‖w‖2 : w ∈ coW k(x)}. If

‖wk‖ ≤ δ,

then stop. Otherwise go to Step 3.

step 3. gk+1 = −‖wk‖−1wk .

step 4. f (x + rgk+1)− f (x) ≤ −cr‖wk‖, then stop. Otherwise go toStep 5.

step 5. sk+1 = s(x , gk+1, r) with respect to the direction gk+1,construct the set W k+1(x) = co

{W k(x)

⋃{sk+1}

}, set

k = k + 1 and go to Step 2.

Asef Nazari Secant Method

Page 51: A Secant Method for Nonsmooth Optimization · 2014. 7. 12. · A Secant Method for Nonsmooth Optimization Asef Nazari CSIRO Melbourne CARMA Workshop on Optimization, Nonlinear Analysis,

OutlineBackground

Secant Method

DefinitionsOptimality Condition and Descent DirectionThe Secant AlgorithmNumerical Results

An algorithm for descent direction (Alg1)

step 1. compute an r -secant s1 = s(x , g1, r) . Set W 1(x) = {s1} andk = 1.

step 2. ‖wk‖2 = min{‖w‖2 : w ∈ coW k(x)}. If

‖wk‖ ≤ δ,

then stop. Otherwise go to Step 3.

step 3. gk+1 = −‖wk‖−1wk .

step 4. f (x + rgk+1)− f (x) ≤ −cr‖wk‖, then stop. Otherwise go toStep 5.

step 5. sk+1 = s(x , gk+1, r) with respect to the direction gk+1,construct the set W k+1(x) = co

{W k(x)

⋃{sk+1}

}, set

k = k + 1 and go to Step 2.

Asef Nazari Secant Method

Page 52: A Secant Method for Nonsmooth Optimization · 2014. 7. 12. · A Secant Method for Nonsmooth Optimization Asef Nazari CSIRO Melbourne CARMA Workshop on Optimization, Nonlinear Analysis,

OutlineBackground

Secant Method

DefinitionsOptimality Condition and Descent DirectionThe Secant AlgorithmNumerical Results

An algorithm for descent direction (Alg1)

step 1. compute an r -secant s1 = s(x , g1, r) . Set W 1(x) = {s1} andk = 1.

step 2. ‖wk‖2 = min{‖w‖2 : w ∈ coW k(x)}. If

‖wk‖ ≤ δ,

then stop. Otherwise go to Step 3.

step 3. gk+1 = −‖wk‖−1wk .

step 4. f (x + rgk+1)− f (x) ≤ −cr‖wk‖, then stop. Otherwise go toStep 5.

step 5. sk+1 = s(x , gk+1, r) with respect to the direction gk+1,construct the set W k+1(x) = co

{W k(x)

⋃{sk+1}

}, set

k = k + 1 and go to Step 2.

Asef Nazari Secant Method

Page 53: A Secant Method for Nonsmooth Optimization · 2014. 7. 12. · A Secant Method for Nonsmooth Optimization Asef Nazari CSIRO Melbourne CARMA Workshop on Optimization, Nonlinear Analysis,

OutlineBackground

Secant Method

DefinitionsOptimality Condition and Descent DirectionThe Secant AlgorithmNumerical Results

An algorithm for descent direction (Alg1)

step 1. compute an r -secant s1 = s(x , g1, r) . Set W 1(x) = {s1} andk = 1.

step 2. ‖wk‖2 = min{‖w‖2 : w ∈ coW k(x)}. If

‖wk‖ ≤ δ,

then stop. Otherwise go to Step 3.

step 3. gk+1 = −‖wk‖−1wk .

step 4. f (x + rgk+1)− f (x) ≤ −cr‖wk‖, then stop. Otherwise go toStep 5.

step 5. sk+1 = s(x , gk+1, r) with respect to the direction gk+1,construct the set W k+1(x) = co

{W k(x)

⋃{sk+1}

}, set

k = k + 1 and go to Step 2.

Asef Nazari Secant Method

Page 54: A Secant Method for Nonsmooth Optimization · 2014. 7. 12. · A Secant Method for Nonsmooth Optimization Asef Nazari CSIRO Melbourne CARMA Workshop on Optimization, Nonlinear Analysis,

OutlineBackground

Secant Method

DefinitionsOptimality Condition and Descent DirectionThe Secant AlgorithmNumerical Results

The secant method (r , δ)-stationary point(Alg 2)

step 1. x0 ∈ Rn and set k = 0.

step 2. Apply Alg 1 at x = xk

Either ‖vk‖ ≤ δ or for the search direction gk = −‖vk‖−1vk

step 3. If ‖vk‖ ≤ δ then stop. Otherwise go to Step 4.

step 4. xk+1 = xk + σkgk , where σk is defined as follows

σk = arg max{σ ≥ 0 : f (xk + σgk)− f (xk) ≤ −c2σ‖vk‖

}.

Set k = k + 1 and go to Step 2.

Asef Nazari Secant Method

Page 55: A Secant Method for Nonsmooth Optimization · 2014. 7. 12. · A Secant Method for Nonsmooth Optimization Asef Nazari CSIRO Melbourne CARMA Workshop on Optimization, Nonlinear Analysis,

OutlineBackground

Secant Method

DefinitionsOptimality Condition and Descent DirectionThe Secant AlgorithmNumerical Results

The secant method (r , δ)-stationary point(Alg 2)

step 1. x0 ∈ Rn and set k = 0.

step 2. Apply Alg 1 at x = xk

Either ‖vk‖ ≤ δ or for the search direction gk = −‖vk‖−1vk

step 3. If ‖vk‖ ≤ δ then stop. Otherwise go to Step 4.

step 4. xk+1 = xk + σkgk , where σk is defined as follows

σk = arg max{σ ≥ 0 : f (xk + σgk)− f (xk) ≤ −c2σ‖vk‖

}.

Set k = k + 1 and go to Step 2.

Asef Nazari Secant Method

Page 56: A Secant Method for Nonsmooth Optimization · 2014. 7. 12. · A Secant Method for Nonsmooth Optimization Asef Nazari CSIRO Melbourne CARMA Workshop on Optimization, Nonlinear Analysis,

OutlineBackground

Secant Method

DefinitionsOptimality Condition and Descent DirectionThe Secant AlgorithmNumerical Results

The secant method (r , δ)-stationary point(Alg 2)

step 1. x0 ∈ Rn and set k = 0.

step 2. Apply Alg 1 at x = xk

Either ‖vk‖ ≤ δ or for the search direction gk = −‖vk‖−1vk

step 3. If ‖vk‖ ≤ δ then stop. Otherwise go to Step 4.

step 4. xk+1 = xk + σkgk , where σk is defined as follows

σk = arg max{σ ≥ 0 : f (xk + σgk)− f (xk) ≤ −c2σ‖vk‖

}.

Set k = k + 1 and go to Step 2.

Asef Nazari Secant Method

Page 57: A Secant Method for Nonsmooth Optimization · 2014. 7. 12. · A Secant Method for Nonsmooth Optimization Asef Nazari CSIRO Melbourne CARMA Workshop on Optimization, Nonlinear Analysis,

OutlineBackground

Secant Method

DefinitionsOptimality Condition and Descent DirectionThe Secant AlgorithmNumerical Results

The secant method (r , δ)-stationary point(Alg 2)

step 1. x0 ∈ Rn and set k = 0.

step 2. Apply Alg 1 at x = xk

Either ‖vk‖ ≤ δ or for the search direction gk = −‖vk‖−1vk

step 3. If ‖vk‖ ≤ δ then stop. Otherwise go to Step 4.

step 4. xk+1 = xk + σkgk , where σk is defined as follows

σk = arg max{σ ≥ 0 : f (xk + σgk)− f (xk) ≤ −c2σ‖vk‖

}.

Set k = k + 1 and go to Step 2.

Asef Nazari Secant Method

Page 58: A Secant Method for Nonsmooth Optimization · 2014. 7. 12. · A Secant Method for Nonsmooth Optimization Asef Nazari CSIRO Melbourne CARMA Workshop on Optimization, Nonlinear Analysis,

OutlineBackground

Secant Method

DefinitionsOptimality Condition and Descent DirectionThe Secant AlgorithmNumerical Results

The secant method (Alg 3)

step 1. x0 ∈ Rn and set k = 0.

step 2. Apply Alg 2 to xk for r = rk and δ = δk . This algorithmterminates after a finite number of iterations p > 0 and as aresult the algorithm finds (rk , δk)-stationary point xk+1.

step 3. Set k = k + 1 and go to Step 2.

Asef Nazari Secant Method

Page 59: A Secant Method for Nonsmooth Optimization · 2014. 7. 12. · A Secant Method for Nonsmooth Optimization Asef Nazari CSIRO Melbourne CARMA Workshop on Optimization, Nonlinear Analysis,

OutlineBackground

Secant Method

DefinitionsOptimality Condition and Descent DirectionThe Secant AlgorithmNumerical Results

The secant method (Alg 3)

step 1. x0 ∈ Rn and set k = 0.

step 2. Apply Alg 2 to xk for r = rk and δ = δk . This algorithmterminates after a finite number of iterations p > 0 and as aresult the algorithm finds (rk , δk)-stationary point xk+1.

step 3. Set k = k + 1 and go to Step 2.

Asef Nazari Secant Method

Page 60: A Secant Method for Nonsmooth Optimization · 2014. 7. 12. · A Secant Method for Nonsmooth Optimization Asef Nazari CSIRO Melbourne CARMA Workshop on Optimization, Nonlinear Analysis,

OutlineBackground

Secant Method

DefinitionsOptimality Condition and Descent DirectionThe Secant AlgorithmNumerical Results

The secant method (Alg 3)

step 1. x0 ∈ Rn and set k = 0.

step 2. Apply Alg 2 to xk for r = rk and δ = δk . This algorithmterminates after a finite number of iterations p > 0 and as aresult the algorithm finds (rk , δk)-stationary point xk+1.

step 3. Set k = k + 1 and go to Step 2.

Asef Nazari Secant Method

Page 61: A Secant Method for Nonsmooth Optimization · 2014. 7. 12. · A Secant Method for Nonsmooth Optimization Asef Nazari CSIRO Melbourne CARMA Workshop on Optimization, Nonlinear Analysis,

OutlineBackground

Secant Method

DefinitionsOptimality Condition and Descent DirectionThe Secant AlgorithmNumerical Results

Convergence Theorem

Theorem

Assume that the function f is locally Lipschitz and the set L(x0) isbounded for starting points x0 ∈ Rn. Then every accumulationpoint of the sequence {xk} belongs to the setX 0 = {x ∈ Rn : 0 ∈ ∂f (x)}.

Asef Nazari Secant Method

Page 62: A Secant Method for Nonsmooth Optimization · 2014. 7. 12. · A Secant Method for Nonsmooth Optimization Asef Nazari CSIRO Melbourne CARMA Workshop on Optimization, Nonlinear Analysis,

OutlineBackground

Secant Method

DefinitionsOptimality Condition and Descent DirectionThe Secant AlgorithmNumerical Results

Results

Asef Nazari Secant Method

Page 63: A Secant Method for Nonsmooth Optimization · 2014. 7. 12. · A Secant Method for Nonsmooth Optimization Asef Nazari CSIRO Melbourne CARMA Workshop on Optimization, Nonlinear Analysis,

OutlineBackground

Secant Method

DefinitionsOptimality Condition and Descent DirectionThe Secant AlgorithmNumerical Results

Results

Asef Nazari Secant Method