Upload
lenhi
View
236
Download
2
Embed Size (px)
Citation preview
Lecture 3
Outline
• Convex Functions
• Examples
• Verifying Convexity of a Function
• Operations on Functions Preserving Convexity
Convex Optimization 1
Lecture 3
Convex Functions
Informally: f is convex when for every segment [x1, x2], as xα =
αx1+(1−α)x2 varies over the line segment [x1, x2], the points (xα, f(xα))
lie below the segment connecting (x1, f(x1)) and (x2, f(x2))
Let f be a function from Rn to R, f : Rn → RThe domain of f is a set in Rn defined by
dom(f) = {x ∈ Rn | f(x) is well defined (finite)}Def. A function f is convex if
(1) Its domain dom(f) is a convex set in Rn and
(2) For all x1, x2 ∈ dom(f) and α ∈ (0,1)
f(αx1 + (1− α)x2) ≤ αf(x1) + (1− α)f(x2)
Convex Optimization 2
Lecture 3
More on Convex Function
Def. A function f is strictly convex when dom(f) is convex and
f(αx1 + (1− α)x2) < αf(x1) + (1− α)f(x2)
for all x1, x2 ∈ dom(f) and α ∈ (0,1)
Def. A function f is concave when −f is convex, i.e.,
(1) Its domain dom(f) is a convex set in Rn and
(2) For all x1, x2 ∈ dom(f) and α ∈ (0,1)
f(αx1 + (1− α)x2) ≥ αf(x1) + (1− α)f(x2)
Def. A function f is strictly concave when −f is strictly convex
Convex Optimization 3
Lecture 3
Examples on R
Convex:
• Affine: ax + b over R for any a, b ∈ R• Exponential: eax over R for any a ∈ R• Power: xp over (0,+∞) for p ≥ 1 or p ≤ 0
• Powers of absolute value: |x|p over R for p ≥ 1
• Negative entropy: x lnx over (0,+∞)
Concave:
• Affine: ax + b over R for any a, b ∈ R• Powers: xp over (0,+∞) for 0 ≤ p ≤ 1
• Logarithm: lnx over (0,+∞)
Convex Optimization 4
Lecture 3
Examples: Affine Functions and Norms
• Affine functions are both convex and concave
• Norms are convex
Examples on Rn
• Affine function f(x) = a′x + b with a ∈ Rn and b ∈ R
• Euclidean, l1, and l∞ norms
• General lp norms
‖x‖p =
(n∑
i=1
|xi|p)1/p
for p ≥ 1
Convex Optimization 5
Lecture 3
Examples on Rm×n
The space Rm×n is the space of m× n matrices
• Affine function
f(X) = tr(ATX) + b =m∑
i=1
n∑j=1
aijxij + b
• Spectral (maximum singular value) norm
f(X) = ‖X‖2 = σmax(X) =√
λmax(XTX)
where λmax(A) denotes the maximum eigenvalue of a matrix A
Convex Optimization 6
Lecture 3
Verifying Convexity of a Function
We can verify that a given function f is convex by
• Using the definition
• Applying some special criteria
• Second-order conditions
• First-order conditions
• Reduction to a scalar function
• Showing that f is obtained through operations preserving convexity
Convex Optimization 7
Lecture 3
Second-Order Conditions
Let f be twice differentiable and let dom(f) = Rn [in general, it is
required that dom(f) is open]
The Hessian ∇2f(x) is a symmetric n × n matrix whose entries are the
second-order partial derivatives of f at x:
[∇2f(x)
]ij
=∂2f(x)
∂xi∂xj
for i, j = 1, . . . , n
2nd-order conditions: For a twice differentiable f with convex domain
• f is convex if and only if
∇2f(x) � 0 for all x ∈ dom(f)
• f is strictly convex if
∇2f(x) � 0 for all x ∈ dom(f)
Convex Optimization 8
Lecture 3
Examples
Quadratic function: f(x) = (1/2)x′Px + q′x + r with a symmetric
n× n matrix P
∇f(x) = Px + q, ∇2f(x) = P
Convex for P � 0
Least-squares objective: f(x) = ‖Ax− b‖2 with an m× n matrix A
∇f(x) = 2AT(Ax− b), ∇2f(x) = 2ATA
Convex for any A
Quadratic-over-linear: f(x, y) = x2/y
∇2f(x, y) = 2y3
[y
−x
] [y
−x
]T
� 0
Convex for y > 0
Convex Optimization 9
Lecture 3
Verifying Convexity of a Function
We can verify that a given function f is convex by
• Using the definition
• Applying some special criteria
• Second-order conditions
• First-order conditions
• Reduction to a scalar function
• Showing that f is obtained through operations preserving convexity
Convex Optimization 10
Lecture 3
First-Order Condition
f is differentiable if dom(f) is open and the gradient
∇f(x) =
(∂f(x)
∂x1,∂f(x)
∂x2, . . . ,
∂f(x)
∂xn
)
exists at each x ∈ domf
1st-order condition: differentiable f is convex if and only if its domain isconvex and
f(x) +∇f(x)T(z − x) ≤ f(z) for all x, z ∈ dom(f)
A first order approximation is a global underestimate of f
Very important property used in algorithm designs and performance analysis
Convex Optimization 11
Lecture 3
Restriction of a convex function to a line
f is convex if and only if domf is convex and the function g : R → R,
g(t) = f(x + tv), domg = {t | x + tv ∈ dom(f)}
is convex (in t) for any x ∈ domf , v ∈ Rn
Checking convexity of multivariable functions can be done by checkingconvexity of functions of one variableExample f : Sn → R with f(X) = − ln detX, domf = Sn
++
g(t) = − ln det(X + tV ) = − ln detX − ln det(I + tX−1/2V X−1/2)
= − ln detX −n∑
i=1
ln(1 + tλi)
where λi are the eigenvalues of X−1/2V X−1/2
g is convex in t (for any choice of V and any X � 0); hence f is concave
Convex Optimization 12
Lecture 3
Operations Preserving Convexity
• Positive Scaling
• Sum
• Composition with affine function
• Pointwise maximum and supremum
• Composition
• Minimization
Convex Optimization 13
Lecture 3
Scaling, Sum, & Composition with Affine Function
Positive multiple For a convex f and λ > 0, the function λf is convex
Sum: For convex f1 and f2, the sum f1 + f2 is convex
(extends to infinite sums, integrals)
Composition with affine function: For a convex f and affine g [i.e.,
g(x) = Ax + b], the composition f ◦ g is convex, where
(f ◦ g)(x) = f(Ax + b)
Examples• Log-barrier for linear inequalities
f(x) = −m∑
i=1
ln(bi−aTi x), domf = {x | aT
i x < bi, i = 1, . . . , m}
• (Any) Norm of affine function: f(x) = ‖Ax + b‖
Convex Optimization 14
Lecture 3
Pointwise maximum
For convex functions f1, . . . , fm, the pointwise-max functionF (x) = max {f1(x), . . . , fm(x)}
is convex (What is domain of F ?)
Examples
• Piecewise-linear function: f(x) = maxi=1,...,m(aTi x + bi) is convex
• Sum of r largest components of a vector x ∈ Rn:
f(x) = x[1] + x[2] + · · ·+ x[r]
is convex (x[i] is i-th largest component of x)
f(x) = max(i1,...,ir)∈Ir
{xi1 + xi2 + · · ·+ xir}
Ir = {(i1, . . . , ir) | i1 < . . . < ir, ij ∈ {1, . . . , m}, j = 1, . . . , n}
Convex Optimization 15
Lecture 3
Pointwise Supremum
Let A ⊆ Rp and f : Rn × Rp → R. Let f(x, z) be convex in x for
each z ∈ A. Then, the supremum function over the set A is convex:
g(x) = supz∈A f(x, z)
Examples
• Set support function is convex for a set C ⊂ Rn,
SC : Rn → R, SC(x) = supz∈C zTx
• Set farthest-distance function is convex for a set C ⊂ Rn,
f : Rn → R, f(x) = supz∈C ‖x− z‖
• Maximum eigenvalue function of a symmetric matrix is convexλmax : Sn → R, λmax(X) = sup‖z‖=1 zTXz
Convex Optimization 16
Lecture 3
Composition with Scalar Functions
Composition of g : Rn → R and h : R → R with dom(g) = Rn and
dom(h) = R:
f(x) = h(g(x))
f is convex if
(1) g is convex, h is nondecreasing and convex
(2) g is concave, h is nonincreasing and convex
Examples
• eg(x) is convex if g is convex
• 1g(x)
is convex if g is concave and positive
Convex Optimization 17
Lecture 3
Composition with Vector Functions
Composition of g : Rn → Rp and h : Rp → R with dom(g) = Rn and
dom(h) = Rp:
f(x) = h(g(x)) = h(g1(x), g2(x), . . . , gp(x))
f is convex if
(1) each gi is convex, h is convex and nondecreasing in each argument
(2) each gi is concave, h is convex and nonincreasing in each argument
Example
•∑m
i=1 egi(x) is convex if gi are convex
Convex Optimization 18
Lecture 3
Extended-Value Functions
A function f is an extended-value function if f : Rn → R ∪ {−∞,+∞}
Example: consider f(x) = infy≥0 xy for x ∈ R
Def. The epigraph of a function f over Rn is the following set in Rn+1:
epif = {(x, w) ∈ Rn+1 | x ∈ Rn, f(x) ≤ w}
General Convex Function Def. A function f is convex if its epigraph
epif is a convex set in Rn+1
This definition is equivalent to the one we have used so far (when reduced
to the function class we have considered thus far). How?
For an f with domain domf , we associate an extended-value function f̃
defined byf̃(x) =
f(x) if x ∈ domf
+∞ otherwise
domf is the projection of epif on Rn; convexity of f by letting w = f(x)
Convex Optimization 19
Lecture 3
Minimization
Let C ⊆ Rn × Rp be a nonempty convex set
Let f : Rn × Rp → R be a convex function [in (x, z) ∈ Rn × Rp]. Then
g(x) = infz∈C
f(x, z) is convex
Example
• Distance to a set: for a nonempty convex C ⊂ Rn,
dist(x, C) = infz∈C ‖x− z‖ is convex
Proof: Let x1, x2 ∈ Rn and α ∈ (0,1) be arbitrary. Let ε > 0 be arbitrarily
small. Then, there exist z1, z2 ∈ C such that f(x1, z1) ≤ g(x1) + ε and
f(x2, z2) ≤ g(x2) + ε. Consider f(αx1 + (1 − α)x2, αz1 + (1 − α)z2)
and use convexity of f and C.
Convex Optimization 20