Dynamic Regret Bounds for Online Nonconvex OptimizationJulie Mulvaney-Kemp, SangWoo Park, Ming Jin, and Javad Lavaei

Abstract—Online optimization problems are well-understoodin the convex case, where algorithmic performance is typicallymeasured relative to the best fixed decision. In this paper, weshed light on online nonconvex optimization problems in whichalgorithms are evaluated against the optimal decision at eachtime using the more useful notion of dynamic regret. The focusis on loss functions which are arbitrarily nonconvex, but haveglobal solutions that are slowly time-varying. We address thisproblem by first analyzing the region around the global solutionat each time to define time-varying target sets, which containthe global solution and exhibit desirable properties under theprojected gradient descent algorithm. All points in a target setsatisfy the proximal Polyak-Łojasiewicz inequality, among otherconditions. Then, we introduce two algorithms and prove that thedynamic regret for each algorithm is bounded by a function of thetemporal variation in the optimal decision. The first algorithmassumes that the decision maker has some prior knowledgeabout the initial objective function. This algorithm ensures thatdecisions are within the target set at every time. The secondalgorithm makes no assumption about prior knowledge. It insteadrelies on random sampling and memory to find and then trackthe target sets over time. In this case, the landscape of the lossfunctions determines the likelihood that the dynamic regret willbe small. Numerical experiments validate these theoretical resultsand highlight the impact of a single low-complexity problem earlyin the sequence.

I. INTRODUCTION

Nonconvex optimization is ubiquitous in real-world appli-cations, such as the training of deep neural nets [1], matrixsensing/completion [2], [3], state estimation of dynamic sys-tems [4], and the optimal power flow problem [5]. Moreover,most of these practical problems are solved sequentially overtime with time-varying input data, leading to online (real-time)versions of the aforementioned examples [4], [6], [7].

In this paper, we study an online nonconvex optimization(ONO) problem whose loss (objective) function changes overdiscrete time periods, namely,

minimizex∈S

ft(x) (1)

where t ∈ Z+ denotes the time and S ⊆ Rn is the time-invariant feasible region. At each time t = 1, . . . , T in thisONO framework, the decision maker first chooses an actionxt ∈ S while oblivious to the loss function ft : S→ R. Oncethe action is played, it is evaluated against ft, which may bechosen by an adversary in response to the action. Then, thedecision maker is granted access to the loss function and itsgradient.

Julie Mulvaney-Kemp, SangWoo Park and Javad Lavaei are with the De-partment of Industrial Engineering and Operations Research at the Universityof California, Berkeley. Emails: {julie_mulvaney-kemp, spark111,lavaei}@berkeley.edu. Ming Jin is with the Department of Electricaland Computer Engineering at Virginia Tech. Email: jinming@vt.edu. Thiswork was supported by grants from ARO, ONR, AFOSR, and NSF.

The performance of a decision maker, or equivalently analgorithm, in online settings is typically evaluated by a metriccalled regret [8]. In this paper, we exclusively focus on thestrictest version of regret, dynamic regret, which is defined as

RegdT (x1, ...,xT ) :=

T∑t=1

ft(xt)−T∑t=1

f∗t , (2)

where f∗t denotes the global optimal objective value of (1).Dynamic regret (also called non-stationary regret) comparesthe decision maker’s actions to an optimal action at each timet. In comparison, static regret (also called stationary regret orsimply regret) compares the decision maker’s actions to thebest fixed action in hindsight:

RegsT (x1, ...,xT ) =

T∑t=1

ft(xt)−miny∈S

T∑t=1

ft(y). (3)

In general, nonconvex optimization problems are NP-hard,and therefore commonly used local search algorithms, such asfirst-order and second-order descent algorithms, may convergeto a spurious local minimum (i.e., a local minimum that is notglobally optimal). As a result, dynamic regret can be arbitrarilyhigh in a general setting due to the inability to efficiently finda near-optimal point xt. The existing works in the literaturehave derived regret bounds in terms of various quantities,such as the regularity of the comparator sequence [9], thetemporal variation in the loss functions [10], the temporalvariation in the gradient of the loss functions [11], and thetemporal variation in the optimal decision (also called pathlength or path variation) [12], [13]. Details on many of thesevariation measures used to bound dynamic regret can be foundin [14], where the online convex optimization problem isanalyzed. The existing regret bounds for ONO either focuson static regret [15]–[18] or require the loss functions to beweakly pseudo-convex which is a restrictive condition thatexcludes spurious local minima [19]. In [20], the authors ofthis paper established probabilistic nonconvexity regret boundsfor a variation of the ONO problem in which ft is known tothe decision maker at time t, future loss functions are unknownbut fixed, the global minima are “sufficiently superior” to alllocal minima, and limit points of a continuous-time projectedgradient algorithm can be found precisely. Finally, [21] and[22] explored how variability in the input data can help ONOsolution trajectories escape non-global local solutions overtime but they did not study dynamic regret and focused onasymptotic regret.

The main goal of this paper is to analyze how the qualityof the obtained solutions evolves in ONO settings where theglobal solution changes slowly over time. To this end, we firstdevelop mathematical tools for characterizing the landscapeof constrained nonconvex optimization problems and analyze

the behavior of the projected gradient descent algorithm onsuch problems. There are many conditions in the literaturethat guarantee linear convergence of local search algorithms. Inthe unconstrained case when S = Rn, the Polyak-Łojasiewicz(PL) condition has been proven to be weaker than other com-mon assumptions (such as strong convexity, essential strongconvexity, weak convexity, and restricted secant inequality)that guarantee linear convergence [23]. Despite its favorablecharacteristics, requiring that a function satisfy the PL condi-tion still significantly restricts the type of nonconvex functionsthat one can study. For instance, functions satisfying the PLcondition cannot have local minima that are not globallyoptimal.

We leverage the generalization of the PL condition forconstrained optimization, called the proximal-PL condition(originally proposed in [23]), to study dynamic regret min-imization in a non-convex setting. The first contribution ofthis paper is to establish a target set for each time instancewith the property that once the algorithm finds a point in thecorresponding target set at a given time, the global minimizersof future problems can be found efficiently. These time-varying target sets are defined with respect to the proximal-PLcondition and the global solution. We show several importantproperties of these sets, including linear convergence to theglobal minimizer and quadratic growth.

The design and regret analysis of two online algorithmsconstitute the second contribution of this paper. Specifically,we equip local search algorithms with memory and randomexploration and establish dynamic regret bounds for eachalgorithm in terms of the path length of the optimal decisionsequence, when the difference between consecutive points inthis sequence is bounded appropriately. The first algorithmassumes that the decision maker has some prior knowledgeabout the initial function and can start at a point that iswithin its target set. This algorithm ensures bounded dynamicregret by producing decisions which track the time-varyingtarget sets. The second algorithm obviates this initial conditionassumption by using random exploration. In this case, dynamicregret depends on when the decision maker first finds a pointwithin the corresponding target set, as after that time alldecisions will track the time-varying target sets. Therefore, therelative volume of the time-varying target sets with respectto the entire feasible domain–a measure of how favorablethe loss function landscape is–influences the likelihood thatthe dynamic regret will be small. In particular, a single low-complexity problem in the sequence can have a large influenceon the outcomes.

The remainder of this paper is organized as follows. InSection II, we analyze the optimization problem for each fixedtime step, focusing on a neighborhood of the global solution.In Section III, we introduce ONO algorithms, derive bounds ontheir dynamic regret, and support the analysis with empiricalresults. Finally, we conclude the paper in Section IV.

A. Notations

Let || · || indicate the `2-norm of a vector and | · | representthe cardinality of a set. The symbols Rn and Z+ denote the

space of n-dimensional real vectors and the set of positiveintegers, respectively. The globally optimal objective value ofthe optimization problem at time t is denoted by f∗t . If thereis a unique global optimum at time t, it will be denoted asx∗t , in which case ft(x∗t ) = f∗t . The indicator function IS(x)returns zero if x belongs to the set S and infinity otherwise.We define the projection operator as follows:

ΠS(x) := argminy∈S

‖x− y‖ (4)

The tangent cone of a convex set S at x is denoted as TS(x).The sublevel set Lt is defined as Lt(α) := {x ∈ Rn|ft(x) <α}. Finally, P[·] denotes the probability of the argument.

II. THEORETICAL RESULTS FOR A FIXED TIME STEP

A. Properties of the Problem Structure

Throughout this paper, we make the following assumptionson the problem structure:

1) The time-invariant feasible region S ⊂ Rn is a compact,convex set known to the decision maker.

2) ft is coercive and differentiable, but potentially nonconvexin x with many local minima, for all t ∈ {1, 2, . . . , T}.

3) ft has a unique global minimum x∗t over S for all t ∈{1, 2, . . . , T}.

4) The magnitude of the gradient is bounded above by apositive constant M1 for all t ∈ {1, 2, . . . , T}. That is,supx∈S,1≤t≤T ||∇ft(x)|| ≤M1.

5) The first derivative of ft is L-Lipschitz continuous on S forall t ∈ {1, 2, . . . , T}, implying the following inequality forsome constant L:

ft(y)− ft(x)≤〈∇ft(x),y − x〉+L

2||y − x||2 ∀x,y∈S.

B. Proximal Polyak-Łojasiewicz Regions

In the context of unconstrained optimization problems, wesay that a differentiable function ft satisfies the Polyak-Lojasiewicz (PL) condition [24] if the following conditionholds for some parameter µ > 0:

1

2||∇ft(x)||2 ≥ µ(ft(x)− f∗t )︸ ︷︷ ︸

PL inequality

∀x ∈ Rn (5)

If a function satisfies the PL condition and the magnitude ofits gradient is small at some x, then the function value at xwill be close to the global minimum. This is the reason whythe PL condition is also referred to as the gradient domina-tion condition [25]. For a general (unconstrained) nonconvexoptimization problem, first-order methods such as gradientdescent may not converge to a global minimizer. However, ifa function ft satisfies the Polyak-Lojasiewicz condition, thenevery stationary point is a global minimizer. Moreover, PLis one of the most general conditions under which gradientdescent offers linear convergence to a global minimizer [23].Note that, in general, functions satisfying the PL conditionmay not have a unique global minima.

The top plot in Figure 1 shows an example of a nonconvexfunction that satisfies the PL condition. On the other hand,

-5 -4 -3 -2 -1 0 1 2 3 4 5

x

0

10

20

30

f1(x)

-10 -5 0 5 10

x

0

10

20

30

40

50

f2(x)

Fig. 1: The top figure shows the nonconvex functionf1(x) = x2 + 3 sin2(x), which satisfies the PL inequality withthe parameter µ = 1/32 for all x ∈ R. The bottom figureshows an example of a nonconvex function that satisfiesthe PL inequality with the parameter µ = 1/32 only forx ∈ {[−55.9,−10.2]∪ [−5.4, 5.4]∪ [10.2, 55.9]}. The functionfor the bottom figure is given below:

f2(x) =

− 1

3πx3 − 5

2x2 − 6πx− 17

6 π2, if x < −π

x2 + 3 sin2(x), if− π ≤ x ≤ π1

3πx3 − 5

2x2 + 6πx− 17

6 π2, if π < x

the function in the bottom plot of Figure 1 manifests spuriouslocal minima and therefore cannot satisfy the PL inequality forall x for any µ > 0. However, for a given µ, we can identifya subset of R that satisfies the PL inequality. The idea offocusing on regions where the PL inequality is satisfied, ratherthan only considering functions satisfying the PL conditionover the entire feasible region, leads to our definition of time-varying target sets in Section II-C.

Next, we return to considering constrained optimizationproblems. Constrained optimization can be cast in the frame-work of unconstrained optimization by appending the objectivefunction with IS, an indicator function of a convex set S. Thisindicator function is non-smooth and convex. Subsequently, anatural generalization of gradient descent to the constrainedcase is the proximal gradient method, whose iteration isdescribed by

xk+1t = argmin

y

[〈∇ft(xkt ), y − xkt 〉+

‖y − xkt ‖2

2s

+ IS(y)− IS(xkt )]

(6)

for every k ∈ Z+, where s is a positive constant. It can beshown that the above algorithm is equivalent to the projectedgradient descent algorithm:

xk+1t = ΠS(xkt − s∇ft(xkt )). (7)

A matching generalization of the PL inequality, namely theproximal-PL inequality, was first proposed in [23].

Definition 1. (Proximal-PL inequality) For a function ft,define the proximal-gradient with parameter β > 0 as

Dt(x, β) = (8)

− 2βminy

[〈∇ft(x),y − x〉+

β‖y − x‖2

2+ IS(y)− IS(x)

]We say that a point x ∈ S satisfies the proximal-PL inequalitywith the parameters µ > 0 and β > 0 if

1

2Dt(x, β) ≥ µ(ft(x)− f∗t ). (9)

While [23] considers functions that satisfy the proximal-PL inequality at all points in S, in this work we insteadidentify a subset of the entire space that satisfies the inequality.Hereby, we define the time-varying proximal-PL region, de-noted Pt(µ, β), as the set of all x ∈ S satisfying the proximal-PL inequality with the parameters µ > 0 and β > 0. That is,

Pt(µ, β) :=

{x ∈ S

∣∣∣ 1

2Dt(x, β) ≥ µ(ft(x)− f∗t )

}. (10)

Note that by virtue of the equivalence between equations(6) and (7), the proximal-gradient can also be expressed asfollows:

Dt(x, β) = −2β

[〈∇f(x),ΠS(x− 1

β∇ft(x))− x〉

+β

2‖ΠS(x− 1

β∇ft(x))− x‖2

]. (11)

C. Time-Varying Regions of Attraction and Target Sets

A proximal-PL region can span over multiple regions ofattraction associated with different local minima. Also, notethat a region of attraction (RoA) is algorithm dependent. In thispaper, we define RoAs with respect to the global minimizerunder the projected gradient descent method and also underthe projected gradient flow system, as a continuous version ofthe former.

Definition 2. (Region of Attraction) The region of attraction ofa global minimizer x∗t of ft based on the projected gradientdescent method (discrete algorithm) with the step size s isdefined as follows:

RADt (s) :={x| limk−→∞xkt = x∗t where

xk+1t = ΠS(xkt − s∇ft(xkt )) and x0

t = x}. (12)

In addition, we define the region of attraction of a globalminimizer based on the projected gradient flow system [26](continuous algorithm) as follows:

RACt :={x| lim`→∞

xt(`) = x∗t , where (13)

xt(`) = ΠTS(xt)[−∇ft(xt(`))], xt(0) = x}.

Next, we define a region that we call the target set. In sub-sequent sections, we will show that if our proposed algorithmenters the target set at any point in time t, then it is possibleto approach the global minimizer and track it henceforth.

Definition 3. (Target set) Let RPDt and RPCt denote thesubsets of the discrete and continuous RoAs that are containedwithin the proximal-PL region at time t:

RPDt (µ, β, s) :={x | xk+1

t = ΠS(xkt − s∇ft(xkt )),

x0 = x, limk−→∞xkt = x∗t and {xkt }∞k=0 ⊂ Pt(µ, β)

}(14)

RPCt (µ, β) :={x | xt = ΠTS(xt)(−∇ft(xt)),xt(0) = x,

lim`→∞

xt(`) = x∗t and xt(`) ∈ Pt(µ, β) ∀` ≥ 0}

(15)

We define our target set for time t to be a subset of a sublevelset around the global minimizer that is contained within bothRPDt and RPCt :

Tt(µ, β, s) := Lt(αt) ∩RADt ∩RACt (16)

where αt is the largest α satisfying the following condition:(Lt(α) ∩RADt ∩RA

Ct

)⊆(RPDt ∩RP

Ct

)(17)

In summary, all points in each target set are feasible, satisfythe proximal-PL inequality, and lead to the global solutionunder the continuous and discrete descent methods initializedat those points. Further, the target set is invariant for bothof these methods. Note that the sets RADt ,RA

Ct ,RP

Dt and

RPCt depend on some or all of the parameters (µ, β, s) butthis dependency has been omitted in order to simplify notation.As indicated by the subscript t, the target set varies over time.

One useful way to measure the size of a target set is withrespect to the global solution.

Definition 4. (Reach) Define the reach of a target set as themaximum distance between the global minimum and any pointin the target set:

ρt(µ, β, s) := maxx∈Tt(µ,β,s)

‖x∗t − x‖. (18)

D. Properties of Target SetsIn [23], the authors showed the linear convergence of

the proximal-gradient algorithm when applied to functionssatisfying the proximal-PL condition. In this paper, we showthat initializing the proximal-gradient algorithm in the corre-sponding target set ensures linear convergence, regardless ofwhether the proximal-PL inequality is satisfied for all feasiblepoints. Additionally, there is an open ball around the globalsolution whose intersection with the feasible set S is alsocontained in the corresponding target set (see Appendix B).

Theorem 1. Given µ > 0, β ≥ L and a fixed instance of t,consider the problem of minimizing ft over S (Problem (1))via the projected gradient descent method (7) with the stepsize s. If x0

t ∈ Tt(µ, β, s), then the projected gradient descentmethod with 0 < s < min( 1

µ ,1β ) converges linearly to the

optimal value f∗t , i.e.,

ft(xNt )− f∗t ≤ (1− µs)N [ft(x

0t )− f∗t ], (19)

where N ∈ {0, 1, 2, . . . } indicates the number of iterations.

Proof. The proof is similar to that of Theorem 5 in [23]. LetFt(x) := ft(x) + IS(x). By using the Lipschitz continuity ofthe gradient of ft, one can write:

Ft(x1t ) = ft(x

1t ) + IS(x0

t ) + IS(x1t )− IS(x0

t )

≤ ft(x0t ) + IS(x0

t ) + 〈∇ft(x0t ),x

1t − x0

t 〉

+L

2||x1

t − x0t ||2 + IS(x1

t )− IS(x0t )

Then, noting that x0t ∈ Tt(µ, β, s) ⊂ S and L ≤ 1/s, we

obtain an upper bound of the form:

Ft(x1t ) ≤ ft(x0

t ) + 〈∇ft(x0t ),x

1t − x0

t 〉+1

2s||x1

t − x0t ||2

+ IS(x1t )− IS(x0

t )

= ft(x0t )−

s

2Dt(x0

t , 1/s)

where the equality follows from the definition of xk+1t and the

proximal-gradient. Finally, we upper bound the equation aboveby using the facts that x0

t satisfies the proximal-PL inequalitywith parameters µ and β and that Dt(x0

t , 1/s) ≥ Dt(x0t , β)

since 1s ≥ β [23]:

Ft(x1t ) ≤ ft(x0

t )− µs[ft(x0t )− f∗t ]

Since x1t is feasible by the definition of projection, we have

ft(x1t ) ≤ ft(x0

t )− µs[ft(x0t )− f∗t ],

which subsequently implies

ft(x1t )− f∗t ≤ (1− µs) [ft(x

0t )− f∗t ]. (20)

Furthermore, by showing the decrease in objective value anddirectly following the definition of the target set, the aboveresults also prove that x1

t ∈ Tt(µ, β, s). In other words, thetarget set is invariant under the projected gradient descentmethod, as was mentioned in Section II-C. Repeating theprocess for N steps, we have the final result:

ft(xNt )− f∗t ≤ (1− µs)N [ft(x

0t )− f∗t ]

Theorem 1 also gives a lower bound on f∗t after N iterations:

f∗t ≥ft(x

Nt )−

(1− µs

)Nft(x

0t )

1−(1− µs

)N , ∀N ∈ Z+ (21)

The next lemma establishes what we will refer to as therobustness property of a target set.

Lemma 1. (Robustness of a target set) Assume there existparameters µ, β, and s > 0 such that αt > 0 for all t ∈{1, . . . , T} (where αt is as defined in (17)). Then, the target setTt(µ, β, s) includes a feasible ball of radius at least r aroundthe global solution for some r > 0. That is, ∃µ, β, s, r >0 : Tt(µ, β, s) ⊇ (B(x∗t , r) ∩ S) for all t ∈ {1, . . . , T}, whereB(x∗t , r) := {y|‖x∗t − y‖2 ≤ r}.

Proof: See Appendix A.For unconstrained problems, a function satisfying the PL

condition implies that it also satisfies the quadratic growthcondition [23]. Next, we prove a similar relationship betweenthe proximal-PL inequality and quadratic growth.

Theorem 2. (Quadratic growth) The following inequalityholds:√

ft(x)− f∗t ≥√µ

2‖x− x∗t ‖ ∀x ∈ RP

Ct (µ, β). (22)

Proof: See Appendix C.While the proof of Theorem 2 relies on the continuous

version of the projected gradient flow algorithm, this paperdoes not require implementing or solving this continuousdynamical system. The algorithms in Section III use thediscrete-time projected gradient descent algorithm.

E. Visualization of a Proximal-PL Region and Target Set

To develop intuition about proximal-PL regions and targetsets, it is beneficial to visualize these sets in an example.Consider the optimization problem

min f(x1, x2) = x41 − 4x3

1 + x21 + 2x1 +

3

2sin(2πx1)

+ x42 − 4x3

2 + x22 + 2x2 +

3

2sin(2πx2) + 28.87

s.t. − 1 ≤ x1 ≤ 3, −1 ≤ x2 ≤ 3 (23)

which is depicted in Figure 2a. This problem has the optimalvalue of 0 at x∗ = (2.75, 2.75) and includes many spuriouslocal solutions.

The proximal-PL region and target set for this problem withthe parameters µ = 0.5, β = 250 > L and s = 1

2β are depictedin Figure 2b and Figure 2c, respectively. The proximal-PLregion includes a neighborhood of the global solution, as wellas points far from the global solution. However, many pointsin the feasible set do not satisfy the proximal-PL inequality, inparticular those near local maxima or saddle points. Observethat the target set is a subset of RAD ∩RAC , RPD ∩RPCand the proximal-PL region. The symmetry in Figure 2b andFigure 2c is a result of the symmetry in the loss function f .

III. ONLINE PROJECTED-GRADIENT DESCENT WITHRANDOM EXPLORATION

In this section, we leverage the results developed in Sec-tion II to study the ONO problem (1). We introduce andanalyze two algorithms for different scenarios:

1) Scenario 1: An initial point in the target region aroundthe global solution x∗1 is known.

2) Scenario 2: No information about the loss functions ortheir minimizers is known in advance.

A. Scenario 1 - Known desirable initial point

Algorithm 1 provides a natural approach to solving theONO problem (1) in the setting where a suitable initial pointis known. At each time t, the decision maker performs Stiterations of projected gradient descent on ft, with the finaliteration becoming the decision maker’s action at t + 1. Theassumption is that the decision maker has enough knowl-edge about the problem at t = 1 to select an initial pointin the corresponding target set and that the change in theglobal optimum between time steps is upper-bounded basedon parameters reflecting the functions’ landscapes. The latterassumption restricts the adversary’s choice of loss functionand can be regarded as requiring the global solution sequenceto have steadiness. This assumption is formalized next.

(a) Topology of the objective function f over the feasible set. Observethat this problem has many local minima.

(b) Points in the grey region satisfy the proximal-PL inequality forthe function f over the set [−1, 3] × [−1, 3] with the parametersµ = 0.5 and β = 250, while those points in the white regions donot. The unique optimal solution x∗ = (2.75, 2.75) is identified bya red star.

(c) Illustration of the target set (yellow) and other sets critical to itsdefinition. The red dashed circle demonstrates the robustness propertyestablished in Lemma 1. The length of the black dashed line is thereach of the target set.

Fig. 2: Visualization of the proximal-PL region and the targetset for the optimization problem (23)

Assumption 1. (Steadiness of global solution) The changein global optimum between consecutive time steps is upper-bounded by r < r, where r is as defined in Lemma 1. That is,for t = 1, . . . , T − 1,

||x∗t+1 − x∗t || ≤ r. (24)

where µ, β, s, and r collectively satisfy the robustness propertyin Lemma 1 and ρt(µ, β, s) is defined in (18). Furthermore,assume that St is large enough to satisfy the inequalities:√

2M1 · ρt(µ, β, s) · (1− µs)St

µ≤ r − r (25a)

St >log(µ)− log(2M2)

log(1−µs). (25b)

The constant M2 defined as

M2 := inf{M ∈ R : ft(x)−f∗t ≤M‖x− x∗t ‖2

∀t = 1, . . . , T, ∀x ∈ S} (26)

exists and is finite because the functions ft are continuousover the bounded set S.

Note that this assumption only limits the change in theglobal minimum; the overall landscape of the function canchange arbitrarily. Under this assumption, we will establish adeterministic dynamic regret bound for Algorithm 1. To aidin establishing this bound, we first prove two lemmas:

i) one showing the convergence in terms of the variables x,ii) another one proving that once the chosen action xt is

within the target region at time t, all successive actionschosen by the algorithm will also lie within the targetregion of their respective time.

Lemma 2. Consider a sequence {xt}Tt=1 generated by Algo-rithm 1. Under Assumption 1, if xt ∈ Tt(µ, β, s) for somet ∈ {1, 2, . . . , T − 1}, then

‖xt+1 − x∗t ‖ ≤ γ‖xt − x∗t ‖. (27)

where

γ = maxt=1,...,T

√2M2(1− µs)St

µ< 1 (28)

Proof: From the convergence rate in Theorem 1 (specifically,equation (20)), we have

ft(xt+1)− f∗t ≤ (1− µs)St[ft(xt)− f∗t

]Applying the quadratic growth inequality from Theorem 2 andtaking the square root of all sides, we obtain

‖xt+1−x∗t ‖≤

√2(ft(xt+1)− f∗t

)µ

≤

√2(1−µs)St

(ft(xt)− f∗t

)µ

Then, using the definition of M2, we arrive at

‖xt+1−x∗t ‖≤

√2M2(1− µs)St

µ︸ ︷︷ ︸=γt

‖xt − x∗t ‖ (29)

Algorithm 1 Online Projected Gradient Descent with Desir-able InitializationRequire: x1 ∈ T1(µ, β, s), 0 < s < min{ 1

µ ,1β }

1: for t = 1, 2, . . . , T do2: Play xt3: Set z0 = xt4: for i = 1, ..., St do5: Observe ∇ft(zi−1)6: Perform projected gradient descent update:

zi = ΠS [zi−1 − s∇ft(zi−1)]7: end for8: Set xt+1 = zSt

9: end for

Then γt < 1 since St > log(µ/(2M2))/ log(1−µs).The above lemma proves that given a sufficiently large St,

we can make γ ∈ [0, 1) arbitrarily close to zero, implying thatthe iterates can become arbitrarily close to the global mini-mizers at different times. The trade-off is between accuracyand computation time, which is driven by St. There is alsoan intuitive trade-off between the step size s and computationtime: smaller step sizes require more algorithmic iterations.

Lemma 3. Consider a sequence {xt}Tt=1 generated by Al-gorithm 1. Under Assumption 1, if xt ∈ Tt(µ, β, s) for anyt ∈ {1, 2, . . . , T − 1}, then xt+1 ∈ Tt+1(µ, β, s).

Proof: It is desirable to show that ‖xt+1 − x∗t+1‖ < r, whichensures that xt+1 ∈ B(x∗t+1, r). By Lemma 1 (the robustnessproperty of target sets), we have B(x∗t+1, r) ⊂ Tt+1(µ, β, s).One can write:

‖xt+1−x∗t+1‖ ≤ ‖xt+1 − x∗t ‖+ ‖x∗t − x∗t+1‖≤‖xt+1−x∗t ‖+ r

≤‖xt+1−x∗t ‖+r−

√2M1ρt(µ, β, s)(1−µs)St

µ

≤

√2(ft(xt+1)− f∗t

)µ

+r−

√2M1ρt(µ, β, s)(1− µs)St

µ

≤

√2(1−µs)St

(ft(xt)− f∗t

)µ

+ r −

√2M1ρt(µ, β, s)(1−µs)St

µ

≤r +

√2M1(1−µs)St

µ

(√‖xt − x∗t ‖ −

√ρt(µ, β, s)

)≤ r

where the second and third inequalities use Assumption 1,the fourth inequality applies Theorem 2, the fifth inequalityis due to Theorem 1, the sixth inequality applies the boundedgradient assumption from Section II-A, and the last inequalityis due to (18).

Now, we present a dynamic regret bound for Algorithm 1.

Corollary 1. Consider a sequence {xt}Tt=1 generated byAlgorithm 1. Under Assumption 1, the dynamic regret satisfiesthe inequality

RegdT (x1,...,xT )≤ M1

1−γ

T∑t=2

‖x∗t−x∗t−1‖+ρ1(µ, β, s)

(1−γ)/M1(30)

Proof: This proof will follow the same line of reasoning asTheorem 1 and Corollary 1 of [14], where a similar resultis proved for strongly convex functions. In the nonconvexsetting considered in this paper, we will utilize Lemma 2 andLemma 3 in our proof.By the Intermediate Value Theorem, there exists y ∈ {z|z =ωxt + (1 − ω)x∗t , 0 ≤ ω ≤ 1} such that ft(x) − ft(x∗t ) =∇ft(y)T (xt − x∗t ). Therefore, by applying the bounded gra-dient assumption in Section II-A, we have

RegdT (x1, ...,xT ) ≤M1

T∑t=1

‖xt − x∗t ‖. (31)

Next we establish an upper bound on the summation in (31):

T∑t=1

‖xt − x∗t ‖ = ‖x1 − x∗1‖+

T∑t=2

‖xt − x∗t ‖

≤ ‖x1 − x∗1‖+

T∑t=2

‖xt − x∗t−1‖+

T∑t=2

‖x∗t − x∗t−1‖

≤ ‖x1 − x∗1‖ − γ‖xT − x∗T ‖+ γ

T∑t=1

‖xt − x∗t ‖

+

T∑t=2

‖x∗t − x∗t−1‖

=⇒T∑t=1

‖xt − x∗t ‖ ≤‖x1 − x∗1‖ − γ‖xT − x∗T ‖

1− γ

+1

1− γ

T∑t=2

‖x∗t − x∗t−1‖ (32)

≤ ρ1(µ, β, s)

1− γ+

1

1−γ

T∑t=2

‖x∗t − x∗t−1‖ (33)

The first inequality invokes the triangle inequality. The secondinequality applies Lemma 2 for each t = 1, . . . , T andre-indexes the summation. This application of Lemma 2 isderived by recursively applying Lemma 3 to the requirementthat x1 ∈ T1(µ, β, s). We rearrange terms to arrive at (32)and then apply the definition of the reach of the target set(18) to achieve the final inequality. Combining (33) with (31)completes the proof.

Observe that the dynamic regret is a function of the temporalvariation in the optimal decision (also called path length orpath variation), a common measure of variation discussed inthe introduction. The path length is weighted by a functionof γ that is large when γ is close to one and approximatelyone when γ is close to 0. Again, this trade-off between thestrength of the dynamic regret bound and computation time isdriven by St.

Algorithm 2 Online Projected Gradient Descent with RandomExploration

Require: x1 ∈ S, M1 = ∅, m = 0, 0 < s < min{ 1µ ,

1β }

1: for t = 1, 2, . . . , T do2: • Play xt3: • Create Wt = {w1

t , . . . ,wqt} by uniformly sampling q

random points from S4: • Set Yt =Wt

⋃Mt

⋃{xt} := {y1

t , ...,yq+m+1t }

5: for k = 1, 2, . . . , q +m+ 1 do6: • Initialize zk0 = ykt , zk∗ = ykt , ck =∞, bk = −∞7: • Set i = 18: while ck − bk > ε or i ≤ St do9: • Observe ∇ft(zki−1)

10: • Compute zki = ΠS[zki−1 − s∇ft(zki−1)

]11: • Observe cki = ft(z

ki )

12: if cki < ck then13: • zk∗ = zki , ck = cki14: end if15: • bki =

(ft(z

ki )−(1−µs)i ft(zk0)

)/(1−(1−µs)i

)16: • Update bk = max{bk, bki }17: • Update i = i+ 118: end while19: • Return Ikt = i20: end for21: • Let K = argmink ck, and set xt+1 = zK∗22: • Store in memory all other points in {zk∗}

q+m+1k=1 which

could be in the proximal PL-region at time t:Mt+1 ={zk∗ :ck≤cK + ε, k∈{1, ..., q+m+1} \K}

23: m = |Mt+1|24: end for

B. Scenario 2 - Blind initialization

The initialization scenario described in Scenario 1 – thata point in the target region is known at the initial time – isdifficult to satisfy in practice. The reason is that the decisionmaker may have no information about how their adversarywill design f1. In this case, it is advantageous to explore thelandscape of ft before selecting decision xt+1. Algorithm 2explores by running the projected gradient descent algorithmfrom multiple initial points, which are sampled uniformly atrandom from S.

The goal of exploration is to find a point in a time-varyingtarget set. The decision maker cannot verify when this occurs,however, since they do not have knowledge of the landscape ofthe function. As a result, Algorithm 2 utilizes memory to makeavailable at time t+1 points which may be in the target set attime t. Once a point in a time-varying target set is sampled,memory ensures that the decision maker has at least one initialpoint in the target sets for each future time step. This trackingguarantee is formalized in the following lemma.

Lemma 4. Consider sequences {xt}Tt=1 and {Yt}Tt=1 gen-erated by Algorithm 2. Under Assumption 1, if zk0 ∈(Tt(µ, β, s) ∩ Yt) for any t ∈ {1, 2, . . . , T − 1}, then zk∗ ∈(Tt+1(µ, β, s) ∩ Yt+1).

Proof: The number of iterations Ikt is at least as large as St.

Therefore, applying the same logic as the proof of Lemma 3,we know that zk

Ikt∈ Tt+1(µ, β, s). By Theorem 1, we have

zk0 ≥ zk1 ≥ · · · ≥ zkSt≥ zk

Iktwith zki = zki+1 only if zki = x∗t .

This implies that zk∗ = zkIkt

. It remains to show that zk∗ ∈Yt+1. If zk∗ = xt+1, then zk∗ ∈ Yt+1. Otherwise, since zk0 ∈Tt(µ, β, s), it holds that ck ≤ f∗t + ε ≤ cK + ε, which implieszk∗ ∈Mt+1 ⊂ Yt+1. As a result, zk∗ ∈ Yt+1, which completesthe proof.

Since Algorithm 1 is a deterministic algorithm, the dynamicregret bound established in Corollary 1 is deterministic too.Algorithm 2 relies on sampling, and therefore its associatedregret bound should be probabilistic. In the following culmi-nating theorem, we provide an upper bound on the dynamicregret accrued using Algorithm 2 and a lower bound on theprobability with which this bound holds.

Theorem 3. Consider a sequence {xt}Tt=1 generated byAlgorithm 2. Under Assumption 1, the dynamic regret satisfiesthe following probabilistic bound for all T ∈ {1, . . . , T}:

P

[RegdT (x1, . . . ,xT ) ≤ RegdT−1(x1, . . . ,xT−1)

+M1ρT (µ, β, s)

(1− γ)+

M1

1− γ

T∑t=T+1

‖x∗t−x∗t−1‖

](34)

≥ 1−T∏t=1

(1−Vol(Tt(µ, β, s))

Vol(S)

)q,

where Vol(·) indicates the volume of the set and γ is definedin Lemma 2. This theorem relates the dynamic regret at timeT to the dynamic regret at an earlier time T , the variationwithin the optimal decision sequence after T , and the relativesizes of the target sets through T .

Proof: The probability that a point located in the time-varyingtarget set Tt(µ, β, s) appears in Yt by time T is related to thevolumes of Tt(µ, β, s) and S because, at each time step, qinitial points are selected from S uniformly at random. Hence,

P[Yt ∩ Tt(µ, β, s) 6=∅ for some t∈{1, . . . , T}

](35)

≥ P[Wt ∩ Tt(µ, β, s) 6=∅ for some t∈{1, . . . , T}

]= P

T⋃t=1

q⋃i=1

wit ∈ Tt(µ, β, s)

= 1−P

[wit /∈Tt(µ, β, s) ∀t=1, . . . , T ,∀i=1, . . . , q

]= 1−

T∏t=1

(1− Vol(Tt(µ, β, s))

Vol(S)

)q(36)

Now, we will show that if Yt ∩ Tt(µ, β, s) 6= ∅ for somet∈{1, . . . , T}, then the dynamic regret is upper bounded bythe expression in (34). Applying Lemma 4, Corollary 1 andDefinition 4 yields that

RegdT (x1, . . . ,xT ) =

T−1∑t=1

(ft(xt)− f∗t ) +

T∑t=T

(ft(xt)− f∗t )

=RegdT−1(x1, . . . ,xT−1) +

T∑t=T

(ft(xt)− f∗t )

≤RegdT−1(x1, . . . ,xT−1) +M1

‖xT−x∗T ‖−γ‖xT−x∗T ‖

(1− γ)

+M1

1− γ

T∑t=T+1

‖x∗t−x∗t−1‖

≤ RegdT−1(x1, . . . ,xT−1) +M1ρT (µ, β, s)

(1− γ)

+M1

1− γ

T∑t=T+1

‖x∗t−x∗t−1‖. (37)

This completes the proof.Observe that the strength of this probabilistic bound depends

on the landscape of loss functions around the global solutionthrough the volume of the target sets. In particular, one cananalyze the role that a “lower-complexity problem” at sometime T plays in determining the complexity of the entireonline nonconvex optimization. As an extreme but importantcase, suppose that there is a time T ∈ {1, . . . , T} such thatfT is convex. Then the dynamic regret bound (34) holdswith probability 1 since Tt(µ, β, s) = S. In other words, theexistence of a single convex problem, in between the sequenceof nonconvex problems, is enough to break down the NP-hardness of solving nonconvex problems for all future times,under the steadiness of the global solution assumption. On theother hand, if the global solution is extremely “sharp” at alltimes, it is unrealistic to expect any algorithm with limitedcomputation time to find the global solution. Thus, dynamicregret could be arbitrarily large in this case. Indeed, the targetset of a sharp minima is small and therefore the probability ofsatisfying the dynamic regret bound in (34) is low, as expected.

Choices for the step size s, number of iterations St, andnumber of samples q, represent trade-offs between regretbound strength and computation time. As discussed in SectionIII-A, a smaller step size requires more algorithmic iterationsto satisfy Assumption 1. Increasing the number of iterationsmay increase the time to execute the while loop (line 8).However, larger values of St improve the upper bound ondynamic regret in (34) by reducing γ. Increasing the numberof random initial points improves the probability with which(34) holds, but also increases computation time.

C. Empirical study of Algorithm 2

The objective of this section is to support the results ofSection III-B through numerical analysis. We will illustratethe performance of Algorithm 2 on online function sequenceswhich satisfy the assumptions in Section II-A and Assump-tion 1 (steadiness of the global solution). To demonstrate therole that a single comparatively low-complexity problem canplay in a sequence of nonconvex problems, we will considertwo cases:A) “No low-complexity problem”: In this case,{ft : R2 → R}40

t=1 each have many local minimaover S = [−1, 3] and the target sets’ volumes representbetween 2.47% and 4.14% of the feasible space. Thegeometry of f1, . . . , f6, which are representative of theentire sequence, is shown in Figure 3.

TABLE I: Parameter and constant values

S µ β s ε r[−1, 3]2 0.5 289 0.0031 0.1 0.29

St(max) L M1 ‖x∗t−x∗

t−1‖7060 289 140 0.22

B) “Lower-complexity problem at time 4”: In this case,{ft : R2 → R}40

t=1 is identical to Case A at every timeperiod except t = 4. The target set corresponding to f4

covers 20.7% of the feasible space. Meanwhile, x∗4 is thesame in both scenarios.

The parameter choices and key problem constants for thesetwo online optimization problems are summarized in Table I.

-1 0 1 2 3-1

0

1

2

3t=1

-1 0 1 2 3-1

0

1

2

3t=2

-1 0 1 2 3-1

0

1

2

3t=3

-1 0 1 2 3-1

0

1

2

3t=4

-1 0 1 2 3-1

0

1

2

3t=5

-1 0 1 2 3-1

0

1

2

3t=6

-1 0 1 2 3-1

0

1

2

3t=7

-1 0 1 2 3-1

0

1

2

3t=8

-1 0 1 2 3-1

0

1

2

3t=9Fig. 3: Contour plots of f1, . . . , f6 for Case A. The red star

marks the unique global minimum of each function.

We conducted 500 trials of Algorithm 2 on Case A and CaseB for 3 different sampling rates: q = 1, q = 2, and q = 5.Figure 4 plots the empirical probability that Yt∩Tt(µ, β, s) 6=∅versus the theoretical lower bound provided in Theorem 3.(Note that, by Lemma 4, this is the same as the probabilitythat Yt∩Tt(µ, β, s) 6=∅ for some t∈{1, . . . , t}.) For the samevalue of q, the two cases are identical for t = 1, 2, 3 anddiverge at t = 4 as a result of the “easy” problem in Case B.These results support Theorem 3. A gap between the lowerbound and observed likelihood of initializing in the targetregion is expected, since the lower bound does not accountfor the possibility that xt or a memory point may be in thesubsequent target set. The dynamic regret and optimality gapover time is shown in Figure 5. Regret accumulates quicklyuntil the target set is found and then accumulates slowly asAlgorithm 2 starts tracking the global solution.

IV. CONCLUSION

In this paper, we defined proximal-PL regions and targetsets, characterized their properties, and used this new knowl-edge to propose and analyze algorithms for online nonconvexoptimization problems. Linear convergence to the global min-imizer and quadratic growth are the two key properties ofthe target sets that we established. Since dynamic regret can

0 10 20 30 40

0

0.2

0.4

0.6

0.8

1

(a) Case A

0 10 20 30 40

0

0.2

0.4

0.6

0.8

1

(b) Case B

Fig. 4: Empirical validation of Theorem 3 probability bound

0 10 20 30 40

40

80

120q=1

q=2

q=5

0 10 20 30 40

4

8

12

q=1

q=2

q=5

(a) Case A

0 10 20 30 40

40

80

120

q=1

q=2

q=5

0 10 20 30 40

4

8

12

q=1

q=2

q=5

(b) Case B

Fig. 5: Regret resulting from Algorithm 2

be arbitrarily large when there are no restrictions on the lossfunctions, we constrain consecutive functions to have globalsolutions which are not too far apart, but do not limit thevariation in the loss functions otherwise. In this setting, wepropose two online algorithms. Algorithm 1 is relevant whenthe decision maker has a good initial point, and it providesa deterministic dynamic regret upper bound as a function ofthe path length of the optimal decision. Algorithm 2 utilizesexploration and memory to be relevant regardless of the initialpoint. It provides a probabilistic dynamic regret upper bound,which is also a function of the path length of the optimaldecision. The strength of this probabilistic bound depends onthe loss function landscapes. For example, the bound holdswith probability 1 in the special case where one of the lossfunctions in the sequence is convex. Empirical studies supportthese bounds.

REFERENCES

[1] H. Li, Z. Xu, G. Taylor, C. Studer, and T. Goldstein, “Visualizing the losslandscape of neural nets,” Advances in Neural Information ProcessingSystems, pp. 6389–6399, 2018.

[2] E. J. Candes and B. Recht, “Exact matrix completion via convexoptimization,” Foundations of Computational mathematics, vol. 9, no. 6,pp. 717–772, 2009.

[3] E. J. Candes and Y. Plan, “Matrix completion with noise,” Proc. of theIEEE, vol. 98, no. 6, pp. 925–936, 2010.

[4] C. V. Rao, J. B. Rawlings, and D. Q. Mayne, “Constrained stateestimation for nonlinear discrete-time systems: Stability and movinghorizon approximations,” IEEE Trans. on Automatic Control, vol. 48,no. 2, 2003.

[5] F. Zohrizadeh, C. Josz, M. Jin, R. Madani, J. Lavaei, and S. Sojoudi,“Conic relaxations of power system optimization: Theory and algo-rithms,” to appear in European Jrnl. of Operational Research, 2020.

[6] M. S. Asif and J. Romberg, “Sparse recovery of streaming signals usingl1-homotopy,” IEEE Trans. on Signal Processing, vol. 62, no. 16, pp.4209–4223, 2014.

[7] Y. Tang, K. Dvijotham, and S. Low, “Real-time optimal power flow,”IEEE Trans. on Smart Grid, vol. 8, no. 6, pp. 2963–2973, 2017.

[8] E. Hazan, Introduction to Online Convex Optimization. Foundationsand Trends in Optimization, 2015, vol. 2, no. 4.

[9] E. C. Hall and R. M. Willett, “Online convex optimization in dynamicenvironments,” IEEE Jrnl. of Selected Topics in Signal Processing,vol. 9, no. 4, pp. 647–662, 2015.

[10] O. Besbes, Y. Gur, and A. Zeevi, “Non-stationary stochastic optimiza-tion,” Operations Research, vol. 63, no. 5, pp. 1227–1244, 2015.

[11] A. Jadbabaie, S. S. Alexander Rakhlin, and K. Sridharan, “Online opti-mization: Competing with dynamic comparators,” Artificial Intelligenceand Statistics, Proc. of Machine Learning Research, vol. 38, pp. 398–406, 2015.

[12] M. Zinkevich, “Online convex programming and generalized infinitesi-mal gradient ascent,” Proc. of the Int. Conf. on Machine Learning, pp.928–936, 2003.

[13] T. Yang, L. Zhang, R. Jin, and J. Yi, “Tracking slowly movingclairvoyant: Optimal dynamic regret of online learning with true andnoisy gradient,” in Proceedings of The 33rd International Conferenceon Machine Learning, 2016.

[14] A. Mokhtari, S. Shahrampour, A. Jadbabaie, and A. Ribeiro, “Online op-timization in dynamic environments: Improved regret rates for stronglyconvex problems,” in 2016 IEEE 55th Conference on Decision andControl (CDC). IEEE, 2016, pp. 7195–7201.

[15] L. Yang, L. Deng, M. H. Hajiesmaili, C. Tan, and W. S. Wong, “Anoptimal algorithm for online non-convex learning,” Proc. of the ACMon Measurement and Analysis of Computing Syst., vol. 25, 2018.

[16] O.-A. Maillard and R. Munos, “Online learning in adversarial lipschitzenvironments,” in Joint european conference on machine learning andknowledge discovery in databases. Springer, 2010, pp. 305–320.

[17] E. Hazan and S. Kale, “Online submodular minimization.” Journal ofMachine Learning Research, vol. 13, no. 10, 2012.

[18] L. Zhang, T. Yang, R. Jin, and Z.-H. Zhou, “Online bandit learningfor a special class of non-convex losses,” in Proceedings of the AAAIConference on Artificial Intelligence, vol. 29, no. 1, 2015.

[19] X. Gao, X. Li, and S. Zhang, “Online learning with non-convex lossesand non-stationary regret,” in International Conference on ArtificialIntelligence and Statistics. PMLR, 2018, pp. 235–243.

[20] S. Park, J. Mulvaney-Kemp, M. Jin, and J. Lavaei, “Diminishingregret for online nonconvex optimization,” in 2021 American ControlConference (ACC), 2021, pp. 978–985.

[21] Y. Ding, J. Lavaei, and M. Arcak, “Escaping spurious local minimumtrajectories in online time-varying nonconvex optimization,” AmericanControl Conference, 2021.

[22] S. Fattahi, C. Josz, R. Mohammadi, J. Lavaei, and S. Sojoudi, “Absenceof spurious local trajectories in time-varying optimization: A control-theoretic perspective,” 2020, iEEE Conference on Control Technologyand Applications (CCTA).

[23] H. Karimi, J. Nutini, and M. Schmidt, “Linear convergence of gradientand proximal-gradient methods under the polyak-łojasiewicz condition,”in Joint European Conference on Machine Learning and KnowledgeDiscovery in Databases. Springer, 2016, pp. 795–811.

[24] B.T.Polyak, “Gradient methods for minimizing functionals,” Zh. Vychisl.Mat. Mat. Fiz., vol. 3, no. 4, pp. 643–653, 1963.

[25] M. Fazel, R. Ge, S. Kakade, and M. Mesbahi, “Global convergence ofpolicy gradient methods for the linear quadratic regulator,” in Interna-tional Conference on Machine Learning, 2018.

[26] A. Hauswirth, S. Bolognani, and F. Dorfler, “Projected dynamicalsystems on irregular, non-euclidean domains for nonlinear optimization,”SIAM Journal on Control and Optimization, vol. 59, no. 1, pp. 635–668,2021.

[27] A. Nagurney and D. Zhang, Projected dynamical systems and variationalinequalities with applications. Springer Science & Business Media,2012, vol. 2.

[28] J.-B. Hiriart-Urruty and C. Lemarechal, Convex analysis and minimiza-tion algorithms I: Fundamentals. Springer science & business media,2013, vol. 305.

APPENDIX

A. Proof of Lemma 1

Lemma 5 (see Appendix B) establishes the existence ofan open set SDt ⊂ RA

Dt (s) such that x∗t ∈ SDt . Then, by

definition of an open set, SDt ⊇ (B(x∗t , r1) ∩ S) for somer1 > 0. It can be concluded from Proposition 8.5 and Lemma8.3 in [26] that the projected gradient flow system describedin (13) converges to the set of critical points of (1) and thesublevel sets of ft are invariant under this system. Define lt asthe second-lowest objective value among all critical points of(1). (f∗t is the lowest objective value among all critical points.)Then x∗t is the only critical point in the sublevel set Lt(l),which implies Lt(l) ⊆ RACt . By continuity of ft and theunique global optimality of x∗t , we have Lt(l) ⊇ (B(x∗t , r2)∩S) for some r2 > 0. Similarly, Lt(αt) ⊇ (B(x∗t , r3) ∩ S) forsome r3 > 0. Therefore,

Tt(µ, β, s)=(Lt(αt) ∩RADt (s) ∩RACt

)⊃(B(x∗t , r1)∩B(x∗t , r2)∩B(x∗t , r3)∩S

)=(B(x∗t , r) ∩ S

)where r = min{r1, r2, r3} > 0. This completes the proof.

B. Capture property

Lemma 5. Let f be a continuously differentiable function ona compact, convex set S. Let {xk} be a sequence of pointsin S satisfying f(xk+1) ≤ f(xk) generated by the projectedgradient descent method xk+1 = ΠS(xk − s∇f(xk)), whichis convergent in the sense that every limit point of suchsequences is a stationary point of minx∈S f(x). Let x∗ bea local minimum of minx∈S f(x), which is the only stationary

point within some open set. Then there exists an open set Bcontaining x∗ such that if xk ∈ B for some k ≥ 0, thenxk ∈ B for all k ≥ k and {xk} → x∗.

Proof: Let ρ > 0 be a constant such that

f(x∗) < f(x), ∀x 6= x∗ with ‖x− x∗‖ ≤ ρ.

For every δ ∈ [0, ρ], define

φ(δ) = min{x|δ≤‖x−x∗‖≤ρ}

f(x)− f(x∗).

Note that φ(δ) is a monotonically non-decreasing function ofδ, and that φ(δ) > 0 for all δ ∈ (0, ρ]. Given any ε ∈ (0, ρ],let r ∈ (0, ε] be such that

‖x− x∗‖ < r ⇒ ‖x− x∗‖+1

β‖∇f(x)‖ < ε.

Consider the open set

B = {x ∈ S | ‖x− x∗‖ < ε, f(x) < f(x∗) + φ(r)}.

We claim that if xk ∈ B for some k, then xk+1 ∈ B. In orderto prove the claim, assume that xk ∈ B. Then,

φ(‖xk − x∗‖) ≤ f(xk)− f(x∗) < φ(r),

where the first inequality is due to φ(‖xk − x∗‖) =min{x|‖xk−x∗‖≤‖x−x∗‖≤ρ} f(x)−f(x∗) ≤ f(xk)−f(x∗) andthe second inequality is due to the fact that xk ∈ B. Since φ(·)is monotonically non-decreasing, the above statement impliesthat ‖xk − x∗‖ < r, which means that

‖xk − x∗‖+1

β‖∇f(xk)‖ < ε

We also know that

‖xk+1 − x∗‖ = ‖(xk+1 − xk) + (xk − x∗)‖≤ ‖xk+1 − xk‖+ ‖xk − x∗‖

= ‖ΠS(xk − 1

β∇f(xk))−ΠS(xk)‖+ ‖xk − x∗‖

≤ ‖(xk − 1

β∇f(xk)

)− xk‖+ ‖xk − x∗‖ (39)

= ‖ 1

β∇f(xk)‖+ ‖xk − x∗‖ < ε

where equation (39) follows from the non-expansive propertyof the projection operator (when projected onto convex sets)and the final inequality follows from applying equation (B).Therefore by induction, this implies that if xk ∈ B for somek, we have xk ∈ B for all k ≥ k. Let B be the closure of B.Since B is compact, the sequence {xk} will have at least onelimit point, which by assumption must be a stationary pointof minx∈S f(x). The only stationary point of minx∈S f(x)within B is x∗ since ‖x− x∗‖ < ε ≤ ρ for all x ∈ B. Hence,xk → x∗.

C. Proof of Theorem 2

Take the function f to be any ft, t ∈ {1, . . . , T}. Definethe function g(x) :=

√f(x)− f∗ and

x(`) = ΠTS(x(`))

(− 1

β∇f(x(`))

), ∀` ≥ 0 (40)

with x(0) = x. Then, by the fundamental theorem of calculus,we have√

f(x)− f∗ = g(x)−g(x∗)

= −∫ ∞

0

dd`g(x(`)) d` = −

∫ ∞0

∇f(x(`))

2g(x(`))· x(`) d` (41)

The following lemma will be used to establish a lowerbound on the term inside the integral.

Lemma 6. Consider the projected gradient flow (40) withx(0) ∈ S. There exists a unique solution x(`) to this projecteddynamical system, and⟨∇f(x(`)), x(`)

⟩=−1

2β· limh→∞

D(x(`), h)− β ‖x(`)‖2

2. (42)

Proof: The existence and uniqueness of the solution of theprojected dynamical system (40) is guaranteed by [27, Thm.2.5] under the assumptions in Section II-A. Let {x(`)}`≥0

denote the unique solution to (40), and

xε(`) := ΠS

(x(`)− ε

β∇f(x(`))

)= argmin

y∈S

[〈y − x(`),∇f(x(`))〉ε+

β‖y − x(`)‖2

2

]Then, it follows from [28, Sec III Prop. 5.3.5] that

limε↓0

xε(`)− x(`)

ε= ΠTS(x(`))

(− 1

β∇f(x(`))

),

where TS(x(`)) is the tangent cone of S at x(`) ∈ S. By thedefinition of the proximal-gradient,

limh→∞

D(x(`), h) = limε↓0D(x(`), β/ε)

= limε↓0

−2β

ε·miny∈S

[〈∇f(x(`)),y−x(`)〉+ β

2ε‖y−x(`)‖2

]= lim

ε↓0

−2β

ε2·miny∈S

[〈∇f(x(`)),y−x(`)〉ε+ β

2‖y−x(`)‖2

]= lim

ε↓0

−2β

ε2[〈∇f(x(`)),xε(`)−x(`)〉ε+ β

2‖xε(`)−x(`)‖2

]= −2β

[〈∇f(x(`)), x(`)〉+ β

2‖x(`)‖2

]where the last equation is due to the continuity of ‖ · ‖2.Rearranging the above equation yields the desired result.

Returning to the proof of Theorem 2, next we establish alower bound on the term inside the integral in (41).

− ∇f(x(`))

2g(x(`))· x(`)

= − 1

2g(x(`))

⟨∇f(x(`)),ΠTS(x(`))

(− 1

β∇f(x(`))

)⟩=

1

2g(x(`))

(1

2βlimh→∞

D(x(`), h)

+β

2

∥∥∥∥ΠTS(x(`))

(− 1

β∇f(x(`))

)∥∥∥∥2)

≥ 1

2g(x(`))

(D(x(`), β)

2β+β

2

∥∥∥∥ΠTS(x(`))

(− 1

β∇f(x(`))

)∥∥∥∥2)

≥ 1

2g(x(`))

(µ

βg(x(`))2 +

β

2

∥∥∥∥ΠTS(x(`))

(− 1

β∇f(x(`))

)∥∥∥∥2)

≥√µ

2

∥∥∥∥ΠTS(x(`))

(− 1

β∇f(x(`))

)∥∥∥∥The first equality follows from the definition of the gradientflow system and the second equality is due to Lemma 6. Thefirst inequality holds because of Lemma 1 of [23]. The secondinequality applies the fact that x(t) satisfies the proximal-PLinequality with the parameters µ and β. The third inequality isthe result of the arithmetic-geometric mean inequality. Finally,substituting this lower bound into (41) gives√

f(x)− f∗ ≥√µ

2

∥∥∥∥∫ ∞0

ΠTS(x(`))

(− 1

β∇f(x(`))

)d`∥∥∥∥

=

õ

2

∥∥∥∥∫ ∞0

x(`) d`∥∥∥∥ =

õ

2‖x− x∗‖

This completes the proof.

Julie Mulvaney-Kemp is a Ph.D. candidate at theUniversity of California, Berkeley in Industrial En-gineering and Operations Research. She holds a B.S.in Systems Engineering from Washington Universityin St. Louis and an M.S. in Industrial Engineeringand Operations Research from the University ofCalifornia, Berkeley. In 2020 she received a “best ofthe best” conference paper award at the IEEE Power& Energy Society General Meeting and the BestStudent Paper Award from the INFORMS Energy,Natural Resources and the Environment Section.

SangWoo Park received the B.S. degree in Environ-mental Engineering from Johns Hopkins University,Baltimore, MD, in 2016 and the M.S. degree in In-dustrial Engineering and Operations Research fromthe University of California, Berkeley, CA, in 2017.He is currently a Ph.D. candidate in the departmentof Industrial Engineering and Operations Researchfrom the University of California, Berkeley, CA. Heis the recipient of the Best Student Paper Award of2020 American Control Conference.

Ming Jin is an assistant professor in the BradleyDepartment of Electrical and Computer Engineeringat Virginia Tech. He was a postdoctoral researcher inthe Department of Industrial Engineering and Opera-tions Research at University of California, Berkeley.He received his doctoral degree from EECS depart-ment at University of California, Berkeley in 2017.He is the recipient of the Siebel scholarship, 2018Best Paper Award of Building and Environment, andthe 2016 Best Paper Award at the International Con-ference on Mobile Ubiquitous Computing, Systems,

Services and Technologies.

Javad Lavaei received the Ph.D. degree in controland dynamical systems from the California Instituteof Technology, Pasadena, CA, in 2011. He is anAssistant Professor in the Department of IndustrialEngineering and Operations Research, University ofCalifornia, Berkeley, Berkeley, CA. Dr. Lavaei re-ceived multiple awards, including the NSF CAREERAward, Office of Naval Research Young InvestigatorAward, and the Donald P. Eckman Award.