STOCHASTIC LINEAR-QUADRATIC CONTROL WITH CONIC CONTROL …€¦ · STOCHASTIC LINEAR-QUADRATIC CONTROL WITH CONIC CONTROL CONSTRAINTS ON AN INFINITE TIME HORIZON∗ XI CHEN† AND

STOCHASTIC LINEAR-QUADRATIC CONTROL WITH CONICCONTROL CONSTRAINTS ON AN INFINITE TIME HORIZON∗

XI CHEN† AND XUN YU ZHOU‡

SIAM J. CONTROL OPTIM. c© 2004 Society for Industrial and Applied MathematicsVol. 43, No. 3, pp. 1120–1150

Abstract. This paper is concerned with a stochastic linear-quadratic (LQ) control problem inthe infinite time horizon where the control is constrained in a given, arbitrary closed cone, the costweighting matrices are allowed to be indefinite, and the state is scalar-valued. First, the (mean-square, conic) stabilizability of the system is defined, which is then characterized by a set of simpleconditions involving linear matrix inequalities (LMIs). Next, the issue of well-posedness of the un-derlying optimal LQ control, which is necessitated by the indefiniteness of the problem, is addressedin great detail, and necessary and sufficient conditions of the well-posedness are presented. On theother hand, to address the LQ optimality two new algebraic equations a la Riccati, called extendedalgebraic Riccati equations (EAREs), along with the notion of their stabilizing solutions, are intro-duced for the first time. Optimal feedback control as well as the optimal value are explicitly derivedin terms of the stabilizing solutions to the EAREs. Moreover, several cases when the stabilizingsolutions do exist are discussed and algorithms of computing the solutions are presented. Finally,numerical examples are provided to illustrate the theoretical results established.

Key words. stochastic linear-quadratic control, infinite time horizon, constrained control, cone,(conic) stabilizability, well-posedness, extended algebraic Riccati equations

AMS subject classifications. 93E20, 93E15, 49K40

DOI. 10.1137/S0363012903429529

1. Introduction. Linear-quadratic (LQ) control, pioneered by Kalman [17] fordeterministic systems and extended to stochastic systems by Wonham [31, 32] andBismut [5], constitutes, in both theory and applications, an extremely important classof control problems. In recent years, there has been considerable renewed interest instochastic LQ control. In particular, the notion of mean-square stabilizability and de-tectability was introduced in [12]. On the other hand, initiated by Chen, Li, and Zhou[10], extensive research has been carried out in the so-called indefinite stochastic LQcontrol, where, quite contrary to the conventional belief, the cost weighting matricesare allowed to be indefinite; see [11, 2, 1, 33]. Moreover, this new theory has foundapplications in financial portfolio selection; see [35, 19, 18]. For systematic accountsof the deterministic and stochastic LQ theory, refer to [4] and [34], respectively.

A key assumption in the LQ theory at large, deterministic and stochastic alike,is that the control variable is unconstrained. This assumption renders the feedbackcontrol constructed via the Riccati equation automatically admissible, and in turn(along with the underlying LQ structure) makes possible the elegant explicit solutionto the optimal LQ control problem. Because of this, the whole conventional LQapproach would collapse in the presence of any control constraint.

∗Received by the editors June 9, 2003; accepted for publication (in revised form) March 8, 2004;published electronically November 17, 2004.

http://www.siam.org/journals/sicon/43-3/42952.html†Center for Intelligent and Networked Systems, Tsinghua University, Beijing, China (bjchenxi@

tsinghua.edu.cn). The work of this author was done while she was a Postdoctoral Fellow at theDepartment of Systems Engineering and Engineering Management, The Chinese University of HongKong.

‡Department of Systems Engineering and Engineering Management, The Chinese University ofHong Kong, Shatin, NT, Hong Kong ([email protected]). This author’s work was supportedby RGC Earmarked grants CUHK4175/00E and CUHK 4234/01E.

1120

STOCHASTIC LQ CONTROL WITH CONIC CONTROL CONSTRAINT 1121

On the other hand, from a practical point of view, LQ control with control con-straints is a well-defined and sensible problem. For example, in many real applicationsthe control variable is required to take only nonnegative values. The mean-varianceportfolio selection problem with short-selling prohibition exemplifies such problems.Other applications include models in medicine, chemistry, and economics where sys-tem inputs are inherently constrained.

There have been some attempts in dealing with deterministic LQ problems withpositive controls or, more general, with controls contained in a given cone. For ex-ample, controllability for linear system x = Ax+Bu with positive/conic controls wasstudied in [26, 7, 25, 14, 8, 22]. These papers investigated the necessary and sufficientconditions of different types of controllability (null-controllability, global controllabil-ity, differential controllability, etc.). Later, conic stabilization was addressed in [23].In a recent work on positive feedback stabilization [30], a stabilizing positive feedbackcontroller was derived based on the pole placement technique.

Deterministic continuous-time LQ optimal control problems with positive con-trollers were studied in [21, 9, 13]. Discrete-time versions can be found in [27, 28].In these works, however, only some necessary and sufficient conditions for optimalitywere derived based on Pontryagin’s maximum principle and/or Bellman’s dynamicprogramming, and some numerical schemes were suggested. The special LQ struc-ture was not fully taken advantage of, and no explicit result comparable to those ofunconstrained control was obtained.

As for the constrained stochastic LQ control, to the best of our knowledge ithas never been studied by anyone else in the literature. A related, albeit specific,problem is the mean-portfolio selection model with no-shorting constraint solved inLi, Zhou, and Lim [18], which was formulated as a stochastic LQ control problemwith positive controls in a finite time horizon. The approach developed in [18] isnevertheless rather ad hoc (via the Hamilton–Jacobi–Bellman equation and viscositysolution theory) and by no means suggests a remedy for a more general problem.More recently, a stochastic LQ control problem with conic control constraint, randomcoefficients, as well as possibly singular cost weighting matrices in a finite time horizonwas solved by Hu and Zhou [15], with explicit solutions based on Tanaka’s formulaand the backward stochastic differential equation theory.

In this paper, we study stochastic LQ control in the infinite time horizon, wherethe control variable is constrained in a cone (which includes the problem with positivecontrols as a special case). Moreover, the problem is allowed to be indefinite in thesense that the cost weighting matrices are possibly indefinite. A main assumption ofthe paper is that the state variable is scalar-valued. Note that this assumption is validin many meaningful practical applications, in particular in the area of finance wherethe one-dimensional wealth process is typically taken as the state. The investigationin this paper centers around several key issues associated with the problem, namely,conic stabilizability, well-posedness, and optimality. Conic stabilizability refers to thequestion of whether the system can be stabilized by a control satisfying the given conicconstraint. It arises from the infinity of the time horizon under consideration, and isquite different from the normal stabilizability for unconstrained control. In this paperwe will derive simple necessary and sufficient conditions for the conic stabilizability.The second issue, well-posedness of the LQ problem, becomes an issue because theproblem is indefinite. To ensure well-posedness the problem data must coordinatewell, which will be characterized in this paper by the nonemptiness of certain sets inthe real space. Finally, for optimality, we aim to obtain explicit solutions comparable

1122 XI CHEN AND XUN YU ZHOU

to those classical unconstrained-control counterparts. To this end, we will introduce,for the first time in this paper, two algebraic equations termed the extended algebraicRiccati equations (EAREs) along with the notion of their stabilizing solutions. Thenit will be shown that the existence of the stabilizing solutions is sufficient for theexistence of optimal feedback control of the constrained LQ problem, and explicitforms of the optimal feedback control as well as the optimal cost value will be derivedin terms of the stabilizing solutions. Furthermore, several important cases, includingthat of the definite LQ control, will be discussed where stabilizing solutions to theEAREs do exist, and algorithms for computing these solutions will be presented. Todemonstrate the theoretical results obtained, numerical examples will be given.

The rest of the paper is organized as follows. In section 2, the constrained stochas-tic LQ control problem is formulated and conic stabilizability of the system defined.As a prelude to the main analysis, two technical lemmas are presented in section 3.The subsequent three sections, sections 4, 5, and 6, are devoted to the three majorissues, namely, stabilizability, well-posedness, and optimality, respectively. Numericalexamples are reported in section 7. Finally, section 8 concludes the paper.

2. Problem formulation.Notation. We make use of the following basic notation in this paper:

Rn : the set of n-dimensional column vectors.R : = R1.Rn

+ : the subset of Rn consisting of elements with nonnegative components.Rm×n : the set of all m× n matrices.Sn×n : the set of all n× n symmetric matrices.

|v| : =√v′v, v ∈ Rn.

x+ : = maxx, 0, x ∈ R.x− : = max−x, 0, x ∈ R.M ′ : the transpose of a matrix M .M > 0 : the square matrix M is positive definite.M ≥ 0 : the square matrix M is positive semidefinite.E[x] : the expectation of a random variable x.

Let (Ω,F ,P;Ft) be a given filtered probability space with a standard Ft-adapted,k-dimensional Brownian motion w(t) ≡ (w1(t), w2(t), . . . , wk(t))

′ on [0,+∞). LetΓ ⊆ Rm be a given closed cone; i.e., Γ is closed, and if u ∈ Γ, then αu ∈ Γ ∀α ≥ 0.Typical examples of such a cone are Γ = Rm

+ , Γ = u ∈ Rm|Mu ≤ 0, and Γ =u ∈ Rm|Mu = 0, where M ∈ Rn×m, or the so-called second-order cone (cf., e.g.,[20, p. 221]) Γ = (t, u) ∈ R × Rm−1|t ≥ |u|. Next, for M ∈ Sm×m if there isδ > 0 so that K ′MK ≥ δ|K|2 ∀0 = K ∈ Γ, then we denote M |Γ > 0. Similarly wewrite M |Γ ≥ 0 if K ′MK ≥ 0 ∀K ∈ Γ. Clearly M > 0 (respectively, M ≥ 0) impliesM |Γ > 0 (respectively, M |Γ ≥ 0). Finally, define the following Hilbert space:

L2F (Γ) =

φ(·, ·) : [0,+∞) × Ω → Γ

∣∣∣φ(·, ·) is Ft-adapted, measurable,

and E

∫ +∞

0

|φ(t, ω)|2dt < +∞

with the norm ‖φ(·, ·)‖ := (E∫ +∞0

|φ(t, ω)|2dt) 12 .


Consider the Ito stochastic differential equation (SDE)

⎧⎪⎪⎪⎨⎪⎪⎪⎩dx(t) = [Ax(t) + Bu(t)]dt +

k∑j=1

[Cjx(t) + Dju(t)]dwj(t), t ∈ [0,∞),

x(0) = x0 ∈ R,

(1)

where A,Cj ∈ R and B,Dj ∈ R1×m. A process u(·) is called a control (with conicconstraint) if u(·) ∈ L2

F (Γ).Definition 2.1. A control u(·) ∈ L2

F (Γ) is called (mean-square, conic) stabilizingwith respect to x0 if the corresponding state x(·) of (1) with the initial state x0 satisfieslimt→+∞ E|x(t)|2 = 0.

Definition 2.2. System (1) is said to be (mean-square, conic) stabilizable ifthere is a feedback control of the form u(t) = K+x

+(t) + K−x−(t), where K+ and

K− are constant vectors with K+ ∈ Γ and K− ∈ Γ, which is (mean-square, conic)stabilizing with respect to every initial state x0.

Now, for any x0 ∈ R, we define the set of admissible controls

Ux0:= u(·) ∈ L2

F (Γ)|u(·) is stabilizing with respect to x0.(2)

If u(·) ∈ Ux0and x(·) is the corresponding solution of (1), then (x(·), u(·)) is called

an admissible pair (with respect to x0). For each (x0, u(·)) ∈ R × Ux0the associated

cost to system (1) is

J(x0;u(·)) := E

∫ +∞

0

[Qx(t)2 + u(t)′Ru(t)]dt,(3)

where Q ∈ R and R ∈ Sm×m. Note that here Q and R are not assumed to benonnegative/positive semidefinite. As a result, J(x0;u(·)) is not necessarily boundedbelow.

The indefinite LQ control problem with conic constraint entails minimizing thecost functional (3), for a given x0, subject to (1) and u(·) ∈ Ux0 . Such a problem isdenoted as problem (LQ). An admissible control u(·) ∈ Ux0 is called optimal (withrespect to x0) if u(·) achieves the infimum of (3), and in this case problem (LQ) isalso referred to as attainable (with respect to x0).

The value function V is defined as

V (x0) := infu(·)∈Ux0

J(x0;u(·)), x0 ∈ R,(4)

where V (x0) is set to be +∞ in the case when Ux0 is empty.Definition 2.3. Problem (LQ) is called well-posed if

V (x0) > −∞ ∀x0 ∈ R.(5)

It is well known that V is a continuous, though not necessarily differentiable,function when problem (LQ) is well-posed. Also note that a well-posed problem isnot necessarily attainable with respect to any x0 (see Example 6.2).


3. Two lemmas. In this section we present two lemmas that are useful in whatfollows.

Lemma 3.1 (Tanaka’s formula). Let X(t) be a continuous semimartingale. Then

dX+(t) = 1(X(t)>0)dX(t) +1

2dL(t),

dX−(t) = −1(X(t)≤0)dX(t) +1

2dL(t),

(6)

where L(·) is an increasing continuous process, called the local time of X(·) at 0,satisfying

∫ t

0

|X(s)|dL(s) = 0, P-a.s.(7)

In particular, X+(t) and X−(t) are semimartingales.Proof. See, for example, [24, Chapter VI, Theorem 1.2 and Proposition 1.3].Lemma 3.2. Let constants N+, N− ∈ R be given. Then for any admissible pair

(x(·), u(·)) with respect to x0, we have, for every t ≥ 0,

E

∫ t

0

[Qx(s)2 + u(s)′Ru(s)]ds

= N+(x+0 )2 + N−(x−

0 )2 − E[N+x+(t)2] − E[N−x

−(t)2]

+ E

∫ t

0

⎧⎨⎩Qx(s)2 +

⎛⎝2A +

k∑j=1

C2j

⎞⎠N+x

+(s)2 +

⎛⎝2A +

k∑j=1

C2j

⎞⎠N−x

−(s)2

+ u(s)′

⎡⎣R + 1(x(s)>0)N+

k∑j=1

D′jDj + 1(x(s)≤0)N−

k∑j=1

D′jDj

⎤⎦u(s)

+ 2

⎛⎝B +

k∑j=1

CjDj

⎞⎠u(s)N+x

+(s) − 2

⎛⎝B +

k∑j=1

CjDj

⎞⎠u(s)N−x

−(s)

⎫⎬⎭ ds.

(8)

Proof. Let x(·) be the solution of (1) under an arbitrary u(·) ∈ Ux0. By

Lemma 3.1, we have

dx+(t) = 1(x(t)>0)[Ax(t) + Bu(t)]dt + 1(x(t)>0)

k∑j=1

[Cjx(t) + Dju(t)]dwj(t)

+1

2dL(t),

dx−(t) = −1(x(t)≤0)[Ax(t) + Bu(t)]dt− 1(x(t)≤0)

k∑j=1

[Cjx(t) + Dju(t)]dwj(t)

+1

2dL(t).


In the above equations, L(·) is the local time as specified in Lemma 3.1. ApplyingIto’s formula, we get

d[N+x+(t)2]

= 2N+x+(t)

⎧⎨⎩1(x(t)>0)[Ax(t)+Bu(t)]dt+1(x(t)>0)

k∑j=1

[Cjx(t)+Dju(t)]dwj(t)+1

2dL(t)

⎫⎬⎭

+ 1(x(t)>0)N+

k∑j=1

[Cjx(t) + u(t)′D′j ][Cjx(t) + Dju(t)]dt

=

⎧⎨⎩N+[2Ax+(t)2+2Bu(t)x+(t)+1(x(t)>0)

k∑j=1

(Cjx(t)+u(t)′D′j)(Cjx(t)+Dju(t))]

⎫⎬⎭dt

+N+

k∑j=1

[2Cjx+(t)2 + 2Dju(t)x+(t)]dwj(t)

=

⎡⎣⎛⎝2A +

k∑j=1

C2j

⎞⎠N+x

+(t)2 + 2Bu(t)N+x+(t) + 2

k∑j=1

CjDju(t)N+x+(t)

+ 1(x(t)>0)N+u(t)′k∑

j=1

D′jDju(t)

⎤⎦dt +

⎡⎣N+(t)

k∑j=1

(2Cjx+(t)2+2Dju(t)x+(t))

⎤⎦dwj(t),

(9)

where we have used the fact that x+(t)dL(t) = 0 by virtue of (7). Similarly, for theconstant N− ∈ R,

d[N−x−(t)2]

=

⎧⎨⎩N−

⎡⎣2Ax−(t)2−2Bu(t)x−(t)+1(x(t)≤0)

k∑j=1

(Cjx(t)+u(t)′D′j)(Cjx(t)+Dju(t))

⎤⎦⎫⎬⎭ dt

+ N−

k∑j=1

[2Cjx−(t)2 − 2Dju(t)x−(t)]dwj(t)

=

⎡⎣⎛⎝2A +

k∑j=1

C2j

⎞⎠N−x

−(t)2 − 2Bu(t)P−x−(t) − 2

k∑j=1

CjDju(t)N−x−(t)

+ 1(x(t)≤0)N−u(t)′k∑

j=1

D′jDju(t)

⎤⎦dt+

⎡⎣N−(t)

k∑j=1

(2Cjx−(t)2−2Dju(t)x−(t))

⎤⎦dwj(t).

(10)

Fix t ≥ 0 and define a sequence of stopping times

τn := inf

⎧⎪⎨⎪⎩r ∈ [0, t] :

∫ r

0

∣∣∣∣∣∣N+(s)

k∑j=1

(2Cjx+(s)2 + 2Dju(s)x+(s))

∣∣∣∣∣∣2

ds

+

∫ r

0

∣∣∣∣∣∣N−(s)

k∑j=1

(2Cjx−(s)2 − 2Dju(s)x−(s))

∣∣∣∣∣∣2

ds ≥ n

⎫⎪⎬⎪⎭ , n = 1, 2, . . . ,


where inf ∅ := t. It is clear that τn ↑ t as n → +∞, due to E∫ t

0(|x(s)|2 + |u(s)|2)ds <

+∞. Now, summing up (9) and (10), taking integration from 0 to τn and then,taking expectation, we obtain (8) with t replaced by τn. Thus, (8) follows by sendingn → +∞ together with Fatou’s lemma.

4. Conic stabilizability. In this section we address the issue of the conic sta-bilizability of system (1). Notice that conic stabilizability is different from the usualstabilizability with unconstrained controls, for clearly the former requires more strin-gent conditions. Here we will give a complete characterization of conic stabilizabilityin terms of simple conditions involving linear matrix inequalities (LMIs).

Introduce a pair of functions F+, F− from Γ to R:

F+(K) := 2A +

k∑j=1

C2j + 2

⎛⎝B +

k∑j=1

CjDj

⎞⎠K + K ′

k∑j=1

D′jDjK,(11)

and

F−(K) := 2A +

k∑j=1

C2j − 2

⎛⎝B +

k∑j=1

CjDj

⎞⎠K + K ′

k∑j=1

D′jDjK.(12)

Theorem 4.1. The following assertions are equivalent.(i) System (1) is mean-square conic stabilizable.(ii) There exist K+ ∈ Γ and K− ∈ Γ such that F+(K+) < 0 and F−(K−) < 0. In

this case the feedback control u(t) = K+x+(t) + K−x

−(t) is stabilizing.(iii) There exist K+ ∈ Γ and K− ∈ Γ such that

⎛⎜⎜⎝2A +

k∑j=1

C2j + 2

⎛⎝B +

k∑j=1

CjDj

⎞⎠K+ K ′

+D′

DK+ −I

⎞⎟⎟⎠ < 0,(13)

⎛⎜⎜⎝2A +

k∑j=1

C2j − 2

⎛⎝B +

k∑j=1

CjDj

⎞⎠K− K ′

−D′

DK− −I

⎞⎟⎟⎠ < 0,(14)

where D ∈ Rm×m satisfies D′D =∑k

j=1 D′jDj. In this case the feedback control

u(t) = K+x+(t) + K−x

−(t) is stabilizing.Proof. Take a feedback control u(t) = K+x

+(t) + K−x−(t) and consider the

corresponding state x(·) with an initial state x0. Note that by standard SDE theory(cf., e.g., [16]) such x(·) uniquely exists. Moreover,

E sup0≤t≤T

|x(t)|pdt < +∞ ∀T > 0 ∀p ≥ 0.(15)


Making use of (9) with N+ = 1 and u(t) = K+x+(t) + K−x

−(t), we obtain

dx+(t)2 =

⎡⎣2A +

k∑j=1

C2j + 2

⎛⎝B +

k∑j=1

CjDj

⎞⎠K+ + K ′

+

k∑j=1

D′jDjK+

⎤⎦x+(t)2dt

+k∑

j=1

(2Cj + 2DjK+)x+(t)2dwj(t)

= F+(K+)x+(t)2dt +

k∑j=1

(2Cj + 2DjK+)x+(t)2dwj(t).

Taking integration and then expectation yields (after a localization argument as inthe proof of Lemma 3.2)

dE[x+(t)2]

dt= F+(K+)E[x+(t)2],(16)

where the expectation of the Ito integral vanishes due to (15). Similarly, we have

dE[x−(t)2]

dt= F−(K−)E[x−(t)2].(17)

Hence, the equivalence between the assertions (i) and (ii) is evident noting thatlimt→+∞ E|x(t)|2 = 0 if and only if limt→+∞ E|x+(t)|2 = 0 and limt→+∞ E|x−(t)|2 =0.

Finally, the equivalence between the assertions (ii) and (iii) follows from Schur’slemma [6].

Conditions (13) and (14) are in terms of LMIs. On the other hand, some of theconic constraint can also be expressed as LMIs, e.g., when Γ = Rm

+ or when Γ isa second-order cone. Hence, Theorem 4.1(iii) provides an easy way of numericallychecking the stabilizability of system (1) due to the availability of many LMI solvers.Note that in general there may exist many feasible solutions to LMIs.

Denote the two sets of feedback gains as

K+ := K ∈ Γ|F+(K) < 0, K− := K ∈ Γ|F−(K) < 0.

Then Theorem 4.1 implies the followingTheorem 4.2. System (1) is conic stabilizable if and only if K+ = ∅ and K− = ∅.Corollary 4.1. If 2A +

∑kj=1 C

2j < 0, then system (1) is conic stabilizable.

Proof. In this case 0 ∈ K+ ∩ K−.Proposition 4.1. We have the following results.(i) K+ and K− are convex sets if Γ is a convex set.

(ii) K+ and K− are bounded if∑k

j=1 D′jDj |Γ > 0.

Proof. (i) Define the following operator M+ from Γ to S(m+1)×(m+1):

M+(K) =

⎛⎜⎜⎝2A +

k∑j=1

C2j + 2

⎛⎝B +

k∑j=1

CjDj

⎞⎠K K ′D′

DK −I

⎞⎟⎟⎠ ,

where D′D =∑k

j=1 D′jDj . By Theorem 4.1, K+ can be equivalently represented as

K+ = K ∈ Γ|M+(K) < 0. Thus the convexity of K+ follows from that of Γ together


with the fact that the operator M+ is affine. Similarly we can show the convexityof K−.

(ii) If K+ is unbounded, then there is a sequence Kn ⊂ K+ so that |Kn| → +∞.

Since∑k

j=1 D′jDj |Γ > 0, we have K ′

n

∑kj=1 D

′jDjKn ≥ δ|Kn|2 → +∞ as n → +∞,

where δ > 0 is some constant. Therefore F+(Kn) → +∞ as n → +∞. This contra-dicts F+(Kn) < 0 ∀n. Hence K+ is bounded. Similarly, K− is bounded.

5. Well-posedness. Since the cost weighting matrices Q and R are allowedto be indefinite, the well-posedness of the problem is no longer automatic or trivial(as opposed to the classical definite case when Q ≥ 0 and R > 0). In fact, the well-posedness for an indefinite LQ control problem is a prerequisite for the optimality andis an interesting problem in its own right. In this section we will carry out an extensiveinvestigation on the well-posedness and some related issues, including necessary andsufficient conditions for the well-posedness in terms of the nonemptiness of certainsets.

5.1. Representation of value function. In this subsection we present thefollowing representation result, which is a key to many results of this paper.

Proposition 5.1. Problem (LQ) is well-posed if and only if the value functioncan be represented as

V (x0) = P+(x+0 )2 + P−(x−

0 )2 ∀x0 ∈ R(18)

for some P+, P− ∈ R.Proof. We prove only the “only if” part, as the “if” part is evident.Assume that problem (LQ) is well-posed. Fix any x > 0, y > 0. Since V (y) >

−∞, for any ε > 0 there is uε(·) ∈ Uy along with the corresponding state xε(·) (withthe initial state y) satisfying

V (y) ≥ J(y;uε(·)) − ε = E

∫ +∞

0

[Qxε(t)2 + uε(t)′Ruε(t)]dt− ε.(19)

Now, as x > 0, the linearity of the dynamics (1) and the conic control constraintensure that xuε(·) ∈ Uxy with the corresponding state xxε(·). Hence it follows from(19) that

V (y) ≥ 1

x2E

∫ +∞

0

[Q|xxε(t)|2 + (xuε(t))′R(xuε(t))]dt− ε ≥ 1

x2V (xy) − ε.(20)

Sending ε → 0 we obtain

V (xy) ≤ V (y)x2 ∀x > 0, y > 0.(21)

Similarly, one can show that

V (xy) ≤ V (−y)x2 ∀x < 0, y > 0.(22)

Now for any x > 0, by (21) we have V (x) ≤ V (1)x2. On the other hand, (21)also implies V (1) = V (x 1

x ) ≤ 1x2V (x). So we have shown that

V (x) = V (1)x2 ∀x > 0.(23)

Similarly, in view of (22) we can prove that

V (x) = V (−1)x2 ∀x < 0.(24)


Finally, the continuity of the value function along with (23) yields

V (0) = 0.(25)

The desired result (18) thus follows from (23)–(25) with P+ := V (1) andP− := V (−1).

Remark 5.1. The preceding proposition, which suggests the form of the valuefunction when the underlying LQ problem is well-posed, is crucial for all the mainresults in this paper. In fact, the proofs for the stabilizability in the previous section,the characterization of the well-posedness in this section, and the optimality in thenext section are all inspired by this result. This also explains why one needs to applyTanaka’s formula to evaluate dx+(t) and dx−(t), as we have seen in the previoussection and will continue to see in the subsequent sections.

Remark 5.2. We saw that the value function for the constrained LQ problem isnot smooth.

5.2. Characterization of well-posedness. Define the following functions fromR to R ∪ −∞:

Φ+(P ) := infK∈Γ

⎡⎣K ′

⎛⎝R + P

k∑j=1

D′jDj

⎞⎠K + 2

⎛⎝B +

k∑j=1

CjDj

⎞⎠PK

⎤⎦ ,

Φ−(P ) := infK∈Γ

⎡⎣K ′

⎛⎝R + P

k∑j=1

D′jDj

⎞⎠K − 2

⎛⎝B +

k∑j=1

CjDj

⎞⎠PK

⎤⎦ .

Remark 5.3. Since 0 ∈ Γ, we must have

Φ+(P ) ≤ 0 Φ−(P ) ≤ 0 ∀P ∈ R.(26)

On the other hand, Φ+(P ) and Φ−(P ) have finite values if (R+P∑k

j=1 D′jDj)|Γ > 0.

Indeed, in this case there exist constants α1 = α1(P ) > 0 and α2 = α2(P ) > 0 suchthat

K ′

⎛⎝R + P

k∑j=1

D′jDj

⎞⎠K + 2

⎛⎝B +

k∑j=1

CjDj

⎞⎠PK ≥ α1|K|2 − α2|K|

= α1|K|(|K| − α2

α1

)∀K ∈ Γ.

If |K| > α2

α1, then the above expression is positive. Taking (26) into consideration

we conclude

Φ+(P ) = infK∈Γ,|K|≤α2

α1

⎡⎣K ′

⎛⎝R + P

k∑j=1

D′jDj

⎞⎠K + 2

⎛⎝B +

k∑j=1

CjDj

⎞⎠PK

⎤⎦ > −∞.

Hence, Φ+(P ) is finite. The same is true for Φ−(P ).Next, we define the following two sets:

P+ :=

⎧⎨⎩P ∈ R

∣∣∣∣∣∣⎛⎝2A +

k∑j=1

C2j

⎞⎠P + Q + Φ+(P ) ≥ 0,

⎛⎝R + P

k∑j=1

D′jDj

⎞⎠∣∣∣∣∣∣Γ

≥ 0

⎫⎬⎭ ,

P− :=

⎧⎨⎩P ∈ R

∣∣∣∣∣∣⎛⎝2A +

k∑j=1

C2j

⎞⎠P + Q + Φ−(P ) ≥ 0,

⎛⎝R + P

k∑j=1

D′jDj

⎞⎠∣∣∣∣∣∣Γ

≥ 0

⎫⎬⎭ .


The following is the main result of the section, which characterizes the well-posednessof problem (LQ) by the nonemptiness of the sets P+ and P−.

Theorem 5.1. Assume that system (1) is conic stabilizable. Then problem (LQ)is well-posed if and only if P+ = ∅ and P− = ∅. Moreover, in this case

V (x0) ≥ P+(x+0 )2 + P−(x−

0 )2 ∀x0 ∈ R, ∀P+ ∈ P+, P− ∈ P−.(27)

Proof. First we prove the “if” part. For any x0 ∈ R let x(·) be the solution of (1)under an arbitrary u(·) ∈ Ux0 . Pick any P+ ∈ P+ and P− ∈ P−. By Lemma 3.2, wehave

E

∫ t

0


= P+(x+0 )2 + P−(x−

0 )2 − E[P+x+(t)2] − E[P−x

−(t)2]

+ E

∫ t

0

⎧⎨⎩Qx(s)2 +

⎛⎝2A +

k∑j=1

C2j

⎞⎠P+x

+(s)2 +

⎛⎝2A +

k∑j=1

C2j

⎞⎠P−x

−(s)2

+ u(s)′

⎡⎣R + 1(x(s)>0)P+

k∑j=1

D′jDj + 1(x(s)≤0)P−

k∑j=1

D′jDj

⎤⎦u(s)

+ 2

⎛⎝B +

k∑j=1

CjDj

⎞⎠u(s)P+x

+(s) − 2

⎛⎝B +

k∑j=1

CjDj

⎞⎠u(s)P−x

−(s)

⎫⎬⎭ ds.

(28)

Denote by ψ(x(s), u(s)) the integrand on the right-hand side of (28) and fix s ∈ [0, t].If x(s) > 0, then write u(s) = Kx(s) (note that K may depend on s). Since u(s) ∈ Γand Γ is a cone, we have K ∈ Γ. Hence at s, bearing in mind that 1(x(s)≤0) = 0, wehave

ψ(x(s), u(s))

= Qx(s)2 +

⎛⎝2A +

k∑j=1

C2j

⎞⎠P+x(s)2 + K ′

⎡⎣R + P+

k∑j=1

D′jDj

⎤⎦Kx(s)2

+ 2

⎛⎝B +

k∑j=1

CjDj

⎞⎠KP+x(s)2

=

⎡⎣⎛⎝2A+

k∑j=1

C2j

⎞⎠P+ + Q + K ′

⎛⎝R + P+

k∑j=1

D′jDj

⎞⎠K + 2

⎛⎝B +

k∑j=1

CjDj

⎞⎠KP+

⎤⎦x(s)2

≥

⎡⎣⎛⎝2A +

k∑j=1

C2j

⎞⎠P+ + Q + Φ+(P+)

⎤⎦x(s)2

≥ 0.(29)

If x(s) < 0, then write u(s) = −Kx(s). Again K ∈ Γ. An argument similar to thatabove yields

ψ(x(s), u(s)) ≥

⎡⎣⎛⎝2A +

k∑j=1

C2j

⎞⎠P− + Q + Φ−(P−)

⎤⎦x(s)2 ≥ 0


at s. Finally, if x(s) = 0 at s, then

ψ(x(s), u(s)) = u(s)′

⎡⎣R + P−

k∑j=1

D′jDj

⎤⎦u(s) ≥ 0.

The preceding analysis shows that it always holds that ψ(x(s), u(s)) ≥ 0 ∀s ∈ [0, t].Consequently, it follows from (28) that

E

∫ t

0

[Qx(s)2 + u(s)′Ru(s)]ds ≥ P+(x+0 )2 + P−(x−

0 )2 − E[P+x+(t)2]

− E[P−x−(t)2].

Letting t → +∞ and noting that u(·) is conic stabilizing, we obtain

J(x0;u(·)) = limt→+∞

E

∫ t

0


≥ P+(x+0 )2 + P−(x−

0 )2.

Since u(·) ∈ Ux0is arbitrary, we conclude V (x0) ≥ P+(x+

0 )2 +P−(x−0 )2 > −∞. Hence

problem (LQ) is well-posed.To prove the “only if” part, suppose that the LQ problem is well-posed. Then by

Proposition 5.1 the value function has the following representation:

V (x) = P+(x+)2 + P−(x−)2 ∀x ∈ R.

We want to show that P+ ∈ P+, P− ∈ P−. To this end, applying the optimalityprinciple of dynamic programming and noting the time-invariance of the underlyingsystem, we obtain

P+(x+0 )2 + P−(x−

0 )2

≤ E

∫ h

0

[Qx(t)2 + u(t)′Ru(t)]dt + P+[x+(h)]2 + P−[x−(h)]2

∀h > 0, ∀u(·) ∈ Ux0 , ∀x0 ∈ R.

(30)

Using the above and applying Lemma 3.2, we obtain

E

∫ h

0

ψ(x(s), u(s))ds ≥ 0 ∀h > 0, ∀u(·) ∈ Ux0 , ∀x0 ∈ R,(31)

where, as before, the mapping ψ is defined via the integrand on the right-hand sideof (28). Set x0 > 0 and take the following control

u1(t) :=

K, 0 ≤ t < 1,K+x

+(t) + K−x−(t), t ≥ 1,

where K ∈ Γ is arbitrarily fixed and K+x+(t)+K−x

−(t) is a conic stabilizing feedbackcontrol which exists due to the stabilizability assumption. Clearly u1(·) ∈ Ux0 , and


let x1(·) be the corresponding state. Since x1(s) → x0 and u1(s) → K as s → 0,P-a.s., we have

ψ(x1(s), u1(s))

→

⎡⎣⎛⎝2A +

k∑j=1

C2j

⎞⎠P+ + Q

⎤⎦x2

0 + 2(B +

∑kj=1 CjDj

)KP+x0

+ K ′

⎛⎝R + P+

k∑j=1

D′jDj

⎞⎠K as s → 0, P-a.s.

Thus appealing to (31), the dominated convergence theorem yields

0 ≤ 1

hE

∫ h

0

ψ(x1(s), u1(s))ds

→

⎡⎣⎛⎝2A +

k∑j=1

C2j

⎞⎠P+ + Q

⎤⎦x2

0 + 2(B +

∑kj=1 CjDj

)KP+x0

+ K ′

⎛⎝R + P+

k∑j=1

D′jDj

⎞⎠K as h → 0.

Letting x0 → 0 we obtain K ′(R+P+

∑kj=1 D

′jDj)K ≥ 0. The arbitrariness of K ∈ Γ

then implies (R+P+

∑kj=1 D

′jDj)|Γ ≥ 0. On the other hand, take x0 > 0 and consider

the following feedback control

u2(t) :=

Kx+(t) + Kx−(t), 0 ≤ t < 1,K+x

+(t) + K−x−(t), t ≥ 1,

where K ∈ Γ and K ∈ Γ are arbitrarily fixed. Let x2(·) be the state under u2(·).Noting x2(s) → x0 and u2(s) = Kx+

2 (s)+Kx−2 (s) → Kx0 as s → 0, P-a.s., we obtain

ψ(x2(s), u2(s))

→

⎡⎣⎛⎝2A +

k∑j=1

C2j

⎞⎠P+ + Q + K ′

⎛⎝R + P+

k∑j=1

D′jDj

⎞⎠K + 2

⎛⎝B +

k∑j=1

CjDj

⎞⎠KP+

⎤⎦x2

0

as s → 0, P-a.s.

An analysis similar to the preceding one leads to⎛⎝2A +

k∑j=1

C2j

⎞⎠P+ +Q+K ′

⎛⎝R + P+

k∑j=1

D′jDj

⎞⎠K+2

⎛⎝B +

k∑j=1

CjDj

⎞⎠KP+ ≥ 0.

Since K ∈ Γ is arbitrary, we arrive at (2A +∑k

j=1 C2j )P+ + Q + Φ+(P+) ≥ 0. So far

we have shown P+ ∈ P+. Similarly, we can prove P− ∈ P−.Finally, the inequality (27) has been proved in the proof of the “if” part.Remark 5.4. From the above proof we see that the stabilizability assumption is

in fact not necessary for the “if” part of the theorem.


Remark 5.5. The above theorem tells that the positive definiteness or positivesemidefiniteness of Q and R are not necessary for problem (LQ) to be well-posed.

Corollary 5.1. If R|Γ ≥ 0 and Q ≥ 0, then problem (LQ) is well-posed.Proof. In this case 0 ∈ P+ ∩ P−.Proposition 5.2. P+ and P− are both convex sets (hence they are both inter-

vals). Moreover, if system (1) is stabilizable and problem (LQ) is well-posed, then P+

and P− each has a finite maximum element.Proof. The convexity of P+ and P− are clear noticing that the functions Φ+(P )

and Φ−(P ) are concave in P . To prove the existence of the finite maximum elements,we note that Proposition 5.1 provides

V (x0) = P ∗+(x+

0 )2 + P ∗−(x−

0 )2 ∀x0 ∈ R(32)

for some P ∗+, P

∗− ∈ R. Moreover, the proof of Theorem 5.1 implies P ∗

+ ∈ P+ andP ∗− ∈ P−. Hence it follows from (27) that P ∗

+ and P ∗− are the maximum elements of

P+ and P−, respectively.Remark 5.6. Proposition 5.2 indicates that if system (1) is stabilizable and prob-

lem (LQ) is well-posed, then the infimum value of problem (LQ) or, equivalently, theP+ and P− in the representation of the value function as stipulated in Proposition 5.1can be obtained by solving the following two mathematical programming problems,respectively:

maximize P

subject to

⎧⎪⎪⎪⎪⎪⎪⎨⎪⎪⎪⎪⎪⎪⎩

⎛⎝2A +

k∑j=1

C2j

⎞⎠ P + Q + Φ+(P ) ≥ 0,

⎛⎝R + P

k∑j=1

D′jDj

⎞⎠∣∣∣∣∣∣Γ

≥ 0,

(33)

and

maximize P

subject to

⎧⎪⎪⎪⎪⎪⎪⎨⎪⎪⎪⎪⎪⎪⎩

⎛⎝2A +

k∑j=1

C2j

⎞⎠ P + Q + Φ−(P ) ≥ 0,

⎛⎝R + P

k∑j=1

D′jDj

⎞⎠∣∣∣∣∣∣Γ

≥ 0.

(34)

5.3. An algorithm. Theorem 5.1 stipulates that it suffices to check the non-emptiness of P+ and P− or the feasibility of the problems (33) and (34) in order toverify the well-posedness of a given LQ problem. However, it is sometimes hard tocheck numerically the aforementioned feasibility because, on one hand, the functionsΦ+(·) and Φ−(·) in general do not have analytical forms, and on the other hand, the

constraint (R+P∑k

j=1 D′jDj)|Γ ≥ 0 is usually very hard to verify for a general cone

Γ (except second-order cones for which the constraint can be reformulated as an LMI;see [29, Theorem 1]).

In this subsection we give an algorithm that can check the well-posedness moredirectly. First we need a lemma.


Lemma 5.1. Assume that problem (LQ) is well-posed. Given K+ ∈ K+ and

K− ∈ K−, set P+ := −Q+K′+RK+

F+(K+) and P− := −Q+K′−RK−

F−(K−) . Then

P+ ≤ P+ ∀P+ ∈ P+ and P− ≤ P− ∀P− ∈ P−.(35)

Proof. By their definitions P+ and P− satisfy, respectively,

⎛⎝2A +

k∑j=1

C2j

⎞⎠ P++Q+K ′

+

⎛⎝R + P+

k∑j=1

D′jDj

⎞⎠K++2

⎛⎝B +

k∑j=1

CjDj

⎞⎠K+P+ = 0

(36)

and⎛⎝2A +

k∑j=1

C2j

⎞⎠ P−+Q+K ′

−

⎛⎝R + P−

k∑j=1

D′jDj

⎞⎠K−−2

⎛⎝B +

k∑j=1

CjDj

⎞⎠K−P− = 0.

(37)

Take a feedback control u(t) = K+x+(t) +K−x

−(t), which is stabilizing by Theorem4.1 (bearing in mind the definitions of K+ and K−), and let x(·) be the correspondingstate with x(0) = x0. Then a similar calculation as in (28) yields

E

∫ t

0


= P+(x+0 )2 + P−(x−

0 )2 − E[P+x+(t)2] − E[P−x

−(t)2] + E

∫ t

0

ψ(x(s), u(s))ds,

(38)

where ψ(x(s), u(s)) is as the integrand on the right-hand side of (28), with P+ and P−replaced by P+ and P−, respectively. However, u(s) = K+x(s) whenever x(s) > 0;hence

ψ(x(s), u(s))

=

⎡⎣⎛⎝2A+

k∑j=1

C2j

⎞⎠P+ + Q+K ′

+

⎛⎝R+P+

k∑j=1

D′jDj

⎞⎠K+ + 2

⎛⎝B+

k∑j=1

CjDj

⎞⎠KP+

⎤⎦x(s)2

= 0

in view of (36). Similarly, based on (37) one can show that ψ(x(s), u(s)) = 0 wheneverx(s) ≤ 0. It then follows from (38) that

E

∫ t

0

[Qx(s)2 + u(s)′Ru(s)]ds = P+(x+0 )2 + P−(x−

0 )2 − E[P+x+(t)2] − E[P−x

−(t)2].

Since u(·) is stabilizing, we have

J(x0;u(·)) = limt→+∞

E

∫ t

0

[Qx(s)2 + u(s)′Ru(s)]ds = P+(x+0 )2 + P−(x−

0 )2.


By virtue of Theorem 5.1, we conclude

P+(x+0 )2 + P−(x−

0 )2 = J(x0;u(·))≥ V (x0) ≥ P+(x+

0 )2 + P−(x−0 )2 ∀P+ ∈ P+, ∀P− ∈ P−.

This proves (35).We now assume that (1) is conic stabilizable. According to Theorem 4.1 there

exist K+ ∈ Γ and K− ∈ Γ such that F+(K+) < 0 and F−(K−) < 0. Take δ > 0sufficiently small with δ < min−F+(K+),−F−(K−). Calculate

P(1)+ := min

F+(K)≤−δ

− Q + K ′RK

F+(K)

, P

(1)− := min

F−(K)≤−δ

− Q + K ′RK

F−(K)

.(39)

In view of Lemma 5.1, P(1)+ (respectively, P

(1)− ) is a (very tight) upper bound of

P+ (respectively, P−) under the well-posedness assumption. As a consequence, if

problem (LQ) is well-posed, then it is necessary that (R + P(1)+

∑kj=1 D

′jDj)|Γ ≥ 0

and (R + P(1)−

∑kj=1 D

′jDj)|Γ ≥ 0. Now, calculate

P (0) := min(R+P

∑k

j=1D′

jDj)|Γ≥0

P.(40)

Then we know that points of P+ (respectively, P−), if any, must lie between P (0)

and P(1)+ (respectively, P

(1)− ). Inspired by the above discussion, we have the following

algorithm.Step 1. Apply Theorem 4.1(iii) to obtain K+, K− ∈ Γ with F+(K+) < 0 and

F−(K−) < 0. Set δ := minε,−F+(K+),−F−(K−), where ε is a very smallnumber allowed by the computer.

Step 2. Calculate P(1)+ and P

(1)− via (39). If either (R + P

(1)+

∑kj=1 D

′jDj)|Γ < 0 or

(R+P(1)−

∑kj=1 D

′jDj)|Γ < 0 holds, stop, and problem (LQ) is not well-posed.

Step 3. Calculate P (0) via (40).

Step 4. If there exists a P ∈ [P (0), P(1)+ ) satisfying L+(P ) ≥ 0, then go to Step 5;

otherwise, stop, and problem (LQ) is not well-posed.

Step 5. If there exists a P ∈ [P (0), P(1)− ) satisfying L−(P ) ≥ 0, then stop, and problem

(LQ) is well-posed; otherwise, stop, and problem (LQ) is not well-posed.

5.4. Well-posedness margin. In view of Remark 5.5, problem (LQ) may stillbe well-posed when R is indefinite or even negative definite. That said, it is clear thatR cannot be too negative for the well-posedness. Therefore, it is interesting to studythe range of R over which problem (LQ) is well-posed, given that all the other datais fixed. Specifically, define

r∗ := infr ∈ R∣∣ problem (LQ) is well-posed for any R ∈ Sm×m with R > rI,

(41)

where inf ∅ := +∞. The value r∗ is called the well-posedness margin. By its verydefinition, r∗ has the following interpretation: Problem (LQ) is well-posed if thesmallest eigenvalue of R, λmin(R), is such that λmin(R) > r∗, and is not well-posedif the largest eigenvalue of R, λmax(R), is such that λmax(R) < r∗.

It follows from Theorem 5.1 that, provided that system (1) is stabilizable, thewell-posedness margin r∗ can be obtained by solving the following nonlinear program


(with P and r being the decision variables):

minimize r

subject to

⎧⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎨⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎩

⎛⎝2A +

k∑j=1

C2j

⎞⎠ P + Q + Φ+(P, r) ≥ 0,

⎛⎝2A +

k∑j=1

C2j

⎞⎠ P + Q + Φ−(P, r) ≥ 0,

⎛⎝rI + P

k∑j=1

D′jDj

⎞⎠∣∣∣∣∣∣Γ

≥ 0,

(42)

where

Φ+(P, r) := infK∈Γ

⎡⎣K ′

⎛⎝rI + P

k∑j=1

D′jDj

⎞⎠K + 2

⎛⎝B +

k∑j=1

CjDj

⎞⎠PK

⎤⎦ ,

Φ−(P, r) := infK∈Γ

⎡⎣K ′

⎛⎝rI + P

k∑j=1

D′jDj

⎞⎠K − 2

⎛⎝B +

k∑j=1

CjDj

⎞⎠PK

⎤⎦ .

Notice, again, that it is hard to solve the preceding mathematical program as, inaddition to the difficulty associated with the last constraint, Φ+(P, r) and Φ+(P, r)in general do not have analytical forms. In the following, we provide an explicit lowerbound of r∗.

Theorem 5.2. Assume that system (1) is stabilizable. Then r := maxr+, r−is a lower bound of the well-posedness margin, where

r+ :=

⎧⎪⎪⎨⎪⎪⎩

λQ

infK∈K+F+(K) − λK ′K if Q ≥ 0,

λQ

supK∈K+F+(K) − λK ′K if Q < 0

(43)

and

r− :=

⎧⎪⎪⎨⎪⎪⎩

λQ

infK∈K−F−(K) − λK ′K if Q ≥ 0,

λQ

supK∈K−F−(K) − λK ′K if Q < 0,

(44)

with λ := infK∈Γ,|K|=1 K′∑k

j=1 D′jDjK ≥ 0.

Proof. Suppose problem (LQ) is well-posed with R = −rI, r ∈ R. Then P+ = ∅.Since system (1) is stabilizable, we can take a K ∈ K+. It follows from Lemma 5.1

that P := −Q+rK′KF+(K) is an upper bound of the nonempty set P+. Because (rI +

P+

∑kj=1 D

′jDj)|Γ ≥ 0 ∀P+ ∈ P+, we conclude (rI + P

∑kj=1 D

′jDj)|Γ ≥ 0, which is

equivalent to r + PK ′∑kj=1 D

′jDjK ≥ 0 for any K ∈ Γ, |K| = 1. Hence r + Pλ ≥ 0.

Substituting P = −Q+rK′KF+(K) we obtain r ≥ λQ

F+(K)−λK′K . Since K ∈ K+ is arbitrary,

we easily obtain that r ≥ r+. Similarly, we have r ≥ r−. Our analysis implies thatproblem (LQ) is not well-posed whenever the largest eigenvalue of R, λmax(R), is suchthat λmax(R) < r. Hence r is a lower bound of the well-posedness margin r∗.


6. Optimality. This section is devoted to solving the optimal LQ control prob-lem under consideration. We will first introduce two algebraic equations, in the spiritof the classical Riccati equation (for the unconstrained LQ problem), along with thenotion of the so-called stabilizing solution. Then the optimality of problem (LQ) isaddressed via the stabilizing solutions of the two algebraic equations.

We impose the following assumptions on the rest of the paper.Assumption 6.1. System (1) is conic stabilizable.Assumption 6.2. Problem (LQ) is well-posed.

6.1. Extended algebraic Riccati equations. In this subsection we definethe two algebraic equations that play a key role in solving problem (LQ). Denote

R := P ∈ R|(R + P∑k

j=1 D′jDj)|Γ > 0 and consider the following two functions

from R to R:

ξ+(P ) := arg minK∈Γ

⎡⎣K ′

⎛⎝R + P

k∑j=1

D′jDj

⎞⎠K + 2

⎛⎝B +

k∑j=1

CjDj

⎞⎠PK

⎤⎦ ,

ξ−(P ) := arg minK∈Γ

⎡⎣K ′

⎛⎝R + P

k∑j=1

D′jDj

⎞⎠K − 2

⎛⎝B +

k∑j=1

CjDj

⎞⎠PK

⎤⎦ .

Note that the minimizers above are uniquely achievable due to a similar argument inRemark 5.3 and the fact that Γ is closed. Moreover, it is evident that both ξ+(·) andξ−(·) are continuous on R.

Define a pair of functions L+ and L− from R to R:

L+(P ) :=

⎛⎝2A +

k∑j=1

C2j

⎞⎠P + Q + Φ+(P ),

L−(P ) :=

⎛⎝2A +

k∑j=1

C2j

⎞⎠P + Q + Φ−(P ).

The two equations

L+(P ) = 0,

⎛⎝R + P

k∑j=1

D′jDj

⎞⎠∣∣∣∣∣∣Γ

> 0,(45)

L−(P ) = 0,

⎛⎝R + P

k∑j=1

D′jDj

⎞⎠∣∣∣∣∣∣Γ

> 0(46)

are called extended algebraic Riccati equations (EAREs). Note that the constraint

(R+P∑k

j=1 D′jDj)|Γ > 0 is part of each of the two equations; so the EAREs are not

exactly equations in a strict sense. Also, being an algebraic equation, each of themmay admit more than one solution, or may admit no solution at all. Note that theEAREs introduced here both reduce to the same stochastic algebraic Riccati equationextensively studied in [2].

Definition 6.1. A solution P of the EARE (45) (respectively, (46)) is called astabilizing solution if ξ+(P ) ∈ K+ (respectively, ξ−(P ) ∈ K−).


It should be noted that the EAREs may not admit any stabilizing solution (seeProposition 6.1).

Before we conclude this subsection, we will present several lemmas.Lemma 6.1. We have the inequalities

(P2 − P1)[F+(ξ+(P2)) − F+(ξ+(P1))] ≤ 0,(47)

(P2 − P1)[F−(ξ−(P2)) − F−(ξ−(P1))] ≤ 0,(48)

L+(P2) − L+(P1) ≤ (P2 − P1)F+(ξ+(P1)),(49)

L−(P2) − L−(P1) ≤ (P2 − P1)F−(ξ−(P1))(50)

for any P1, P2 ∈ R.Proof. Denoting v1 := ξ+(P1) and v2 := ξ+(P2), we have

v′2

⎛⎝R + P1

k∑j=1

D′jDj

⎞⎠ v2 + 2

⎛⎝B +

k∑j=1

CjDj

⎞⎠P1v2 − Φ+(P1) ≥ 0,

v′1

⎛⎝R + P2

k∑j=1

D′jDj

⎞⎠ v1 + 2

⎛⎝B +

k∑j=1

CjDj

⎞⎠P2v1 − Φ+(P2) ≥ 0.

Then, adding the two inequalities, we get

⎡⎣v′2

⎛⎝R + P1

k∑j=1

D′jDj

⎞⎠ v2 + 2

⎛⎝B +

k∑j=1

CjDj

⎞⎠P1v2

⎤⎦− Φ+(P2)

≥ Φ+(P1) −

⎡⎣v′1

⎛⎝R + P2

k∑j=1

D′jDj

⎞⎠ v1 + 2

⎛⎝B +

k∑j=1

CjDj

⎞⎠P2v1

⎤⎦ .

Recall that the infimum in Φ+(Pi) is achieved by vi, i = 1, 2; hence the above yields

(P2 − P1)

⎡⎣v′2 k∑

j=1

D′jDjv2 + 2

⎛⎝B +

k∑j=1

CjDj

⎞⎠ v2

⎤⎦

≤ (P2 − P1)

⎡⎣v′1 k∑

j=1

D′jDjv1 + 2

⎛⎝B +

k∑j=1

CjDj

⎞⎠ v1

⎤⎦ .

This is equivalent to (47). Similarly we can prove (48).


Next, we calculate

L+(P2) − L+(P1)

=

⎛⎝2A +

k∑j=1

C2j

⎞⎠ (P2 − P1) + Φ+(P2) − Φ+(P1)

≤

⎛⎝2A +

k∑j=1

C2j

⎞⎠ (P2 − P1) + v′1

⎛⎝R + P2

k∑j=1

D′jDj

⎞⎠ v1

+ 2

⎛⎝B +

k∑j=1

CjDj

⎞⎠P2v1 − Φ+(P1)

= (P2 − P1)

⎡⎣⎛⎝2A +

k∑j=1

C2j

⎞⎠+ v′1

k∑j=1

D′jDjv1 + 2

⎛⎝B +

k∑j=1

CjDj

⎞⎠ v1

⎤⎦

= (P2 − P1)F+(ξ+(P1)).

This proves (49). Similarly we can show (50).Lemma 6.2. Assume that P1 ∈ R and P2 ≥ P1. If ξ+(P1) ∈ K+ (respectively,

ξ−(P1) ∈ K−), then ξ+(P2) ∈ K+ (respectively, ξ−(P2) ∈ K−).Proof. Since P2 ≥ P1 it follows from (47) of Lemma 6.1 that

F+(ξ+(P2)) − F+(ξ+(P1)) ≤ 0.

As F+(ξ+(P1)) < 0 we have F+(ξ+(P2)) < 0, implying ξ+(P2) ∈ K+. Similarly wecan prove the assertion for K−.

Lemma 6.3. If there exists P(0)+ ∈ R (respectively, P 0

− ∈ R) with ξ+(P(0)+ ) ∈ K+

(respectively, ξ−(P 0−) ∈ K−), then L+(·) (respectively, L−(·)) is strictly decreasing on

[P(0)+ ,+∞) (respectively, [P 0

−,+∞)).

Proof. Take P2 > P1 ≥ P(0)+ . It follows from Lemma 6.2 that F+(ξ+(P1)) < 0.

On the other hand, it is clear that P1, P2 ∈ R. Hence Lemma 6.1 yields

L+(P2) − L+(P1) ≤ (P2 − P1)F+(ξ+(P1)) < 0.

This proves that L+(P2) < L+(P1). We can prove the assertion for L−(P ) in a similarmanner.

6.2. Optimality of the LQ problem via EAREs. In this subsection we provethat stabilizing solutions of (45) and (46), if any, lead to a complete and explicitsolution to problem (LQ).

Theorem 6.2. If the EAREs (45) and (46) admit stabilizing solutions P ∗+ and

P ∗− respectively, then the feedback control

u∗(t) = ξ+(P ∗+)x+(t) + ξ−(P ∗

−)x−(t)(51)

is optimal for problem (LQ) with respect to any initial state x0. Moreover, the valuefunction is

V (x0) = P ∗+(x+

0 )2 + P ∗−(x−

0 )2 ∀x0 ∈ R.(52)


Proof. Since P ∗+ solves (45), we have

P ∗+ = −

Q + ξ+(P ∗+)′Rξ+(P ∗

+)

F+(ξ+(P ∗+))

.

Similarly,

P ∗− = −

Q + ξ−(P ∗−)′Rξ−(P ∗

−)

F−(ξ−(P ∗−))

.

Moreover, ξ+(P ∗+) ∈ K+, ξ−(P ∗

−) ∈ K− as both P ∗+ and P ∗

− are stabilizing solutions.Thus the proof of Lemma 5.1 yields V (x0) ≤ J(x0;u

∗(·)) = P ∗+(x+

0 )2 + P ∗−(x−

0 )2.On the other hand, P ∗

+ ∈ P+ and P ∗− ∈ P−. Hence it follows from Theorem 5.1

that V (x0) ≥ P ∗+(x+

0 )2 + P ∗−(x−

0 )2. Therefore, V (x0) = J(x0;u∗(·)) = P ∗

+(x+0 )2 +

P ∗−(x−

0 )2.The above proof has also shown the following result.Corollary 6.1. If the EAREs (45) and (46) admit stabilizing solutions P ∗

+ andP ∗−, respectively, then P ∗

+ = maxP |P ∈ P+ and P ∗− = maxP |P ∈ P−. As a

result, (45) and (46) each has at most one stabilizing solution.Corollary 6.1 guarantees that any stabilizing solution is the maximal solution of

the respective EAREs. This result is in parallel with the unconstrained case (see, e.g.,[3, Theorem 2.3]).

Note that the converse of Theorem 6.2 is not necessarily true. The followingexample shows that the existence of a solution to the EAREs is not necessary for theLQ problem to be attainable with respect to any initial state.

Example 6.1. Consider the LQ problem

minimize J(x0;u(·)) = E

∫ +∞

0

[|x(t)|2 − |u(t)|2]dt

subject to

dx(t) = [−x(t) + u(t)]dt + [−x(t) + u(t)]dw(t),x(0) = x0,

(53)

where all the variables are scalar-valued and Γ = R. This example was originallydiscussed in [33, Example 6.1, p. 817]. It was verified in [33] that the system isstabilizable, and the LQ problem is attainable with respect to any x0 (in fact thereare infinitely many optimal feedback controls). But both EAREs (45) and (46) in thiscase reduce to −p + 1 = 0, −1 + p > 0, which clearly admits no solution at all.

In spite of the preceding remarks and example, the following result shows thatunder an additional assumption, the EAREs indeed admit stabilizing solutions ifproblem (LQ) is attainable.

Theorem 6.3. Assume that there exist P+ ∈ P+ and P− ∈ P− such that (R +

P∑k

j=1 D′jDj)|Γ > 0 for P = P+, P−. If problem (LQ) is attainable with respect to

any x0 ∈ R, then the EAREs (45) and (46) admit stabilizing solutions P ∗+ and P ∗

−,respectively. Moreover, any optimal control with respect to a given x0 must be uniqueand represented by the feedback control (51).

Proof. The proof of Proposition 5.2 yields that, under Assumptions 6.1 and 6.2,the value function can be represented as (32), where P ∗

+ and P ∗− are the maximum

elements of P+ and P−, respectively. Moreover, by the assumption we have⎛⎝R + P ∗

+

k∑j=1

D′jDj

⎞⎠∣∣∣∣∣∣Γ

> 0,

⎛⎝R + P ∗

−

k∑j=1

D′jDj

⎞⎠∣∣∣∣∣∣Γ

> 0.(54)


Now, for any x0 ∈ R let x∗(·) be the solution of (1) under an optimal control u∗(·) ∈Ux0 which exists by the attainability assumption. Then a similar calculation to thatof (28) leads to

E

∫ t

0

[Qx∗(s)2 + u∗(s)′Ru∗(s)]ds

= P ∗+(x+

0 )2 + P ∗−(x−

0 )2 − E[P ∗+x

∗+(t)2] − E[P ∗−x

∗−(t)2] + E

∫ t

0

ψ(x∗(s), u∗(s))ds,

(55)

where ψ(x∗(s), u∗(s)) is the same as the integrand on the right-hand side of (28), withP+ and P− replaced by P ∗

+ and P ∗−, respectively. Letting t → +∞ and noting that

u∗(·) is stabilizing, we obtain

V (x0) ≡ J(x0;u∗(·)) = lim

t→+∞E

∫ t

0

[Qx∗(s)2 + u∗(s)′Ru∗(s)]ds

= P ∗+(x+

0 )2 + P ∗−(x−

0 )2 + E

∫ +∞

0

ψ(x∗(s), u∗(s))ds.

Recalling that V (x0) = P ∗+(x+

0 )2 + P ∗−(x−

0 )2 and that ψ(x∗(s), u∗(s)) ≥ 0 (via thesame proof of Theorem 5.1), we conclude

ψ(x∗(s), u∗(s)) = 0, a.e. s ∈ [0,+∞), P-a.s.

Fix s ∈ [0,+∞), satisfying the above equality. If x∗(s) > 0, then we can writeu∗(s) = K(s)x∗(s), where K(s) ∈ Γ. Going through the same analysis as in (29), weobtain

0 = ψ(x∗(s), u∗(s))

=

⎡⎣⎛⎝2A +

k∑j=1

C2j

⎞⎠P ∗

+ + Q + K(s)′

⎛⎝R + P ∗

+

k∑j=1

D′jDj

⎞⎠K(s)

+ 2

⎛⎝B +

k∑j=1

CjDj

⎞⎠K(s)P ∗

+

⎤⎦x∗(s)2

≥

⎡⎣⎛⎝2A +

k∑j=1

C2j

⎞⎠P ∗

+ + Q + Φ+(P ∗+)

⎤⎦x∗(s)2

≥ 0.

Thus, all the inequalities above become equalities and, noting that x∗(s) = 0, onehas K(s) = ξ+(P ∗

+) and L+(P ∗+) = 0. As a result, u∗(s) ≡ K(s)x∗(s) = ξ+(P ∗

+)x∗(s)at s when x∗(s) > 0. Similarly, we can prove that u∗(s) = −ξ−(P ∗

−)x∗(s) at s whenx∗(s) ≤ 0, and L−(P ∗

−) = 0. To summarize, we have shown that any optimal controlu∗(·) can be represented by (51), and hence the uniqueness of optimal control follows.On the other hand, we have also proved that P ∗

+ and P ∗− are solutions to the EAREs

(45) and (46), respectively. Moreover, they must be stabilizing solutions becauseu∗(·), which is now represented by (51), is a stabilizing control.

Remark 6.1. Theorem 6.3 shows that the existence of stabilizing solutions to theEAREs (45) and (46) is almost necessary for the attainability of problem (LQ). Theonly exception, as also demonstrated by Example 6.1, is the “singular” case when(R + P

∑kj=1 D

′jDj)|Γ = 0 for all elements P in at least one of the sets P+ and P−.


6.3. Existence of stabilizing solutions to EAREs. Theorem 6.2 asserts thatif one can find stabilizing solutions to the EAREs, then the original optimal LQ controlcan be solved completely and explicitly in terms of obtaining the optimal feedbackcontrol as well as the value function. The next natural questions are, then, whendo the EAREs admit stabilizing solutions, and how do we find them? These are theissues that we are going to address in this subsection. Indeed, we will identify anddiscuss three cases when the EAREs do have the stabilizing solutions.

Theorem 6.4. If there exist P(0)+ , P

(0)− satisfying

L+(P(0)+ ) ≥ 0,

⎛⎝R + P

(0)+

k∑j=1

D′jDj

⎞⎠∣∣∣∣∣∣Γ

> 0,(56)

L−(P(0)− ) ≥ 0,

⎛⎝R + P

(0)−

k∑j=1

D′jDj

⎞⎠∣∣∣∣∣∣Γ

> 0,(57)

and

F+(ξ+(P(0)+ )) < 0, F−(ξ−(P

(0)− )) < 0,(58)

then the EAREs (45) and (46) admit unique stabilizing solutions P ∗+ and P ∗

−, respec-tively.

Proof. If L+(P(0)+ ) = 0, then P

(0)+ is the stabilizing solution to (45) and we are

done. So let us assume that L+(P(0)+ ) ≡ (2A +

∑kj=1 C

2j )P

(0)+ + Q + Φ+(P

(0)+ ) > 0,

namely,

F+(ξ+(P(0)+ ))P

(0)+ + Q + ξ+(P

(0)+ )′Rξ+(P

(0)+ ) > 0.

Since F+(ξ+(P(0)+ )) < 0, we have

P(0)+ < −

Q + ξ+(P(0)+ )′Rξ+(P

(0)+ )

F+(ξ+(P(0)+ ))

:= P(1)+ .(59)

By Lemma 6.2, we have ξ+(P(1)+ ) ∈ K+ because P

(1)+ > P

(0)+ . Moreover,

L+(P(1)+ ) ≡

⎛⎝2A +

k∑j=1

C2j

⎞⎠P

(1)+ + Q + Φ+(P

(1)+ )

≤

⎛⎝2A +

k∑j=1

C2j

⎞⎠P

(1)+ + Q + ξ+(P

(0)+ )′

⎛⎝R + P

(1)+

k∑j=1

D′jDj

⎞⎠ ξ+(P

(0)+ )

+ 2

⎛⎝B +

k∑j=1

CjDj

⎞⎠P

(1)+ ξ+(P

(0)+ )

= 0,

(60)


where the last inequality was due to the very definition of P(1)+ . Now, if L+(P

(1)+ ) = 0,

then P(1)+ is the stabilizing solution of (45). If L+(P

(1)+ ) < 0, then, noting that L+(·)

is a strictly decreasing (by Lemma 6.3) continuous function on the interval [P(0)+ , P

(1)+ ]

along with L+(P(0)+ ) > 0, we conclude that there exists a unique P ∗

+ ∈ (P(0)+ , P

(1)+ )

such that L+(P ∗+) = 0. Clearly (R+P ∗

+

∑kj=1 D

′jDj)|Γ > 0, and ξ+(P ∗

+) ∈ K+ thanksto Lemma 6.2. Hence P ∗

+ is the stabilizing solution of (45).The proof for the existence of the stabilizing solution to the EARE (46) is com-

pletely analogous. Finally, the uniqueness of the stabilizing solutions follows fromCorollary 6.1.

Theorem 6.5. If there exist P(0)+ , P

(0)− satisfying

L+(P(0)+ ) > 0,

⎛⎝R + P

(0)+

k∑j=1

D′jDj

⎞⎠∣∣∣∣∣∣Γ

> 0,(61)

and

L−(P(0)− ) > 0,

⎛⎝R + P

(0)−

k∑j=1

D′jDj

⎞⎠∣∣∣∣∣∣Γ

> 0,(62)

then the EAREs (45) and (46) admit unique stabilizing solutions P ∗+ and P ∗

−, respec-tively.

Proof. Take K+ ∈ K+, which exists by the stabilizability assumption. Set

P(1)+ := −

Q + K ′+RK+

F+(K+).(63)

It follows from Lemma 5.1 that P(1)+ ≥ P

(0)+ since P

(0)+ ∈ P+. Hence⎛

⎝R + P(1)+

k∑j=1

D′jDj

⎞⎠∣∣∣∣∣∣Γ

> 0.

Moreover,

L+(P(1)+ ) ≡

⎛⎝2A +

k∑j=1

C2j

⎞⎠P

(1)+ + Q + Φ+(P

(1)+ )

≤

⎛⎝2A +

k∑j=1

C2j

⎞⎠P

(1)+ + Q + K ′

+

⎛⎝R + P

(1)+

k∑j=1

D′jDj

⎞⎠K+

+ 2

⎛⎝B +

k∑j=1

CjDj

⎞⎠P

(1)+ K+

= 0.

(64)

Applying Lemma 6.1 and noting that L+(P(0)+ ) > 0 we get

0 < L+(P(0)+ ) − L+(P

(1)+ ) ≤ (P

(0)+ − P

(1)+ )F+(ξ+(P

(1)+ )).(65)


Hence F+(ξ+(P(1)+ )) < 0 or ξ+(P

(1)+ ) ∈ K+. Now set

P(2)+ := −

Q + ξ+(P(1)+ )′Rξ+(P

(1)+ )

F+(ξ+(P(1)+ ))

.(66)

Again by Lemma 5.1 we obtain P(2)+ ≥ P

(0)+ and, therefore, (R+P

(2)+

∑kj=1 D

′jDj)|Γ >

0. On the other hand, the fact that L+(P(1)+ ) ≤ 0 can be rewritten as P

(2)+ ≤ P

(1)+ in

view of the relation (66). Moreover, analysis similar to that for P(1)+ above leads to

L+(P(2)+ ) ≤ 0, and ξ+(P

(2)+ ) ∈ K+ or F+(ξ+(P

(2)+ )) < 0.

In general, we construct iteratively the following sequence:

P(i+1)+ := −

Q + ξ+(P(i)+ )′Rξ+(P

(i)+ )

F+(ξ+(P(i)+ ))

, i = 1, 2, . . . .(67)

An induction argument shows that P(0)+ ≤ · · · ≤ P

(i+1)+ ≤ P

(i)+ · · · ≤ P

(1)+ , (R +

P(i)+

∑kj=1 D

′jDj)|Γ > 0, L+(P

(i)+ ) ≤ 0, and ξ+(P

(i)+ ) ∈ K+, i = 1, 2, . . . . Since the

sequence P (i)+ is decreasing with a lower bound P

(0)+ , there exists P ∗

+ ∈ R so

that P ∗+ = limi→∞ P

(i)+ . Moreover it is clear that (R + P ∗

+

∑kj=1 D

′jDj)|Γ ≥ (R +

P(0)+

∑kj=1 D

′jDj)|Γ > 0. On the other hand, L+(P ∗

+) ≤ 0 since each L+(P(i)+ ) ≤ 0.

Thus an argument similar to (65) yields ξ+(P ∗+) ∈ K+ or F+(ξ+(P ∗

+)) < 0. As a

result, we can pass the limit in (67) to obtain P ∗+ = −Q+ξ+(P∗

+)′Rξ+(P∗+)

F+(ξ+(P∗+

)) , which is

equivalent to L+(P ∗+) = 0. This shows that P ∗

+ is the stabilizing solution to (45).Similarly we can prove that (46) admits the stabilizing solution.

Remark 6.2. Recall that Theorem 5.1 characterizes the well-posedness of problem(LQ) by the nonemptiness of the sets P+ and P−. Theorems 6.4 and 6.5 spell outtwo important cases when the EAREs (45) and (46) have stabilizing solutions and,therefore, problem (LQ) can be completely solved with explicit solutions. These twocases are specified in terms of the existence of certain “special elements” of the setsP+ and P−. Specifically, the case with Theorem 6.4 is one when each of P+ andP− has a “stabilizing element” in the sense that (58) holds. On the other hand,Theorem 6.5 asserts that the nonemptiness of the interiors of P+ and P− is sufficientfor the existence of stabilizing solutions to the EAREs. In view of the fact thatthe nonemptiness of P+ and P− is the minimum requirement for the underlying LQproblem to be meaningful, the sufficient conditions respectively given in Theorems6.4 and 6.5 are very mild indeed.

Remark 6.3. The proof of Theorem 6.5 constitutes an algorithm for finding thestabilizing solutions to the EAREs. In fact it is given by the iterative scheme (67)with an initial point (63). On the other hand, although the proof of Theorem 6.4 hasnot given an explicit algorithm for computing the stabilizing solutions, one can use amiddle-point algorithm to find them based on the proof. Alternatively, one may use

the same iterative scheme (67) with the initial point, P(1)+ , given by (59). It can be

proved, using almost the same analysis as that in the proof of Theorem 6.5, that theconstructed sequence converges to the desired point, P ∗

+. The only argument thatneeds to be modified is that for proving ξ+(P ∗

+) ∈ K+. In this case, ξ+(P ∗+) ∈ K+ is

seen from the fact that P ∗+ ≥ P

(0)+ and ξ+(P

(0)+ ) ∈ K+ as well as from Lemma 6.2.

Finally we present the results on the definite case Q ≥ 0 and R|Γ ≥ 0 (includingthe so-called singular case when R is allowed to be singular).


Theorem 6.6. Assume Q ≥ 0 and R|Γ ≥ 0. Then the EAREs (45) and (46)admit unique stabilizing solutions P ∗

+ and P ∗−, respectively, under one of the following

additional conditions:(i) Q > 0 and R|Γ > 0.

(ii) Q = 0, R|Γ > 0, and 2A +∑k

j=1 C2j = 0.

(iii) Q > 0, R|Γ ≥ 0, and∑k

j=1 D′jDj |Γ > 0.

Proof. (i) In this case P(0)+ = P

(0)− = 0 satisfy the assumption of Theorem 6.5.

(ii) If 2A +∑k

j=1 C2j < 0, then take P

(0)+ = P

(0)− = 0. We see that L+(P

(0)+ ) =

L−(P(0)− ) = Q = 0 and F+(ξ+(P

(0)+ )) = F−(ξ−(P

(0)− )) = 2A +

∑kj=1 C

2j < 0. Thus

the assumption of Theorem 6.4 is satisfied.If 2A+

∑kj=1 C

2j > 0, due to the stabilizability assumption there is 0 = K+ ∈ K+.

Set P(1)+ := −K′

+RK+

F+(K+) . As F+(K+) < 0, R|Γ > 0, and K+ = 0, we have P(1)+ > 0.

Define

P(i+1)+ := −

ξ+(P(i)+ )′Rξ+(P

(i)+ )

F+(ξ+(P(i)+ ))

, i = 1, 2, . . . .(68)

Then an analysis similar to that in the proof of Theorem 6.5 leads to 0 ≤ · · · ≤P

(i+1)+ ≤ P

(i)+ · · · ≤ P

(1)+ , (R + P

(i)+

∑kj=1 D

′jDj)|Γ > 0, L+(P

(i)+ ) ≤ 0, and ξ+(P

(i)+ ) ∈

K+, i = 1, 2, . . . . Hence there exists P ∗+ ≥ 0 so that P ∗

+ = limi→∞ P(i)+ . Moreover

(R+P ∗+

∑kj=1 D

′jDj)|Γ ≥ R|Γ > 0. Note that at this point we can no longer apply the

same argument as that used in the proof of Theorem 6.5 to conclude F+(ξ+(P ∗+)) < 0,

because the element 0, which substitutes the point P(0)+ of Theorem 6.5, does not

satisfy L+(0) > 0. To get around this, let us suppose F+(ξ+(P ∗+)) = 0 (recall that

it always holds that F+(ξ+(P ∗+)) ≤ 0 since F+(ξ+(P

(i)+ )) < 0). Multiplying (68) by

F+(ξ+(P(i)+ )) and then passing to the limit, we obtain ξ+(P ∗

+)′Rξ+(P ∗+) = 0, resulting

in ξ+(P ∗+) = 0. Thus F+(ξ+(P ∗

+)) = 2A+∑k

j=1 C2j > 0, which is a contradiction. This

proves that F+(ξ+(P ∗+)) < 0. The rest of the proof is the same as that of Theorem

6.5.(iii) First note that for any P > 0,

0 ≥ Φ+(P )

≥ infK∈Γ

K ′RK + P infK∈Γ

⎡⎣K ′

k∑j=1

D′jDjK + 2

⎛⎝B +

k∑j=1

CjDj

⎞⎠K

⎤⎦

= P infK∈Γ

⎡⎣K ′

k∑j=1

D′jDjK + 2

⎛⎝B +

k∑j=1

CjDj

⎞⎠K

⎤⎦ .

Hence limP→0+ Φ+(P ) = 0. Since Q > 0, we have, by the definition of L+(·), that

there exists P(0)+ > 0 so that L+(P

(0)+ ) > 0. On the other hand, it is clear that

(R + P(0)+

∑kj=1 D

′jDj)|Γ ≥ P

(0)+

∑kj=1 D

′jDj

∣∣∣Γ> 0. Consequently Theorem 6.5 ap-

plies.One may be curious about what happens if Q = 0, R|Γ > 0, and 2A+

∑kj=1 C

2j = 0

(refer to Theorem 6.6(ii)). It turns out that in this case the EAREs never admitstabilizing solutions.


Proposition 6.1. Neither (45) nor (46) admits any stabilizing solution when

Q = 0, R|Γ > 0, and 2A +∑k

j=1 C2j = 0.

Proof. By Assumption 6.1, there exists 0 = K+ ∈ K+ and 0 = K− ∈ K− sinceF+(0) = F−(0) = 0. Denote Kε

+ := εK+ and Kε− := εK− for ε ∈ (0, 1]. Then Kε

+ ∈ Γand Kε

− ∈ Γ.

For any fixed P+ > 0, set ε := min−P+F+(K+)

2K′+RK+

, 1∈ (0, 1]. Then

L+(P+) = Φ+(P+)

≤ (Kε+)′

⎛⎝R + P+

k∑j=1

D′jDj

⎞⎠Kε

+ + 2P+

⎛⎝B +

k∑j=1

CjDj

⎞⎠Kε

+

≤ ε

⎧⎨⎩εK ′

+RK+ + P+

⎡⎣K ′

+

k∑j=1

D′jDjK+ + 2

⎛⎝B +

k∑j=1

CjDj

⎞⎠K+

⎤⎦⎫⎬⎭

≤ ε

[−P+F+(K+)

2K ′+RK+

(K ′+RK+) + P+F+(K+)

]

=ε

2P+F+(K+)

< 0.

This implies that there exists no positive solution to the EARE (45).

Next, for any fixed P+ < 0 with (R + P+

∑kj=1 D

′jDj)|Γ > 0, set

ε := min

−|P+|F−(K−)

2K ′−RK−

, 1

∈ (0, 1].

Then

L+(P+) = Φ+(P+)

≤ (Kε−)′

⎛⎝R + P+

k∑j=1

D′jDj

⎞⎠Kε

− + 2P+

⎛⎝B +

k∑j=1

CjDj

⎞⎠Kε

−

≤ ε

⎧⎨⎩εK ′

−RK− + |P+|

⎡⎣K ′

−

k∑j=1

D′jDjK− − 2

⎛⎝B +

k∑j=1

CjDj

⎞⎠K−

⎤⎦⎫⎬⎭

≤ ε

[−|P+|F−(K−)

2K ′−RK−

(K ′−RK−) + |P+|F−(K−)

]

=ε

2|P+|F−(K−)

< 0.

Hence there is no negative solution to the EARE (45).Finally, when P+ = 0, we do have L+(P+) = 0 but F+(ξ+(P+)) = F+(0) = 0. So

P+ = 0 is not a stabilizing solution either.Similarly, we can prove the nonexistence of a stabilizing solution to (46).Although the conclusion of Proposition 6.1 does not necessarily lead to the nonex-

istence of optimal feedback control for the corresponding LQ problem (refer to sec-tion 6.2), the following example shows that the latter could indeed occur.


Example 6.2. Consider the LQ problem

minimize J(x0;u(·)) = E

∫ +∞

0

r|u(t)|2dt

subject to

dx(t) = [ax(t) + bu(t)]dt + cx(t)dw(t),x(0) = x0,

(69)

where all the variables are scalar-valued, Γ = R, 2a + c2 = 0, b > 0, and r > 0. It iseasy to verify that the problem is stabilizable and well-posed. Take a feedback controluε(t) = −εxε(t) for ε > 0. Under this control the state satisfies

E|xε(t)|2 = e(2a+c2−2bε)tx20 = e−2bεtx2

0.

Hence uε is stabilizing. Moreover, the cost under this control is

J(x0;uε(·)) = ε2r2E

∫ +∞

0

|xε(t)|2dt =r2x2

0

2bε.

Letting ε → 0 we see that V (x0) = 0 ∀x0 ∈ R. Note that this value cannot be

attained if x0 = 0, for whenever E∫ +∞0

r|u∗(t)|2dt = 0 it is necessary that u∗(t) = 0,a.e. t ≥ 0. However, this control, u∗(t), is not stabilizable when x0 = 0. In otherwords, the LQ problem is not attainable with respect to x0 = 0.

Remark 6.4. Theorem 6.6(i),(ii) and Proposition 6.1 together with Example 6.2give a complete answer to the question of optimality for problem (LQ) in the classicaldefinite case Q ≥ 0 and R|Γ > 0. Moreover, Theorem 6.6(iii) addresses the casewhen R is possibly singular. Note that this case occurs often in financial applications(where typically R = 0).

Remark 6.5. In view of Theorem 6.2, under the respective assumptions of The-orems 6.4, 6.5, and 6.6, problem (LQ) has the optimal feedback control (51) and thevalue function (52). Moreover, as per Remark 5.6, in these cases the stabilizing so-lutions P ∗

+ and P ∗− can also be obtained, in addition to the preceding algorithms, by

solving the mathematical programs (33) and (34) if the corresponding constraints aretractable.

7. Numerical examples. To numerically calculate the optimal solution to prob-lem (LQ) one needs to carry out two steps: the first is to check the conic stabiliz-ability and the well-posedness, and the second is to find the stabilizing solutions tothe EAREs. The procedures for the first step are depicted in sections 4 and 5.3,whereas that for the second part is described in section 6. Here we give an exampleto illustrate the whole process (where we used the computing tool Scilab to carry outall the calculations).

Example 7.1. Consider problem (LQ) with m = k = 3, Γ = R3+, and the

dynamics coefficients as follows:

A = 2.00, B = (−50 − 100 200),C1 = −0.84, D1 = (6.85 − 8.78 0.68),C2 = −3.78, D2 = (11.22 13.24 14.53),C3 = 0.849, D3 = (−1.98 − 5.44 − 2.32).

The eigenvalues of∑3

j=1 D′jDj are 1.5880509, 126.23912, and 547.84943. So

∑3j=1 D

′jDj

is positive definite in this case. The cost parameters are

Q = 10, R =

⎛⎝0 0 0

0 −5.0 00 0 4.0

⎞⎠ .


Hence this is an indefinite LQ problem.To solve this problem, we first apply Theorem 4.1(iii) (note that in this example

the constraint K ∈ R3+ is also an LMI) to obtain stabilizing feedback gains

K+ =

⎛⎝0.5435353

0.53072890.0701460

⎞⎠ , K− =

⎛⎝0.0816967

0.05625700.7185273

⎞⎠ .

Next we use the algorithm in section 5.3 to find out that problem (LQ) is well-

posed. Furthermore, when P(0)+ = P

(0)− = 0.1, the eigenvalues of R+0.1∗

∑3j=1 D

′jDj

are 1.641309, 10.293806, and 54.632545; hence R+0.1∗∑3

j=1 D′jDj > 0. On the other

hand, L+(0.1) = 1.6073676 > 0 and L−(0.1) = 4.0651986 > 0. According to Theorem6.5, problem (LQ) has an optimal feedback control. Now we use the algorithm givenin the proof of Theorem 6.5 to obtain the optimal control and optimal value. First,

set the initial values P(1)+ and P

(1)− by using K+, K− and formulas similar to (63),

respectively. They are

P(1)+ = 1.1814412, P

(1)− = 13.167238.

By the iterative formula (67), we obtain

P ∗+ = 0.1225762, ξ+(P ∗

+) =

⎛⎝0.2889240

0.49187550

⎞⎠ ,

P ∗− = 0.1561608, ξ−(P ∗

−) =

⎛⎝ 0

00.5875815

⎞⎠ .

Therefore, the optimal feedback control is

u∗(t) =

⎛⎝0.2889240

0.49187550

⎞⎠x+(t) +

⎛⎝ 0

00.5875815

⎞⎠x−(t),

with the optimal cost

J∗(x0) = 0.1225762 ∗ (x+0 )2 + 0.1561608 ∗ (x−

0 )2.

In the next example we demonstrate the calculation of a lower bound of thewell-posedness margin (refer to section 5.4).

Example 7.2. Using the same values of the coefficients A, B, Cj , Dj , j = 1, 2, 3,and Q as in Example 7.1, we want to compute a lower bound of the well-posednessmargin r∗.

According to Theorem 5.2, we first calculate λ = 176.7313. Next we have r+ =−9.434553 and r− = −6.4967141. Hence, the lower bound of the well-posednessmargin is r = −6.4967141. Note that, as seen in Example 7.1, the problem is stillwell-posed when one of the eigenvalues of R is −5.

8. Conclusion. In this paper, we studied an indefinite stochastic LQ controlproblem in the infinite time horizon with conic control constraint. Several key is-sues, including conic stabilizability, well-posedness, and optimality were addressed


with complete solutions. In particular, two algebraic equations, the EAREs, werenewly introduced, in lieu of the classical algebraic Riccati equation, whose stabiliz-ing solutions give rise to the explicit forms of the optimal feedback control and thevalue function. It was also seen that the representation of the value function givenby Proposition 5.1 served as the technical key to all the main results of this paper,which motivated the utilization of the cerebrated Tanaka formula.

It should be stressed again that the approach of this paper crucially dependedon the special structure of the problem. One main assumption is that the state ofthe system is one-dimensional. While the conclusion of Proposition 5.1 appears tohold, mutatis mutandis, for the problem with multidimensional state variable, it seemsthat an analogy of Lemma 3.2, if any, would be far more complicated. This makesthe multidimensional problem very challenging. Another structural property of themodel is that the dynamics of the system is homogeneous (in state and control) andthe cost contains no first-order term of the state variable as well as no control-statecross term. As a result, our approach will fail, say, for the case when there is no stateand control dependent noise, and with fixed variance. Solving these kinds of problemscalls for different techniques. Finally, an even more difficult problem is the stochasticLQ control with state constraint.

Acknowledgment. We thank the three anonymous reviewers for their carefulreading of an earlier version of the paper and for their constructive comments thatled to an improved version.

REFERENCES

[1] M. Ait Rami, X. Chen, J. B. Moore, and X. Y. Zhou, Solvability and asymptotic behaviorof generalized Riccati equations arising in indefinite stochastic LQ controls, IEEE Trans.Automat. Control, AC-46 (2001), pp. 428–440.

[2] M. Ait Rami and X. Y. Zhou, Linear matrix inequalities, Riccati equations, and indefi-nite stochastic linear quadratic control, IEEE Trans. Automat. Control, AC-45 (2000),pp. 1131–1143.

[3] M. Ait Rami, X. Y. Zhou, and J. B. Moore, Well-posedness and attainability of indefi-nite stochastic linear quadratic control in infinite time horizon, Systems Control Lett., 41(2000), 123–133.

[4] B. D. O. Anderson and J. B. Moore, Optimal Control—Linear Quadratic Methods, Prentice-Hall, Englewood Cliffs, NJ, 1989.

[5] J.-M. Bismut, Linear quadratic optimal stochastic control with random coefficients, SIAM J.Control Optim., 14 (1976), pp. 419–444.

[6] S. Boyd, L. El Ghaoui, E. Feron, and V. Balakrishnan, Linear Matrix Inequalities inSystem and Control Theory, SIAM, Philadelphia, 1994.

[7] R. F. Brammer, Controllability in linear autonomous systems with positive controllers, SIAMJ. Control, 10 (1972), pp. 339–353.

[8] R. F. Brammer, Differential controllability and the solution of linear inequalities-Part I, IEEETrans. Automat. Control, AC-20 (1975), pp. 128–131.

[9] S. L. Campbell, On positive controllers and linear quadratic optimal control problems, Inter-nat. J. Control, 36 (1982), pp. 885–888.

[10] S. Chen, X. Li, and X. Y. Zhou, Stochastic linear quadratic regulators with indefinite controlweight costs, SIAM J. Control Optim., 36 (1998), pp. 1685–1702.

[11] S. Chen and J. Yong, Stochastic linear quadratic optimal control problems, Appl. Math.Optim., 43 (2001), pp. 21–45.

[12] M. D. Fragoso, O. L. V. Costa, and C. E. de Souza, A new approach to linearly perturbedRiccati equations arising in stochastic control, Appl. Math. Optim., 37 (1998), pp. 99–126.

[13] W. P. M. H. Heemels, S. J. L. van Eijndhoven, and A. A. Stoorvogel, Linear quadraticregulator problem with positive controls, Internat. J. Control, 70 (1998), pp. 551–578.

[14] M. Heymann and R. J. Stern, Controllability of linear systems with positive controls: Geo-metric considerations, J. Math. Anal. Appl., 52 (1975), pp. 36–41.


[15] Y. Hu and X. Y. Zhou, Constrained Stochastic LQ Control with Random Coefficients, andApplication to Mean-Variance Portfolio Selection, Preprint, 2003.

[16] N. Ikeda and S. Watanabe, Stochastic Differential Equations and Diffusion Processes, North-Holland/Kodansha, Tokyo, 1981.

[17] R. E. Kalman, Contribution to the theory of optimal control, Bol. Soc. Mat. Mexicana (2), 5(1960), pp. 102–119.

[18] X. Li, X. Y. Zhou, and A. E. B. Lim, Dynamic mean-variance portfolio selection with no-shorting constraints, SIAM J. Control Optim., 40 (2002), pp. 1540–1555.

[19] A. E. B. Lim and X. Y. Zhou, Mean-variance portfolio selection with random parameters ina complete market, Math. Oper. Res., 27 (2002), pp. 101–120.

[20] Y. Nesterov and A. Nemirovskii, Interior Point Polynomial Algorithms in Convex Program-ming, SIAM, Philadelphia, 1994.

[21] M. Pachter, The linear-quadratic optimal control problem with positive controllers, Internat.J. Control, 32 (1980), pp. 589–608.

[22] M. Pachter and D. H. Jacobson, Control with conic control constraint sets, J. Optim. TheoryAppl., 25 (1978), pp. 117–123.

[23] M. Pachter and D. H. Jacobson, Stabilization conic control constraint sets, Internat. J.Control, 29 (1979), pp. 125–132.

[24] D. Revuz and M. Yor, Continuous Martingales and Brownian Motion, 3rd ed., Springer-Verlag, Berlin, 1999.

[25] S. H. Saperstone, Global controllability of linear systems with positive controls, SIAM J.Control, 11 (1973), pp. 417–423.

[26] S. H. Saperstone and J. A. Yorke, Controllability of linear oscillatory systems using positivecontrols, SIAM J. Control, 9 (1971), pp. 253–262.

[27] H. Sissaoui and W. D. Collins, The numerical solution of the linear regulator problem withpositive control, IMA J. Math. Control Inform., 5 (1988), pp. 191–201.

[28] P. O. M. Sockaert and J. B. Rawlings, Constrained linear quadratic regulation, IEEE Trans.Automat. Control, AC-43 (1998), pp. 1163–1169.

[29] J. F. Sturm and S. Zhang, On cones of nonnegative quadratic function, Math. Oper. Res.,28 (2003), pp. 246–267.

[30] F. Willems, W. P. M. H. Heemels, B. de Jager, and A. A. Stoorvogel, Positive feedbackstabilization of centrifugal compressor surge, Automatica, 38 (2002), pp. 311–318.

[31] W. M. Wonham, Optimal stationary control of a linear system with state-dependent noises,SIAM J. Control, 5 (1967), pp. 486–500.

[32] W. M. Wonham, On a matrix Riccati equation of stochastic control, SIAM J. Control, 6 (1968),pp. 681–697; erratum, SIAM J. Control, 7 (1969), p. 365.

[33] D. D. Yao, S. Zhang, and X. Y. Zhou, Stochastic linear-quadratic control via semidefiniteprogramming, SIAM J. Control Optim., 40 (2001), pp. 801–823.

[34] J. Yong and X. Y. Zhou, Stochastic Controls. Hamiltonian Systems and HJB Equations,Springer-Verlag, New York, 1999.

[35] X. Y. Zhou and D. Li, Continuous-time mean-variance portfolio selection: A stochastic LQframework, Appl. Math. Optim., 42 (2000), pp. 19–33.

Documents

STOCHASTIC LINEAR-QUADRATIC CONTROL WITH CONIC CONTROL …€¦ · STOCHASTIC LINEAR-QUADRATIC CONTROL WITH CONIC CONTROL CONSTRAINTS ON AN INFINITE TIME HORIZON∗ XI CHEN† AND