We estimate MS2 n. If K.-- 2 ~P' then i ~ l

n n--[

< I+,m+,Mm,+, I < +,+,+: m. I ~;i+] ~,n ~"-I. j=O

Hence, if

n ~ l i~l j=O

then it follows f rom the Chebyshev inequality P{] Sn l > eK~}~ MS2,/e2K~ and the B o r e l - C a n t e l l i lemma, that 0n -- 0wi th probabili ty 1. In par t icu lar , condition (28) holds if ~(i) = iq, where q _> 0, and ~(i) = O(i-~/), ~ > 0. I f - 0 . 5 < q < 0, condition (28) holds if fi(i} = O(i-7}, where -~ = 1 + e for some e > 0.

1 .

2. 3.

4. 5.

L I T E R A T U R E C I T E D

M. B. Nevel 'son and R. Z. Khas 'minski i , Stochastic Approximation and Recur ren t Est imat ion [in Rus- sian], Nauka, Moscow (1972}. Yu. A. Rozanov, Stat ionary Random P r o c e s s e s [in Russian], Nauka, Moscow (1963}. I. A. Ibragimov and Yu. V. Linnik, Independent and Stationarily Connected Random Variables [in Rus- sian], Nauka, Moscow (1965}. J. Lamper t i , Probability. A Survey of the Mathematical Theory, W. A. Benjamin (1966). E. J. Hannah, Multiple Time Ser ies , Wiley (1970}.

S U F F I C I E N T C O N D I T I O N S F O R C O N V E R G E N C E O F

S T O C H A S T I C A P P R O X I M A T I O N A L G O R I T H M S F O R

R A N D O M P R O C E S S E S W I T H C O N T I N U O U S T I M E

O. Y u . K u l ' c h i t s k i i UDC 519.2:681.513.6

I N T R O D U C T I O N

In the present ar t ic le we consider the convergence of s tochast ic approximation procedures for random processes with continuous time, as applied to the solution of the extremal problem

f (x*) -~ min f (x), (1) xEX

where X c-- R. is a convex set Rn, x E R n is the vector of pa ramete r s with respec t to which the minimization is per formed, and f(x) is a downwards convex function in X. This function can be wri t ten in the form

/(x}----'= ~ :(x, u) dp(u)~M{:(x, u)}, (2) R,n

where p(u), u E R m is an absolutely continuous function, having all the propert ies of a distr ibution function; the function f(x, u) is integrable with respec t to measure dp(u) for any x ~ Rn. It is a lso assumed that, fl)r any g ivenx ERn, u ~R m, t -> 0, we can measure or evaluate the vec tor funct iony(x, u, t) ERn, such that, fl)r any

x ~ R n a n d t - > 0 ,

y (x, u, t) dp(u) = M{y (x, u,t)}=/~ (x) + g(x, 0, (3) Rm

where fx(x) ~ Rn is the gradient or some support functional of f(x) at the point x E Rn, g{x, t) E Rn is a con- tinuous function with respec t to t -> 0, satisfying the following limit re la t ion for any x E Rn: lira [rg (x, t)II ==0.

Transla ted f rom Kibernetika, No. 6, pp. 114-126, November-December , 1979. Original ar t ic le sub- mitted February 18, 1975.

0011-4235/79/1506- 0901 $07.50 �9 1980 Plenum Publishing Corporat ion 901

Given the p r o b a b i l i t y s p a c e (fl, 92, P ) , and the r a n d o m p r o c e s s u(t , ~) = u t, m e a s u r a b l e in the s p a c e , and s t a t i o n a r y in the n a r r o w s e n s e , w i th u n i v a r i a t e d i s t r i b u t i o n func t ion P(u). In the con tex t of s o l u t i o n of our p r o b l e m , the s t o c h a s t i c a p p r o x i m a t i o n p r o c e s s w i th con t inuous t i m e is de f ined by the s y s t e m of s t o c h a s t i c

d i f f e r e n t i a l equa t ions

x(t , co)= - - a( t )y(x( t , co), u(t, co), t), x (0) = x 0, (4)

w h e r e the func t ion a(t) > 0 is i n t e g r a b l e in any f in i t e i n t e r v a l .

I t w i l l be a s s u m e d th roughou t tha t s y s t e m (4) has a unique s o l u t i o n , a b s o l u t e l y con t inuous w i th p r o b - a b i l i t y 1, for a l l t _> O. Su f f i c i en t cond i t i ons u n d e r wh ich t h e s e a s s u m p t i o n s a r e s a t i s f i e d m a y be found e . g . ,

in [11 .

The c o n v e r g e n c e of p r o c e s s (4) to a s o l u t i o n x* of the e x t r e m a l p r o b l e m (1) has b e e n d i s c u s s e d by a num- b e r of a u t h o r s u n d e r c e r t a i n s i m p l i f y i n g a s s u m p t i o n s r e g a r d i n g f(x, u), y{x, u, t). F o r i n s t a n c e , D r i m l and Nedota in [2] p r o v e d the c o n v e r g e n c e , w i th p r o b a b i l i t y 1, of the p r o c e s s x t to x* as t - - ~ for the c a s e n = 1, g(x , t) = 0, a(t) = 1 / ( t + 1). In [3], S a k r i s o n s t u d i e d the c o n v e r g e n c e of p r o c e s s x t to x* as t - - ~ in the mean . In [4] Dupac p r o v e d the c o n v e r g e n c e in the m e a n s q u a r e s e n s e of p r o c e s s x t to x* as t - - ~ for a con t inuous ana log of the K i e f e r - W o l f o w i t z p r o c e d u r e in the c a s e when the v e c t o r func t ion y(x, u, t) is s p e c i f i e d l i n e a r l y a s a func t ion of u ~ R m wi th r e s p e c t to r e R l. The r e s u l t s of [2, 4] a r e d e a l t w i th in [5].

The p roof s of c o n v e r g e n c e in [2-4] a r e b a s e d on the e r g o d i c p r o p e r t i e s of the r a n d o m p r o c e s s y (x , u{t, w), t) for any x ~ R n. A p r o o f of the c o n v e r g e n c e of x t to x* as t - - ~ in the m e a n s q u a r e s e n s e , b a s e d on the u se of a p r o c e s s u(t , w), s a t i s f y i n g cond i t ions of u n i f o r m l y s t r o n g i n t e r m i x i n g , was o f f e r e d by K r a s u l i n a in [6] fo r the c a s e n = 1, g(x, t) = 0 and add i t i ve no ise . In the p r e s e n t a r t i c l e we s h a l l e x a m i n e the c o n v e r g e n c e cond i t i ons fo r p r o c e s s x*( t , ~) to x* as t - - ~ , w i th a r b i t r a r y n < ~ , depend ing on the " c o m p l e x i t y " of func t ion y ( x , u, t) and the cond i t ions u n d e r w h i c h the r a n d o m p r o c e s s u(t , ~) i s r e g u l a r .

The c o n v e r g e n c e of d i s c r e t e a l g o r i t h m s of a n a n a l o g o u s type was e a r l i e r d i s c u s s e d , e . g . , in [13].

1. REGULAR PROCESSES AND SOME OF THEIR P R O P E R T I E S

Let ut = u(t, w), t -> 0 be a stationary in the narrow sense random process with values in R m. Through- out, we shall understand by 9J (a, b) the minimal a-algebra with respect to which the random variables u (t, ~0), t E [a, b].

We s h a l l make use of p r o c e s s e s u t wh ich s a t i s f y the cond i t i ons of s t r o n g i n t e r m i x i n g (SI), u n i f o r m l y s t r o n g i n t e r m i x i n g (USI), and to ta l r e g u l a r i t y . Below we quote s o m e s p e c i a l p r o p e r t i e s of p r o c e s s ut , r e - q u i r e d fo r our l a t e r t r e a t m e n t .

Def in i t i on 1.1. The s t a t i o n a r y p r o c e s s ut s a t i s f i e s the SI cond i t i on [7] if

(~) ----- sup sup J P (AB) -- P (A) P (B) j --+ 0 t~O AEgJ.(O,t)

B~I(t+~,~) (I. I) ! as T--~OO

where c~ (T) is the intermixing coefficient, characterizing the "rate" of intermixing of process ut, which has the following properties.

LEMMA 1. I. Let the random variables ~ and H be measurable with respect to 92 (0, t) and 92 (t + z, oo), T --> 0, respectively.

1. If

p(l-k~ ~ q( 1+ 6 ) M{]~] "}~<ci < oo, M{]~I : } ~<c2<oo

1 1 for c e r t a i n S > 0, p > 1, q > 1, -~--}--~-~--- 1, then

C 1 14-6, [ M {~1} - - M {~} M {~l} [ ~ (4 + 3 t + c, H-~ I )) (~)~+~i (1.2) p-l-6 c2q-F6, O~ ,

w h e r e 61 = 5 ( q / p ) .

902

2. I f ] ~ } ~ < c , < o o , f l l t ~ c 2 < o c , then

I M {~) - - M {~} M {~l} ] < 4c,c# (T). (1.3)

The p roo f of the s e c o n d p a r t of the l e m m a , and of the f i r s t p a r t wi th p = q = 2, is to be found in [7]; the p roof is s i m i l a r f o r o the r p and q.

THEOREM 1.1. Le t the fol lowing eondi t ions be sa t i s f ied .

1. Func t ion 0(x, t) is d i f fe ren t iab le wi th r e s p e c t to x ~ Rn fo r a l l t >_ 0.

2. ~0(n), u ~ R m, is a m e a s u r a b l e function.

3. The r a n d o m p r o c e s s tit s a t i s f i e s condi t ion (1.1).

4. o = (1.4)

fo r 0 <- z - t) w h e r e e1(~', t) is in tegrab le wi th r e s p e c t to v in the in te rva l [0, t] fo r any t > O,

I _I_~i. for some 8>0, p> I, q>l, -~+ q

Then

(u,)I ~ "/}<c~<oo

t 6t 1 l + S t 1 + 6 I

I M {0 (x,, t) ~p (u,)} - - M {0 (x,, t)} M {q, (u,)} i < ~ a (r) c~ 2+8' (t - - r) [4 + 3 (cf +6 (r, t) d2 +6i + c~ +8 (r, t)cg +6''-7 )] d*, 0

(1.5)

(1.6)

w h e r e 5" 1 = 5(q /p) .

H e r e and hence fo r th , the c i r c l e o over a r a n d o m var i ab le denotes the cen t e r i ng opera t ion : ~ = ~ - M{~ ~. By 0x(X, t) we unde r s t and the vec to r c o m p o s e d of the de r iva t ives of 0{x, t) wi th r e s p e c t to the xi:

o~ (x) = ~ 0 (x, t ) , . . . , 0 (x, t)] ~.

Proof . It follows f r o m Eq. (4) that

t

i~ (x, t) = I a (3) [O[(x~, t) y (x,, t~, r)] ~ dr.

F r o m this , and e s t i m a t e s (1.2), (1.4), (1.5), we obtain

t �9 M r ]M{O(xL, t)r =IM{~)(xt, t);p(uL)}l<.~a(T)l {[0x(x,,t)

0

t 6j l 1 + 5 , 1 + 8 1

X y(x,, u,, T)] ~ ~ (ut) t dr ~ ~ a (T) ~'+~'(t - - r) [4 -{- 3 (ct p+---K (3, t) c q+~' + ct '+6 (% t) d +6' )] dr. 0

QED.

THEOREM 1.2. abi l i ty 1

Le t condi t ions 1-3 of T h e o r e m 1.1 be sa t i s f i ed , and m o r e o v e r , le t us have with p r o b -

r 1 I 0x (x,, t) y (x~, u~, T) ] ~ -~- c, (r, t) < oo ( : .7)

l~(u,) I ~< �89 c2 < oo. (1.8)

l M 0 (x,, t) ~p (ut)) - - M {O (xt, t)) M {r (u3}] ~ cz I a (T) c, ~ , t) ~ (t - - T) aT. 0

The proof , which is ba sed on e s t i m a t e s (1.3), (1.7), (1.8), is s i m i l a r to the p roof of T h e o r e m 1.1.

(1.9)

for T ~ [0, t],

The n,

903

Def in i t i on 1.2. p a s t p o w e r q if

We s a y tha t the s t a t i o n a r y p r o c e s s ut s a t i s f i e s the USI cond i t i on of fu tu re p o w e r p and

sup sup I M {~l} - - M {~} M {~}1 , , ~ (t (z, p) --~ 0 (1 .10 ) t~o ~,~ - - --

M" { I ~_I"} M" {I n I"}

a s v - - ~ , w h e r e the r a n d o m v a r i a b l e s ~ and g a r e m e a s u r a b l e w i th r e s p e c t to ~.I(O, t), 2 ( t ~ , oo), and

I l

M"{l.~l"}<oo, M" {Inl~}<oo, p ~ l , I I=i" q>z,~+u

H e r e , by M~ ~1} ~ we u n d e r s t a n d vrai sup] ~(to)] in a c c o r d a n c e wi th the p a s s a g e to the l i m i t [8]: 6}

I

lira M q { [ ~ (to) I q} -~-- vrai sup I ~ (to) l-

I f r e l a t i o n (1.10) ho lds fo r p = - , q = 1, the r a n d o m p r o c e s s ut w i l l be s a i d to s a t i s f y the USI cond i t i on of the p a s t , o r i f p = t , q = ~ , the USI c o n d i t i o n of [7], the func t ion ~(T, 1) be ing deno ted s i m p l y by go(T); i f p = q = 2, then the p r o c e s s s a t i s f i e s the cond i t ions of t o t a l r e g u l a r i t y [9, 10].

�9 THEOREM 1.3. L e t cond i t ions 1 and 2 of T h e o r e m 1.1 be s a t i s f i e d and, in add i t i on , the fo l lowing c o n d i - t ions :

1. R a n d o m p r o c e s s ut s a t i s f i e s the USI of fu tu re p o w e r p > 1 a n d p a s t power q = p / ( p - 1)

I T 2. M p { I[0x (x~, t) y (x~. u~, t ) r [p} < oo, (1.11)

1

M ~ { ~ (u~)I ~} < oo (1.12)

fo r a l l T ~ [ 0 , t] , t > 0. Then,

t 1

T 0 T [ M {0 (x t, t ) , (ut) } - - M {0 (x t, t)} M {r (ut)} [ ~ ~ a (T) qD (t - - x, p) M { I[ x (x,, t) y (xT, u~, T)J~ I p} dr. (1 .13) 0

The p roo f , w h i c h is b a s e d on e s t i m a t e s (1 .10)- (1 .12) , is s i m i l a r to the p r o o f of T h e o r e m 1.1.

LEMMA 1.2. P r o c e s s u t s a t i s f i e s the USI cond i t i on i f arid only i f

P (AB) P (A) P (B) sup sup ---- % (T) --~ 0, T--~ oo. (1.14) t>~0 A~(0.t~ P (A)

BE~( t+ '~ ,*a}

H e r e , go10") -< g0(T) ___ 2gol(z) , and p r o c e s s ut s a t i s f i e s the USI cond i t i on of fu tu re p o w e r p > 1 and p a s t power !

l ] q = p / ( p - 1) > 1 w i th func t ion (p(% p) ~2qDT-(~), i . e . , g i v e n any p > 1, q > 1, -~ +-~------ l , and r a n d o m v a r i a b l e s

1 1

, ~7, m e a s u r a b l e wi th r e s p e c t to 9~ (0, t). 92 (t + ~, ao), r e s p e c t i v e l y , and such tha t M" {t ~ I p} < co, M Q {I ~11q}< ~o we have the e s t i m a t e

1 ! I

I M {~l} - - M {~} M {~} [ ~ 2q~ p (~) M ~ {[ ~ I p} M r {[ 'I [q}. (1.15)

Proof. We know [7] that, from (1.14) we obtain (1.15) with go(T) = got(T), while irom (1.10) and (1.15) with p = 1 we obtain go(T) <-- 2gol0-). In addition, taking the upper bound in (1.10) with respect to ~=~A, A@ 9~(0,t),

= ~B, B @ 9/(t ~- ,, oo), we obtain gol 0") < go0"), where

1~ f-- toEA, XA (to) ~ for to ~ A.

LEMMA 1.3. P r o c e s s tl t s a t i s f i e s the USI cond i t i on of the p a s t i f and only i f

sup sup ] P (AB) - - P (A) P(B) ] ~ r (T) --~ O, ~ - ~ oo. ~o A~(O.0 P (B)

(1.16)

QED.

904

H e r e , q~2(T) --< ~0(r, ~) --< 2 ~ ( r ) , and p r o c e s s u t s a t i s f i e s the USI condi t ion of fu ture power p > 1 and pas t power q _> 1 wi th the. funct ion r p) --< 2r i . e . , for any p > 1, q = p / ( p -- 1) > 1, and for r a n d o m va r i ab le s ~,~ and

1 1

~, m e a s u r a b l e wi th r e s p e c t to 9~ (0, t) and ~ (t q- ~, oo), r e s p e c t i v e l y , and such that M p {] ~ I ~} < co, M r {] '1 ]q} < co, we have the e s t i m a t e

1 1 I

[ M (~ l} - - M {~} M {~1} ] ~ 2(P "~ (% co) M ~ {] ~ [P} M ~-{I ~1 Iq} �9 (1 .17)

COROLLARY. A s s e r t i o n (1.13) of T h e o r e m 1.3 r e m a i n s in fo rce for any p > 1, q = 1 / ( p - 1), and under the conditions:

1) P r o c e s s u t s a t i s f i e s the USI condi t ion of the pas t , while ins tead of the funct ion ~o(T, p) in (1.13) we take the funct ion 2r ~);

2) p r o c e s s u t s a t i s f i e s the USI condi t ion of the pas t , whi le ins tead of funct ion ~(T, p) in (1.13) we take the funct ion 2~ /P(T) .

THEOREM 1.4. Le t the fol lowing condi t ions hold:

1. The funct ion 0(x, u , t) is d i f fe ren t iab le wi th r e s p e c t to x ~ R n for any u ~ R m, t > 0, and is such that the v e c t o r funct ion 0x(x, u, L) is m e a s u r a b l e wi th r e s p e c t to u e R m.

0~ (x, v. t) y (x, u~, ~) I < oo for any v ~ Rm, t >~ 0, �9 ~ [0, t]. 2, sup vraisup[ r

3. The s t a t i o n a r y pirocess ut s a t i s f i e s the USI condi t ion of the past. Then,

1M {O (xt, ut, t)/u~ = v} - - M {0 (xt, v, t)} I ~ i "a (~) q~ (t - - x, oo) vrai sup [ O r (x,, v, t) y (x,, u~, %) [ dr. (1.19) 0

Proof . I t follows f r o m Eq. (4) that

i 0, (x~, ut, t) y (x~, u~, ~) d~ + 0 (Xo, u,, t). 0(x, , u , t) ----- - - a(~) r (1.20) 0

Given any Bore l se t A ~ R m, le t

I I for u(t,(0)EA,

ZA(~ 0 for u(t,o))~A.

We take v E R m and choose A such that v E A. On the bas is of (1.10), wi th q = 1, we have the e s t i m a t e

T I M {0~ r (x~, v. t) y (x~, u~, ~) XA} - - M { 9x (x~, v, t) y (x,, u,, T)} P (A) I ~ 2M {] ~A 1} q~ (t - - X, oo) vrai sup [O r (x,, v, t)

X y (x~, u~, T) I ----- 2P (A) (1 - - P (A)) q~ (t - - x, co) vrai sup [ 0 [ (x,, v, t) y (x~, u~, ~) 1. (1.21) o)

Afte r dividing inequal i ty (1.21) by P(A) and pas s ing to the l imi t as A - - v, (1.21) t r a n s f o r m s

]M {O r (x,, u,, t) y (x,, eh, ~)/ut ----- v} - - M (0 r (x~, v, t) y (x~, u~, ~)} J ~ 2q~ (t - - ~, co) vrai sup ] O r (x~, v, t) y (x~, u~, ~) 1. (1.22) fo

Inequal i ty (1.22) in con junc t ion wi th Eq. (1.20) g ives us (1.19). QED.

LEMMA 1.4. Condi t ion (1.16) is equivalent to the condi t ion

sup sup v r a i s u p I P ( B / ~ i ( O , t ) ) - - P ( B ) l _ ~ . q ~ t ( ~ ) - , . O as ~ - , . o o . (1.23) t>~0 BE~{(t--F~, ao ) co

The p r o o f may be found in [7].

LEMMA 1.5. If p r o c e s s u t s a t i s f i e s the USI condi t ion, then, g iven any t 1 < t, and r a n d o m var i ab les ~ and ~7, m e a s u r a b l e , r e s p e c t i v e l y wi th r e s p e c t to ~(0, t) and ~[(t-k ~, co), with probabi l i ty 1 we have the following e s t i m a t e , for a n y p - 1, q = p / ( p - 1 ) > 1:

I I I

]M {[~/9[ (0, t0} - - M {E/g[ (0, t,)} M {~1} [ ~ 2r p (~)M p {1 ~ [P/9.1(0, t~)} vrai sup M q { ] ~11q/~ (0, tO}, (1.24) m

if the r i gh t s ide of (1.24) is meaningful .

905

Proo f . Le t ~N(00) and ~N(W) be r a n d o m va r i ab le s of the f o r m

N~--I N ~ - - I

~ (~o) = ~ ~.,x.~, ((o), %, (o)) - ~ ~,x~ (0% ~=--N' i=--N ~

where

~ ( ( o ) = { ~ ~or (o~c, for (o (~ C,

A s--- {(o i i- l-1 : ~- <~(~o)<~ ~ / ,

Bj= i.: i <~l(o)) <~-L-~-}, Noting that , by v i r tue of (1.23), we have the inequal i ty

i ~' = "E'

/ ~=E"

N2- -1

sup ~ l P (B/A,, 91 (0, t,)) - - P (Bj) I ~< 2q) (~), I i ~ , g ~ ,

we obta in for the r a n d o m va r i ab l e s i N and ~N the e s t i m a t e

m !_ I M g,v%,/~ (0, t,)} - - M 1~/~ (0, tO} M {nM}l = ~: ~,P "(A,/~ (0, t,))

An t t " N2 1 N~, M ~ [P(B,'A~, 91(O,t~))--P(Bj)]~t~P~-(A,/91(O, tO)I~MT{l[~lP/91(O,t~)} ( ~ P(A/91(O, t,) I pjl q

I=--N*--I i=--NL-I ] I=--N --1

q !

N(P(Bj/A~, ~l(O, ti))-I-P(B.~)) IP(B/A~, ~(O, tO)--P(Bj)I ~ 2 3 ~ ~ ( '0M {I~NIP/9~(0, ti)} \ [/=--N'--!

l 1 1 I

~- -- M T t P 0 X (M {I ~I~Iq/~ (O, tO} -F M {I ~INIq}) ~ 2 ( p " (x) {I~N] /~ ( ,t0}vraisup Mq {ln~lq/~(O, tO}. (9

The proof of the l e m m a is obta ined in the gene ra l ease by pas sage to the l imi t as N ~ .o in (1.25). QED.

THEOREM 1.5. Le t condit ion 1 of T h e o r e m 1.4 be sa t i s f ied and, in addit ion, the condi t ions:

�9 1 1. vrai sup I O~ (x, u~, t) y~ (x, u, "0 [ ~ -~ L (x, u, t, ~) < oo,

o)

w h e r e L(x, u, t , T) is m e a s u r a b l e wi th r e s p e c t to x ~ R n, u (: R m, ~- ~: [0, t].

2. P r o c e s s ut s a t i s f i e s the USI condit ion. Then,

I M {M {0 (x, ut, t)} I ~ , } - - M {0 (x . n,, t)} I ~< i a (~) q~ (t - - x) M {L (x v u, , t, ~)} d~. O

P r o o L Let U(T), ~" t [0, t], be a m e a s u r a b l e de t e rmina t e funct ion, U(T) ~: R m, x(t) (: R n is the so lu t ion of Eq. (4) c o r r e s p o n d i n g to funct ion ut - u(t). Then, us ing (1.20), we e a n w r i t e

0

u~ ----- u ( '0}-- M {'0~ (x (~) u . t) y (x 0:), u t~), "0) I~,(~)_~} d'~.

Using inequal i ty (1.24) of L e m m a 1.5, and condi t ions (1.26), we can a s s e r t that

I M{ 0~ (x ('0, ut, t) y (x (~), u(~), ~ ) / x = x (~),

u. ---- u (T)} - - M {0~ (x (x), ut, t) y (x (T), u (~), v)} J ~ q) (t - - ~) L (x (T), u (~), t, T).

(1.26)

(1.27)

Subst i tu t ion of (1.27) in (1.26) proves the t heo rem.

906

THEOREM 1.6. Le t the condi t ions of T h e o r e m 1.5 hold, and le t

1 vrai sup ] 0 (x, u,, t) I ~ -~- K (x, t). co

Then, g iven any t 1 _< t, we have

t

IM {[0 (x,, u,,t) M{0 (x, u , t)} l~=~tl/9~ (O, t,)}l:<~q~(t--tJK(xt, t) + I q~(t-- ~) a( '0 M{L (x~, u~, t, "0/~I (0, t0} d~:. (1.28) t t

Proof . F r o m (1.20) we obtain the e x p r e s s i o n , ana logous to (1.26),

I M {(o (x~, u . t) - - a,I {0 (x, u . t)} I.=~)/~a (o, t0}[ ~<l M {o (xt, u,, t)pa (0, t,)}--/vI {o (x, u . t)} I~=.t I+

t

+ ~ i M {M {[0~ (x,, u~, t) - - M {0~ (x, ut, t)} I~ffi, I /9/(0, ,)} y dx,, u,, z)/9~ (0, tt)}l a (z) d * . It

(1.29)

Using r e l a t i o n (1.24) of L e m m a 1.5 wi th ~ = 1, ~ l~0(x , ut, t) and ~l=0~(x, uv t)yt (x, n, z), r e s p e c t i v e l y , we obta in

I M {0 (x, ut 0/9I (0. tO} - - M {0 (x, u t, t)}l <~q~ (t - - tO K (x, t), [ M {0~ (x, ut, t)

XYi (x, u, T)/~ (0, "0}--M{0~ (x, u v t)} Yi (x, u, z) ] ~ r (t - - ~) L (x, u, t, ~). (1.30)

Subst i tu t ion of inequal i t ies (1.30) in (1.29) gives us (1.28). QED.

The r e s u l t s obta ined in the p r e s e n t s e c t i o n in fac t point to the poss ib i l i ty of e s t ima t ing , with a c e r t a i n g u a r a n t e e d a c c u r a c y , the condi t ional and uncondi t ional m a t h e m a t i c a l expec ta t ions of the r a n d o m p r o c e s s 8(Xt, ut, t), in the s a m e way as if p r o c e s s e s x t and ut w e r e independent . E s t i m a t e s of this kind a r e g iven in Theo- r e m s 1 .1-1 .3 fo r the weakes t c l a s s e s of r e g u l a r r a n d o m p r o c e s se s u t and fo r funct ions 0 (x, u, t), 0 (x, t)r (u, t). The e s t i - ma tes of the m a t h e m a t i c a l expec ta t ions of funct ions 0(x, u, t) of a m o r e gene ra l type, g iven in T h e o r e m s 1.4- 1.6, demand s t r eng t he ne d r e g u l a r i t y p r o p e r t i e s of the p r o c e s s ut , namely , the p r o p e r t i e s of USI o r USI of the past . We can use these e s t i m a t e s to prove the c o n v e r g e n c e of va r ious v e r s i o n s of s t o c h a s t i c a p p r o x i m a t i o n p r o c e d u r e wi th cont inuous t ime.

2 . C O N D I T I O N S F O R C O N V E R G E N C E I N T H E M E A N . I

The type of the r e g u l a r p r o c e s s u t emp loyed in the s t o c h a s t i c a p p r o x i m a t i o n a l g o r i t h m (4) has an i m - por tan t inf luence on the condi t ions under which the a l g o r i t h m c o v e r g e s to the e x t r e m u m point. The d i f fe rence in the c o n v e r g e n c e condi t ions depends on the r a t e s of d e c r e a s e of functions a(t) and on the admis s ib l e sl~ruc- tu re of v e c t o r funct ion y 0 t , it, t).

Be fo re p rov ing our c o n v e r g e n c e t h e o r e m s , we r e q u i r e s o m e p r e l i m i n a r y l e m m a s .

LEMMA 2.1. Le t the d i f fe ren t iab le funct ions 7r and v(t), t ~ [0, T] , be such that

v(t) --~ f(t) ~.~ - - , (v (t), t) + 7(t), v(0) ~ v ~ 0 . (2.1)

v(t) -~- - - , ( v (t), t) -{- 7(t), v(0) ~---vo> / O, (2.2)

w h e r e f(t), ~ (t) a r e bounded fo r al l t -> 0, and in tegrab le in any finite in te rva l [0, T], 7 (t)1> 0, ~ 7 (t)dt < oo, 0

0 f~ v ~ 0 , t />0 , , ( o , t ) ~ %(v,t)>/O f~ v > 0 , t ~ O ,

$0(v, t) is bounded wi th r e s p e c t to t > 0 for any g iven v > 0, and is in tegrab le in the finite in t e rva l [0, T] , while i t s a t i s f i e s a Lipsehi tz condi t ion wi th r e s p e c t to 0 < v <- c < ~o. (Here and below, 0 < v < c < oo is a cons tan t , whose value is not impor tan t . ) Then, v(t) _< v(t) fo r a l l t ___ 0.

Proof . I t follows obvious ly f r o m the condi t ions of the l e m m a that v(t) < c < ~o and 0 -< v(t) < c < ~ for al l t > 0. We sha l l use E u l e r ' s method wi th s tep A for a p p r o x i m a t i o n of the so lu t ion of Eqs. (2.1), (2.2) in the in te rva l [0, T]. Then, fo r kAE [0, T] , we have

v t ((k + 1) A) ~--- vt (kA) + f (kA) A ~ vi (kA) - - , (v, (kA, kA), A + 7 (kA) A, (2.3)

((k + 1) A) ~---v~ (kA)-- , (vt (kA), kA) A+u (kA) A. (2.4)

907

We know [11], g i v e n any e > 0, we can f ind b 1 > 0 such tha t , if A < 51, then [v(leA)--v~(kA)l~e and. Iv(kh) -

v~ (kA)I~8 fo r a l l kA ~ [0, T].

L e t 62 �9 0 be s u c h tha t , for A -< 52, the funct ion v - ~(v, t )A is m o n o t o n i c a l l y n o n d e c r e a s i n g wi th r e s p e c t to v fo r a n y t ~ [0, T] a n d v r [0, c]. Then , i t c an be s e e n f r o m (2.3), (2.4) tha t , f o r A _< m i n (61, 52), we have v~(A) _< Vl(A) and v~((k + 1)A) _ v~((k + 1)A), if v~{kA) _< v~((k + 1)A). Consequen t ly , v~(kA) _< v~(kA) fo r all kA [0, T] a n d v(kA) _< v(kA) + 2e. S ince e > 0 is a r b i t r a r y , w e c a n c l a i m tha t v(t) <_ 7r for any f in i te t >_ 0. QED.

LEMMA 2.2. A s s u m e tha t , fo r t e [0, ~o), the func t ion v(t) s a t i s f i e s the e q u a t i o n

v (t) = - - a (t) • (v (t)) -]- b (t), v (0) ~ v o >/O, (2.5)

w h e r e a(t) �9 0, b(t) > 0 a r e bounded func t ions fo r t -> 0, and i n t e g r a b l e in any f in i te i n t e r v a l [0, T], wh i l e

ao r

~ ~ ( t ) d t = oo, j" b(t) d~< o o, 0 0

0 f~ ~<~0, ~ (D = Xo(~ )>0 f~ ~ > 0 ,

~t0(D is con t inuous w i t h r e s p e c t to ?, ~ [0, oo). Then , v(t) ~ 0 a s t ~ .o.

P roof . I t i m m e d i a t e l y fo l lows f r o m (2.5) tha t 0 <- v(t) _< c < ~o and

t t

,, (t) = ~, (0) = I ~ (~) ~ (o (~)) d~ + o ( b (~) d~, 0 0

i . e . , v(t) c a n be w r i t t e n a s the s u m of a m o n o t o n i c a l l y i n c r e a s i n g , u p p e r - b o u n d e d , and a m o n o t o n i c a l l y d e - c r e a s i n g , l o w e r - b o u n d e d , funct ion . Hence v(t) has a f in i te l i m i t a s t ~ ~o. But then , v(t) ~ 0 a s t ~ ~o. O t h e r -

co

w i s e , ~a(~)u(v(~))d~---oo. QED. 0

THEOREM 2.1. L e t the fo l lowing cond i t i ons hold.

1. P r o b l e m (1) i s s o l v a b l e wi thou t c o n s t r a i n t s , i . e . , X = Rn.

2. The funct ion fix) is downwards convex and r e a c h e s i t s m i n i m u m wi th r e s p e c t to x e R n in a bounded s e t X * ~ R~, i . e . , f o r x * ~ X * , we have [Ix*l[ < R < ~r

3. T h e r e e x i s t s a func t ion n(h) of one v a r i a b l e ~, de f ined and d o w n w a r d s convex for ~ -> 0, and such tha t ~t(h) _> 0 fo r h > 0, wh i l e fo r any X E R n we have

u ( rain II x - - x* II ~) ~< (x - - x*) ~ f~ (x), (2.6) x*EX*

w h e r e fx(x) is the g r a d i e n t o r a s u p p o r t func t iona l of func t ion f ~ ) a t the point x E R n.

4. The r a n d o m p r o c e s s u t s a t i s f i e s the SI cond i t i on (1.1).

5. The v e c t o r func t ion y{x, u, t) c a n be w r i t t e n in the f o r m

N

y (x, u, t)----z ~ (x, t )+H (u, t ) + ~ z ~ (x, t) hk (u, t), k==l

(2.7)

w h e r e the v e c t o r func t ion H(u, t) ~ R n and the func t ions hk(u , t ) , k : 1, 2 , . . . , N, a r e m e a s u r a b l e wi th r e s p e c t to u fo r any t >_ 0 and s a t i s f y the e s t i m a t e s

N 8

"+~/M {11 n (ut, 0 II ~+6} + ~ ~+~/M {I h~ (u,, t)f~+6~ ~< c < oo, k = l

q+~/M{[I H (art)I[ q+6' } + a~ VM{!hk(u~,t)]q-~,} ~ c < o o ' (2.8)

fo r a l l t -> 0, w h i l e the v e c t o r func t ions z k ( x , t) E R n s a t i s f y , fo r a l l x E Rn, the e s t i m a t e s

1 V

iizO(x, 011~c ( l+ l lx l l 2 - (a -~ -~ )~ ) , I i z~ (x , t ) l t~c ( l+ l lx l l~ - ( l%--~)T) , k = 1 , 2 ..... N,

908

i Iz~(x , t ) l l . l l zO(x, t ) l t<c( l+l lx l l~-( ' -~)v) , / ~ = I , 2 . . . . . At,

~ o ~ (,, 011 ,,z0(,, o,, <c( , + , , , , ' - ( ' ~ ) ~ ) (~.~)

and the quant i t ies p, q, 6, 51, 3' s a t i s fy the r e l a t ions p, q > 1, 1 / p + 1 / q = 1, 6, 7 > 0, 5~ = ~kl/p.

6. Fo r the v e c t o r funct ion g(x, t) we have the inequal i ty

II g (x, t)ll <~ d (t) (] + II x It). (2 .10)

7. The funct ions a(t), c~(t), d(t) a r e bounded for t -> 0 and in tegrab le in any finite t ime in te rva l , wlhile they

sa t i s fy the condi t ions a(t) -> 0, m(t) -> 0, d(t) - 0, and

S a(t)dt=oo, I a(t)d(t)dt < cr 0 0

a (t) dt a2-~, (t - - x) a (x) dx < oo. 0 0

(2.11)

T h e n , v v (t) = M { rain II xt - - x* IIv}-~0 a s t - - ~ . x*EX*

Proo f . We put

v (0 = M { ,rain, e (11 x~ - x* II)},

w h e r e e(X) = AT if y -> 2, and

?

e (;9= ~-~ r'~ X z for 0..<~,..< V ~ '

i f 0 < T < 2 .

Obvious ly , the c o n v e r g e n c e to z e r o of v(t) impl ies the c o n v e r g e n c e to z e r o of vy(t) , and v ice v e r s a .

Denote by x*(t} ~X* the v e c t o r such that rain Ilx (t) - - z*ll-----IIx(t) --x*(t) ll. If x(t) ~ X * , then the ve loc i ty x*EX*

v e c t o r x*(t) l ies in the suppor t plane to the s e t X* at the point x*(t). Consequent ly , (x(t) - x*(t))Tx*(t) =- 0. Us ing th is , we obta in f r o m Eq. (4):

t l x t - - x ~ t l ( x t - - x t ) [ z~

_o(,)ZM { *'"'* k,=l

(x, - - x;) ~ z k (xt, t) hh (ut, t)}. (2.12)

It follows f r o m T h e o r e m 1.1 and condi t ions (2.8), (2.9) that

M ] 8" (11 xe - - x7 II)

II x~ - - x~' II r . ( . , ,)} - M { *" ("*'-*: '')

II x, - - x, II t

~ - ' ~ (t + v (z)) dr, <~c i a(~) --~)(I 0

(x~ --x~) ~} M {n (u,, t)}

(2.13)

and for i= 1 , 2 , . . . , Nwehave

II xt - - x; It (xt --x~)' z k (x t, t) h~ (u t, t) - - M e" (tILl xtX'-- x;XTIIll)

l i " N M{h~ (u,, t)} ~ c a (~) ~z 2+~, (t - - 1:) (1 + v (z)) tiT.

(x, -- x])' z* (x,, t)}

(2.14)

909

A f t e r subs t i tu t ing (2.13) and (2.14) in (2.12) and r eca l l i ng (2.6), we get

{e " II x, - - x; It } ' ~' * S a (r) ~2-'~'~ - - r) (1 dr v (t) ~ < - - a (t) .vl II xt - - x, II (xt - - xD' (f,, (xO + g (xt, t)) + ca (0 (t + v (~))

0

[ t 6, ] { C ( l lxt__x; l l ) a ('0 r 2+6, r) (1 + M <~ca(t) d( t ) ( l +o( t ) ) + ~ ( t - - v(~))d~ - -a ( t ) I l x t _ x T i I

o

(11 x t - x; It')}. (2.15)

' . () 8'I XT) 2 The funct ion v t ( X ) = - - - - - y ~ X V has a nonnegat ive de r iva t ive for X _> 0, and the re ex i s t P0 > 0 and X 0 > 0

s u c h that /~ ' (~,) _> #0 for X -> A 0. Hence a downwards convex funct ion xt(X) , pos i t ive for ~ > 0. ex i s t s , such that xiiX) -< V(A) for X _> 0. On e s t ima t ing the f u n c t i o n # (M in (2.15) wi th the aid of xl(X) and us ing J e n s e n ' s inequal - i ty, we obta in

' ~'. (2.16) ~, it) < a It) ,,, (v it)) + ca it) [d it) i l + v it)). + S a (r) ~,~+~, it - ~) ( 1 + v (r)) d'c. O

In view of condi t ions (2.11), funct ion v(t) is upper -bounded . Using this , we can a s s u m e that v(t) s a t i s f i e s the inequal i ty

b (t) <~-- a (t) ~ (v (t)) + cb (t), v (0) ~ vo >I O,

w h e r e

t 81 ] b ( t ) = a (t) ~ a (r) czV+~ (t - - r) dr + d (t) .

0

We now in t roduce the funct ion v(t), def ined by the equat ion

it) = - - ait) • (if(t)) + cb it), ~(o) = vo 1> 0.

By L e m m a 2.1, we have v(t) <_ v( t ) , and by L e m m a 2.2, "v(t) --* 0 as t - - ~o. QED.

R e m a r k . Condi t ions (2.6) and (2.9) can only be compat ib le i f , fo r su f f i c i en t l y l a rge 0 < R < 0% we have II z~ t)ll > c l lx l l . Hence T cannot be g r e a t e r than (p + & ) / ( p - 1 + 5).

THEOREM 2.2. Le t the fol lowing condi t ions hold.

1. The s e t X in p r o b l e m (1) is bounded and defined in the f o r m

x = {x 6 R. : , (x) ~< 0},

w h e r e $(x) is downwards convex.

2. The funct ion f(x) is downwards convex in the se t {x : r -< p}, r e a c h i n g its m i n i m u m in X in a se t X* such that , g iven any x* E X*, we have r < 0, w h e r e p > 0 is a constant .

3. The re ex i s t s a downwards convex funct ion ~(M, X > 0, s a t i s fy ing condi t ion (2.6) for al l x : ~0t) <- P,

4. The r a n d o m p r o c e s s u t sa t i s f i e s the s t r o n g in te rmix ing condit ion.

5. The v e c t o r f unc t i ony (x , u, t) is f o r m e d as fol lows:

y(x, u, t) = F (q' (x), p) Yt (x, u, t) + ( l - - p (~2 (x), p)) ~, (x), (2.17)

1 for X ~ 0 ,

I ~ ( ~ , p ) = / - - l ( ~ - - p ) for 0 < ~ < p ,

/ 0 for X ~ p .

It can be w r i t t e n in the f o r m

N y, (x, u, t) ----- ~ z k (x, t) hh (u, t), (2.18)

~ 1

910

/g

w h e r e hk(U, t) , k = 1, 2, . . . , N, a r e m e a s u r a b l e w i t h r e s p e c t to u for a l l t > 0, such tha t ~ l h a ( u t , t)k< c < o o

w i t h p r o b a b i l i t y 1 fo r t ~ 0, v.k(x, t) ~ R n a r e d i f f e r e n t i a b l e w i th r e s p e c t to x t~ Rn f o r any x ~ R n : r _< p,, and any t > 0, and a r e v e c t o r func t ions s u c h tha t , fo r r _ p, t -> 0, we have

IIz~(x't) ll-l- Ox~ (x , t ) ~ < c < ~ . r = l

(2.19)

6. The v e c t o r func t ion g{x, t) is s u c h tha t

] [ g ( x , t ) l l ~ d ( t ) for ~p(x)~p, t ~ 0 . (2.20)

7. F u n c t i o n s a( t ) , o~(t), d(t) a r e i n t e g r a b l e in any f in i te i n t e r v a l and s a t i s f y the cond i t i ons a(t) _~ 0, ~(t) -

O, d(t) -> O, and la(t) dt -~- oo , 0

i a (t) d (t) at < ~ , 0

t

0 0

(2.2Z)

Then , g iven any T > 0, we o b t a i n

vv (t) ~- M/m~n [[ xt - - � 9 II*) ~ 0 as t ~ .

P r o o f . I t c a n be a s s u m e d wi thou t l o s s of g e n e r a l i t y that ,~(x t) _< p wi th p r o b a b i l i t y 1. S ince the s e t {x : ~(x) _<p } is bounded, th i s i s e q u i v a l e n t to the c o n v e r g e n c e to z e r o of M { l l x t - x ~ l l ~ / ) fo r any T > 0. Hence i t is s u f f i c i e n t to p r o v e the t h e o r e m , e . g . , in the c a s e T = 2.

U s i n g the s a m e no ta t i on as w a s u s e d in the p r o o f of T h e o r e m 2.1 , we o b t a i n the i n e q u a l i t y , s i m i l a r to (2 . I2) ,

N * T ~a (t) ~ - - a (t) ~ M{(x t - - xt)rzk(xt, t) ~ (~p (xt), p) h a (u t, t)} - - a (t) M {(x t - - x ~) Cx (xt) ( 1 - - p. (i[, (xt) , p))}.

By T h e o r e m 1.2 and c o n d i t i o n (2.19), we have

(2.22)

t * k r ~ ~ (t - - ~) a (~) d~. I M {(xt - - x ~ ) r z k (Xt, t) 11 (lp (Xt) , p) h k (tit, t ) } - - M {(x, - - xt) z (xt, t) .a (r (xt), r')} M {hh (U,, t)} ~< C

0

(2.23)

Subs t i t u t i ng ( 2 . 2 3 ) i n (2.22) and r e c a l l i n g (2.6), we g e t

( ' ) v (l) ~ a (t) M {n (xt) } --}- ca (t) d (t) Jr- ~ o~ (t - - ~) a (~) d~

w h e r e

(x) = ~ (~ (x), P) ~ (]l x - -x~ II~)-t-(l--.uO~" (x), P)) (x - - Xx) 'I'. (x),

I] x - - x x I[ = rain I[ x - - x* II- x*EX*

(2.24)

The func t ion u(x) i s con t inuous and nonnega t i ve fo r ~(x) _ p, and v a n i s h e s only fo r x ~ X*. Hence t h e r e e x i s t s a nonnega t ive func t ion >tl00, downwards convex fo r ~ > 0, which s a t i s f i e s the concl i t ion

u,(minjl x - - x * H ~ ) ~ a ( x ) for r (2 ,25) x*EX* *

U s i n g (2.25), we c a n w r i t e i nequa l i t y (2.24) a s

b (t) ~ - - a (t) • (v (t)) "k" ca (t) d (t) -I- ~ a (~) ~ (t - - ~) d . 0

(2.26)

U s i n g L e m m a s 2.1 and 2 .2 , i t fo l lows f r o m (2.26) tha t v(t) - - 0 a s t ~ ~. QED.

911

3 . C O N D I T I O N S F O R C O N V E R G E N C E I N T H E M E A N . I I

By us ing in a l g o r i t h m (4) a s t o c h a s t i c p r o c e s s tha t s a t i s f i e s c e r t a i n un i fo rmly s t rong in t e rmix ing cond i - t ions , we can p rove the c o n v e r g e n c e of the a l g o r i t h m in the c a s e of a w i d e r c l a s s of funct ions y(x , u, t) that is admi t t ed by T h e o r e m s 2.1 and 2.2.

THEOREM 3.1. Le t the fol lowing condi t ions be sa t i s f ied .

1. Condit ions 1-3 , 5 (with 5 = 0), and 6 of T h e o r e m 2.1.

2. R a n d o m p r o c e s s ut s a t i s f i e s the USI condi t ion of fu ture power p _> 1 and pas t power q = p / ( p - 1).

3. The funct ions a ( t ) , ~(t) , d(t) a r e bounded for t _> 0, a r e in tegrab le in any finite in te rva l , and sa t i s fy the condi t ions a(t) >- 0, ~(t) -> 0, d(t) >_ 0,

co oo ~ a (t) dt = ~ , ~ a (t) d (t) dt < ,,,,, 0 0

t

a (t) dt~ cp (t - - ~, p) a (~) d'r < c o .

0 0

T h e n , v~ (t) ---= M {rain II x~ - - x* 11~}-+ 0 a s t - - ~ . x*EX*

The p roo f of this t h e o r e m is based on T h e o r e m 1.3 and is s i m i l a r to the p roo f of T h e o r e m 2.1.

R e m a r k . Ana lys i s of condi t ions (2.9) shows that they impose v e r y weak cons t r a in t s on the vec to r func- t ions zk(x, t), k = 0, 1 . . . . . N w i t h p = 1. In this ca se the conve rgence to z e r o of w/(t) as t ~ ~ holds for any "y>0 .

THEOREM 3.2. Let the fol lowing condi t ions hold.

1. Condit ions 1-3 and 6 of T h e o r e m 2.2.

2. The r a n d o m p r o c e s s ut s a t i s f i e s the USI condi t ion of the past .

3. The vec to r funct ion y{x, u, t) is w r i t t e n in the f o r m (2.17), w h e r e yl(x, u, t) is d i f ferent iable wi th r e - spec t to x e R n for r _< p for any u ~ R m, t _> 0, is m e a s u r a b l e wi th r e s p e c t to u ~ R m along wi th its d e r i v a - t ives (~/~xi)yl(x, u, t), and is such that

fo r al l r {x) _< p.

It Yi (x, u~, ~)Jl vraisup (l[ Yl (x, ut, t) ]1 -~ ~X~

4. The funct ions a(t), (p(t), d(t) a r e in tegrab le in any finite in terval . They sa t i s fy the condi t ions

Then, for any ~/> 0, we have

ao ~a

~ a ( t ) d t = ~ , ~a( t )d ( t )d t < ~ , 0 0

t

i a (t) dt ~ q~ (t - - T) a Q;) d z < oo. 0 0

vv ( t ) - - - - -M{min l l x t - - x* l l~} - -~O as t -+oo. x*EX*

Proof . It is suf f ic ien t to prove the t h e o r e m for ~ = 2. We have the re la t ion , s i m i l a r to (2.22),

b (t) -~ - - a (t) M {(xt-- x;)rY (xt, Ut, t) ~ (• (Xt), [,)} - - a (t)/~4 {(X/ - - x t )T~x (Xt) (1 - - ~t (4' (Xt), P)))"

F r o m T h e o r e m 1.4 and condi t ion (3.1) we have

I 114 {(X t - - x;)Ty (Xt, U,, D ~t(r P)} - / 1 4 {M {(x t - - x;)Ty (X t, V, t)} lyric} I ---- I M ((xt - - X;) r

X Y (Xl, Ur t) 9 0P (xt), p)} - - M {(x t - - x;) r (fx (xt)~- g (xt, t))} I ~ i a (~) q~ (t - - ~, oo) 0

(3.1)

(3.2)

912

I) } M vrai,sup ~ [(x~ - - x~)ry (x;, v, t) ~ (tp (x~), p)]ry (x~, u~, ~:) "~"t o (3.3)

Subs t i tu t ing (3.3) in (3.2), and noting the e x i s t e n c e of the funct ion nl(k) , d e s c r i b e d in the p r o o f of ' t h e o -

r e m 2.2, we ob ta in the inequal i ty

( o ) v (t) ~ - - a (t) ~q (v (t)) -]- ca (t) d (t) -b ~ a (~) ep (t - - "c, oo) d~ . (3.4)

The c o n v e r g e n c e to z e r o of v(t) as t ~ ~ fol lows f r o m (3.4) and condi t ion 4 of the t h e o r e m , us ing L e m m a s 2.1

and 2.2. QED.

R e m a r k . By us ing a p r o c e s s ut that s a t i s f i e s the USI condi t ion of the pas t , we c a n d i spense in T h e o r e m 3.2 wi th the s p e c i a l s t r u c t u r e of funct ion yl(x, u, t), p r e v i o u s l y s p e c i f i e d by r e l a t i o n (2.18). The r e q u i r e m e n t tha t the s e a r c h d o m a i n be bounded r e m a i n s e s s e n t i a l he r e .

T H E O R E M 3.3. Le t the fol lowing condi t ions hold.

1. Condi t ions 1-3 and 6 of T h e o r e m 2.1.

2. R a n d o m p r o c e s s ut s a t i s f i e s the USI condi t ion.

3. y(x , u, t) is d i f f e r en t i ab l e wi th r e s p e c t to x ~ R n for any u ~ R m and t _> 0, is m e a s u r a b l e wi th r e s p e c t to u ~ R m a long wi th i ts f i r s t d e r i v a t i v e s wi th r e s p e c t to x , n a m e l y x - ( a /0x i )y (x , u, t), and is such tha t

vrai sup M {yr (x, v, t) y (x, u t, t)} I~=.t ~ c (1-HI x ]l z) (3.5) (i)

fo r a l l x e R n, T ~ [ 0 , t].

Then, g iven any T > 0, we have

v v (t) -~ M {min II xt - - x* II v} --+- 0 x*EX*

as t--~ oo.

P roo f . We can p r o v e , in the s a m e way as r e l a t i o n (2.12), the r e l a t i o n

{ e ' ( I ]x t - -x ; l l ) ( x , _ x : ) r y(xt, ut, t)}. b (t) = - - a (t) M II x ~ - - x ; II

F r o m T h e o r e m 1.5 and e s t i m a t e (3.5) we have

(3.6)

I (11 x~ - - x7 II) ! ~' (11 x, - - x~ It) - - x~) T q,, (x,) M

t

"{- g(xt't))l I ~ . fa (T)eP( t - -~)M{vra ' , . sup ~ t s' ([[x--x'l,)ll x - - x* U (x--x*)rY (x' ut ' t)

0

t

X y (x, u, z)I} d'~ ~ c ~ a(z)~ (t-- 'c)(1-k-v(T)) d'~. 0

(3.7)

Subst i tu t ing (3.7) in (3.6), and noting the e x i s t e n c e of the funct ion ~ ( k ) d e s c r i b e d in the p roof of T h e o r e m

2.1 , we ob ta in the inequal i ty l

v (t) ~ - a (t) x (v (t)) + ca (t) [d (t) (I + v (t)) + f a (~) q) (t - - ~) (I + v ('~)) d'c. (3.8) 0

The c o n v e r g e n c e of v(t) to z e r o as t - - ~o fol lows f r o m inequal i ty (3.8) and L e m m a s 2.1 and 2 .2 , in the s a m e way as in the p r o o f of T h e o r e m 2.1. QED.

4 . C O N V E R G E N C E W I T H P R O B A B I L I T Y 1

The above condi t ions , fo r c o n v e r g e n c e in the m e a n of s t o c h a s t i c a p p r o x i m a t i o n a l g o r i t h m s , have a s e r i - ous i nhe ren t d r a w b a c k , n a m e l y , that we have to i m p o s e condi t ion 3 of T h e o r e m 2.1. This condi t ion s e r i o u s l y r e s t r i c t s the c l a s s of a d m i s s i b l e convex funct ions f(x) for which the c o n v e r g e n c e can be p roved . In p a r t i c u l a r , the condi t ion d e m a n d s tha t the g r ad i en t of f(x) have def ini te s m o o t h n e s s p r o p e r t i e s in the ne ighborhood of the e x t r e m u m - though it is we l l known that this is not the c a s e in a wide r ange of m a t h e m a t i c a l p r o g r a m m i n g

913

p r o b l e m s . A w a y to o v e r c o m e this d i f f icul ty is to confine o u r s e l v e s to p rov ing the c o n v e r g e n c e wi th p robab i l i t y 1 of type (4) s t o c h a s t i c a p p r o x i m a t i o n a l g o r i t h m s .

LEMMA 4.1. Given the s e q u e n c e {~n(~O)}n=0 of r a n d o m v a r i a b l e s , not c o n v e r g e n t to any l im i t for any o ) E D ~ ~ , P(D) = P0 > 0, and g iven the n u m b e r s equence {hk)k~_-0 s u c h that hk ~ ~ as k - - ~. Then t he r e ex i s t s a s equence of s u b s c r i p t s {nk}~=_ 0 such tha t ) 'k -< nk+t - nk < ~ and such that the s u b s e q u e n c e {~nk (W)~k=0 is not c o n v e r g e n t to any l im i t as k - - ~ in the s e t D * ~ D : P { D * ) > 0 .

P roo f . We r e n u m b e r by s u p e r s c r i p t r = 0, 1, . . . a l l the s e q u e n c e s {[~k (r176 for which h k _< nk+ 1 -

n k < ~ o

A s s u m e tha t the l e m m a is fa l se ; then fo r o ~ E D ~ D, P I ~ - ] D ~ I = P o and t he r e n e c e s s a r i l y /rm-0 ] - -

e x i s t a t l e a s t two s e q u e n c e s , s a y {~, (~)}~_--o and {~ (~)}~=o, fo r wh ich P ~ ' (~) ~a ~r' (~), CO E 1"-1 D~ > O, s ince o t h e r - r = 0 )

w i s e the s e q u e n c e {~ (r 0 would be c o n v e r g e n t wi th p rohab i l i t y 1 in the s e t D to ~r(w). But then, the s equence r3, c o m p o s e d of s u c c e s s i v e t e r m s of s e q u e n c e s r~ and r~ a l t e r n a t e l y , wi l l not be c o n v e r g e n t in a s e t of f ini te p robab i l i t y . QED.

LEMMA 4.2. Le t the nonnegat ive r a n d o m p r o c e s s v t = v( t , w), M { v 0) < 0% m a t c h e d wi th the f ami ly of a - a l g e b r a s {9 (0, t)}t=0 fo r any t~ < t, s a t i s f y the inequal i ty

t

M {vt/91 (0, h)} ~ ca (t) ~ q~ (t - - ~) a (~)(1 Jr M {vJgA (0, tl)}) d~ + a (t) d (t) tt

X(1 -t- M {vt/91 (0, tl)}) -~- a (t) b (t) (1 + vt,) (p(t --t~), (4.1)

w h e r e funct ions a(t) , ~0(t), d(t), b(t) s a t i s fy the condi t ions of T h e o r e m 4.1. Then, wi th p robab i l i t y 1, v t has a f ini te l im i t as t ~ ~.

~Proof. I t fo l lows f r o m (4.1) tha t

w h e r e

w h e r e

g {vt/~ (0, tl)} <~ ~t,

t t

= vt,e', + ~ e �9 ~ (T, t 0 d~, tl

(4.2)

[~ (t, tO ---- a (0 d (t) + a (t) c ~ q) (t - - "0 a ('0 d~ + a (0 b (t) q) (t - - h). 0

A p a r t i c u l a r consequence of (4.2) is that p r o c e s s M { v t } is uppe r -bounded .

A s s u m e tha t the l e m m a is f a l s e , and tha t p r o c e s s vt does not c o n v e r g e wi th p robab i l i t y 1 to a f ini te l im i t as t Then t h e r e is a s equence of ins tan t s { k}k=0 such tha t {Vtk}k=0 is not c o n v e r g e n t wi th p robab i l i t y 1 to a f ini te l imi t . Us ing L e m m a 4.1 , we c a n a s s u m e tha t ins tan t s t k s a t i s f y the condi t ion

tk+l

Ir t~

I t c a n ea s i l y be s e e n that the s equence of r a n d o m v a r i a b l e s

| | t,+ ( ~ + , ' ~ - , , ~ ,4 , ~ ~ ~ct.,,~a, ~ ~(e,t,~do+ S ~(e.tkml

is a nonnegat ive s u p e r m a r t i n g a l e and i s t h e r e f o r e c o n v e r g e n t wi th p robab i l i t y 1 to a f ini te l i m i t [12]. This c o n t r a d i c t i o n p r o v e s the l e m m a .

THEOREM 4.1. Le t the fol lowing condi t ions be sa t i s f i ed .

914

1. Condi t ion 1.2 of T h e o r e m 2.1.

2. R a n d o m p r o c e s s ut s a t i s f i e s the USI condit ion.

3. The v e c t o r funct ion y(x , it, t) 6 Rn is d i f fe rent iab le wi th r e s p e c t to x for all u ~ R m and is m e a s u r a b l e wi th r e s p e c t to u r R m a long wi th its f i r s t de r iva t ives wi th r e s p e c t to x , such that

�9 M {y (x, u,, t)} = V (x) [~ (x) + g (x, t), (4 .3)

- n

( I) Ily(x, u~,'Ollvraisup .llY(X'Ut' t)ll + .-o~y(x, ut, 0 ~ c ( l + l lx l l ~)

Xvrai sup [ (x - - x * ) r y (X, U t, t) [ ~ b (t) (1 + II x ID (4.4)

fo r al l x ERn, w h e r e T(x) is a cont inuous funct ion o f x ~ R n such that O < T(x) -< F < ~o fo r a l l x ERn, while the v e c t o r funct ion g(x, t) s a t i s f i e s condi t ion 6 of T h e o r e m 2.1.

4. Funct ions a(t) > 0, r >_ 0, b(t) _> 0, d(t) _> 0 a r e bounded for t >_ 0, a r e in tegrab le in any finite t ime in te rva l , and sa t i s fy the condi t ions a(t)b(t) - - 0 as t - - ~o,

e? q) dt < oo, a (t) dt = oo, a if) d (t) dt < oo, 0 0 0

, (4 .5 )

Then, r a n d o m p r o c e s s x t is c o n v e r g e n t to the s e t X* wi th probabi l i ty 1, i .e . , v t = rnin[I xt-- x* 11 ~ 0 wi th p robab i l i ty 1 as t ~ ~. ~*~x*

Proof . Using the s a m e nota t ion as in the p roof of T h e o r e m 2.1, we obta in f r o m Eq. (4) the r e l a t i on

v't = - - 2a (t) (x: - - xi)ry (xt, ut, t). (4.6)

F r o m T h e o r e m 1.6 and r e l a t i o n (4.5) we have

I tl4 {(xt - - x~)ry (xt, ut, 0 - - (xt - - x~) r (V (xt) h(x,)

t

+ g (xt, t))/~ (0, t~)} I ~< c ~ (p (t - - 1:) a (~) (1 + M {v~/~ (0, t~)}) dr + r (t - - t,) b (t) (l + vt,). i t

Subst i tu t ion of (4.7) in (4.6) gives

M Ivd~ (0, t3} ~ - - 2M {(xt - - x~)r'r (xt) T~ (xO/~i(O, tO} + 2a (0 d (t) (1 + M {vd~l (0, q)})

t

+ e (t - - t,) a (t) b (t) (1 + v 0 + 2c,, (0 ~ �9 (t - - ~) a (T) (1 + M 1v~/9i (O,t3}) d~. tt

(4.7)

(4.8)

Since 7 (x) (x--Zx . ) r /x (x) > 0 for a l l x e R n, the inequal i ty (4.8) is m e r e l y s t r eng thened if we neglect its l a s t t e r m s . Consequent ly , by L e m m a 4.2, p r o c e s s v t c o n v e r g e s wi th probabi l i ty 1 to a finite l imi t as t - - .o.

On a v e r a g i n g throughout inequal i ty (4.8), as t - - ~ we get

a (t) M {'}, (xt) (x t - - x;)Tf (X,)} dt ~ M {v,} + 2c ~ a (t) d/~ (p(t - - "~) a ('~) (1 - - M { v~}) dr 0 0 0

-F 2 ~ a (t) b (t) (p (t) dt + 2 ~ a (t) d (t) (1 -}-M{v,})dt. (4.9) 0 0

t ~r The r igh t s ide of (4.9) is upper -bounded . Hence the re ex is t s an infinite sequence of ins tants { k}k=0 such * Y that t k - - ~o as k - .% and 7 (xek)- ( x , k - xt) ix (xt 0 --+ 0 wi th probabi l i ty 1 as k - .o. But the funct ion T(X){x --x*) T x

fxC() can only tend to z e r o i f x - - ~ o r i f x - - X * . But v t is bounded with probabi l i ty 1, and hence the re only r e - mains the c a s e v~----]1 xt~--x;k I]--+0 as t k - - ~ wi th probabi l i ty 1. Since v t has a finite l imi t wi th probabi l i ty 1,

this l imi t mus t be ze ro . QED.

THEOREM 4.2. Le t the fol lowing condi t ions hold:

915

1 .

downwards convex and di f ferent iable in R n.

2. f(x) is downwards in the s e t X and r ea ches its min imum in X in the se t X*.

3. The vec tor func t iony(x , u, t) in (4) is obtained as follows:

1 (r (x)) y (x, u, t) = y, (x, u, t) -[- I1 r (x) II ~ % (x~ 1 * [ (x) yt (x, u, t ) I - ,

w h e r e

The se t X in p r o b l e m (1) is bounded and defined in the f o r m X ---- {x : x E R~, * (x) ~ 0}, where r {x) is

(4.10)

{~ for X~0 , ! - -X for X~0, 1 (X) = ]~,i- =

for ~ 0 , 0 for X ~ O ,

while , for al l x E X, the vec to r function yl (x, u, t) ~ Rn sa t i s f i e s condition 3 of T h e o r e m 4.1, and V(x) --- 1.

4. Conditions 2.4 of T h e o r e m 4.1. Then, if x 0 ~ X, the r andom process xt is convergent to the se t X* with probabi l i ty 1, i ,e . , vt ----- min [I x, - - x*]l~-~0 with probabi l i ty 1 as t ~ ,o.

x*EX*

Proof . The r andom p roces s xt , defined by (4) and (4.10), is bounded, s ince x 0 ~X, and if ~J(xt) = 0, then

(x,) = - - a (t) ( r y~ (xt, u,, t) + l r (x0 y, (xt, ut, t ) I - ) ~< 0.

We have the following re la t ion , analogous to (4.6):

�9 - 1 (r x v t = - - 2a (t) (x, - - x~) r (Yi (x. u,, t) ar ]]~P~--~t) H~ ~p (xt) t q~r (xt) y~ (x,. u t, t)]-), (4.11)

F r o m T h e o r e m 1.6 and re la t ion {4.10) we have

t

I M {(x t - - x*) r [Yx (xt, at, t) - - [x (xt) - - g (xt, t)]lg[ (0, ti)} ] ~ ci ~ (p (t - - ~) a (~) d~ W (~' ( t . tx) b (t) c 2. tl

(4.12)

Substituting (4.12) in (4.11) and noting that ( x t - x ; ) r , ( x t ) ~ 0 for al l t _> 0, we obtain the in tegrodi f feren- tial inequality

t

M {vt/~ (0, tx)} ~ - - 2a (t) M {(x, - - x;)r/. (x0} -}- 2a (t) Cl S q~ (t - - ~) a (~) d~ + 2a (t) q~ (t - - tl) b (t) c2 -}- a (t) d (t) c3. tl

The r e s t of the proof is s i m i l a r to the proof of T h e o r e m 4.1.

C ONC L U S I O N S

The t r ea tmen t of the p resen t a r t i c l e is not exhaust ive as r ega rd s s ta t i s t i ca l opt imizat ion a lgor i thms in the p resence of c o r r e l a t e d d is turbances . Other aspec t s of this topic a re cons idered in [14-17, 18, 19].

L I T E R A T U R E C I T E D

1. R . Z . Khas~minskii , Stability of Sys tems of Different ial Equations under Random Dis turbances of The i r P a r a m e t e r s [in Russian] , F izmatg iz , Moscow (1969).

2. M. Dr iml and T. Nedota, "Stochast ic approximat ions for continuous r andom p r o c e s s e s , " Transac t ions of the Second Prague Conference on Informat ion Theory , Prague (1960).

3. D . T . Sakr ison, "A continuous K ie f e r -Wol fov i t z procedure for r andom p r o c e s s e s , " Ann. Math. Stat., 3._.55, No. 2 (1964).

4. V. Dupac, "On the K ie f e r -Wol fov i t z approx imat ion method," Casopic Pest . Math., 82 (1957). 5. M. T. Wasan, Stochast ic Approximat ion, Cambridge Univ. (1969). 6. T . P . Krasul ina , "On s tochas t ic approx imat ion of r andom p r o c e s s e s with continuous t ime," Teor .

Veroyatn. Ee P r i m en . , 16, No. 4 (1971). 7. I . A . Ib r ag imov et al . , Independent and Stat ionari ly Connected Variables [in Russ ian] , Nauka, Moscow

(1965). 8. L . V . Kantorovieh and G. P. Akilov, Functional Analys is in Normed Spaces [in Russian] , F izmatgiz ,

Moscow (1959). 9. Yu. A. Rozanov, Stat ionary Random P r o c e s s e s [in Russian] , Nauka, Moscow (1963).

10. A . I . Ib rag imov and Yu. A. Rozanov, Gauss ian Random P r o c e s s e s [in Russian] , F izmatg iz , Moscow (1970).

916

11. V V. Stepanov, Course of Different ia l Equations [in Russian] , F izmatg iz , Moscow (1959). 12. J . L . Doob, Stochast ic P r o c e s s e s , Wiley (1953). 13. Yu. M . ' E r m o l ' e v , nOn method of genera l i zed s tochas t ic gradients and s tochas t ic q u a s i - F e j e r sequences,"

No. 2 (1969). 14. O. Yu. Kul 'chi tsk i i , "Algor i thms of s t a t i s t i ca l opt imizat ion with continuous t ime and methods of studying

the i r convergence by in tegrodif ferent ia l inequal i t ies ," Matemat ika , No. 3 (1976). 15. O. Yu. Kul 'chi tsk i i , "Non-Markov a lgor i thms of s ta t i s t i ca l opt imizat ion with continuous t ime. I, II ,"

Avtom. Te lemekh . , Nos. 5, 6 (1978). 16. O. Yu. Kul 'chi tsk i i , "On suff icient conditions for convergence of r andom s e a r c h a lgor i thms in con-

tinuous t ime , " in: P r o b l e m s of Random Search [in Russ ian] , No. 5, Z ina tne ,R iga (1976). 17. O. Yu. Kul 'chi tsk i i , "On convergence of an a lgo r i thm of s ta t i s t i ca l opt imizat ion with continuous t ime,"

in: Cyberne t ics Topics . Adaptive Sys tems [in Russ ian] , Nauka, Moscow (1977). 18. T . P . Krasu l ina , "Some r e m a r k s on s tochas t ic approx imat ion p r o c e s s e s , " Avtom. Telemekh. , No. 7

(1975). 19. A . N . Borodin, "Stochast ic approx imat ion procedure in the case of observa t ions sa t is fying a condition

of weak dependence," Teor . Veroyatn. Ee P r i m e n . , No. 1 (1979).

L I M I T T H E O R E M S F O R R A N D O M P R O C E S S E S , A N D

S T O C H A S T I C M A R K O V R E C U R R E N T P R O C E D U R E S

Y u . M. K a n i o v s k i i UDC 519.21

We shal l be concerned below with the asympto t i c p roper t i e s of r e c u r r e n t a lgor i thms such as were con- s ide red in [1-4], studying them f r o m the standpoint of l imi t t heo rems for r andom p r o c e s s e s ; our methods a r e based on [5].

A s s u m e that, in probabi l i ty space (~2, F, P), we have a sequence of independent r andom vec to rs {~(s, x)}, s -> 1, x EX, where X is a c losed convex se t in Eucl idean space R N. We a s s u m e that }(s, x), s -> 1, a r e Borel functions with r e s p e c t to x , and that, for s -> so, we have the re la t ions

M~ (s, x) = cJ~ (x) + ~,w (s, x),

(s, x) - - M~ (s, x) = ~ z (s, x),

where {Cs}, { a s } , {'~s} a r e posi t ive number sequences , and {fs(x)}, {w(s, x)} a r e sequences of vec tor func- tions (we cons ider column vec tors ) . Let each of the equations fs (x) = 0 have a unique solut ion x~ ~ X, and let there ex is t a point x* E In tX such that II x~-x*ll:~< ~ 0 , il z - x* II >~ ~, > 0 for zC X \ I n t X .

S-->m

For finding x*, we cons ider the sequence

(i)

where ,~X(" ) is the p ro jec to r onto X, /3 s is the s c a l a r s tep mul t ip l ie r , fls ~ 0 as s --~o, u(s, x) is the n o r m a l - izing fac tor , }s = }(s, xS), x ~ i s independent of ~(s, x), s -> 1. We a s s u m e with r e g a r d to the normal iz ing fac tor that

inf infu(s,x)----d(L):>0, LE(0, oo). s~so ilXl| ~;L

uKs, x)(tl f~(x)ll + ll~(s, x)tl) -<_ k (1 § !1 x 11), (2)

u (s, x) = M I[ z (s, x)ll 2 ~ k (t q- t l x ]t2q - a~ [I x - - x:112), (3)

w h e r e 2 2 ~ s ~ s % = O(Cs).

Trans la t ed f r o m Kibernet ika , No. 6, pp. 127-130, N o v e m b e r - D e c e m b e r , 1979. Original a r t i c le sub- mit ted June 24, 1977.

0011-4235/79/1506-0917507.50 �9 1980 Plenum Publishing Corpora t ion 917