Markov theory

], ,oi

Mp",RKOVTHEORY

744 ~ ~Q~ a4dtUlt~"1'7,,~ U

.. 14tUe ., ~ '* 'N41foi '/411'8ti/".

DEFINITION 3.1:

A stochastic process, {x( t ), t E T}, is a collectionof randomvariables. That is, for each t E T:. X(t) is a random variable. Theindex t is often referred to as time and asa result, we refer to X( t)as the state of the process at.time ~..The set T is called the indexset of the process.

DEFINITION 3.2:

When T is a countable set, the stochastic process is said to be adiscrete-time process. [f T is an interval of the real line, thestochasticprocess is said to be continuous time- process.

DEFINITION: 3.3: .

The state space of a stochasticprocess is defined as the set ofall possible values that the random variables X(t) can assulne.

THUS,A STOCHASTICPROCESSISA fAMILY Of RANDOMVARIABLESTHATDESCRIBESTHE EVOLUTIONTHROUGH

, TIMEOF SOME (PHYSICAL) PROCESS.

1111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111III111III11

MARKOV THEORY EDGAR L. DE CASTRO PAGE 1

.... ..

DISCRETE-TIME PROCESSES

DEFINITION 3.4:

An epoch is a point in time at which the system is observed. Thestates correspond ,to the possible conditions observed. Atransition is a change of state. A record of the observed statesthrough time is caned a realization of the process.

DEFINITION 3.5:

A transition diagram is a pictorial map in which the states arerepresented by points and transition by arrows.

oTRANSITION DIAGRAM FOR THREE STATES

1111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111IIIIII11

MARKOVTHEORY EDGARL. DECASTRO PAGE2

" ::,.:.' .... .. ,'.',., '. '

,", <

DEFINITION 3.6:

The process of transition can be visualized as a random walk ofthe particle over the transition diagram. A virtual transition isone where the new state is the same as the old. A real transitionis a genuine ?hange of state.

THE RANDOM WALK MODEL

Consider a discrete time process whose state space is given by theintegers i = O,:f: 1,:f: 2, The discrete time process is said to

be a random walk, if for some number 0 < P < 1,

lj,i+l =P =1..1li,i-I i = 0,:1:1,:J:2,. ..

The random walk may be thought of as being a model for anindividual walking on a straight line who at each point of timeeither takes one step to the right with probabilityp and one step tothe left with probability 1 - p.

I1I11111I11III1111111111111111111111111111111111111111I11I1III11111111111111111111111111111111111111111111111II1111I1111I111111111111111111111111111111111111111111I11I1I1I1IIII1111


" " , . ,, ,.. "

. t:,'. ." .,

THE MARKOV CHAIN

DEFINITION 3.7:

A markov chain is a discrete time stochastic process in whichthe current state of each random variable Xi depends only on theprevious state. The word chain suggests the linking of the randomvariables to their immediately adjacent neighbors in the sequence.Markov is the Russian mathematician who developed the processaround the beginning of the 20th century.

TRANSITION PROBABILITY (Pij) - the probability of atransition from state i to state j after one period.

.

TRANSITION MATRIX (P) - the matrix of transitionprobabilities.

..t:",',.."

1111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111 II 111111111111111111111111111111111111111111111111111111111111111111111111111111111111111


.. ...........

.'. .', .',' .. ..".

PII PI2 ... PIn

P =I P21P22 ... P2n. . . .. . . . .. . . .

. PDI Pn2... Pnn

ASSUMPTIONS OF THE MARKOV CHAIN

t. THEMARKOV ASSUMPTION,The knowledge of the state at any time is sufficient to predictthe future of the process. Or, given the present, the future isindependent of the parts and the process is "forgetful."

2. THE STATIONARITYASSUMPTION

The probability mechanism is assumed as stable.

CHAPMAN-KOLMOGOROV EQUATIONS

Let PDI1)= the n step transition probability, i.e., the probabilitythat a process in state i will be in state j after nadditional transitions.

pJn) = P{Xn+m = jlXnl = i}, n > 0, i,j > 0

The Chapman-Kolmogorov equations provide a method forcalculating these n-step transition probabilities.

00

P(n+m)- ~' p.(n)n(~n)\in m> 0 all i,Jo

ij - L... Ik rkJ ' -,k=O '

Formally, we derive:

11111111111111111111111111111111111111111111111111111111IIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIII!IIIIIIIIIIIIIIIIIIIII1II1I11I11III1I1111111111111111111111111111111111111111111111I1111111

MARKOV THEORY EDGAR r..,DE CASTRO PAGE 5

" ' . '. ., " " ' .

. .:. -.:', "

, ,

00.

. LP{Xn+m = j,Xn = kixo = i}k=O

00

= LP{Xn+m = jlXn = k,Xo = i}P{Xn = klXo = i}k=O

00- " n(m) p (n)- .i- rkj ik

k=O .

If we let p~n) denote the matrix of n-step transition probabiIitiesp,(n) then

1] ,

p(n+m) =p(n) - p(trt)

where the dot represents matrix multiplication.particular:

Hence, in

p(2) =p(l+l) = p- p = p2

And by induction:

p(o) = p(n-l+1) = pn-I - p =pO

That is, the n-step transition matrix is obtained by multiplyingmatrix P by itself n times. Therefore the N-step transition matrix isgivenby:

1II111I1I111III111111111111111111111111I1111111111111111111111111111111111111111111111111111111111111111111111I1I11111I11111II111111111111111111111111111111111111111IIIIIIIIIIilili

MARKOV THEORY EDGARL. DECASTRO PAGE 6

0.

FIRST PASSAGEPROBABILITIES

AND FIRST RETURN

Let f~N) = first passage probability= probability of reaching state j from state i for

the first time in N steps.

f~N)= first return probability if i = j

fi~N) =P{XN =j,XN-I :I:j,XN-2 :I:j,...,Xf :I:jlXo = i}

f.(I) = p..lJ IJ

N-lf,(N)= p.(N)- ~ f.(k)p(N-k)IJ IJ ~ IJ JJ

k=1

111111111111111111111111111111111111111111111111111111" 11111111111" 11111111111111111111111111111111111" 111111111111111111111111111111111111111""" 11/1111111" 11111111111I11111

fv1ARKOVTHEORY EDGAR L. DE CASTRO PAGE 7

.'. I"... .. ".:'.. . .

rp(N) p(N) ...pCl'J) l

11112 In

(N) p(N) p(N)p(N) =I P21

...22 2n. . . .. . . .. . . .

CLAS~IFICATION OF STATES

For fixed i andj, the fi~N) are nonnegative numbers such that

When the sum does equal 1, fi~N) can be considered as aprobability distribution for the random variable: first passage time

If i =j and

00

L f(N) - 1IJ -

N=1

then state i is caned a recurrent state because this conditionimplies that once the process is in state i, it will return to state i.

A special case of the recuuent state is the absorbingstate. A stateis said to be an absorb,lng state if the one step transitionprobability Pij = 1. Thus, if a state is absorbing, the process winnever leave once it enters. If

00

L f(N) < tN=] 1J

then state i is called a transient state because this condition impliesthat once the process is in state i, there is a strictly positiveprobability that it will never return to i.

I1111II1I11111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111I1111111111111111111111111111111111111111I1I1II1II1IIIII11


'", ..

.:

Let Mij =expected first passage time from i to j

0000

if L fi(N) < 1N=1 ~

00

if L fDN) =1N=1

[Mijexists only if the states are recurrent]

Whenever

00

~ f.(N)- 1£.oJ 1J -

N=l

Then

Moo= 1+ '" P'kMk'

1J £.. I Jk*j

When j = i, the expected first passage time is caUed the firstrecurrence time. If Mii = 00, it is called a null recurrentstate, If Mii< 00, it is called a positive recurrent state, In afinite Markov chain, there are no null recurrent states (onlypositive recurrent states and transient states).

111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111I111I11III11


.'." .

State j is accessible from i if Pij > 0 for some n > o. If j isaccessible to i and i is accessible to j, then the two statescommunicate. In general :

(1) any state communicates with itself.(2) if state i communicates with state j, then state j communicates

with state i.

(3) if state i communicates with state j and state j communicateswith state k, then state i communicates with state k.

If all states communicate, the Markov chain is Irreducible. In afmite Markov chain, the members of a class are either all transientstates or all positive recurrent states. A state i is said to have a

. period t (t > 1) if Pj]N) =0 whenever n is not divisible by t, and tis the largest integerwith this property.If a state has a period 1, itis called aperiodic state. If state i in a class is aperiodic, then.all states in the class are aperiodic. Positive recurrent states thatare aperiodicare calledergodic states.

11111111111111111111111111111111111111 n 11111111111" 1111111111" II" 111111" II" II iIIlllll"" !l1I1I111111" n II111I "" 11111111111111111" 11111" I" 1111111111111111" I111III1IIII


. . .

. ..:. '."," .",'I

ERGODIC MARKOV CHAINS

STEADY STATEPROBABILITIES)

PROBABILITIES (LIMITING

Let 7tj = lim p.(N)N~oo IJ

As N grows large:

7t1 7t2

pN ~ 17tI 7t2. .. .. .

... 7tn

... 7tn... ...

... 7tn

As long as the process is ergodic, such Iin1itexists.

p(I'l) =p(N-I) 8 P

Jim peN)= Jim p(N-I). pN~oo N~oo

...... 7t 'it-- 1t2 ... 7t

l11 1 n. ..... =. . . . . p

. ... : . ,:J... 1tnJ L7t1 7t2

1t=1t8P

1tT = pT 81t

[This system possesses an infinite number of solutions.]

1111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111I11I11111I1III11111111111I11111111111111111111111111111I1I1IIIII1


',' : '.. ".

. ::".,'. ." 'I

The nonnalizing equation

L 1ti =1al1 i

is used to identifY the one solution which wiUprobability distribution.

qualify as a

ABSORBING MARKOV CHAINS

IIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIII!11111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111II1I1111111II11111


Let

PH Pl2 ...'Plk I Pl,k+l

... ...

P21 P22 ... P2k I P2,k+l... ...

. . . .I

. . . .. . . .

Pkl Pk2 ...Pkk I Pk,k +1

... ...

- - - - I - - -0 0 ... 0

I1 ... 0

0 0 ... 0I

0 ... 0.

I

. . .. . . .. . . .0 0 . . . 0

I0 ... I

The partitioned matrix is given by:

Q I I( ..,

P=I- -

J0 I

Let eij = mean number of times that transient state j is occupiedgiven the initial state in i before absorption

E = corresponding matrix

Then,

k

i:l: j: eij"= L ~vevjv=l

k

i =j :eij = 1 + L Pjvevjv=l

In matrix form :

E = I + QE

E - QE = I

(I-Q)E=I

E=(I-Q)-l

Let di = total number of transitions until absorption

k

di = Leijj=l

111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111II1I11

MARKOVTHEORY EDGARL. DE CASTRO PAGE 13

.":.":

ABSORPTION PROBABILITY - probability of entering anabsorbing state

Let Aij = probability that the process even enters absorbing state jgiven that the initial state is i.

k

Aij = Pij + L PivAvjv=l

In matrix form

A = matrix of Aij (not necessarily square)

[where the number of rows is the number of transient states andthe number of columns is the number of absorbing states]

Examining matrix A

A=R+QA

A-QA=R(1- Q)A =R

A = [I - Q ]-1 R

CONDITIONAL MEAN FIRST PASSAGE TIME - number oftransitions which will occur before an absorbing state is entered

A..M..=A..+ ~ P.kA k.M k.

U IJ 1J £.- I'" J Jk=#j

111II1111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111!11111111111111111111IIII1I111111111


. '. . .. .',.

.,.'

Education

Markov theory