Intro. to Stochastic Processes

Cheng-Fu Chou, CMLab, CSIE, NTU

Intro. to Stochastic Processes

Cheng-Fu Chou

Cheng-Fu Chou, CMLAB, CSIE, NTU

P. 2

Outline

Stochastic Process Counting Process Poisson Process Markov Process

Cheng-Fu Chou, CMLAB, CSIE, NTU

P. 3

Stochastic Process A stochastic process N= {N(t), t T} is a

collection of r.v., i.e., for each t in the index set T, N(t) is a random variable– t: time– N(t): state at time t– If T is a countable set, N is a discrete-time

stochastic process– If T is continuous, N is a continuous-time stoc.

proc.

Cheng-Fu Chou, CMLAB, CSIE, NTU

P. 4

Counting Process A stochastic process {N(t) ,t 0} is said to be a

counting process if N(t) is the total number of events that occurred up to time t. Hence, some properties of a counting process is– N(t) 0– N(t) is integer valued– If s < t, N(t) N(s)– For s < t, N(t) – N(s) equals number of events

occurring in the interval (s, t]

Cheng-Fu Chou, CMLAB, CSIE, NTU

P. 5

Counting Process Independent increments

– If the number of events that occur in disjoint time intervals are independent

Stationary increments– If the dist. of number of events that occur in

any interval of time depends only on the length of time interval

Cheng-Fu Chou, CMLAB, CSIE, NTU

P. 6

Poisson Process Def. A: the counting process {N(t), t0} is said to

be Poisson process having rate,>0 if– N(0) = 0;– The process has independent-increments– Number of events in any interval of length t is

Poisson dist. with mean t, that is for all s, t 0.

( )[ ( ) ( ) ]!

= 0,1, 2,...

nt tP N t s N s n e

nn

Cheng-Fu Chou, CMLAB, CSIE, NTU

P. 7

Poisson Process Def. B: The counting process {N(t), t 0} is said

to be a Poisson process with rate , >0, if – N(0) = 0– The process has stationary and independent

increments– P[N(h) = 1] = h +o(h)– P[N(h) 2] = o(h)– The func. f is said to be o(h) if – Def A Def B, i.e,. they are equivalent.– We show Def B Def A – Def A Def B is HW

0( )lim 0h

f hh

Cheng-Fu Chou, CMLAB, CSIE, NTU

P. 8

Important Properties Property 1: mean number of event for any t 0,

E[N(t)]=t.

Property 2: the inter-arrival time dist. of a Poisson process with rate is an exponential dist. with parameter

Property 3: the superposition of two independent Poisson process with rate 1 and 2 is a Poisson process with rate 1+2

Cheng-Fu Chou, CMLAB, CSIE, NTU

P. 9

Properties (cont.) Property 4: if we perform Bernoulli trials to make

independent random erasures from a Poisson process, the remaining arrivals also form a Poisson process

Property 5: the time until rth arrival , i.e., r is known as the rth order waiting time, is the sum of r independent experimental values of and is described by Erlan pdf.

Cheng-Fu Chou, CMLAB, CSIE, NTU

P. 10

Ex 1 Suppose that X1 and X2 are independent

exponential random variables with respective means 1/1 and 1/2;What is P{X1 < X2}

Cheng-Fu Chou, CMLAB, CSIE, NTU

P. 11

21

1

0

)(1

10

10 2

0 112121

}{

}|{}{

21

12

1

1

dxe

dxee

dxeXxP

dxexXXXPXXP

x

xx

x

x

Cheng-Fu Chou, CMLAB, CSIE, NTU

P. 12

Conditional Dist. Of the Arrival Time

Suppose we are told that exactly one event of a Poisson process has taken place by time t, what is the distribution of the time at which the event occurred?

Cheng-Fu Chou, CMLAB, CSIE, NTU

P. 13

t][0,over ddistributeuniformly isevent theof time theSo,

1} P{N(t)t)}[s,in events 0 s)}P{[0,in event 1{

1} P{N(t)t)}[s,in events 0 s),[0,in event 1{}1)({

}1)(,{}1)(|{

)(

ts

teese

P

PtNP

tNsxPtNsxP

t

sts

Cheng-Fu Chou, CMLAB, CSIE, NTU

P. 14

Ex 2 Consider the failure of a link in a communication

network. Failures occur according to a Poisson process with rate 4.8 per day. Find– P[time between failures 10 days]– P[5 failures in 20 days]– Expected time between 2 consecutive failures– P[0 failures in next day]– Suppose 12 hours have elapsed since last

failure, find the expected time to next failure

Cheng-Fu Chou, CMLAB, CSIE, NTU

P. 15

hours 5 .5 4.

hours 5 .35!

20)*(4.8 .2

1 .1

8.4

20*8.45

10*8.4

e

e

e

Cheng-Fu Chou, CMLAB, CSIE, NTU

P. 16

Markov Process P[X(tn+1) Xn+1| X(tn)= xn, X(tn-1) = xn-1,…X(t1)=x1] =

P[X(tn+1) Xn+1| X(tn)=xn]– Probabilistic future of the process depends only

on the current state, not on the history– We are mostly concerned with discrete-space

Markov process, commonly referred to as Markov chains

– Discrete-time Markov chains– Continuous-time Markov chains

Cheng-Fu Chou, CMLAB, CSIE, NTU

P. 17

DTMC Discrete Time Markov Chain:

– P[Xn+1 = j | Xn= kn, Xn-1 = kn-1,…X0= k0] = P[Xn+1 = j | Xn = kn]

discrete time, discrete space A finite-state DTMC if its state space is finite A homogeneous DTMC if P[Xn+1 = j | Xn= i ] does not

depend on n for all i, j, i.e., Pij = P[Xn+1 = j | Xn= i ], where Pij is one step transition prob.

Cheng-Fu Chou, CMLAB, CSIE, NTU

P. 18

Definition P = [ Pij] is the transition matrix

– A matrix that satisfies those conditions is called a stochastic matrix

– n-step transition prob.

00 01 0

10 11 1

0

... ...

... ...... ... ... ... ...

... ... ...... ... ... ... ...

where 0 and 1

j

j

i ij

ij ijj

p p pp p p

Pp p

p p

0[ | ]

, , 0, is the prob. of going from state

to in step

nij n

nij

p P x j x i

i j n p i

j n

Cheng-Fu Chou, CMLAB, CSIE, NTU

P. 19

Chapman-Kolmogorov Eq. Def.

Proof:

( )

For all n 0, m 0, i , j I

in matrix form where =[ ]

n m n mij ik kj

k I

n m n m n nij

p p p

P P P P p

Cheng-Fu Chou, CMLAB, CSIE, NTU

P. 20

Question We have only been dealing with conditional prob.

but what we want is to compute the unconditional prob. that the system is in state j at time n, i.e.

0

0 0 0

0 0

0

( ) ( )So, given the initial dist. of ,i.e.,

( ) ( ) and 1

we can get

[ ] ( | ) ( )

( )

n n

i I

n ni I

nij

i I

j p x jx

i p x i

p x j p x j x i i

p i

Cheng-Fu Chou, CMLAB, CSIE, NTU

P. 21

Result 1 For all n 1, n = 0Pn, where m = (m(0),m(1),

…) for all m 0. From the above equ., we deduce that n+1 = nP. Assume that limn n(i) exists for all i, and refer it as (i). The remaining question is how to compute

– Reachable: a state j is reachable from i. if

– Communicate: if j is reachable from i and if i is reachable form j, then we say that i and j communicate (i j)

0 for some 1nijp n

Cheng-Fu Chou, CMLAB, CSIE, NTU

P. 22

Result 1 (cont.) Irreducible:

– A M.C. is irreducible if i j for all i,j I

Aperiodic:– For every state iI, define d(i) to be largest common

divisor of all integer n, s.t.,

0 if ( ) 1 then the state is aperiodicnijp d i

Cheng-Fu Chou, CMLAB, CSIE, NTU

P. 23

Result 2 Invariant measure of a M.C., if a M.C. with

transition matrix P is irreducible and aperiodic and if the system of equation =P and 1=1 has a strict positive solution then (i) = limn n(i) independently of initial dist.– Invariant equ. : =P – Invariant measure

Cheng-Fu Chou, CMLAB, CSIE, NTU

P. 24

Gambler’s Ruin Problem Consider a gambler who at each play of game has

probability p of winning one unit and probability q=1-p of losing one unit. Assuming that successive plays of the game are independent, what is the probability that, starting with i units, the gambler’s fortune will reach N before reaching 0?

Cheng-Fu Chou, CMLAB, CSIE, NTU

P. 25

Ans If we let Xn denote the player’s fortune at time n,

then the process {Xn, n=0, 1,2,…} is a Markov chain with transition probabilities:– p00 =pNN =1– pi,i+1 = p = 1-pi,i-1

This Markov chain has 3 classes of states: {0},{1,2,…,N-1}, and {N}

Cheng-Fu Chou, CMLAB, CSIE, NTU

P. 26

Let Pi, i=0,1,2,…,N, denote the prob. That, starting with i, the gambler’s fortune will eventually reach N.

By conditioning on the outcome of the initial play of the game we obtain– Pi = pPi+1 + qPi-1, i=1,2, …, N-1Since p+q =1Pi+1 – Pi = q/p(Pi-Pi-1),Also, P0 =0, soP2 – P1 = q/p*(P1-P0) = q/p*P1

P3 - P2 =q/p*(P2-P1)= (q/p)2*P1

Cheng-Fu Chou, CMLAB, CSIE, NTU

P. 27

1/2 p if 0

1/2p if pq-1

N as that,Note

2/1p if 1

2/1p if )/(1)/(1

obtain we,1 using Now,

1qp if

1qp if

)/(1)/(1

i

1

1

1

i

N

N

i

i

P

N

pqpq

P

P

iP

Ppqpq

P

Cheng-Fu Chou, CMLAB, CSIE, NTU

P. 28

If p > ½, there is a positive prob. that the gambler’s fortune will increase indefinitely

Otherwise, the gambler will, with prob. 1, go broke against an infinitely rich adversary.

Cheng-Fu Chou, CMLAB, CSIE, NTU

P. 29

CTMC Continuous-time Markov Chain

– Continuous time, discrete state– P[X(t)= j | X(s)=i, X(sn-1)= in-1,…X(s0)= i0]

= P[X(t)= j | X(s)=i]– A continuous M.C. is homogeneous if

o P[X(t+u)= j | X(s+u)=i] = P[X(t)= j | X(s)=i] = Pij[t-s], where t > s

– Chapman-Kolmogorov equ. For all t > 0, s > 0, i , j I

( ) ( ) ( ) ij ik kjk I

p t s p t p s

Cheng-Fu Chou, CMLAB, CSIE, NTU

P. 30

CTMC (cont.)(t)=(0)eQt

– Q is called the infinitesimal generator– Proof:

Cheng-Fu Chou, CMLAB, CSIE, NTU

P. 31

Result 3 If a continuous M.C. with infinitesimal generator Q

is irreducible and if the system of equations Q = 0, and 1=1, has a strictly positive solution then (i)= limt p(x(t)=i) for all iI, independently of the initial dist.