Upload
tess
View
52
Download
1
Embed Size (px)
DESCRIPTION
Intro. to Stochastic Processes. Cheng-Fu Chou. Outline. Stochastic Process Counting Process Poisson Process Markov Process. Stochastic Process. A stochastic process N = {N(t), t T} is a collection of r.v., i.e., for each t in the index set T, N(t) is a random variable t: time - PowerPoint PPT Presentation
Citation preview
Cheng-Fu Chou, CMLab, CSIE, NTU
Intro. to Stochastic Processes
Cheng-Fu Chou
Cheng-Fu Chou, CMLAB, CSIE, NTU
P. 2
Outline
Stochastic Process Counting Process Poisson Process Markov Process
Cheng-Fu Chou, CMLAB, CSIE, NTU
P. 3
Stochastic Process A stochastic process N= {N(t), t T} is a
collection of r.v., i.e., for each t in the index set T, N(t) is a random variable– t: time– N(t): state at time t– If T is a countable set, N is a discrete-time
stochastic process– If T is continuous, N is a continuous-time stoc.
proc.
Cheng-Fu Chou, CMLAB, CSIE, NTU
P. 4
Counting Process A stochastic process {N(t) ,t 0} is said to be a
counting process if N(t) is the total number of events that occurred up to time t. Hence, some properties of a counting process is– N(t) 0– N(t) is integer valued– If s < t, N(t) N(s)– For s < t, N(t) – N(s) equals number of events
occurring in the interval (s, t]
Cheng-Fu Chou, CMLAB, CSIE, NTU
P. 5
Counting Process Independent increments
– If the number of events that occur in disjoint time intervals are independent
Stationary increments– If the dist. of number of events that occur in
any interval of time depends only on the length of time interval
Cheng-Fu Chou, CMLAB, CSIE, NTU
P. 6
Poisson Process Def. A: the counting process {N(t), t0} is said to
be Poisson process having rate,>0 if– N(0) = 0;– The process has independent-increments– Number of events in any interval of length t is
Poisson dist. with mean t, that is for all s, t 0.
( )[ ( ) ( ) ]!
= 0,1, 2,...
nt tP N t s N s n e
nn
Cheng-Fu Chou, CMLAB, CSIE, NTU
P. 7
Poisson Process Def. B: The counting process {N(t), t 0} is said
to be a Poisson process with rate , >0, if – N(0) = 0– The process has stationary and independent
increments– P[N(h) = 1] = h +o(h)– P[N(h) 2] = o(h)– The func. f is said to be o(h) if – Def A Def B, i.e,. they are equivalent.– We show Def B Def A – Def A Def B is HW
0( )lim 0h
f hh
Cheng-Fu Chou, CMLAB, CSIE, NTU
P. 8
Important Properties Property 1: mean number of event for any t 0,
E[N(t)]=t.
Property 2: the inter-arrival time dist. of a Poisson process with rate is an exponential dist. with parameter
Property 3: the superposition of two independent Poisson process with rate 1 and 2 is a Poisson process with rate 1+2
Cheng-Fu Chou, CMLAB, CSIE, NTU
P. 9
Properties (cont.) Property 4: if we perform Bernoulli trials to make
independent random erasures from a Poisson process, the remaining arrivals also form a Poisson process
Property 5: the time until rth arrival , i.e., r is known as the rth order waiting time, is the sum of r independent experimental values of and is described by Erlan pdf.
Cheng-Fu Chou, CMLAB, CSIE, NTU
P. 10
Ex 1 Suppose that X1 and X2 are independent
exponential random variables with respective means 1/1 and 1/2;What is P{X1 < X2}
Cheng-Fu Chou, CMLAB, CSIE, NTU
P. 11
21
1
0
)(1
10
10 2
0 112121
}{
}|{}{
21
12
1
1
dxe
dxee
dxeXxP
dxexXXXPXXP
x
xx
x
x
Cheng-Fu Chou, CMLAB, CSIE, NTU
P. 12
Conditional Dist. Of the Arrival Time
Suppose we are told that exactly one event of a Poisson process has taken place by time t, what is the distribution of the time at which the event occurred?
Cheng-Fu Chou, CMLAB, CSIE, NTU
P. 13
t][0,over ddistributeuniformly isevent theof time theSo,
1} P{N(t)t)}[s,in events 0 s)}P{[0,in event 1{
1} P{N(t)t)}[s,in events 0 s),[0,in event 1{}1)({
}1)(,{}1)(|{
)(
ts
teese
P
PtNP
tNsxPtNsxP
t
sts
Cheng-Fu Chou, CMLAB, CSIE, NTU
P. 14
Ex 2 Consider the failure of a link in a communication
network. Failures occur according to a Poisson process with rate 4.8 per day. Find– P[time between failures 10 days]– P[5 failures in 20 days]– Expected time between 2 consecutive failures– P[0 failures in next day]– Suppose 12 hours have elapsed since last
failure, find the expected time to next failure
Cheng-Fu Chou, CMLAB, CSIE, NTU
P. 15
hours 5 .5 4.
hours 5 .35!
20)*(4.8 .2
1 .1
8.4
20*8.45
10*8.4
e
e
e
Cheng-Fu Chou, CMLAB, CSIE, NTU
P. 16
Markov Process P[X(tn+1) Xn+1| X(tn)= xn, X(tn-1) = xn-1,…X(t1)=x1] =
P[X(tn+1) Xn+1| X(tn)=xn]– Probabilistic future of the process depends only
on the current state, not on the history– We are mostly concerned with discrete-space
Markov process, commonly referred to as Markov chains
– Discrete-time Markov chains– Continuous-time Markov chains
Cheng-Fu Chou, CMLAB, CSIE, NTU
P. 17
DTMC Discrete Time Markov Chain:
– P[Xn+1 = j | Xn= kn, Xn-1 = kn-1,…X0= k0] = P[Xn+1 = j | Xn = kn]
discrete time, discrete space A finite-state DTMC if its state space is finite A homogeneous DTMC if P[Xn+1 = j | Xn= i ] does not
depend on n for all i, j, i.e., Pij = P[Xn+1 = j | Xn= i ], where Pij is one step transition prob.
Cheng-Fu Chou, CMLAB, CSIE, NTU
P. 18
Definition P = [ Pij] is the transition matrix
– A matrix that satisfies those conditions is called a stochastic matrix
– n-step transition prob.
00 01 0
10 11 1
0
... ...
... ...... ... ... ... ...
... ... ...... ... ... ... ...
where 0 and 1
j
j
i ij
ij ijj
p p pp p p
Pp p
p p
0[ | ]
, , 0, is the prob. of going from state
to in step
nij n
nij
p P x j x i
i j n p i
j n
Cheng-Fu Chou, CMLAB, CSIE, NTU
P. 19
Chapman-Kolmogorov Eq. Def.
Proof:
( )
For all n 0, m 0, i , j I
in matrix form where =[ ]
n m n mij ik kj
k I
n m n m n nij
p p p
P P P P p
Cheng-Fu Chou, CMLAB, CSIE, NTU
P. 20
Question We have only been dealing with conditional prob.
but what we want is to compute the unconditional prob. that the system is in state j at time n, i.e.
0
0 0 0
0 0
0
( ) ( )So, given the initial dist. of ,i.e.,
( ) ( ) and 1
we can get
[ ] ( | ) ( )
( )
n n
i I
n ni I
nij
i I
j p x jx
i p x i
p x j p x j x i i
p i
Cheng-Fu Chou, CMLAB, CSIE, NTU
P. 21
Result 1 For all n 1, n = 0Pn, where m = (m(0),m(1),
…) for all m 0. From the above equ., we deduce that n+1 = nP. Assume that limn n(i) exists for all i, and refer it as (i). The remaining question is how to compute
– Reachable: a state j is reachable from i. if
– Communicate: if j is reachable from i and if i is reachable form j, then we say that i and j communicate (i j)
0 for some 1nijp n
Cheng-Fu Chou, CMLAB, CSIE, NTU
P. 22
Result 1 (cont.) Irreducible:
– A M.C. is irreducible if i j for all i,j I
Aperiodic:– For every state iI, define d(i) to be largest common
divisor of all integer n, s.t.,
0 if ( ) 1 then the state is aperiodicnijp d i
Cheng-Fu Chou, CMLAB, CSIE, NTU
P. 23
Result 2 Invariant measure of a M.C., if a M.C. with
transition matrix P is irreducible and aperiodic and if the system of equation =P and 1=1 has a strict positive solution then (i) = limn n(i) independently of initial dist.– Invariant equ. : =P – Invariant measure
Cheng-Fu Chou, CMLAB, CSIE, NTU
P. 24
Gambler’s Ruin Problem Consider a gambler who at each play of game has
probability p of winning one unit and probability q=1-p of losing one unit. Assuming that successive plays of the game are independent, what is the probability that, starting with i units, the gambler’s fortune will reach N before reaching 0?
Cheng-Fu Chou, CMLAB, CSIE, NTU
P. 25
Ans If we let Xn denote the player’s fortune at time n,
then the process {Xn, n=0, 1,2,…} is a Markov chain with transition probabilities:– p00 =pNN =1– pi,i+1 = p = 1-pi,i-1
This Markov chain has 3 classes of states: {0},{1,2,…,N-1}, and {N}
Cheng-Fu Chou, CMLAB, CSIE, NTU
P. 26
Let Pi, i=0,1,2,…,N, denote the prob. That, starting with i, the gambler’s fortune will eventually reach N.
By conditioning on the outcome of the initial play of the game we obtain– Pi = pPi+1 + qPi-1, i=1,2, …, N-1Since p+q =1Pi+1 – Pi = q/p(Pi-Pi-1),Also, P0 =0, soP2 – P1 = q/p*(P1-P0) = q/p*P1
P3 - P2 =q/p*(P2-P1)= (q/p)2*P1
Cheng-Fu Chou, CMLAB, CSIE, NTU
P. 27
1/2 p if 0
1/2p if pq-1
N as that,Note
2/1p if 1
2/1p if )/(1)/(1
obtain we,1 using Now,
1qp if
1qp if
)/(1)/(1
i
1
1
1
i
N
N
i
i
P
N
pqpq
P
P
iP
Ppqpq
P
Cheng-Fu Chou, CMLAB, CSIE, NTU
P. 28
If p > ½, there is a positive prob. that the gambler’s fortune will increase indefinitely
Otherwise, the gambler will, with prob. 1, go broke against an infinitely rich adversary.
Cheng-Fu Chou, CMLAB, CSIE, NTU
P. 29
CTMC Continuous-time Markov Chain
– Continuous time, discrete state– P[X(t)= j | X(s)=i, X(sn-1)= in-1,…X(s0)= i0]
= P[X(t)= j | X(s)=i]– A continuous M.C. is homogeneous if
o P[X(t+u)= j | X(s+u)=i] = P[X(t)= j | X(s)=i] = Pij[t-s], where t > s
– Chapman-Kolmogorov equ. For all t > 0, s > 0, i , j I
( ) ( ) ( ) ij ik kjk I
p t s p t p s
Cheng-Fu Chou, CMLAB, CSIE, NTU
P. 30
CTMC (cont.)(t)=(0)eQt
– Q is called the infinitesimal generator– Proof:
Cheng-Fu Chou, CMLAB, CSIE, NTU
P. 31
Result 3 If a continuous M.C. with infinitesimal generator Q
is irreducible and if the system of equations Q = 0, and 1=1, has a strictly positive solution then (i)= limt p(x(t)=i) for all iI, independently of the initial dist.