Upload
others
View
6
Download
0
Embed Size (px)
Citation preview
1
EP2200 Queuing theory and teletraffic systems 3rd lecture
Markov chains Birth-death process - Poisson process Discrete time Markov chains
Viktoria Fodor
KTH EES
2 EP2200 Queuing theory and teletraffic systems
Outline for today
โข Markov processes
โ Continuous-time Markov-chains
โ Graph and matrix representation
โข Transient and steady state solutions
โข Balance equations โ local and global
โข Pure Birth process โ Poisson process as special case
โข Birth-death process as special case
โข Outlook: Discrete time Markov-chains
3 EP2200 Queuing theory and teletraffic systems
Markov processes
โข Stochastic process
โ pi(t)=P(X(t)=i)
โข The process is a Markov process if the future of the process depends on the
current state only - Markov property
โ P(X(tn+1)=j | X(tn)=i, X(tn-1)=l, โฆ, X(t0)=m) = P(X(tn+1)=j | X(tn)=i)
โ Homogeneous Markov process: the probability of state change is unchanged
by time shift, depends only on the time interval
P(X(tn+1)=j | X(tn)=i) = pij(tn+1-tn)
โข Markov chain: if the state space is discrete
โ A homogeneous Markov chain can be represented by a graph:
โข States: nodes
โข State changes: edges 1 0 M
4 EP2200 Queuing theory and teletraffic systems
Continuous-time Markov chains (homogeneous case)
โข Continuous time, discrete space stochastic process, with Markov property, that is:
โข State transition can happen in any point of time
โข Example:
โ number of packets waiting at the output buffer of a router
โ number of customers waiting in a bank
โข The time spent in a state has to be exponential to ensure Markov property:
โ the probability of moving from state i to state j sometime between tn and tn+1 does not depend on the time the process already spent in state i before tn.
1101
011
),)(|)((
))(,)(,)(|)((
nnnn
nnn
ttttitXjtXP
mtXltXitXjtXP
t0 t1 t2 t3 t4 t5
stat
es
5 EP2200 Queuing theory and teletraffic systems
Continuous-time Markov chains (homogeneous case)
โข State change probability depends on the time interval: P(X(tn+1)=j | X(tn)=i) = pij(tn+1-tn)
โข Characterize the Markov chain with the state transition rates instead:
ijij
qii
q
ji,ฮt
i)j|X(t)ฮt)P(X(tlim
0ฮtij
q
Transition rate matrix Q:
MMMMM
MM
M
qqq
q
qqq
)1(0
)1(
00100
Q
- rate (intensity) of state change
- defined to easy calculation later on
0 1
q01=4
q10=6
66
44Q
6 EP2200 Queuing theory and teletraffic systems
Outline for today
โข Markov processes
โ Continuous-time Markov-chains
โ Graph and matrix representation
โข Transient and steady state solutions
โข Balance equations โ local and global
โข Pure Birth process โ Poisson process as special case
โข Birth-death process as special case
โข Outlook: Discrete time Markov-chains
7
t
j
jiji
j
jijii
ij
iiij
j
jij
ij
jijiiiii
tij
jij
ij
ijiii
ijt
ij
eptptpdt
tpd
qtpdt
tdp
t
tqtp
t
tpttp
qqttqtpttqtptqtptpttp
t
tttqtptqtptpttp
ttqitXjttXPt
itXjttXPq
)0()(,)()(
)()()(
)()()(
)()()()()()()()(
0)(
lim),()()()()(
)())(|)(())(|)((
lim
0
0
EP2200 Queuing theory and teletraffic systems
Transient solution
โข The transient - time dependent โ state probability distribution
โข p(t)={p0(t), p1(t), p2(t),...} โ probability of being in state i at time t, given p(0).
Transient solution
leaves the state arrives to the state
8 EP2200 Queuing theory and teletraffic systems
Example โ transient solution
0 1
q01=4
q10=6
0.2 0.4 0.6 0.8 1
0.2
0.4
0.6
0.8
1
0.2 0.4 0.6 0.8 1
0.2
0.4
0.6
0.8
1
A: p0(0)=0, p1(0)=1 B: p0(0)=1, p1(0)=0
p0(t)
p1(t)
p0(t)
p1(t)
Transient
state
Stationary / steady
state
66
44Q teptp Q )0()(
dt
tpdtp
)()( Q
9 EP2200 Queuing theory and teletraffic systems
โข Matrix exponential
โข Matrix exponentials are defined as:
๐๐ = 1
๐!๐๐
โ
๐=0
And are difficult to calculate.
โข Therefore, for the small case on the 2-state MC I solved the original set of
differential equations instead:
Transient solution - comment
1)()(
)()()('
)()()('
)()(
10
1110101
1010000
tptp
qtpqtptp
qtpqtptp
tpdt
tpdQ
โข To solve linear differential equations is
still tough. See relevant course books.
โข Fortunately, math programs can do it for
you (I used Mathematica).
10 EP2200 Queuing theory and teletraffic systems
โข Def: stationary state probability distribution (stationary solution)
โ exists
โ p is independent from p(0)
โข The stationary solution p has to satisfy:
Note: the rank of QMM is M-1!
)(lim tppt
Stationary solution (steady state)
1)(,0)(
)( tpdt
tpdtp iQ
4.0,6.0
1,0,066
44,
10
1010
pp
pppp0 1
q01=4
q10=6
MMMMM
MM
M
qqq
q
qqq
)1(0
)1(
00100
Q
11 EP2200 Queuing theory and teletraffic systems
Important theorems โ without the proof
โข Stationary solution exists, if
โ The Markov chain is irreducible (there is a path between any two states) and
โ has positive solution
โข Equivalently, stationary solution exists, if
โ The Markov chain is irreducible
โ For all states: the mean time to return to the state is finite
โข Finite state, irreducible Markov chains always have stationary solution.
โข Markov chains with stationary solution are also ergodic:
โ pi gives the portion of time a single realization spends in state i, and
โ the probability that one out of many realizations are in state i at arbitrary
point of time
1,0 1Q pp
Stationary solution (steady state)
12 EP2200 Queuing theory and teletraffic systems
Outline for today
โข Markov processes
โ Continuous-time Markov-chains
โ Graph and matrix representation
โข Transient and steady state solutions
โข Balance equations โ local and global
โข Pure Birth process โ Poisson process as special case
โข Birth-death process as special case
โข Outlook: Discrete time Markov-chains
13 EP2200 Queuing theory and teletraffic systems
Balance equations
โข How can we find the stationary solution? pQ=0
โข Global balance conditions
โ In equilibrium (for the stationary solution)
โ the transition rate out of a state โ or a group of states - must equal the transition rate into the state (or states)
โข flow in = flow out
โ defines a global balance equation
221332112
332221112
11412221
22111412
0
:2 State
)(
)(0
:1 State
0
pqpqpq
pqpqpq
pqqpq
pqpqq
p
Q
flow in flow out
P1 P2 P3
P4
q12
q43 q14
q32 q21
4343
3232
2121
14121412
00
00
00
0)(
qqqq
Q
14 EP2200 Queuing theory and teletraffic systems
Group work
โข Global balance equation for state 1 and 2:
P1 p2 P3
P4
q12
q43 q14
q32 q21
โข Is there a global balance equation for the circle around states 1 and 2?
221332112
332221112
11412221
22111412
0
:2 State
)(
)(0
:1 State
0
pqpqpq
pqpqpq
pqqpq
pqpqq
p
Q
15 EP2200 Queuing theory and teletraffic systems
Balance equations
โข Local balance conditions in equilibrium
โ the local balance means that the total flow from one part of the chain
must be equal to the flow back from the other part
โ for all possible cuts
โ defines a local balance equation
โ The local balance equation is the same as a global balance equation
around a set of states!
P1 p2 P3
P4
q12
q43 q14
q32 q21
16 EP2200 Queuing theory and teletraffic systems
Balance equations
โข Set of linear equations instead of a matrix equation
โข Global balance :
โ flow in = flow out around a state
โ or around many states
โข Local balance equation:
โ flow in = flow out across a cut
โข M states
โ M-1 independent equations
โ pi=1
P1 p2 P3
P4
q12
q43 q14
q32 q21
221332112
3322211120
pqpqpq
pqpqpq
Qp
0
flow in flow out
P1 p2 P3
P4
q12
q43 q14
q32 q21
332443 pqpq
17 EP2200 Queuing theory and teletraffic systems
Outline for today
โข Markov processes
โ Continuous-time Markov-chains
โ Graph and matrix representation
โข Transient and steady state solutions
โข Balance equations โ local and global
โข Pure Birth process โ Poisson process as special case
โข Birth-death process as special case
โข Outlook: Discrete time Markov-chains
18 EP2200 Queuing theory and teletraffic systems
Pure birth process
โข Continuous time Markov-chain, infinite state space โข Transitions occur only between neighboring states
โ State independent birth intensity: ii ,
โข No stationary solution
โข Transient solution:
โ pk(t)=P(system in state k at time t)
โ number of events (births) in an interval t
k-1 k k+1
k-1= k=
000
00
00
0
Q
19 EP2200 Queuing theory and teletraffic systems
Pure birth process โข Transient solution โ number of events (births) in an interval (0,t]
โข Pure birth process gives Poisson process! โ time between state transitions is Exp(ฮป)
k-1 k k+1
k-1= k=
tk
kkkk
tt
t
k
ek
ttptptptp
tetptpetptptptp
etptptp
kforpptptp
!
)()()()()('
)()()(')()()('
)()()('
00)0(,1)0(,)()('
1
111101
000
0
Q
000
00
00
0
Q
20 EP2200 Queuing theory and teletraffic systems
1. Pure birth process with intensity
2. The number of events in period (0,t] has Poisson distribution with parameter
3. The time between events is exponentially distributed with parameter
tetXP 1)(
Equivalent definitions of Poisson process
pure birth process
number of events
Poisson distribution
time between events
exponential
previous slide
previous lecture
check in the binder
21 EP2200 Queuing theory and teletraffic systems
Pure death process
โข Continuous time Markov-chain, infinite state space โข Transitions occur only between neighboring states
โ State independent death intensity: 0, ii
โข No stationary solution
โข Pure death process gives Poisson process until reaching state 0
โข Time between state transitions is Exp(ยต)
k-1 k k+1
22 EP2200 Queuing theory and teletraffic systems
Outline for today
โข Markov processes
โ Continuous-time Markov-chains
โ Graph and matrix representation
โข Transient and steady state solutions
โข Balance equations โ local and global
โข Pure Birth process โ Poisson process as special case
โข Birth-death process as special case
โข Outlook: Discrete time Markov-chains
23 EP2200 Queuing theory and teletraffic systems
Birth-death process โข Continuous time Markov-chain โข Transitions occur only between neighboring states
ii+1 birth with intensity ฮปi
modells population ii-1 death with intensity ฮผi (for i>0)
2222
1111
00
22120
12110
02010
)(0
0)(
0
j
j
j
qqq
qqq
qqq
Q
k-1 k k+1
k+1
k
k
k-1
โข State holding time โ length of time spent in a state k โ Until transition to states k-1 or k+1 โ Minimum of the times to the first birth or first deaths minimum of two
Exponentially distributed random variables: Exp(k+k)
24 EP2200 Queuing theory and teletraffic systems
B-D process - stationary solution
k-1 k
k+1
k
k
k-1
Cut 1 Cut 2
k+1
โข Local balance equations, like for general Markov-chains
โข Stability: positive solution for p (since the MC is irreducible)
,
1
1
1
,
:
:
1 1
1
0
0
0
1
1
00
1
10
1
1
1
1
111
11
11
k i
ik
i
k
i
ik
ik
kk
k
kk
kkk
k
kkkkkk
k
k
kkkkkk
p
p
ppp
ppppp
pppp
2 Cut
1 Cut
Group work: stationary solution for state independent transition rates:
., ii
25 EP2200 Queuing theory and teletraffic systems
Markov-chains and queuing systems โข Why do we like Poisson and B-D processes?
How are they related to queuing systems? โ If arrivals in a queuing system can be modeled as Poisson
process also as a pure birth process
โ If services in a queuing systems can be modeled with exponential service times also as a (pure) death process
โ Then the queuing system can be modeled as a birth-death process
ฮป
ฮผ ฮผ
ฮป
26 EP2200 Queuing theory and teletraffic systems
Summary โ Continuous time Markov-chains
โข Markovian property: next state depends on the present state only
โข State lifetime: exponential โข State transition intensity matrix Q โข Stationary solution: pQ=0, or balance equations โข Poisson process
โ pure birth process () โ number of events has Poisson distribution, E[X]=t โ interarrival times are exponential E()=1/
โข Birth-death process: transition between neighboring states
โข B-D process may model queuing systems!
27 2G1318 Queuing theory and teletraffic systems
Discrete-time Markov-chains (detour)
โข Discrete-time Markov-chain: the time is discrete as well
โ X(0), X(1), โฆ X(n), โฆ
โ Single step state transition probability for homogeneous MC: P(X(n+1)=j | X(n)=i) = pij, n
โข Example
โ Packet size from packet to packet
โ Number of correctly received bits in a packet
โ Queue length at packet departure instants โฆ (get back to it at non-Markovian queues)
1 0 M pm1
p01
pm0
p1m
p0m
p10
28 2G1318 Queuing theory and teletraffic systems
โข Transition probability matrix:
โ The transitions probabilities can be represented in a matrix
โ Row i contains the probabilities to go from i to state j=0, 1, โฆM
โข Pii is the probability of staying in the same state
ip
pp
pp
ppp
j
ij
MMM
M
1,
0
1110
00100
P
Discrete-Time Markov-chains
1 0 M pm1
p01
pm0
p1m
p0m
p10
29 2G1318 Queuing theory and teletraffic systems
โข The probability of finding the process in state j at time n is denoted by:
โ pj(n) = P(X(n) = j)
โ for all states and time points, we have:
โข The time-dependent (transient) solution is given by:
)()(
1
)(
0
)( n
M
nnn pppp
1)0()1()()1(
)()1(
nnnn
ji
ij
n
jiii
n
i
pppp
ppppp
PPPP
Discrete-Time Markov-chains
1 0 M pm1
p01
pm0
p1m
p0m
p10
30
โข Steady (or stationary) state exists if
โ The limiting probability vector exists
โ And is independent from the initial probability vector
โข Stationary state probability distribution is give by:
โข Note also:
โ The probability to remain in a state j for m time units has geometric distribution
โ The geometric distribution is a memoryless discrete probability distribution (the only one)
2G1318 Queuing theory and teletraffic systems
M
n
nppppp 10
)(lim
1,0
M
j
jppp P
jj
m
jj pp 11
P)()1( nn pp
Discrete-Time Markov-chains
31 EP2200 Queuing theory and teletraffic systems
Summary
โข Continuous-time Markov chains
โข Balance equations (global, local)
โข Pure birth process and Poisson process
โข Birth-death process
โข Discrete time Markov chains