16
048866: Packet Switch Architectures Dr. Isaac Keslassy Electrical Engineering, Technion [email protected] http:// Review #3: Discrete-Time Markov Chains

048866: Packet Switch Architectures Dr. Isaac Keslassy Electrical Engineering, Technion [email protected] isaac/ Review

  • View
    231

  • Download
    1

Embed Size (px)

Citation preview

Page 1: 048866: Packet Switch Architectures Dr. Isaac Keslassy Electrical Engineering, Technion isaac@ee.technion.ac.il isaac/ Review

048866: Packet Switch Architectures

Dr. Isaac KeslassyElectrical Engineering, Technion

[email protected]

http://comnet.technion.ac.il/~isaac/

Review #3: Discrete-Time Markov Chains

Page 2: 048866: Packet Switch Architectures Dr. Isaac Keslassy Electrical Engineering, Technion isaac@ee.technion.ac.il isaac/ Review

Spring 2006 048866 – Packet Switch Architectures 2

Simple DTMCs

“States” can be labeled (0,)1,2,3,… At every time slot a “jump” decision is

made randomly based on current state

10

p

q

1-q1-p

10

2

a

d fc b

e

(Sometimes the arrow pointing back to the same state is omitted)

Page 3: 048866: Packet Switch Architectures Dr. Isaac Keslassy Electrical Engineering, Technion isaac@ee.technion.ac.il isaac/ Review

Spring 2006 048866 – Packet Switch Architectures 3

1-D Random Walk

Time is slotted The walker flips a coin every time slot to

decide which way to go

X(t)

p1-p

Page 4: 048866: Packet Switch Architectures Dr. Isaac Keslassy Electrical Engineering, Technion isaac@ee.technion.ac.il isaac/ Review

Spring 2006 048866 – Packet Switch Architectures 4

Single Server Queue

Consider a queue at a supermarket In every time slot:

A customer arrives with probability p The HoL customer leaves with probability q

Bernoulli(p)Geom(q)

Page 5: 048866: Packet Switch Architectures Dr. Isaac Keslassy Electrical Engineering, Technion isaac@ee.technion.ac.il isaac/ Review

Spring 2006 048866 – Packet Switch Architectures 5

Birth-Death Chain

Can be modeled by a Birth-Death Chain (aka. Geom/Geom/1 queue)

Want to know: Queue size distribution Average waiting time, etc.

0 1 2 3

Page 6: 048866: Packet Switch Architectures Dr. Isaac Keslassy Electrical Engineering, Technion isaac@ee.technion.ac.il isaac/ Review

Spring 2006 048866 – Packet Switch Architectures 6

Discrete Time Markov Chains

Markov property (memoryless): “Future” is independent of “Past” given “Present”

A sequence of random variables {Xn} is called a Markov chain if it has the Markov property:

States are usually labeled {0,1,2,…} State space can be finite or infinite

Page 7: 048866: Packet Switch Architectures Dr. Isaac Keslassy Electrical Engineering, Technion isaac@ee.technion.ac.il isaac/ Review

Spring 2006 048866 – Packet Switch Architectures 7

Transition Probability

Probability to jump from state i to state j

Assume stationary: independent of time Transition probability matrix:

P = (pij) Two state MC:

Page 8: 048866: Packet Switch Architectures Dr. Isaac Keslassy Electrical Engineering, Technion isaac@ee.technion.ac.il isaac/ Review

Spring 2006 048866 – Packet Switch Architectures 8

Stationary Distribution

Define

Then k+1 = k P ( is a row vector)

Stationary Distribution:

if the limit exists.

If exists, we can solve it by

Page 9: 048866: Packet Switch Architectures Dr. Isaac Keslassy Electrical Engineering, Technion isaac@ee.technion.ac.il isaac/ Review

Spring 2006 048866 – Packet Switch Architectures 9

Balance Equations

These are called balance equations Transitions in and out of state i are balanced

Page 10: 048866: Packet Switch Architectures Dr. Isaac Keslassy Electrical Engineering, Technion isaac@ee.technion.ac.il isaac/ Review

Spring 2006 048866 – Packet Switch Architectures 10

In General

If we partition all the states into two sets, then transitions between the two sets must be “balanced”. Equivalent to a bi-section in the state

transition graph This can be easily derived from the Balance

Equations

Page 11: 048866: Packet Switch Architectures Dr. Isaac Keslassy Electrical Engineering, Technion isaac@ee.technion.ac.il isaac/ Review

Spring 2006 048866 – Packet Switch Architectures 11

Conditions for to Exist (I)

Definitions: State j is reachable by state i if

State i and j communicate if they are reachable by each other

The Markov chain is irreducible if all states communicate

Page 12: 048866: Packet Switch Architectures Dr. Isaac Keslassy Electrical Engineering, Technion isaac@ee.technion.ac.il isaac/ Review

Spring 2006 048866 – Packet Switch Architectures 12

Conditions for to Exist (I) (cont’d)

Condition: The Markov chain is irreducible

Counter-examples:

21 43

32p=1

1

Page 13: 048866: Packet Switch Architectures Dr. Isaac Keslassy Electrical Engineering, Technion isaac@ee.technion.ac.il isaac/ Review

Spring 2006 048866 – Packet Switch Architectures 13

Conditions for to Exist (II)

The Markov chain is aperiodic: Counter-example:

10

2

1

0 01 1

0

Page 14: 048866: Packet Switch Architectures Dr. Isaac Keslassy Electrical Engineering, Technion isaac@ee.technion.ac.il isaac/ Review

Spring 2006 048866 – Packet Switch Architectures 14

Conditions for to Exist (III)

The Markov chain is positive recurrent: State i is recurrent if it will be re-entered

infinitely often:

Otherwise transient If recurrent

• State i is positive recurrent if E(Ti)<1, where Ti is time between visits to state i

• Otherwise null recurrent

Page 15: 048866: Packet Switch Architectures Dr. Isaac Keslassy Electrical Engineering, Technion isaac@ee.technion.ac.il isaac/ Review

Spring 2006 048866 – Packet Switch Architectures 15

Irreducible Ergodic Markov Chain

The Markov chain is ergodic if it is positive recurrent and aperiodic.

In an irreducible ergodic Markov chain, if k+1 = k P, then:

is independent of the initial conditions

(j) is the limiting probability that the process will be in state j at time n. It is also equal to the long-run proportion of time that the

process will be in state j (ergodicity). It is called the stationary probability.

Page 16: 048866: Packet Switch Architectures Dr. Isaac Keslassy Electrical Engineering, Technion isaac@ee.technion.ac.il isaac/ Review

Spring 2006 048866 – Packet Switch Architectures 16

Irreducible Ergodic Markov Chain

If f is a bounded function on the state space:

Let mjj be the expected number of transitions until the Markov chain, starting in state j, returns to state j. Then mjj=1/(j)

1

0

( )lim ( ) ( )

N

nn

Nj

f Xf j j

N

References: books on stochastic processes (e.g., Ross)