Chapter 4: Stochastic ProcessesChapter 4: Stochastic ProcessesPoisson Processes and Markov ChainsPoisson Processes and Markov Chains

Presented by Vincent BuhrPresented by Vincent Buhr

OverviewOverview

The Homogeneous Poisson ProcessThe Homogeneous Poisson Process

The Poisson and Binomial DistributionsThe Poisson and Binomial Distributions

The Poisson and Gamma DistributionsThe Poisson and Gamma Distributions

The Pure-Birth ProcessThe Pure-Birth Process

Finite Markov ChainsFinite Markov Chains

ModelingModeling

The Poisson ProcessThe Poisson Process

A sequence of events occurring during a time interval A sequence of events occurring during a time interval forms a homogeneous Poisson process if two conditions forms a homogeneous Poisson process if two conditions are met:are met:

The occurrence of any event in the time interval (a,b) is The occurrence of any event in the time interval (a,b) is independent of the occurrence of any event in the time interval independent of the occurrence of any event in the time interval (c,d)(c,d)

There is a constant There is a constant λλ such that for any sufficiently small time such that for any sufficiently small time interval (t,t+h), the probability that an event occurs in the time interval (t,t+h), the probability that an event occurs in the time interval is independent of t, and is interval is independent of t, and is λλh+o(h)h+o(h)

Condition 2 ensures that the probability of an event Condition 2 ensures that the probability of an event occurring within an interval is proportional to the length occurring within an interval is proportional to the length of the intervalof the intervalWith these two conditions, the number of events N that With these two conditions, the number of events N that occur up to any time t has a Poisson distribution with occur up to any time t has a Poisson distribution with parameter parameter λλtt

The Poisson Process (cont)The Poisson Process (cont)

PPjj(t) is the probability that N=j at time t(t) is the probability that N=j at time t

At time 0 the value of N is necessarily 0, so PAt time 0 the value of N is necessarily 0, so P00(0)=1 and (0)=1 and PPii(0)=0 for all i>0(0)=0 for all i>0

At time t+h, N=0 only if no events occur in the interval At time t+h, N=0 only if no events occur in the interval (0,t) or the interval (t,t+h), so:(0,t) or the interval (t,t+h), so:

At time t+h, N=1 can happen one of two ways, either At time t+h, N=1 can happen one of two ways, either N=1 at time t and no events occur in the interval (t,t+h) N=1 at time t and no events occur in the interval (t,t+h) or N=0 at time t and one event occurs in the interval or N=0 at time t and one event occurs in the interval (t,t+h), which gives:(t,t+h), which gives:

The Poisson Process (cont)The Poisson Process (cont)

Finally, N>1 can occur at time t+h one of three waysFinally, N>1 can occur at time t+h one of three ways N=j at time t and no events occur in the interval (t,t+h)N=j at time t and no events occur in the interval (t,t+h) N=j-1 at time t and one event occurs in the interval (t,t+h)N=j-1 at time t and one event occurs in the interval (t,t+h) N=j-k, where k > 1, and more than one event occurs in the N=j-k, where k > 1, and more than one event occurs in the

interval (t,t+h)interval (t,t+h)

These possibilities give us an equation that looks a lot These possibilities give us an equation that looks a lot like 4.3:like 4.3:

The only difference between the equations is the order The only difference between the equations is the order of o(h), so we can use 4.4 for all j > 0of o(h), so we can use 4.4 for all j > 0

The Poisson Process (cont)The Poisson Process (cont)

After subtracting PAfter subtracting P00(t) from both sides of (4.2) and P(t) from both sides of (4.2) and P jj(t) (t) from both sides of (4.4), dividing through by h, and from both sides of (4.4), dividing through by h, and letting h → 0 we get two equations:letting h → 0 we get two equations:

(4.5) has the solution:(4.5) has the solution:

And since we know that PAnd since we know that P00(0)=1 we can infer that C=1, (0)=1 we can infer that C=1, so:so:

The Poisson Process (cont)The Poisson Process (cont)

Using (4.9) and mathematical induction we can Using (4.9) and mathematical induction we can prove that the solution to (4.6) is:prove that the solution to (4.6) is:

Which is the Poisson distribution with Which is the Poisson distribution with parameter parameter λλt as we set out to provet as we set out to prove

The Poisson and Binomial The Poisson and Binomial DistributionsDistributions

Under special circumstances the binomial Under special circumstances the binomial distribution can be made to approximate the distribution can be made to approximate the Poisson distributionPoisson distributionIf n → + Inf., p → 0, and np = If n → + Inf., p → 0, and np = λλ then for any y then for any y the binomial probability approaches the the binomial probability approaches the Poisson probabilityPoisson probabilityTo prove this, first we write the binomial To prove this, first we write the binomial probability equation as:probability equation as:

The Poisson and Binomial The Poisson and Binomial Distributions (cont)Distributions (cont)

If we then fix y and If we then fix y and λλ, and write p as , and write p as λλ / n then / n then as n approaches infinity each term in (4.12) as n approaches infinity each term in (4.12) has a finite limithas a finite limit

Terms of the form (n-i) Terms of the form (n-i) λλ / n approach / n approach λλ

With these limits (4.12) approachesWith these limits (4.12) approaches

Which is the Poisson probabilityWhich is the Poisson probability

The Poisson and Gamma The Poisson and Gamma DistributionsDistributions

Equation (4.9) hints at a connection between the Poisson Equation (4.9) hints at a connection between the Poisson distribution and the exponential distributiondistribution and the exponential distributionThe random time until the first event occurs in a Poisson process The random time until the first event occurs in a Poisson process with parameter with parameter λλ is given by the exponential distribution with is given by the exponential distribution with parameter parameter λλTo prove this we can let F(t) be the probability that the first event To prove this we can let F(t) be the probability that the first event occurs before time t, which means that the density function for occurs before time t, which means that the density function for the time until the first occurrence is the derivative of F(t)the time until the first occurrence is the derivative of F(t)From (4.9)From (4.9)

So:So:

Which is the exponential distributionWhich is the exponential distribution

The Poisson and Gamma The Poisson and Gamma Distributions (cont)Distributions (cont)

Additionally, the distribution of the time between successive events is Additionally, the distribution of the time between successive events is also given by the exponential distributionalso given by the exponential distributionThis means that the random time until the kth event occurs is the sum This means that the random time until the kth event occurs is the sum of k independent exponentially distributed times, which has the of k independent exponentially distributed times, which has the gamma distributiongamma distribution

To prove this, let tTo prove this, let t00 be some fixed value of t be some fixed value of t

Then, if the time until the kth event occurs exceeds tThen, if the time until the kth event occurs exceeds t00, the number of , the number of events occurring before time tevents occurring before time t00 is less than k is less than k

So the probability that k-1 or less events occur before time tSo the probability that k-1 or less events occur before time t0 0 is equal is equal to the probability that the time until the kth event occurs exceeds tto the probability that the time until the kth event occurs exceeds t00

This leads us to the following equality:This leads us to the following equality:

The RHS of this equality is essentially the gamma distributionThe RHS of this equality is essentially the gamma distribution

The Pure-Birth ProcessThe Pure-Birth Process

When deriving the Poisson distribution we assumed that the When deriving the Poisson distribution we assumed that the probability of an event in a time interval is independent of the probability of an event in a time interval is independent of the number of events that have occurred up to time tnumber of events that have occurred up to time tThis assumption does not always hold in biological applicationsThis assumption does not always hold in biological applicationsIn the pure-birth process it is assumed that given the value of a In the pure-birth process it is assumed that given the value of a random variable at time t is j, the probability that it increases to j+1 random variable at time t is j, the probability that it increases to j+1 in a given time interval (t,t+h) is in a given time interval (t,t+h) is λλjjhhThe Poisson case arises when The Poisson case arises when λλj j is independent of j and is just is independent of j and is just written as written as λλAs with the Poisson process we can arrive at a set of differential As with the Poisson process we can arrive at a set of differential equations for the probability that the random variable takes the equations for the probability that the random variable takes the value j at time tvalue j at time t

The Pure-Birth Process (cont)The Pure-Birth Process (cont)

One example of an application of the Pure-Birth One example of an application of the Pure-Birth process is the Yule process, where it is process is the Yule process, where it is assumed that assumed that λλjj= j= jλλ The motivation for this process arises from The motivation for this process arises from

populations where if the size of the population is j the populations where if the size of the population is j the probability that it increases to size j+1 is proportional probability that it increases to size j+1 is proportional to jto j

For this case, the solution to the differential For this case, the solution to the differential equations given before is:equations given before is:

The Pure-Birth Process (cont)The Pure-Birth Process (cont)

Another example of the application of the Pure-Birth Another example of the application of the Pure-Birth process comes from polymerase chain reaction (PCR)process comes from polymerase chain reaction (PCR)In PCR, sequential additions of base pairs to a primer In PCR, sequential additions of base pairs to a primer occur to create the productoccur to create the productFor this process, For this process, λλjj=m-j, which implies that once the =m-j, which implies that once the length of the product reaches m no further increase in length of the product reaches m no further increase in length is possiblelength is possibleWith this condition, the solution is:With this condition, the solution is:

Neither this example nor the last follow the Poisson Neither this example nor the last follow the Poisson distribution, which shows the importance of verifying the distribution, which shows the importance of verifying the event independence assumptionevent independence assumption

Introduction to Finite Markov Introduction to Finite Markov ChainsChains

A Markov chain process occupies one of a finite number A Markov chain process occupies one of a finite number of discrete states at a given time unitof discrete states at a given time unitIn a time step from t to t+1 the process either stays in the In a time step from t to t+1 the process either stays in the same state or moves to another state in a probabilistic same state or moves to another state in a probabilistic way (as opposed to deterministic)way (as opposed to deterministic)A simple Markov chain has two basic properties:A simple Markov chain has two basic properties:

The Markov property- The probability that the process changes The Markov property- The probability that the process changes from Efrom Ejj to E to Ekk in one time step depends only on the current state in one time step depends only on the current state EEjj and not on any past states and not on any past states

The temporally homogeneous transition probabilities property- If The temporally homogeneous transition probabilities property- If at time t the process is in state Eat time t the process is in state E jj, the probability that it changes , the probability that it changes to Eto Ekk in one time step is independent of t in one time step is independent of t

Some Markov processes ignore one or both of these Some Markov processes ignore one or both of these properties, but we will assume both holdproperties, but we will assume both hold

Transition Probabilities and the Transition Probabilities and the Transition Probability MatrixTransition Probability Matrix

If at time t a Markovian random variable is in state EIf at time t a Markovian random variable is in state E jj the the

probability that at time t+1 it is in state Eprobability that at time t+1 it is in state Ekk is denoted by is denoted by

ppjkjk, which is the transition probability from E, which is the transition probability from Ejj to E to Ekk

This notion implicitly contains both the properties mentioned This notion implicitly contains both the properties mentioned beforebefore

A transition probability matrix P of a Markov chain A transition probability matrix P of a Markov chain contains all of the transition probabilities of that chaincontains all of the transition probabilities of that chain

Transition Probabilities and the Transition Probabilities and the Transition Probability Matrix (cont)Transition Probability Matrix (cont)It is also assumed that there is a initial probability It is also assumed that there is a initial probability distribution for the states in the processdistribution for the states in the process

This means that there is a probability This means that there is a probability ππi i that at the initial time that at the initial time point the Markovian random variable is in state Epoint the Markovian random variable is in state E ii

To find the probability that the Markov chain process is in To find the probability that the Markov chain process is in state Estate Ejj two time steps after being in state E two time steps after being in state Eii you must you must consider all the possible intermediate steps after one time consider all the possible intermediate steps after one time step that the process could be instep that the process could be in

This can also be done for the whole process at once by This can also be done for the whole process at once by matrix multiplication, the notation Pmatrix multiplication, the notation Pnn is used to denote an is used to denote an n-step transition probability matrixn-step transition probability matrix

Markov Chains with Absorbing Markov Chains with Absorbing StatesStates

A Markov chain with an absorbing state can be A Markov chain with an absorbing state can be recognized by the appearance of a 1 along the recognized by the appearance of a 1 along the main diagonal of its transition probability matrixmain diagonal of its transition probability matrix

A Markov chain with an absorbing state will A Markov chain with an absorbing state will eventually enter that state and never leave iteventually enter that state and never leave it

Markov chains with absorbing states bring up Markov chains with absorbing states bring up new questions, which will be addressed later, new questions, which will be addressed later, but for now we will only consider Markov chains but for now we will only consider Markov chains without absorbing stateswithout absorbing states

Markov Chains with No Absorbing Markov Chains with No Absorbing StatesStates

In addition to having no absorbing states, the Markov In addition to having no absorbing states, the Markov models that we will consider are also finite, aperiodic, and models that we will consider are also finite, aperiodic, and irreducibleirreducible

Finite means that there are a finite number of possible statesFinite means that there are a finite number of possible states Aperiodic means that there is no state such that a return to that Aperiodic means that there is no state such that a return to that

state is possible only tstate is possible only t00, 2t, 2t00, 3t, 3t00, … transitions later, where t, … transitions later, where t00 > 1 > 1

Irreducible means that any state can eventually be reached from Irreducible means that any state can eventually be reached from any other state, but not necessarily in one stepany other state, but not necessarily in one step

Stationary DistributionsStationary Distributions

Let the probability that at time t a Markov chain process Let the probability that at time t a Markov chain process is in state Eis in state Ejj be be φφjj

This means that the probability that at time t+1 the This means that the probability that at time t+1 the process is in state Eprocess is in state Ejj is given by is given by

If we assume that these two probabilities are equal then If we assume that these two probabilities are equal then we get:we get:

If this is the case, then the process is said to be If this is the case, then the process is said to be stationary, that is, from time t onwards, the probability of stationary, that is, from time t onwards, the probability of the process being in state Ethe process being in state Ejj does not change does not change

Stationary Distributions (cont)Stationary Distributions (cont)

If the row vector If the row vector φφ’ is defined by:’ is defined by:

Then we get the following from (4.25)Then we get the following from (4.25)

The row vector must also satisfyThe row vector must also satisfy

With these equations we can find the stationary With these equations we can find the stationary distribution when it existsdistribution when it exists Note that (4.27) generates one redundant equation Note that (4.27) generates one redundant equation

that can be omittedthat can be omitted

Stationary Distribution ExampleStationary Distribution ExampleWe are given a Markov chain with the following transition probability matrixWe are given a Markov chain with the following transition probability matrix

Using (4.27) and (4.28) we can form a set of equations to solveUsing (4.27) and (4.28) we can form a set of equations to solve

The solution to these equations is:The solution to these equations is:

This means that over a long time period a random variable with the given This means that over a long time period a random variable with the given transition matrix should spend about 24.14% of the time in state Etransition matrix should spend about 24.14% of the time in state E 11, 38.51% of , 38.51% of the time in state Ethe time in state E22, etc., etc.

Stationary Distribution Example Stationary Distribution Example (cont)(cont)

With matrix multiplication we can see how quickly the With matrix multiplication we can see how quickly the Markov chain process would reach the stationary Markov chain process would reach the stationary distributiondistribution

From this it appears that the stationary distribution is From this it appears that the stationary distribution is approximately reached after 16 time stepsapproximately reached after 16 time steps

The Graphical Representation of a The Graphical Representation of a Markov ChainMarkov Chain

It can be convenient to represent a Markov chain by a directed It can be convenient to represent a Markov chain by a directed graph, using the states as nodes and the transition probabilities as graph, using the states as nodes and the transition probabilities as edgesedges

Additionally, start and end states can be added as neededAdditionally, start and end states can be added as neededThe graph structure without probabilities added is called the topologyThe graph structure without probabilities added is called the topologyThese definitions are used later in the book to discuss hidden These definitions are used later in the book to discuss hidden Markov modelsMarkov models

ModelingModeling

While the homogeneous Poisson process has many While the homogeneous Poisson process has many uses in Bioinformatics, the two assumptions made at the uses in Bioinformatics, the two assumptions made at the beginning of the chapter (homogeneity and beginning of the chapter (homogeneity and independence) do not always holdindependence) do not always holdSimilarly, the assumptions made by Markov chain Similarly, the assumptions made by Markov chain processes do not always holdprocesses do not always holdFor example, from analyzing DNA, it has become For example, from analyzing DNA, it has become apparent that the probability that the nucleotide a is apparent that the probability that the nucleotide a is followed by g depends to some extent on the location of followed by g depends to some extent on the location of a gene in a chromosome, also the nucleotides preceding a gene in a chromosome, also the nucleotides preceding a may have an affect on the probability of g following aa may have an affect on the probability of g following aHowever, the memoryless Markov chain property is often However, the memoryless Markov chain property is often assumed even when its applicability is uncertainassumed even when its applicability is uncertain

Modeling (cont)Modeling (cont)

In general, mathematical models often make simplifying In general, mathematical models often make simplifying assumptions about properties of the events being assumptions about properties of the events being modeledmodeledThere are few cases where we could predict outcomes There are few cases where we could predict outcomes of events with exact accuracy, but even if we could, of events with exact accuracy, but even if we could, modeling may be more desirable due to the complexity modeling may be more desirable due to the complexity of the phenomena being analyzedof the phenomena being analyzedIt is often necessary to find a middle ground between It is often necessary to find a middle ground between easily solveable, simple models, and combersome, easily solveable, simple models, and combersome, complex onescomplex onesThe key to this is knowing that a model only needs to The key to this is knowing that a model only needs to capture enough of the true complexity of the situation to capture enough of the true complexity of the situation to serve our purposesserve our purposes

Modeling (cont)Modeling (cont)

Finding the middle ground may not always be easy either Finding the middle ground may not always be easy either however, since benchmarks on model performances are however, since benchmarks on model performances are not often easily available, and subjectivity may come into not often easily available, and subjectivity may come into playplayAn example of modeling simplification comes from the An example of modeling simplification comes from the early version of BLAST, which assumed that nucleotides early version of BLAST, which assumed that nucleotides are identically and independently distributed along a are identically and independently distributed along a DNA sequenceDNA sequenceWe now know that this is not true, but the BLAST We now know that this is not true, but the BLAST procedure does work, in that it captures enough of procedure does work, in that it captures enough of biological reality to be effectivebiological reality to be effectiveAnother aspect of modeling is that we often assume that Another aspect of modeling is that we often assume that a model will become more refined as it is used and we a model will become more refined as it is used and we learn more about the reality of the phenomena; models learn more about the reality of the phenomena; models are rarely said to be in their final versionsare rarely said to be in their final versions