Embed Size (px)
DESCRIPTION
Stick-Breaking Constructions. Patrick Dallaire June 10 th , 2011. Outline. Introduction of the Stick-Breaking process. Outline. Introduction of the Stick-Breaking process Presentation of fundamental representation. Outline. Introduction of the Stick-Breaking process - PowerPoint PPT Presentation
Citation preview
STICK-BREAKING CONSTRUCTIONSPatrick DallaireJune 10th, 2011
Outline Introduction of the Stick-Breaking
process
Outline Introduction of the Stick-Breaking
process Presentation of fundamental
representation
Outline Introduction of the Stick-Breaking
process Presentation of fundamental
representation The Dirichlet process The Pitman-Yor process The Indian buffet process
Outline Introduction of the Stick-Breaking
process Presentation of fundamental
representation The Dirichlet process The Pitman-Yor process The Indian buffet process
Definition of the Beta process
Outline Introduction of the Stick-Breaking
process Presentation of fundamental
representation The Dirichlet process The Pitman-Yor process The Indian buffet process
Definition of the Beta process A Stick-Breaking construction of Beta
process
Outline Introduction of the Stick-Breaking
process Presentation of fundamental
representation The Dirichlet process The Pitman-Yor process The Indian buffet process
Definition of the Beta process A Stick-Breaking construction of Beta
process Conclusion and current work
The Stick-Breaking process
The Stick-Breaking process
Assume a stick of unit length
The Stick-Breaking process
Assume a stick of unit length
The Stick-Breaking process
Assume a stick of unit length At each iteration, a part of the remaining
stick is broken by sampling the proportion to cut
The Stick-Breaking process
Assume a stick of unit length At each iteration, a part of the remaining
stick is broken by sampling the proportion to cut
The Stick-Breaking process
Assume a stick of unit length At each iteration, a part of the remaining
stick is broken by sampling the proportion to cut
The Stick-Breaking process
Assume a stick of unit length At each iteration, a part of the remaining
stick is broken by sampling the proportion to cut
The Stick-Breaking process
Assume a stick of unit length At each iteration, a part of the remaining
stick is broken by sampling the proportion to cut
The Stick-Breaking process
Assume a stick of unit length At each iteration, a part of the remaining
stick is broken by sampling the proportion to cut
The Stick-Breaking process
Assume a stick of unit length At each iteration, a part of the remaining
stick is broken by sampling the proportion to cut
The Stick-Breaking process
Assume a stick of unit length At each iteration, a part of the remaining
stick is broken by sampling the proportion to cut
The Stick-Breaking process
Assume a stick of unit length At each iteration, a part of the remaining
stick is broken by sampling the proportion to cut
The Stick-Breaking process
Assume a stick of unit length At each iteration, a part of the remaining
stick is broken by sampling the proportion to cut
The Stick-Breaking process
Assume a stick of unit length At each iteration, a part of the remaining
stick is broken by sampling the proportion to cut
The Stick-Breaking process
Assume a stick of unit length At each iteration, a part of the remaining stick
is broken by sampling the proportion to cut How should we sample these proportions?
Beta random proportions Let be the proportion to cut at
iteration
Beta random proportions Let be the proportion to cut at
iteration The remaining length can be expressed
as
Beta random proportions Let be the proportion to cut at
iteration The remaining length can be expressed
as
Thus, the broken part is defined by
Beta random proportions Let be the proportion to cut at
iteration The remaining length can be expressed
as
Thus, the broken part is defined by
We first consider the case where
Beta distribution The Beta distribution is a density
function on
Parameters and control its shape
The Dirichlet process
The Dirichlet process Dirichlet processes are often used to
produce infinite mixture models
The Dirichlet process Dirichlet processes are often used to
produce infinite mixture models Each observation belongs to one of the
infinitely many components
The Dirichlet process Dirichlet processes are often used to
produce infinite mixture models Each observation belongs to one of the
infinitely many components The model ensures that only a finite
number of components have appreciable weight
The Dirichlet process A Dirichlet process, , can be constructed
according to a Stick-Breaking process
Where is the base distribution and is a unit mass at
Construction demo
Construction demo
Construction demo
Construction demo
Construction demo
Construction demo
Construction demo
Construction demo
Construction demo
Construction demo
Construction demo
Construction demo
Construction demo
Construction demo
Construction demo
Construction demo
The Pitman-Yor process
The Pitman-Yor process A Pitman-Yor process, , can be
constructed according to a Stick-Breaking process
Where and
Evolution of the Beta cuts The parameter controls the speed at
which the Beta distribution changes
Evolution of the Beta cuts The parameter controls the speed at
which the Beta distribution changes The parameter determines initial
shapes of the Beta distribution
Evolution of the Beta cuts The parameter controls the speed at
which the Beta distribution changes The parameter determines initial
shapes of the Beta distribution When , there is no changes over
time and its called a Dirichlet process
Evolution of the Beta cuts The parameter controls the speed at
which the Beta distribution changes The parameter determines initial
shapes of the Beta distribution When , there is no changes over
time and its called a Dirichlet process
MATLAB DEMO
The Indian Buffet process
The Indian Buffet process The Indian Buffet process was initially
used to represent latent features
The Indian Buffet process The Indian Buffet process was initially
used to represent latent features Observations are generated according to
a set of unknown hidden features
The Indian Buffet process The Indian Buffet process was initially
used to represent latent features Observations are generated according to
a set of unknown hidden features The model ensure that only a finite
number of features have appreciable probability
The Indian Buffet process
Recall the basic Stick-Breaking process
The Indian Buffet process
Recall the basic Stick-Breaking process
The Indian Buffet process
Recall the basic Stick-Breaking process Here, we only consider the remaining
parts
The Indian Buffet process
Recall the basic Stick-Breaking process Here, we only consider the remaining
parts
The Indian Buffet process
Recall the basic Stick-Breaking process Here, we only consider the remaining
parts Each value corresponds to a feature
probability of appearance
Summary
Summary The Dirichlet process induces a
probability over infinitely many classes
Summary The Dirichlet process induces a
probability over infinitely many classes This is the underlying de Finetti mixing
distribution of the Chinese restaurant process
De Finetti theorem It states that the distribution of any
infinitely exchangeable sequence can be written
where is the de Finetti mixing distribution
Summary The Dirichlet process induces a
probability over infinitely many classes This is the underlying de Finetti mixing
distribution of the Chinese restaurant process
The Indian Buffet process induces a probability over infinitely many features
Summary The Dirichlet process induces a
probability over infinitely many classes This is the underlying de Finetti mixing
distribution of the Chinese restaurant process
The Indian Buffet process induces a probability over infinitely many features
Its underlying de Finetti mixing distribution is the Beta process
The Beta process
The Beta process This process
Beta with Stick-Breaking The Beta distribution has a Stick-
Breaking representation which allows to sample from
Beta with Stick-Breaking The Beta distribution has a Stick-
Breaking representation which allows to sample from
The construction is
Beta with Stick-Breaking
Beta with Stick-Breaking
Beta with Stick-Breaking
Beta with Stick-Breaking
Beta with Stick-Breaking
Beta with Stick-Breaking
Beta with Stick-Breaking
Beta with Stick-Breaking
Beta with Stick-Breaking
Beta with Stick-Breaking
Beta with Stick-Breaking
Beta with Stick-Breaking
Beta with Stick-Breaking
Beta with Stick-Breaking
Beta with Stick-Breaking
Beta with Stick-Breaking The Beta distribution has a Stick-
Breaking representation which allows to sample from
The construction is
The Beta process A Beta process is defined as
as , and is a Beta process
Stick-Breaking the Beta process
The Stick-Breaking construction of the Beta process is such that
Stick-Breaking the Beta process
Expending the first terms
Conclusion We briefly described various Stick-Breaking
constructions for Bayesian nonparametric priors
These constructions help to understand the properties of each process
It also unveils connections among existing priors
The Stick-Breaking process might help to construct new priors
Current work Applying a Stick-Breaking process to
select the number of support points in a Gaussian process
Defining a stochastic process for unbounded random directed acyclic graph
Finding its underlying Stick-Breaking representation
