Some standard univariate probability distributions

• Characteristic function, moment generating function, cumulant generating functions

• Discrete distribution

• Continuous distributions

• Some distributions associated with normal

• References

Characteristic function, moment generating function, cumulant generating functions

Characteristic function is defined as an expectation value of the function - e(itx)

Moment generating function is defined as (an expectation of e(tx)):

Moments can be calculated in the following way. Obtain derivative of M(t) and take the value of it at t=0

Cumulant generting function is defined as logarithm of the characteristic function

∫∞

∞−

= dxxfitxetC )()()(

∫∞

∞−

= dxxftxetM )()()(

0

)()(

=

=t

n

nn

dttMd

xE

))(log(... tCfgc =

Discrete distributions: Binomial

Let us assume that we carry out experiment and the result of the experiment can be “success” or “failure”. The probability of “success” in one experiment is p. Then probability of failure is q=1-p. We carry out experiments n times. Distribution of k successes is binomial:

Characteristic function:

Moment generating function:

€

p(k) = P(X = k) =n

k

⎛

⎝ ⎜

⎞

⎠ ⎟pk (1− p)n−k =

n!

k!(n − k)!pk (1− p)n−k

npitpetC )1)(()( −+=

nptpetM )1)(()( −+=

Example and mean valuesAs the number of trials become increases the distribution becomes more symmetric and

dense.

Calculate the probability of 2 or 3 successes if the probability of success is p=0.2 and the number of trials is n=3. Compare it with the the case when p=0.5 and n=3.

Mean value is np. Variance is npq=np(1-p).

If the number of trials is 10 and p = 0.2 then average number of successes is 2.

P=0.2, n=10 P=0.2, n=100P=0.5, n=10 P=0.5, n=100

Discrete distributions: Poisson

When the number of the trials (n) is large and the probability of successes (p) is small and np is finite and tends to as n goes to infinity then the binomial distribution converges to Poisson distribution:

Poisson distribution is used to describe the distribution of an event that occurs rarely (rare events) in a short time period. It is used in counting statistics to describe the number of registered photons.

Characteristic function is:

What is the moment generating function?

0 ,,0,1,2,k ,!

)()( >=−= k

ekpk

))1)((()( −= iteetC

Exampleλ=1, λ=5 and λ=10. As λ increases the distribution becomes more and more symmetric.Expected values is λ and variance is λ. Variance and mean are equal to each other.Exercise: Assume that the distribution of the number accidents is Poisson. If the average number of accidents in one day is 3 then what is the probability of three accidents happening in one day? What is the probability of at least three accidents in one day.

λ=1λ=5 λ=10

Discrete distributions: Negative Binomial

Consider an experiment: Probability of “success” is p and probability of failure is q=1-p. We carry out the experiment until k-th success. We want to find the probability of j failures before having kth success. (It is called sequential sampling. Sampling is carried out until stopping rule - k successes - is satisfied). If we have j failures then it means that the number of trials is k+j. Last trial was success. Then the probability that we will have j failures is:

It is called negative binomial because coefficients have the same from as those of the terms of the negative binomial series: p-k=(1-q)-k

Characteristic function is:

What is the moment generating function?

,,,0,1,2,j ,11

)()( 1 =⎟⎟⎠

⎞⎜⎜⎝

⎛ −+=⎟⎟

⎠

⎞⎜⎜⎝

⎛ −+=== − jkjk qp

jjk

pqpjjk

jXPjp

kk itqeptC −−= ))(1()(

Example, mean and variance

As the number of required successes increases the distribution becomes more and more symmetric. Mean value is kq/p and variance is kq(q+1)/p.Let us say we have an unfair coin. Probability of throwing head is 0.2. We throw the coin until we have 2 heads. What is the probability that we will achieve it in 4 trials?What is the average number of trials before we reach 2 heads?

k=10,p=0.2 k=10,p=0.5k=50,p=0.2. x axis is between 0 and 500 k=50,p=0.5

Continuous distributions: uniform

The simplest form of the continuous distribution is the uniform with density:

Cumulative distribution function is:

Moments and other properties are calculated easily.

otherwise0

if1

)(⎪⎩

⎪⎨⎧ ≤≤

−= bxaabxf

⎪⎩

⎪⎨

⎧

>

≤≤−−

<

=

bx

bxaab

bxax

xF

1

0

)(

Continuous distributions: exponential

Density of random variable with an exponential distribution has the form:

One of the origins of this distribution:From Poisson type random processes. If the probability distribution of j(t) events

occurring during time interval [0;t) is a Poisson with mean value t then probability of time elapsing till the first event occurs has the exponential distribution. Let Trdenotes time elapsed until r-th event

Putting r=1 we get e(- t). Taking into account that P(T1>t) = 1-F1(t) and getting its derivative wrt t we arrive to the exponential distribution

Characteristic function is:

€

f (t) = λe(−λ t) 0 < t < ∞

)())(( tTPrtjP r >=<

€

c(u) = (1−iu

λ)−1

Example, mean varianceAs lambda becomes larger, fall of the distribution becomes sharper.Mean value is 1/λ and variance is (λ+1)/λ2

If average waiting time is 1min then what is probability that first event will happen within 1 minute:

Small exercise: What is the probability that the first event will happen after 2 minutes?

λ=1

€

P(t <1) = F(1) = 1e−xdx =1− e−1 = 0.630

1

∫

Continuous distributions: Gamma

Gamma distribution can be considered as a generalisation of the exponential distribution. It has the form:

It is probability of time - t elapsing before exactly r events happens

Characteristic function of this distribution is:

If there are r independently and identically exponentially distributed random variables then the distribution of their sum is Gamma.

Sometimes for gamma distribution 1/λ instead of λ is written. Implementation in R uses this form. r is called shape and 1/λ is called scale parameter.

∞<<−−

=−

tr

tettf

rr

r 0 ,)!1(

)()(

1 λλ

riuuc −−= )1()(

Gamma distribution

As the shape parameter increases the centre of the distribution shifts to the left and it becomes more symmetric.Mean value is r/λ and variance is r(λ+1)/λ2

Continuous distributions: Normal

Perhaps the most popular and widely used continuous distribution is the normal distribution. Main reason for this is that usually an observed random variable is the sum of many random variables. According to the central limit theorem under some conditions (for example: random variables are independent. first and second and third moments exist and finite then distribution of the sum of these random variables converges to normal distribution)

Density of the normal distribution has the form

There are many tables for the normal distribution.

Its characteristic function is:€

f (x) =1

2πσe(−

(x − μ)2

2σ 2)

€

c(t) = e(itμ −t 2σ 2

2)

Central limit theorem

Let us assume that we have n independent random variables {Xi}, i= 1,..,n. If first, second and third moments (this condition can be relaxed) are finite then the sum of these random variables for sufficiently large n will be approximately normally distributed.

Because of this theorem, in many cases assumption that observations or errors are distributed with normal distribution is sufficiently good and tests based on this assumption give satisfactory results.

Exponential family

Exponential family of distributions has the form

Many distributions are special case of this family.

Natural exponential family of distributions is the subclass of this family:

Where A() is natural parameter.

If we use the fact that distribution should be normalised then characteristic function of the natural exponential family with natural parameter A() = can be derived to be:

Try to derive it. Hint: use the normalisation factor. Find D and then use expression of characteristic function and D.

This distribution is used for fitting generlised linear models.

€

f (x) = e(A(θ)(B(x) + C(x)) /G(ϕ ) + D(θ,ϕ ))

€

f (x) = e((A(θ)x + C(x)) /G(ϕ ) + D(θ,ϕ ))

€

C(t) = e(D(θ,ϕ ) − D(A−1(A(θ) + itG(ϕ )),ϕ ))

Exponential family: ExamplesMany well known distributions belong to this family (All distributions mentioned in

this lecture are from the exponential family).

Binomial

Poisson

Gamma

Normal

€

A( p) = ln(p

1− p), C(x) = ln(

n!

x!(n − x)!), D(p,ϕ ) = n ln(1− p), G(ϕ ) =1

€

A(λ ) = ln λ , C(x) = −ln(x!), D(λ ,ϕ ) = −λ , G(ϕ ) =1

€

A(λ ) = −λ , C(t) = (r −1)ln(t), D(λ ,ϕ ) = r lnλ + ln(r −1)!, G(ϕ ) =1

€

A(μ) = μ, C(x) = −x 2

2, D(μ,σ ) = −

μ 2

2σ 2− ln( 2πσ ), G(ϕ ) = σ 2

Continuous distributions: 2

Random variables with normal distribution are called standardized if their mean is 0 and variance is 1.

Sum of n standardized, independent normal random variables is 2 with n degrees of freedom.

Density function is:

If there are p linear restraints on the random variables then degree of freedom becomes n-p.

Characteristic function for this distribution is:

2 is used widely in statistics for such tests as goodness of fit of model to experiment.

∞<≤−Γ

=−

x0 ,)2

1(

)2

(2

1)(

12

1

2

1

n

nxxe

nxf

nittC 2

1

)21()(−

−=

Continuous distributions: t and F-distributions

Two more distributions are closely related with normal distribution. We will give them when we will discuss sample and sampling distributions. One of them is Student’s t-distribution. It is used to test if mean value of the sample is significantly different from a give value. Another and similar application is for tests of differences of means of two different samples.

Fisher’s F-distribution is the distribution of the ratio of the variances of two different samples. It is used to test if their variances are different. One of the important application is in ANOVA.

Reference

Johnson, N.L. & Kotz, S. (1969, 1970, 1972) Distributions in Statistics, I: Discrete distributions; II, III: Continuous univariate distributions, IV: Continuous multivariate distributions. Houghton Mufflin, New York.

Mardia, K.V. & Jupp, P.E. (2000) Directional Statistics, John Wiley & Sons.

Jaynes, E (2003) The Probability theory: Logic of Science