MAS113 Introduction to Probability and Statisticsjonathanjordan.staff.shef.ac.uk › IntroPS › slides8.pdfMoment Generating Functions The Central Limit Theorem (CLT) MAS113 Introduction

Moment Generating FunctionsThe Central Limit Theorem (CLT)

MAS113 Introduction to Probability and

Statistics

Dr Jonathan Jordan

School of Mathematics and Statistics, University of Sheffield

2019–20

Dr Jonathan Jordan MAS113 Introduction to Probability and Statistics


Moments

Let X be a random variable.

Two important associated numerical quantities are theexpected values E (X ) and E (X 2), from which we cancompute the variance, using

Var(X ) = E (X 2)− E (X )2.

More generally we might try to find the nth moment E (X n)for n ∈ N.

For example E (X 3) and E (X 4) give information about theshape of the distribution and are used to calculate quantitiescalled the skewness and kurtosis.



Moments



Var(X ) = E (X 2)− E (X )2.





Moments



Var(X ) = E (X 2)− E (X )2.





Moments



Var(X ) = E (X 2)− E (X )2.





Moment generating function

We can try to calculate moments directly, but there is also auseful shortcut.

The moment generating function (or mgf) is defined for allt ∈ R by:

MX (t) = E (etX ).



Moment generating function

We can try to calculate moments directly, but there is also auseful shortcut.

The moment generating function (or mgf) is defined for allt ∈ R by:

MX (t) = E (etX ).



Moment generating function cont.

SoMX (t) =

∑x∈RX

etxpX (x)

if X is discrete and

MX (t) =

∫ ∞−∞

etx fX (x)dx ,

if X is continuous.



Moment generating function cont.

SoMX (t) =

∑x∈RX

etxpX (x)

if X is discrete and

MX (t) =

∫ ∞−∞

etx fX (x)dx ,

if X is continuous.



Example

Let X ∼ Bernoulli(p).

Then X only takes two values: 1 with probability p, and 0with probability 1− p.

SoMX (t) = pet.1 + (1− p)et.0 = pet + 1− p.



Example






Example






Graph of mgf of Bernoulli(1/3)

−1.0 −0.5 0.0 0.5 1.0

0.0

0.5

1.0

1.5

t

mgf

at t



Differentiation

Now differentiate to get

d

dtMX (t) = E (XetX ),

and sod

dtMX (t)

∣∣∣∣t=0

= E (X ).

Differentiating again gives

d2

dt2MX (t) = E (X 2etX ),

and henced2

dt2MX (t)

∣∣∣∣t=0

= E (X 2).



Differentiation


d


and sod

dtMX (t)

∣∣∣∣t=0

= E (X ).


d2


and henced2

dt2MX (t)

∣∣∣∣t=0

= E (X 2).



Differentiation


d


and sod

dtMX (t)

∣∣∣∣t=0

= E (X ).


d2


and henced2

dt2MX (t)

∣∣∣∣t=0

= E (X 2).



Differentiation


d


and sod

dtMX (t)

∣∣∣∣t=0

= E (X ).


d2


and henced2

dt2MX (t)

∣∣∣∣t=0

= E (X 2).



All moments

In fact you can find all the moments by this procedure:

dn

dtnMX (t)

∣∣∣∣t=0

= E (X n).



Poisson example

Example

Find the moment generating function of a Poisson randomvariable with parameter λ. Verify that the mean is λ.



Series expansion

Another way of seeing how the mgf of X contains informationabout all the moments comes from writing the seriesexpansion for the exponential function

etX =∞∑n=0

tn

n!X n.

It then follows that:

MX (t) =∞∑n=0

tn

n!E (X n).



Series expansion


etX =∞∑n=0

tn

n!X n.


MX (t) =∞∑n=0

tn

n!E (X n).



Series expansion


etX =∞∑n=0

tn

n!X n.


MX (t) =∞∑n=0

tn

n!E (X n).



Series expansion


etX =∞∑n=0

tn

n!X n.


MX (t) =∞∑n=0

tn

n!E (X n).



Uniform example

Example

Let X have a uniform distribution on [0, 1]. Find an expressionfor the moment generating function MX (t), write it as a seriesexpansion, and hence find the moments E (X ), E (X 2) andE (X 3).



M.g.f. of sum

Our main purpose in introducing mgfs at this stage is to give arough idea of the proof of the central limit theorem. The nextresult is very useful for that, and also has other applications.

Theorem

If X and Y are independent, then for all t ∈ R

MX+Y (t) = MX (t)MY (t).



M.g.f. of sum

Our main purpose in introducing mgfs at this stage is to give arough idea of the proof of the central limit theorem. The nextresult is very useful for that, and also has other applications.

Theorem

If X and Y are independent, then for all t ∈ R

MX+Y (t) = MX (t)MY (t).



Sum of i.i.d. r.v.s

It follows (by using mathematical induction) that ifS(n) = X1 + X2 + · · ·+ Xn is a sum of i.i.d random variableshaving common mgf MX then

MS(n)(t) = MX (t)n.



Normal m.g.f.

Here is the key example that we need.

Lemma

If X ∼ N(0, 1) then

MX (t) = e12t2



Normal m.g.f.

Here is the key example that we need.

Lemma

If X ∼ N(0, 1) then

MX (t) = e12t2



General normal m.g.f.

To get the moment generating function of a general normaldistribution N(µ, σ2), we can use the following result.

Theorem

If X has moment generating function MX (t) and Y = aX + b,then Y has moment generating function MY (t) = etbMX (at).

This tells us that the mgf of a N(µ, σ2) random variable is

exp

(µt +

1

2σ2t2

).





Theorem



exp

(µt +

1

2σ2t2

).





Theorem



exp

(µt +

1

2σ2t2

).



Sum of normals

In fact it can be shown that the mgf uniquely determines thedistribution of a random variable, so X ∼ N(µ, σ2) is the onlyrandom variable with its mgf.

Using this fact we can show that the sum of n i.i.d. normaldistributions is itself normal.

We have

MS(n)(t) = [eµt+ 12σ2t2

]n

= enµt+ 12nσ2t2

,

so S(n) ∼ N(nµ, nσ2), and X (n) ∼ N(µ, σ2/n).



Sum of normals



We have

MS(n)(t) = [eµt+ 12σ2t2

]n

= enµt+ 12nσ2t2

,




Sum of normals



We have

MS(n)(t) = [eµt+ 12σ2t2

]n

= enµt+ 12nσ2t2

,




Sum of normals



We have

MS(n)(t) = [eµt+ 12σ2t2

]n

= enµt+ 12nσ2t2

,




Sum of normals



We have

MS(n)(t) = [eµt+ 12σ2t2

]n

= enµt+ 12nσ2t2

,




Sum of Poissons

Before moving onto the Central Limit Theorem we will giveone other application of m.g.f.s of sums.

Theorem

Let X and Y be independent random variables each withPoisson distributions with parameters λ and µ respectively.Then X + Y has a Poisson distribution with parameter λ + µ.



Sum of Poissons

Before moving onto the Central Limit Theorem we will giveone other application of m.g.f.s of sums.

Theorem

Let X and Y be independent random variables each withPoisson distributions with parameters λ and µ respectively.Then X + Y has a Poisson distribution with parameter λ + µ.



Central Limit Theorem – Introduction

We finish with another important result, which tells us aboutthe distribution of X (n) for large n. It could be argued thatthis is the most important result in the whole of probability(and statistics).

The law of large numbers tells us that X (n) tends to µ asn→∞. But the central limit theorem gives far moreinformation.

It tells us about the behaviour of the distribution of thefluctuations of X (n) around µ, as n→∞. As hinted earlier,these are always normally distributed!















Central Limit Theorem – Statement

Theorem

(The central limit theorem)Let X1,X2, . . . be a sequence of i.i.d random variables, eachwith mean µ and variance σ2. For any −∞ ≤ a < b ≤ ∞,

limn→∞

P

(a ≤ X (n)− µ

σ/√n≤ b

)=

1√2π

∫ b

a

exp

(−1

2z2

)dz .



Central Limit Theorem – Summary

In other words, the distribution of X (n) tends to a normaldistribution with mean µ and variance σ2/n, as n→∞.

So for large n, we have, approximately

X (n) ∼ N

(µ,σ2

n

),

S(n) ∼ N(nµ, nσ2

).






X (n) ∼ N

(µ,σ2

n

),


).






X (n) ∼ N

(µ,σ2

n

),


).



Re-arranging

Notice that the right hand side of the CLT is P(a ≤ Z ≤ b)where Z ∼ N(0, 1) is the standard normal.

We can also rewrite the left hand side in terms of S(n) bymultiplying top and bottom by n. Then we get anotherequivalent form of the central limit theorem (CLT) which isoften seen in books:

limn→∞

P

(a ≤ S(n)− nµ

σ√n

≤ b

)= P(a ≤ Z ≤ b).



Re-arranging



limn→∞

P

(a ≤ S(n)− nµ

σ√n

≤ b

)= P(a ≤ Z ≤ b).



Re-arranging



limn→∞

P

(a ≤ S(n)− nµ

σ√n

≤ b

)= P(a ≤ Z ≤ b).



Validity

As long as X1,X2, . . . have the same distribution (and areindependent) the CLT is valid.

It doesn’t matter what that distribution is.

It can be discrete, or continuous – uniform, Poisson, Bernoulli,exponential, etc etc.



Validity






Validity






Outline proof of the CLT

The full proof of the CLT is beyond the scope of this course.However, we will give an outline, based on moment generatingfunctions.

We will aim to establish

limn→∞

P

(a ≤ S(n)− nµ

σ√n

≤ b

)= P(a ≤ Z ≤ b).



Outline proof of the CLT

The full proof of the CLT is beyond the scope of this course.However, we will give an outline, based on moment generatingfunctions.

We will aim to establish

limn→∞

P

(a ≤ S(n)− nµ

σ√n

≤ b

)= P(a ≤ Z ≤ b).



Normal approximation to binomial

We investigate the normal approximation to the binomialdistribution.

If we take X1,X2, . . . to be Bernoulli random variables withcommon parameter p, then S(n) is Binomial with parametersn and p.



Normal approximation to binomial

We investigate the normal approximation to the binomialdistribution.

If we take X1,X2, . . . to be Bernoulli random variables withcommon parameter p, then S(n) is Binomial with parametersn and p.



de Moivre–Laplace

Since E (S(n)) = np and Var(S(n)) = np(1− p) we get thefollowing:

Corollary

(de Moivre–Laplace central limit theorem)If X1,X2, . . . are Bernoulli with common parameter p, then

limn→∞

P

(a ≤ S(n)− np√

np(1− p)≤ b

)=

1√2π

∫ b

a

exp

(−1

2z2

)dz



de Moivre–Laplace

Since E (S(n)) = np and Var(S(n)) = np(1− p) we get thefollowing:

Corollary

(de Moivre–Laplace central limit theorem)If X1,X2, . . . are Bernoulli with common parameter p, then

limn→∞

P

(a ≤ S(n)− np√

np(1− p)≤ b

)=

1√2π

∫ b

a

exp

(−1

2z2

)dz



Continuity correction

Because the binomial distribution is discrete, while the normaldistribution is continuous, we can get greater accuracy in theCLT by using a continuity correction.

Assume we are trying to approximate a binomial randomvariable X ∼ Bin(n, p) by a normal random variable Y ; byCorollary 36 we will have Y ∼ N(np, np(1− p)).

Because X takes integer values, it in fact makes sense to usethe value of Y rounded to the nearest integer; call this Y .

So then we approximate P(X = x) by P(Y = x), which isequal to P(x − 0.5 < Y ≤ x + 0.5).
























Continuity correction cont.

This implies that we should use the following approximationsto various probabilities involving the binomial:

P(X ≤ x) ≈ P(Y ≤ x + 0.5),

P(X < x) ≈ P(Y ≤ x − 0.5),

P(x1 ≤ X ≤ x2) ≈ P(x1 − 0.5 < Y ≤ x2 + 0.5).





P(X ≤ x) ≈ P(Y ≤ x + 0.5),

P(X < x) ≈ P(Y ≤ x − 0.5),

P(x1 ≤ X ≤ x2) ≈ P(x1 − 0.5 < Y ≤ x2 + 0.5).





P(X ≤ x) ≈ P(Y ≤ x + 0.5),

P(X < x) ≈ P(Y ≤ x − 0.5),

P(x1 ≤ X ≤ x2) ≈ P(x1 − 0.5 < Y ≤ x2 + 0.5).



Example

Example

A woman claims to have psychic abilities, in that she canpredict the outcome of a coin toss. If she is tested 100 times,but she is really just guessing, what is the probability that shewill be right 60 or more times?



Application to finance

Imagine that we are keeping track of the price of a stock,which we will label St at time t.

Assume time here is measured in some small units such asseconds.

It is usual to think of the change in price from time t − 1 to tmultiplicatively, so that we can write St = St−1Rt , where Rt isa random variable.

(One advantage of thinking in this way is that as long as theRt are not negative the modelled stock price will not gonegative.)
























Evolution of prices

As a starting point, assume that the Rt are independent andidentically distributed, and that S0 is known.

We thus can write

St = S0

t∏k=1

Rt .

Taking logs, we have

log St = log S0 +t∑

k=1

logRt .



Evolution of prices


We thus can write

St = S0

t∏k=1

Rt .



k=1

logRt .



Evolution of prices


We thus can write

St = S0

t∏k=1

Rt .



k=1

logRt .



CLT

Assume that E (logRt) = µ and Var(logRt) = σ2.

Then, by the Central Limit Theorem, if t is large we expectlog St to have an approximately normal distribution with meanS0 + tµ and variance tσ2.

A random variable S such that log S has a normal distributionhas a lognormal distribution, so this argument suggests that St

should have an approximately lognormal distribution.



CLT







CLT







Model

In fact, a standard model for stock prices, which you will meetif you take later courses in finance, assumes that the prices St

behave as what is known as geometric Brownian motion,which implies that St has exactly a lognormal distribution,with the mean and variance of log St being as suggested in theprevious paragraph.


Documents

MAS113 Introduction to Probability and Statisticsjonathanjordan.staff.shef.ac.uk › IntroPS › slides8.pdfMoment Generating Functions The Central Limit Theorem (CLT) MAS113 Introduction