Central Limit Theorem - Dartmouth College

Central Limit Theorem

§ number of measurements needed to estimate the mean

§ convergence of the estimated mean § a better approximation than the

Chebyshev Inequality (LLN).

Xingru [email protected]

XC 2020


𝑋! 𝑋"

𝑋# 𝑋$

𝑋% ⋯

𝑋& 𝑋'normal

density function

+

+

+

+

+

XC 2020


𝑋! 𝑋"

𝑋# 𝑋$

𝑋% ⋯

𝑋& 𝑋'normal

density function

++

+

+

+

If 𝑆' is the sum of 𝑛 mutually independent random variables, then the distribution function of 𝑆' is well-approximated by a normal density function.

XC 2020

CENTRAL LIMIT THEOREM FOR BERNOULLI TRIALS

binomial distribution

XC 2020

Bernoulli Trials

A Bernoulli trials process is a sequence of 𝑛 chance experiments such that

§ Each experiment has two possible outcomes, which we may call success and failure.

§ The probability 𝑝 of success on each experiment is the same for each experiment, and this probability is not affected by any knowledge of previous outcomes. The probability 𝑞 of failure is given by 𝑞 = 1 − 𝑝.

50%Toss a Coinhead & tail

60%Weather Forecast

not rain & rain

12.5%Wheel of Fortune50 points & others

𝒑Bernoulli Trialsoutcome 1 & 2

XC 2020

Binomial Distribution

§ Let 𝑛 be a positive integer and let 𝑝 be a real number between 0 and 1.

§ Let 𝐵 be the random variable which counts the number of successes in a Bernoulli trials process with parameters 𝑛 and 𝑝.

§ Then the distribution 𝑏(𝑛, 𝑝, 𝑘) of 𝐵 is called the binomial distribution.


𝑏 𝑛, 𝑝, 𝑘 =𝑛𝑘𝑝(𝑞')(

1 2 3 4 5

✗ ✓ ✗ ✗ ✓

XC 2020


§ Consider a Bernoulli trials process with probability 𝑝 for success on each trial.

§ Let 𝑋* = 1 or 0 according as the 𝑖th outcome is a success or failure.

§ Let 𝑆' = 𝑋& + 𝑋% +⋯+ 𝑋'. Then 𝑆' is the number of successes in 𝑛 trials.

§ We know that 𝑆' has as its distribution the binomial probabilities 𝑏(𝑛, 𝑝, 𝑘).

XC 2020

𝑛 increases

drifts off to the right

flatten out

XC 2020

The Standardized Sum of 𝑆4





§ The standardized sum of 𝑆' is given by

𝑆'∗ =,!)'-'-.

.

𝜇 = ⋯ 𝜎% = ⋯

XC 2020







𝑆'∗ =,!)'-'-.

.

§ 𝑆'∗ always has expected value 0 and variance 1.𝜇 = 0

𝜎% = 1

XC 2020


=𝑆'∗ =𝑆' − 𝑛𝑝𝑛𝑝𝑞

𝑏(𝑛, 𝑝, 1)𝑏(𝑛, 𝑝, 2)

𝑏(𝑛, 𝑝, 3)𝑏(𝑛, 𝑝, 4)

𝑏(𝑛, 𝑝, 5)

𝑏(𝑛, 𝑝, 0)

⋯𝑏(𝑛, 𝑝, 𝑛)

𝑏(𝑛, 𝑝, 𝑘)

0 1 2 3 4 5 … 𝑛

0 − 𝑛𝑝𝑛𝑝𝑞






… 𝑛 − 𝑛𝑝𝑛𝑝𝑞

𝑘

𝑥XC 2020

𝑆'∗ =𝑆' − 𝑛𝑝𝑛𝑝𝑞

𝜇 = 0

𝜎% = 1

XC 2020

standardized sum

standard normal distribution

shapes: sameheights: different

why?

XC 2020

standardized sum standard normal distribution

shapes: sameheights: different why?

standardized sum: ∑(/0' ℎ* = 1

standard normal distribution: ∫)1

21𝑓 𝑥 𝑑𝑥 = 1

XC 2020

standardized sum: ∑(/0' ℎ* = 1

standard normal distribution: ∫)1

21𝑓 𝑥 𝑑𝑥 = 1 ≈ ∑(/0' 𝑓 𝑥* 𝑑𝑥

C(/0

'

ℎ* = 1 C(/0

'

𝑓 𝑥* 𝑑𝑥 ≈ 1

=ℎ( = 𝑓 𝑥( 𝑑𝑥

=𝑓 𝑥( =ℎ(𝑑𝑥

XC 2020

=ℎ) = 𝑏(𝑛, 𝑝, 𝑘)=𝑆'∗ =𝑆' − 𝑛𝑝𝑛𝑝𝑞 =𝑑𝑥 = 𝑥)*& − 𝑥) =

1𝑛𝑝𝑞

𝑏(𝑛, 𝑝, 1)𝑏(𝑛, 𝑝, 2)

𝑏(𝑛, 𝑝, 3)𝑏(𝑛, 𝑝, 4)

𝑏(𝑛, 𝑝, 5)

𝑏(𝑛, 𝑝, 0)

⋯𝑏(𝑛, 𝑝, 𝑛)

0 1 2 3 4 5 … 𝑛








=𝑘 = 𝑛𝑝𝑞𝑥) + 𝑛𝑝 𝑎 : the integer nearest to 𝑎

XC 2020

ℎ( = 𝑏(𝑛, 𝑝, 𝑘)

𝑑𝑥 = 𝑥(2& − 𝑥( =1𝑛𝑝𝑞

𝑘 = 𝑛𝑝𝑞𝑥( + 𝑛𝑝

=𝑓 𝑥( = 3"45= 𝑛𝑝𝑞 𝑏(𝑛, 𝑝, 𝑛𝑝𝑞𝑥( + 𝑛𝑝 )

XC 2020

Central Limit Theorem for Binomial Distributions

§ For the binomial distribution 𝑏(𝑛, 𝑝, 𝑘) we have

lim'→1

𝑛𝑝𝑞𝑏 𝑛, 𝑝, 𝑛𝑝 + 𝑥 𝑛𝑝𝑞 = 𝜙(𝑥),

where 𝜙(𝑥) is the standard normal density.

§ The proof of this theorem can be carried out using Stirling’s approximation.

Stirling’s FormulaThe sequence 𝑛! is asymptotically equal

to 𝑛'𝑒)' 2𝜋𝑛.

XC 2020

Approximating Binomial Distributions

=lim'→1

𝑛𝑝𝑞𝑏 𝑛, 𝑝, 𝑛𝑝 + 𝑥 𝑛𝑝𝑞 = 𝜙(𝑥)

?𝑏 𝑛, 𝑝, 𝑘 ≈ ⋯

normalbinomial

binomial normal

XC 2020


=lim'→,

𝑛𝑝𝑞𝑏 𝑛, 𝑝, 𝑛𝑝 + 𝑥 𝑛𝑝𝑞 = 𝜙(𝑥) ?𝑏 𝑛, 𝑝, 𝑘 ≈ ⋯

𝑘 = 𝑛𝑝 + 𝑥 𝑛𝑝𝑞

𝑥 =𝑘 − 𝑛𝑝𝑛𝑝𝑞

𝑏 𝑛, 𝑝, 𝑘 ≈𝜙(𝑥)𝑛𝑝𝑞

𝑏(𝑛, 𝑝, 𝑘) ≈1𝑛𝑝𝑞

𝜙(𝑘 − 𝑛𝑝𝑛𝑝𝑞

)

XC 2020

Example

Toss a coin

Find the probability of exactly 55 heads in 100 tosses of a coin.

=lim'→,




)

XC 2020

Example

Toss a coin




)𝑛 = 100 𝑝 =12

𝑛𝑝 = 50 𝑛𝑝𝑞 = 5

𝑘 = 55

𝑥 =𝑘 − 𝑛𝑝𝑛𝑝𝑞 = 1 𝑏(100,

12, 55) ≈

15𝜙(1)

XC 2020

Example

Toss a coin


𝑏(𝑛, 𝑝, 𝑘) ≈1𝑛𝑝𝑞 𝜙(

𝑘 − 𝑛𝑝𝑛𝑝𝑞 )

𝑏 100,12, 55 ≈

15𝜙 1 =

15(12𝜋

𝑒)&/%)

Standard normal distribution 𝒁

𝜇 = 0 and 𝜎 = 1

𝑓8 𝑥 = &%9:

𝑒) 5); #/%:# = &%9𝑒)5#/%.

XC 2020

𝑏(𝑛, 𝑝, 1)𝑏(𝑛, 𝑝, 2)

𝑏(𝑛, 𝑝, 3)𝑏(𝑛, 𝑝, 4)

𝑏(𝑛, 𝑝, 5)

𝑏(𝑛, 𝑝, 0)

⋯𝑏(𝑛, 𝑝, 𝑛)

0 1 2 3 4 5 … 𝑛








A specific outcome

XC 2020

𝑏(𝑛, 𝑝, 1)𝑏(𝑛, 𝑝, 2)

𝑏(𝑛, 𝑝, 3)𝑏(𝑛, 𝑝, 4)

𝑏(𝑛, 𝑝, 5)

𝑏(𝑛, 𝑝, 0)

⋯𝑏(𝑛, 𝑝, 𝑛)

0 1 2 3 4 5 … 𝑛








Outcomes in an Interval

XC 2020

Central Limit Theorem for Binomial Distributions

§ Let 𝑆' be the number of successes in n Bernoulli trials with probability 𝑝 for success, and let 𝑎 and 𝑏 be two fixed real numbers.

§ Then

lim'→21

𝑃 𝑎 ≤ ,!)'-'-.

≤ 𝑏 = ∫<=𝜙 𝑥 𝑑𝑥 = NA(𝑎, 𝑏).

§ We denote this area by NA(𝑎, 𝑏).

𝑎 ≤𝑆' − 𝑛𝑝𝑛𝑝𝑞

= 𝑆'∗ ≤ 𝑏 𝑎 𝑛𝑝𝑞 + 𝑛𝑝 ≤ 𝑆' ≤ 𝑏 𝑛𝑝𝑞 + 𝑛𝑝

XC 2020

lim'→21

𝑃 𝑎 ≤ ,!)'-'-.

≤ 𝑏 = NA(𝑎, 𝑏).

𝑎 ≤𝑆' − 𝑛𝑝𝑛𝑝𝑞

= 𝑆'∗ ≤ 𝑏 𝑎 𝑛𝑝𝑞 + 𝑛𝑝 ≤ 𝑆' ≤ 𝑏 𝑛𝑝𝑞 + 𝑛𝑝

𝑖 ≤ 𝑆' ≤ 𝑗 𝑎 =𝑖 − 𝑛𝑝𝑛𝑝𝑞

≤ 𝑆'∗ ≤𝑗 − 𝑛𝑝𝑛𝑝𝑞

= 𝑏

lim'→21

𝑃 𝑖 ≤ 𝑆' ≤ 𝑗 = lim'→21

𝑃 *)'-'-.

≤ 𝑆'∗ ≤>)'-'-.

= NA(*)'-'-.

, >)'-'-.

).


XC 2020

𝑖 ≤ 𝑆' ≤ 𝑗 𝑎 =𝑖 − 𝑛𝑝𝑛𝑝𝑞

≤ 𝑆'∗ ≤𝑗 − 𝑛𝑝𝑛𝑝𝑞

= 𝑏

lim'→21

𝑃 𝑖 ≤ 𝑆' ≤ 𝑗 = lim'→21

𝑃 *)'-'-.

≤ 𝑆'∗ ≤>)'-'-.

= NA(*)'-'-.

, >)'-'-.

).

Approximating Binomial Distributions (more accurate)

lim@→BC

𝑃 𝑖 ≤ 𝑆@ ≤ 𝑗 = NA(DE!"E@F

@FG,HB!"E@F

@FG).

𝑖 𝑗

XC 2020

Example

Toss a coin

A coin is tossed 100 times. Estimate the probability that the number of heads lies between 40 and 60 (including the end points).

=lim'→21

𝑃 𝑖 ≤ 𝑆' ≤ 𝑗 = NA(𝑖 − 12 − 𝑛𝑝

𝑛𝑝𝑞 ,𝑗 + 12 − 𝑛𝑝

𝑛𝑝𝑞 )

XC 2020

Toss a coin


𝑛 = 100 𝑝 =12

𝑛𝑝 = 50 𝑛𝑝𝑞 = 5

𝑖 = 40

𝑖 − 12 − 𝑛𝑝𝑛𝑝𝑞

= −2.1

𝑗 = 60

𝑗 + 12 − 𝑛𝑝𝑛𝑝𝑞

= 2.1

the outcome will not deviate by more than two standard deviations from the expected value

XC 2020

Toss a coin


𝑖 = 40

𝑖 − 12 − 𝑛𝑝𝑛𝑝𝑞 = −2.1

𝑗 = 60

𝑗 + 12 − 𝑛𝑝𝑛𝑝𝑞 = 2.1

=lim$→&'

𝑃 𝑖 ≤ 𝑆$ ≤ 𝑗 = NA(𝑖 − 12 − 𝑛𝑝

𝑛𝑝𝑞,𝑗 + 12 − 𝑛𝑝

𝑛𝑝𝑞)

=lim$→&'

𝑃 40 ≤ 𝑆$ ≤ 60 = NA −2.1,2.1= 2NA 0, 2.1

2𝜎 2.1𝜎

XC 2020

Approximating Binomial Distribution

𝑺𝒏∗ = 𝒙lim$→'


𝑺𝒏 = 𝒌

𝑏(𝑛, 𝑝, 𝑘) ≈1𝑛𝑝𝑞 𝜙(

𝑘 − 𝑛𝑝𝑛𝑝𝑞 )

𝒂 ≤ 𝑺𝒏∗ ≤ 𝒃

lim'→*,

𝑃 𝑎 ≤ -!.'/'/0

≤ 𝑏 = NA(𝑎, 𝑏).

𝒊 ≤ 𝑺𝒏 ≤ 𝒋

lim$→&'

𝑃 𝑖 ≤ 𝑆$ ≤ 𝑗 = NA(()!")$*

$*+,,&!")$*

$*+).

CLT

XC 2020

CENTRAL LIMIT THEOREM FOR DISCRETE INDEPENDENT TRIALS

any independent trials process such that the individual trials have finite variance

XC 2020


§ Consider an independent trials process with common distribution function 𝑚 𝑥defined on the integers, with expected value 𝜇 and variance 𝜎%.

§ Let 𝑆' = 𝑋& + 𝑋% +⋯+ 𝑋' be the sum of 𝑛 independent discrete random variables of the process.


𝑆'∗ =,!)';':#

.

§ 𝑆'∗ always has expected value 0 and variance 1.

𝐸(𝑆') = ⋯

𝑉(𝑆') = ⋯

XC 2020

independent trials process

𝑃(𝑆' = 𝑘)

𝑘 𝑥)=𝑘 − 𝑛𝜇𝑛𝜎%

=𝑆'∗ =𝑆' − 𝑛𝜇𝑛𝜎%

normal

𝑛𝜎-𝑃(𝑆$ = 𝑘)

XC 2020

Approximation Theorem

§ Let 𝑋&, 𝑋%, ⋯, 𝑋' be an independent trials process and let 𝑆' = 𝑋& + 𝑋% +⋯+ 𝑋'.

§ Assume that the greatest common divisor of the differences of all the values that the 𝑋( can take on is 1.

§ Let 𝐸 𝑋( = 𝜇 and 𝑉 𝑋( = 𝜎%.

§ Then for 𝑛 large,

𝑃(𝑆' = 𝑘) ≈ &':#

𝜙(()';':#

).

§ Here 𝜙(𝑥) is the standard normal density.

XC 2020

Central Limit Theorem for a Discrete Independent Trials Process

§ Let 𝑆' = 𝑋& + 𝑋% +⋯+ 𝑋' be the sum of n discrete independent random variables with common distribution having expected value 𝜇 and variance 𝜎%.

§ Then, for 𝑎 < 𝑏,

lim'→21

𝑃 𝑎 ≤ ,!)';':#

≤ 𝑏 = &%9∫<

= 𝑒)5#/%𝑑𝑥 = NA(𝑎, 𝑏).

XC 2020

Approximating a Discrete Independent Trials Process

𝑃(𝑆' = 𝑘) ≈1𝑛𝜎%

𝜙(𝑘 − 𝑛𝜇𝑛𝜎%

)

lim'→*,

𝑃 𝑐 ≤ 𝑆' ≤ 𝑑 =

NA(1.'2'3"

, 4.'2'3"

).

𝑺𝒏 = 𝒌

𝒄 ≤ 𝑺𝒏 ≤ 𝒅

CLT

XC 2020

Approximating a Discrete Independent Trials Process

lim'→*,

𝑃 𝑖 ≤ 𝑆' ≤ 𝑗 =

NA(5.#".'/

'/0,6*#".'/

'/0).

𝒊 ≤ 𝑺𝒏 ≤ 𝒋

𝑃(𝑆' = 𝑘) ≈1𝑛𝜎%

𝜙(𝑘 − 𝑛𝜇𝑛𝜎%

)

𝑺𝒏 = 𝒌



)

𝑺𝒏 = 𝒌

lim'→*,

𝑃 𝑐 ≤ 𝑆' ≤ 𝑑 =

NA(1.'2'3"

, 4.'2'3"

).

𝒄 ≤ 𝑺𝒏 ≤ 𝒅

Bernoulli

XC 2020

ExampleA surveying instrument makes an error of -2, -1, 0, 1, or 2 feet with equal probabilities when measuring the height of a 200-foot tower.

XC 2020

Find the expected value and the variance for the height obtained using this instrument once.

Example

height = error + 200

error -2 -1 0 1 2

probability15

15

15

15

15

𝐸 error =15−2 − 1 + 0 + 1 + 2 = 0

𝑉 error =15(−2)%+(−1)%+0% + 1% + 2% = 2

𝐸 height= 0 + 200 = 200

𝑉 height = 2

XC 2020

Estimate the probability that in 18 independent measurements of this tower, the average of the measurements is between 199 and 201, inclusive.

Example

𝜇 = 200 𝜎% = 2

=lim'→21

𝑃 𝑐 ≤ 𝑆' ≤ 𝑑 = NA(𝑐 − 𝑛𝜇𝑛𝜎%

,𝑑 − 𝑛𝜇𝑛𝜎%

).

199 ≤𝑆'𝑛=𝑆'18

≤ 201𝑛 = 18

XC 2020

Example

𝜇 = 200 𝜎% = 2

=lim'→*,

𝑃 𝑐 ≤ 𝑆' ≤ 𝑑 = NA(𝑐 − 𝑛𝜇𝑛𝜎%

,𝑑 − 𝑛𝜇𝑛𝜎%

).

199 ≤𝑆'𝑛=𝑆'18

≤ 201𝑛 = 18

𝑃 199 ≤𝑆'18

≤ 201 = 𝑃 3582 ≤ 𝑆' ≤ 3618

≈ NA3582 − 3600

36,3618 − 3600

36= NA(−3, 3).

XC 2020

03

01

02

§ independent

Discrete Random Variables

§ independent§ identical

Bernoulli Trials

§ independent§ identical

Discrete Trials


XC 2020


§ Let 𝑋&, 𝑋%, ⋯, 𝑋' be a sequence of independent discrete random variables. There exist a constant 𝐴, such that 𝑋* ≤ 𝐴 for all 𝑖.

§ Let 𝑆' = 𝑋& + 𝑋% +⋯+ 𝑋' be the sum. Assume that 𝑆' → ∞.

§ For each 𝑖, denote the expected value and variance of 𝑋* by 𝜇* and 𝜎*%, respectively.

§ Define the expected value and variance of 𝑆' to be 𝑚' and 𝑠'%, respectively.

§ For 𝑎 < 𝑏,

lim'→21

𝑃 𝑎 ≤ ,!)@!A!

≤ 𝑏 = &%9∫<

= 𝑒)5#/%𝑑𝑥 = NA(𝑎, 𝑏).

XC 2020

CENTRAL LIMIT THEOREM FOR CONTINUOUS INDEPENDENT TRIALScontinuous random variables with a common density function

XC 2020


§ Let 𝑆' = 𝑋& + 𝑋% +⋯+ 𝑋' be the sum of 𝑛 independent continuous random variables with common density function 𝑝 having expected value 𝜇 and variance 𝜎%.

§ Let

𝑆'∗ =,!)';':#

.

§ Then we have, for all 𝑎 < 𝑏,

lim'→21

𝑃 𝑎 ≤ 𝑆'∗ ≤ 𝑏 = &%9∫<

= 𝑒)5#/%𝑑𝑥 = NA(𝑎, 𝑏).

XC 2020

Example

§ Suppose a surveyor wants to measure a known distance, say of 1 mile, using a transit and some method of triangulation.

§ He knows that because of possible motion of the transit, atmospheric distortions, and human error, any one measurement is apt to slightly in error.

§ He plans to make several measurements and take an average.

§ He assumes that his measurements are independent random variables with common distribution of mean 𝜇 = 1 and standard deviation 𝜎 =0.0002.

XC 2020

Example

§ He can say that if 𝑛 is large, the average ,!

'has a density function that

is approximately normal, with mean 1mile, and standard deviation 0.000%

'miles.

§ How many measurements should he make to be reasonably sure that his average lies within 0.0001 of the true value?

XC 2020

Example

𝜇 = 1

Expected Value

𝜎 = 0.0002

Standard Deviation𝑆' = 𝑋& + 𝑋% +⋯+ 𝑋'

Sum of 𝒏 independent measurements

§ He can say that if 𝑛 is large, the average ,!' has a density function that is approximately normal, with mean 1 mile, and standard deviation 0.000%

'miles.

𝜎U = (0.0002)UVariance 𝐴' =

𝑋& + 𝑋% +⋯+ 𝑋'𝑛

𝐸 𝐴' = 𝜇𝐷 𝐴' =

𝜎𝑛

𝑉 𝐴' =𝜎%

𝑛

Average of 𝒏 independent measurements

XC 2020

Example

Chebyshev inequality

LLN

𝑆'∗ =𝑆' − 𝑛𝜇𝑛𝜎%

CLT


XC 2020

𝝈𝟐

𝜺𝟐

𝑷(|𝑿 − 𝝁|≥ 𝜺)

inequalityChebyshev𝑷(|𝑿 − 𝝁| ≥ 𝜺) ≤

𝝈𝟐

𝜺𝟐

This is a sample text. Insert your desired text

here.

This is a sample text. Insert your desired text here.

Let 𝑋 be a discrete random variable with expected value 𝜇 = 𝐸(𝑋) and variance 𝜎% = 𝑉(𝑋), and let 𝜀 > 0 be any positive number. Then

not necessarily positive

XC 2020

Example


LLN


𝜇 = 1 𝜎% = (0.0002)%

=𝑃(|𝑋 − 𝜇| ≥ 𝜀) ≤𝜎%

𝜀%

𝑃 𝐴' − 𝜇 ≥ 0.0001 ≤𝜎%

𝑛𝜀%=

0.0002 %

𝑛 0.0001 % =4𝑛

XC 2020

Example


LLN


=𝑃(|𝑋 − 𝜇| ≥ 𝜀) ≤𝜎%

𝜀%

𝑃 𝐴' − 𝜇 ≥ 0.0001 ≤𝜎%

𝑛𝜀%=4𝑛= 0.05 𝑛 = 80

XC 2020

Example


CLT

=lim'→21

𝑃 𝑎 ≤ 𝑆'∗ ≤ 𝑏 = NA(𝑎, 𝑏)

𝑃 𝐴' − 𝜇 ≤ 0.0001 = 𝑃 𝑆' − 𝑛𝜇 ≤ 0.5𝑛𝜎

= 𝑃𝑆' − 𝑛𝜇𝑛𝜎%

≤ 0.5 𝑛 = 𝑃 −0.5 𝑛 ≤ 𝑆'∗ ≤ 0.5 𝑛

≈ NA −0.5 𝑛, 0.5 𝑛

𝜇 = 1 𝜎% = (0.0002)%

XC 2020

Example


CLT

=lim'→21

𝑃 𝑎 ≤ 𝑆'∗ ≤ 𝑏 = NA(𝑎, 𝑏)

𝑃 𝐴' − 𝜇 ≤ 0.0001 ≈ NA −0.5 𝑛, 0.5 𝑛 = 0.95

0.5 𝑛 = 2 𝑛 = 16

XC 2020

Example


LLN


CLT


𝑛 = 80 𝑛 = 16

XC 2020

Documents

Central Limit Theorem - Dartmouth College