7/30/2019 Sys 6005 Weak Lln

1/13

Topic #5:

The Weak Law of Large Numbers

Randy CogillSYS 6005 - Stochastic SystemsFall 2010

7/30/2019 Sys 6005 Weak Lln

2/13

1

Relative frequency

Suppose I have a coin that lands on heads with probability p

If I flip n times, let k denote the number of heads

Intuitively, kn p as n

However, making this notion precise is tricky

The weak law of large numbers gives one way of doing this

7/30/2019 Sys 6005 Weak Lln

3/13

2

Setting up the weak law

The weak law says the following:

For large n, kn is likely to be close to p

What do we mean by close and likely?

For any measure of closeness > 0, kn is likely to satisfy

k

np

for large enough n

For the likely part, lets precisely specify the random quantity...

7/30/2019 Sys 6005 Weak Lln

4/13

7/30/2019 Sys 6005 Weak Lln

5/13

4

Setting up the weak law (cont.)

For any > 0, P(|Zn p| ) is close to 1 for large n

Precisely, for any > 0,

limn

P(|Zn p| ) = 1

More generally, the weak law of large numbers says the following:

Let X1, . . . , X n be IID random variables

Each Xi has E[Xi] = and var(Xi) <

Let Zn =1n

ni=1 Xi

For any > 0, limnP(|Zn | ) = 1

7/30/2019 Sys 6005 Weak Lln

6/13

5

Probability bounds

Suppose we know the CDF of each Xi...

...how can we evaluate P(|Zn | )?

In general, this can get messy

Suppose instead we could find some sequence b1, b2, . . . with: P(|Zn | ) bn for all n

limn bn = 1

How de we find the bounds P(|Zn | ) bn?

Can find bounds by a surprisingly simple and powerful technique...

7/30/2019 Sys 6005 Weak Lln

7/13

6

Probability bounds (cont.)

Suppose I had functions g1 and g2 with g1(x) g2(x) for all x

For any random variable X, we have E[g1(X)] E[g2(X)]

Can get some useful bounds from clever choices of g1 and g2

7/30/2019 Sys 6005 Weak Lln

8/13

7

Markovs inequality

Suppose X is a random variable taking nonnegative values

For some given x, consider the function

g2(z) =

0 if z < x1 if z x

For this function, E[g2(X)] = P(X x)

For given x, consider the function g1(z) =1xz

Since g1(z) g2(z) for all z 0,

P(X x) 1

x

E[X]

7/30/2019 Sys 6005 Weak Lln

9/13

8

Chebyshevs inequality

Markovs inequality gives bound in terms of the mean

Chebyshevs inequality gives a related bound in terms of variance

Now suppose X can take any real value

Suppose X has mean E[X] = and variance var(X) = 2

Chebyshevs inequality will give a bound on P(|X | > )

7/30/2019 Sys 6005 Weak Lln

10/13

9

Chebyshevs inequality

For some given > 0, consider the function

g2(z) =

1 if z <

1 if z > + 0 if z +

For this function, E[g2(X)] = P(|X | > )

For given x, consider the function g1(z) =12

(z )2

Since g1(z) g2(z) for all z 0,

P(|X | > ) 2

2

7/30/2019 Sys 6005 Weak Lln

11/13

10

The Weak Law of Large Numbers

Theorem:

Let X1, . . . , X n be independent identical RVs

Each Xk has mean and variance 2

Define Zn asZn =

1

n(X1 + + Xn)

For any > 0,

limn

P(|Zn | ) = 1

7/30/2019 Sys 6005 Weak Lln

12/13

7/30/2019 Sys 6005 Weak Lln

13/13

12

Law of Large Numbers (cont.)

Proof (part 2): By Chebyshevs inequality

P(|Zn | > )

E[(Zn )2]

2

=2

n2

Therefore,

limn

P(|Zn | ) limn

1

2

n2

= 1