Statistics and Econometrics I
Asymptotic Theory
Shiu-Sheng Chen
Department of Economics
National Taiwan University
September 13, 2016
Shiu-Sheng Chen (NTU Econ) Statistics and Econometrics I September 13, 2016 1 / 30
Asymptotic Theory: Motivation
Asymptotic theory or large sample theory aims at answering the
question: what happens as we gather more and more data?
In particular, given random sample, {X1, X2, X3, . . . , Xn}, and
statistic:
Tn = t(X1, X2, . . . , Xn),
what is the limiting behavior of Tn as n −→∞?
Shiu-Sheng Chen (NTU Econ) Statistics and Econometrics I September 13, 2016 2 / 30
Asymptotic Theory: Motivation
Why asking such a question?
For instance, given random sample {Xi}ni=1 ∼i.i.d. N(µ, σ2), we know
that
X̄n ∼ N(µ,σ2
n
)However, if {Xi}ni=1 ∼i.i.d. (µ, σ2), what is the distribution of X̄n?
I We don’t know, indeed.
Is it possible to find a good approximation of the distribution of X̄n
as n −→∞?
Shiu-Sheng Chen (NTU Econ) Statistics and Econometrics I September 13, 2016 3 / 30
Part I
Preliminary Knowledge
Shiu-Sheng Chen (NTU Econ) Statistics and Econometrics I September 13, 2016 4 / 30
Preliminary Knowledge
Preliminary Knowledge
Limit
Markov Inequality
Chebyshev Inequality
Shiu-Sheng Chen (NTU Econ) Statistics and Econometrics I September 13, 2016 5 / 30
Preliminary Knowledge
Limit of a Real Sequence
Definition (Limit)
If for every ε > 0, and an integer N(ε),
|bn − b| < ε, ∀ n > N(ε)
then we say that a sequence of real numbers {b1, . . . , bn} converges to a
limit b.
It is denoted by
limn→∞
bn = b
Shiu-Sheng Chen (NTU Econ) Statistics and Econometrics I September 13, 2016 6 / 30
Preliminary Knowledge
Markov Inequality
Theorem (Markov Inequality)
Suppose that X is a random variable such that P (X ≥ 0) = 1. Then for
every real number m > 0,
P (X ≥ m) ≤ E(X)
m
Shiu-Sheng Chen (NTU Econ) Statistics and Econometrics I September 13, 2016 7 / 30
Preliminary Knowledge
Chebyshev Inequality
Theorem (Chebyshev Inequality)
Let Y ∼ (E(Y ), V ar(Y )). Then for every number ε > 0,
P(∣∣Y − E(Y )
∣∣ ≥ ε) ≤ V ar(Y )
ε2
Proof: Let X = [Y − E(Y )]2, then
P (X ≥ 0) = 1
and
E(X) = V ar(Y )
Hence, the result can be derived by applying the Markov Inequality.
Shiu-Sheng Chen (NTU Econ) Statistics and Econometrics I September 13, 2016 8 / 30
Part II
Modes of Convergence
Shiu-Sheng Chen (NTU Econ) Statistics and Econometrics I September 13, 2016 9 / 30
Modes of Convergence
Types of Convergence
For a random variable, we consider three modes of convergence:
Converge in Probability
Converge in Distribution
Converge in Mean Square
Shiu-Sheng Chen (NTU Econ) Statistics and Econometrics I September 13, 2016 10 / 30
Modes of Convergence
Converge in Probability
Definition (Converge in Probability)
Let {Yn} be a sequence of random variables and let Y be another random
variable. For any ε > 0,
P (|Yn − Y | < ε) −→ 1, as n −→∞
then we say that Yn converges in probability to Y , and denote it by
Ynp−→ Y
Equivalently,
P (|Yn − Y | ≥ ε) −→ 0, as n −→∞
Shiu-Sheng Chen (NTU Econ) Statistics and Econometrics I September 13, 2016 11 / 30
Modes of Convergence
Converge in Probability
{Xi}ni=1 ∼i.i.d. Bernoulli(0.5) and then compute Yn = X̄n =∑
iXi
n
In this case, Ynp−→ 0.5
Total Flips
X
0 200 400 600 800 1000
0.2
0.4
0.6
0.8
1.0
Shiu-Sheng Chen (NTU Econ) Statistics and Econometrics I September 13, 2016 12 / 30
Modes of Convergence
Converge in Distribution
Definition (Converge in Distribution)
Let {Yn} be a sequence of random variables with distribution function
FYn(y), (denoted by Fn(y) for simplicity). Let Y be another random
variable with distribution function, FY (y). If
limn→∞
Fn(y) = FY (y) at all y for which FY (y) is continuous
then we say that Yn converges in distribution to Y .
It is denoted by
Ynd−→ Y
FY (y) is called the limiting distribution of Yn.
Shiu-Sheng Chen (NTU Econ) Statistics and Econometrics I September 13, 2016 13 / 30
Modes of Convergence
Converge in Mean Square
Definition (Converge in Mean Square)
Let {Yn} be a sequence of random variables and let Y be another random
variable. If
E(Yn − Y )2 −→ 0, as n −→∞.
Then we say that Yn converges in mean square to Y .
It is denoted by
Ynms−→ Y
It is also called converge in quadratic mean.
Shiu-Sheng Chen (NTU Econ) Statistics and Econometrics I September 13, 2016 14 / 30
Part III
Important Theorems
Shiu-Sheng Chen (NTU Econ) Statistics and Econometrics I September 13, 2016 15 / 30
Important Theorems
Theorems
Theorem
Ynms−→ c if and only if
limn→∞
E(Yn) = c, and limn→∞
V ar(Yn) = 0.
Proof. It can be shown that
E(Yn − c)2 = E([Yn − E(Yn)]2) + [E(Yn)− c]2
Shiu-Sheng Chen (NTU Econ) Statistics and Econometrics I September 13, 2016 16 / 30
Important Theorems
Theorems
Theorem
If Ynms−→ Y then Yn
p−→ Y
Proof: Note that P (|Yn − Y |2 ≥ 0) = 1, and by Markov Inequality,
P (|Yn − Y | ≥ k) = P (|Yn − Y |2 ≥ k2) ≤ E(|Yn − Y |2)
k2
Shiu-Sheng Chen (NTU Econ) Statistics and Econometrics I September 13, 2016 17 / 30
Important Theorems
Weak Law of Large Numbers, WLLN
Theorem (WLLN)
Given a random sample {Xi}ni=1 with σ2 = V ar(X1) <∞. Let X̄n
denote the sample mean, and note that E(X̄n) = E(X1) = µ. Then
X̄np−→ µ
Proof: (1) By Chebyshev Inequality (2) By Converge in Mean Square
Sample mean X̄n is getting closer (in probability sense) to the
population mean µ as the sample size increases.
That is, if we use X̄n as a guess of unknown µ, we are quite happy
that the sample mean makes a good guess.
Shiu-Sheng Chen (NTU Econ) Statistics and Econometrics I September 13, 2016 18 / 30
Important Theorems
WLLN for Other Moments
Note that the WLLN can be thought as∑ni=1Xi
n=X1 +X2 + · · ·Xn
n
p−→ E(X1)
Hence, ∑ni=1X
2i
n=X2
1 +X22 + · · ·X2
n
n
p−→ E(X21 )
Shiu-Sheng Chen (NTU Econ) Statistics and Econometrics I September 13, 2016 19 / 30
Important Theorems
An Application of WLLN
Example: Assume Wn ∼ Binomial(n, µ), and let Yn = Wnn . Then
Ynp−→ µ
I Why?
I Since Wn =∑iXi, Xi ∼i.i.d.Bernoulli(µ) with E(X1) = µ,
V ar(X1) = µ(1− µ), the result follows by WLLN.
Shiu-Sheng Chen (NTU Econ) Statistics and Econometrics I September 13, 2016 20 / 30
Important Theorems
An Application of WLLN
Example: Assume Wn ∼ Binomial(n, µ), and let Yn = Wnn . Then
Ynp−→ µ
I Why?
I Since Wn =∑iXi, Xi ∼i.i.d.Bernoulli(µ) with E(X1) = µ,
V ar(X1) = µ(1− µ), the result follows by WLLN.
Shiu-Sheng Chen (NTU Econ) Statistics and Econometrics I September 13, 2016 20 / 30
Important Theorems
Central Limit Theorem, CLT
Theorem (CLT)
Let {Xi}ni=1 be a random sample, where E(X1) = µ <∞,
V ar(X1) = σ2 <∞, then
Zn =X̄n − E(X̄n)√V ar(X̄n)
=
√n(X̄n − µ)
σ
d−→ N(0, 1)
If a random sample is taken from any distribution with mean µ and
variance σ2, regardless of whether this distribution is discrete or
continuous, then the distribution of the random variable Zn will be
approximately the standard normal distribution in large sample.
Shiu-Sheng Chen (NTU Econ) Statistics and Econometrics I September 13, 2016 21 / 30
Important Theorems
CLT
Using notation of asymptotic distribution,
X̄n − µ√σ2
n
∼A N(0, 1),
Or
X̄n ∼A N(µ,σ2
n
),
where ∼A represents asymptotic distribution, and A represents
Asymptotically
Shiu-Sheng Chen (NTU Econ) Statistics and Econometrics I September 13, 2016 22 / 30
Important Theorems
An Application of CLT
Example: Assume {Xi} ∼i.i.d.Bernoulli(µ), then
X̄n − µ√µ(1−µ)
n
d−→ N(0, 1).
I Why?
I Since E(X̄n) = µ, and V ar(X̄n) = σ2
n = µ(1−µ)n
Shiu-Sheng Chen (NTU Econ) Statistics and Econometrics I September 13, 2016 23 / 30
Important Theorems
Continuous Mapping Theorem
Theorem (CMT)
Given Ynp−→ Y , and g(·) is continuous, then
g(Yn)p−→ g(Y ).
Proof: omitted here.
Examples: if Ynp−→ Y , then
I 1Yn
p−→ 1Y
I Y 2n
p−→ Y 2
I√Yn
p−→√Y
Shiu-Sheng Chen (NTU Econ) Statistics and Econometrics I September 13, 2016 24 / 30
Important Theorems
Theorem
Theorem
Given Wnp−→W and Yn
p−→ Y , then
Wn + Ynp−→W + Y
WnYnp−→WY
Proof: omitted here.
Shiu-Sheng Chen (NTU Econ) Statistics and Econometrics I September 13, 2016 25 / 30
Important Theorems
Slutsky Theorem
Theorem
Given Wnd−→W and Yn
p−→ c, where c is a constant. Then
Wn + Ynd−→W + c
WnYnd−→ cW
WnYn
d−→ Wc for c 6= 0
Proof: omitted here.
Shiu-Sheng Chen (NTU Econ) Statistics and Econometrics I September 13, 2016 26 / 30
Important Theorems
The Delta Method
Theorem
Given√n(Yn − θ)
d−→ N(0, σ2). Let g(·) be differentiable, and g′(θ) 6= 0
exists, then√n(g(Yn)− g(θ))
d−→ N(0, [g′(θ)]2σ2).
Proof: (sketch) Given 1st-order Taylor approximation
g(Yn) ≈ g(θ) + g′(θ)(Yn − θ),
then √n(g(Yn)− g(θ))
g′(θ)≈√n(Yn − θ)
d−→ N(0, σ2)
Shiu-Sheng Chen (NTU Econ) Statistics and Econometrics I September 13, 2016 27 / 30
Important Theorems
Example
Given {Xi}ni=1 ∼i.i.d. (µ, σ2), find the asymptotic distribution ofX̄n
1−X̄n.
I Note that by CLT,
√n(X̄n − µ)
d−→ N(0, σ2)
I Hence, by the Delta method,
g(X̄n) =X̄n
1− X̄n, g(µ) =
µ
1− µ, g′(µ) =
1
(1− µ)2
√n
(X̄n
1− X̄n− µ
1− µ
)d−→ N
(0,
1
(1− µ)4σ2
)
Shiu-Sheng Chen (NTU Econ) Statistics and Econometrics I September 13, 2016 28 / 30
Important Theorems
Applications: Limiting Property of χ2 Random Variable
Theorem
Given Wn ∼ χ2(n), and let Xn = Wnn . Then
Xnp−→ 1
Shiu-Sheng Chen (NTU Econ) Statistics and Econometrics I September 13, 2016 29 / 30