# Nonparametric estimation of - Department of ... Nonparametric estimation of s concave and log-concave densities: alternatives to maximum likelihood Jon A. Wellner University of Washington,

• View
2

0

Embed Size (px)

### Text of Nonparametric estimation of - Department of ... Nonparametric estimation of s concave and...

• Nonparametric estimation of

s−concave and log-concave densities:

alternatives to maximum likelihood

Jon A. Wellner

University of Washington, Seattle

Cowles Foundation Seminar, Yale University

November 9, 2016

• Cowles Foundation Seminar, Yale University

Based on joint work with:

• Qiyang (Roy) Han • Charles Doss • Fadoua Balabdaoui • Kaspar Rufibach • Arseni Seregin

• Outline

• A: Log-concave and s−concave densities on R and Rd

• B: s−concave densities on R and Rd

• C: Maximum Likelihood for log-concave and s−concave densities

B 1: Basics

B 2: On the model

B 3: Off the model

• D. An alternative to ML: Rényi divergence estimators

B 1. Basics

B 2. On the model

B 3. Off the model

• E. Summary: problems and open questions

Cowles Foundation Seminar, Yale, November 9, 2016 1.2

• A. Log-concave densities on R and Rd

If a density f on Rd is of the form

f(x) ≡ fϕ(x) = exp(ϕ(x)) = exp (−(−ϕ(x))) where ϕ is concave (so −ϕ is convex), then f is log-concave. The class of all densities f on Rd of this form is called the class of log-concave densities, Plog−concave ≡ P0.

Properties of log-concave densities:

• Every log-concave density f is unimodal (quasi concave).

• P0 is closed under convolution.

• P0 is closed under marginalization.

• P0 is closed under weak limits.

• A density f on R is log-concave if and only if its convolution with any unimodal density is again unimodal (Ibragimov, 1956).

Cowles Foundation Seminar, Yale, November 9, 2016 1.3

• • Many parametric families are log-concave, for example:

B Normal (µ, σ2)

B Uniform(a, b)

B Gamma(r, λ) for r ≥ 1

B Beta(a, b) for a, b ≥ 1

• tr densities with r > 0 are not log-concave.

• Tails of log-concave densities are necessarily sub-exponential.

• Plog−concave = the class of “Polyá frequency functions of order 2”, PF2, in the terminology of Schoenberg (1951) and

Karlin (1968). See Marshall and Olkin (1979), chapter 18,

and Dharmadhikari and Joag-Dev (1988), page 150. for nice

introductions.

Cowles Foundation Seminar, Yale, November 9, 2016 1.4

• B. s− concave densities on R and Rd

If a density f on Rd is of the form

f(x) ≡ fϕ(x) =

 (ϕ(x))1/s, ϕ convex, if s < 0 exp(−ϕ(x)), ϕ convex, if s = 0 (ϕ(x))1/s, ϕ concave, if s > 0,

then f is s-concave.

The classes of all densities f on Rd of these forms are called the classes of s−concave densities, Ps. The following inclusions hold: if −∞ < s < 0 < r

• Properties of s-concave densities:

• Every s−concave density f is quasi-concave.

• The Student tν density, tν ∈ Ps for s ≤ −1/(1 + ν). Thus the Cauchy density (= t1) is in P−1/2 ⊂ Ps for s ≤ −1/2.

• The classes Ps have interesting closure properties under convolution and marginalization which follow from the Borell-Brascamp-Lieb inequality: let 0 < λ < 1, −1/d ≤ s ≤ ∞, and let f, g, h : Rd → [0,∞) be integrable functions such that

h(λx+ (1− λ)y) ≥Ms(f(x), g(y);λ) for all x, y ∈ Rd

where

Ms(a, b;λ) = (λa s + (1− λ)bs)1/s, M0(a, b;λ) = aλb1−λ.

Then∫ Rd h(x)dx ≥Ms/(sd+1)

(∫ Rd f(x)dx,

∫ Rd g(x)dx;λ

) .

Cowles Foundation Seminar, Yale, November 9, 2016 1.6

• Special Case: s = 0: Prékopa-Leindler inequality.

Let 0 < λ < 1 and let f, g, h : Rd → [0,∞). if

h(λx+ λy) ≥ f(x)λg(y)1−λ for all x, y ∈ Rd,

then ∫ Rd h(x)dx ≥

(∫ Rd f(x)dx

)λ (∫ Rd g(x)dx

)1−λ .

Proposition. (a) Marginalization preserves log-concavity.

(b) Convolution preserves log-concavity.

Proof. (a) Suppose that h̃(x, y) is a log-concave density on

Rm × Rn. Thus

h̃(λ(x, y) + (1− λ)(x′, y′)) ≥ h̃(x, y)λh̃(x′, y′)1−λ.

Cowles Foundation Seminar, Yale, November 9, 2016 1.7

• We want to show that

f̃(y) ≡ ∫ Rm

h̃(x, y)dx is log-concave.

With h(x) ≡ h̃(x, λy + λy′), f(x) ≡ h̃(x, y), g(x) ≡ h̃(x, y′),

h(λx+ λx′) = h̃(λx+ λx′, λy + λy′)

≥ h̃(x, y)λh̃(x′, y′)λ = f(x)λg(x′)λ.

Thus Prékopa-Leindler yields

f̃(λy + (1− λ)y′) ≥ (∫

Rm h̃(x, y)dx

)λ (∫ Rm

h̃(x, y′)dx )1−λ

= f̃(y)λf̃(y′)1−λ.

(b) If X ∼ f and Y ∼ g are independent, f, g log-concave, then (X,Y ) ∼ f ·g ≡ h is log-concave. Since log-concavity is preserved under affine transforms (X−Y,X+Y ) has a log-concave density. By (a), X + Y is log-concave. �

Cowles Foundation Seminar, Yale, November 9, 2016 1.8

• • If f ∈ Ps and s > −1/(d+ 1), then Ef‖X‖ −1/d, then ‖f‖∞ 0 and b ∈ R

f(x) ≤ exp(−a‖x‖+ b) for all x ∈ R.

• If f ∈ Ps and s > −1/d, then with r ≡ −1/s > d, then for some a, b > 0

f(x) ≤ (b+ a‖x‖)−r for all x ∈ R.

• If s < −1/d then there exists f ∈ Ps with ‖f‖∞ =∞.

• If −1/d < s ≤ −1/(d + 1), then there exists f ∈ Ps with Ef‖X‖ =∞.

Cowles Foundation Seminar, Yale, November 9, 2016 1.9

• -4 -2 2 4 0.02

0.04

0.06

0.08

0.10

f(x) = 1

√ rBeta(r/2,1/2)

( r

r + x2

)(1+r)/2 ,

with r = .05, so f ∈ P−1/(1+.05)

Cowles Foundation Seminar, Yale, November 9, 2016 1.10

• -4 -2 2 4 0.05

0.10

0.15

0.20

0.25

0.30

tr densities, r ∈ {0.1,0.2, . . . ,1.0}

Cowles Foundation Seminar, Yale, November 9, 2016 1.11

• -3 -2 -1 0 1 2 3

1

2

3

4

f(x) = 1

2Γ(1/2) |x|−1/2exp(−|x|), with f ∈ P−2.

Cowles Foundation Seminar, Yale, November 9, 2016 1.12

• C. Maximum Likelihood:

0-concave and s-concave densities

MLE of f and ϕ: Let C denote the class of all concave function ϕ : R→ [−∞,∞). The estimator ϕ̂n based on X1, . . . , Xn i.i.d. as f0 is the maximizer of the “adjusted criterion function”

`n(ϕ) = ∫

logfϕ(x)dFn(x)− ∫ fϕ(x)dx

=

 ∫ ϕ(x)dFn(x)−

∫ eϕ(x)dx, s = 0,∫

(1/s)log(−ϕ(x))+dFn(x)− ∫

(−ϕ(x))1/s+ dx, s < 0,

over ϕ ∈ C.

Cowles Foundation Seminar, Yale, November 9, 2016 1.13

• 1. Basics

• The MLE’s for P0 exist and are unique when n ≥ d+ 1.

• The MLE’s for Ps exist for s ∈ (−1/d,0) when

n ≥ d (

r

r − d

) where r = −1/s. Thus n→∞ as −1/s = r ↘ d.

• Uniqueness of MLE’s for Ps?

• MLE ϕ̂n is piecewise affine for −1/d < s ≤ 0.

• The MLE for Ps does not exist if s < −1/d. (Well known for s = −∞ and d = 1.)

Cowles Foundation Seminar, Yale, November 9, 2016 1.14

• 2. On the model

• The MLE’s are Hellinger and L1− consistent.

• The log-concave MLE’s f̂n,0 satisfy∫ ea|x||f̂n,0(x)− f0(x)|dx→a.s. 0.

for a < a0 where f0(x) ≤ exp(−a0|x|+ b0).

• The s−concave MLE’s are computationally awkward; log is “too aggressive” a transform for an s−concave density. [Note that ML has difficulties even for location t− families: multiple roots of the likelihood equations.]

• Pointwise distribution theory for f̂n,0 when d = 1; no pointwise distribution theory for f̂n,s when d = 1; no pointwise distribution theory for f̂n,0 or f̂n,s when d > 1.

• Global rates? H(f̂n,s, f0) = Op(n−2/5) for −1 < s ≤ 0, d = 1.

Cowles Foundation Seminar, Yale, November 9, 2016 1.15

• 3. Off the model Now suppose that Q is an arbitrary probability measure on Rd with density q and X1, . . . , Xn are i.i.d. q.

• The MLE f̂n for P0 satisfies:∫ Rd |f̂n(x)− f∗(x)|dx→a.s. 0

where, for the Kullback-Leibler divergence

K(q, f) = ∫ qlog(q/f)dλ,

f∗ = argminf∈P0(Rd)K(q, f)

is the “pseudo-true” density in P0(Rd) corresponding to q. In fact: ∫

Rd ea‖x‖|f̂n(x)− f∗(x)|dx→a.s. 0

for any a < a0 where f ∗(x) ≤ exp(−a0‖x‖+ b0).

Cowles Foundation Seminar, Yale, November 9, 2016 1.16

• • The MLE f̂n for Ps does not behave well off the model. Retracing the basic arguments of Cule and Samworth (2010)

leads to negative conclusions. (How negative remains to be

pinned down!)

Conclusion: Investigate alternative methods for estimation in

the larger classes Ps with s < 0! This leads to the proposals by Koenker and Mizera (2010).

Cowles Foundation Seminar, Yale, November 9, 2016 1.17

• D. An alternative to ML:

Rényi divergence estimators

0. Notation and Definitions

• β = 1 + 1/s < 0, α−1 + β−1 = 1.

• C(X) = all continuous functions on conv(X).

• C∗(X) = all signed Radon measures on C(X) = dual space of C(X).

• G(X) ##### Nonparametric estimation of multivariate elliptic densities via hbattey/BLvAccepted20130821c.pdf · PDF file 2019-03-03 · Nonparametric estimation of multivariate elliptic densities
Documents ##### Nonparametric Estimation of a Nonseparable Demand Function ... uctp39a/BHP_NonSep_February_26_2016.pdf · PDF file Nonparametric Estimation of a Nonseparable Demand Function under
Documents ##### Nonparametric Estimation: Part II - Stanford University yjhan/nonparametric... · PDF file Nonparametric Estimation: Part II Regression in Transformed Space Yanjun Han Department
Documents ##### NONPARAMETRIC ESTIMATION OF CONCAVE ... ... 2016/10/19  · JOURNAL OF APPLIED ECONOMETRICS J. Appl. Econ. 22: 795–816 (2007) Published online 13 March 2007 in Wiley InterScience
Documents ##### Nonparametric Covariance Function Estimation for ... my2550/papers/covfunest.pdfNonparametric Covariance Function Estimation for Functional and ... we consider nonparametric covariance
Documents ##### Nonparametric estimation of diffusions: a differential equations
Documents ##### Nonparametric Structural Estimation via Continuous Location Shifts
Documents ##### Nonparametric Identification and Estimation of - UCLA Economics
Documents ##### Nonparametric estimation of the incubation time distribution
Documents ##### Kaplan, 1958. Nonparametric Estimation From Incomplete Observations
Documents Documents