Asymptotic behaviour of the marginal likelihood integral ... · BIC and RLCT q-ﬁbers of GMMs Asymptotics for GMMs The case of zero covariances Asymptotic behaviour of the marginal

BIC and RLCT q-fibers of GMMs Asymptotics for GMMs The case of zero covariances

Asymptotic behaviour of the marginal likelihood integralfor general Markov models

Piotr ZwiernikIPAM −→ TU Eindhoven

(With special thanks to Shaowei Lin.)

Singular Learning Theory,AIM, Palo Alto, 13 Dec 2011

1 / 22Asymptotic behaviour of the marginal likelihood integral for general Markov models


Outline of the talk

The asymptotic behavior of the marginal likelihood integral andthe real log-canonical threshold.

The real log-canonical threshold for general Markov models.

q-fibers for (binary) general Markov models.

Polyhedral geometry for the deepest singularity.



Discrete algebraic statistical model

random variable: X ∈ X , |X | <∞, p = (px)x∈X

probability simplex: ∆|X |−1 = p ∈ RX :∑

x∈X px = 1, px ≥ 0

model: p : Θ→ ∆|X |−1,M = p(Θ), p polynomial map

the true distribution of X: q ∈ ∆|X |−1



Asymptotics of the marginal likelihood

random sample: X(N) = (X1, . . . ,XN)

marginal likelihood: ZN =∫

ΘL(θ; X(N))ψ(θ)dθ

stochastic complexity: FN = − log ZN ; entropy:S = −∑x∈X qx log qx

Bayesian Information Criterion (BIC)

If p−1(q) = θ lies in the interior of Θ and the Jacobian of p has fullrank then, as N →∞,

EFN = NS +d2

log N + O(1).



The general case (S. Watanabe)

Kullback-Leibler distance: K(θ) =∑m

i=1 qi log qipi(θ)

K(θ) ≥ 0 on ΘK(θ) = 0 only if p(θ) = q

zeta function on C: ζ(z) =∫

ΘK(θ)−zψ(θ)dθ

real log-canonical threshold: rlctΘ(K;ψ) is the smallest pole of ζits multiplicity: multΘ(K;ψ)

Theorem

With some compactness assumptions, as N →∞ thenEFN = NS + rlctΘ(K;ψ) log N + (multΘ(K;ψ)− 1) log log N + O(1)



Asymptotics and q-fibers

RLCTΘ(K) = minθ∈Θ0 RLCTΘ0(K)

important distinction: if W0 neighbourhood of θ0 in Rd thenRLCTW0 (K) = RLCTθ0 (K)

RLCTΘ(K(θ)) = RLCTΘ(∑

x∈X (px(θ)− qx)2)

q-fiber: Θ = p−1(q)

θ0 /∈ Θ =⇒ rlctθ0(∑

x∈X (px(θ)− qx)2) =∞



The binary tripod tree model

b

bc

b

b

1

2

3h

1

X1 ⊥⊥ X2 ⊥⊥ X3|H or

a Bayesian network a tripod tree

seven free parameters

the codimension is zero



Central moment parametrization

MT :

µ12 = 14(1−s2) η1η2,

µ13 = 14(1−s2) η1η3,

µ23 = 14(1−s2) η2η3,

µ123 = 14(1−s2)s η1η2η3



Finite q-fibers

µ2123 + 4µ12µ13µ23 = ( 1

4 (1− s2)sη1η2η3)2 + 4( 14 (1− s2))3(η1η2η3)2

= ( 14 (1− s2)η1η2η3)2(s2 + 1− s2) = ( 1

4 (1− s2)η1η2η3)2

µ2123

µ2123+4µ12µ13µ23

=( 1

4 (1−s2)sη1η2η3)2

( 14 (1−s2)η1η2η3)2 = s2

µ2123+4µ12µ13µ23

µ223

=( 1

4 (1−s2)η1η2η3)2

( 14 (1−s2)η2η3)2 = η2

1 , and so on

well defined for q ∈MT such that µ12µ13µ23 6= 0



Submodels and singularities

(draw four models,show that the fiber for X1 ⊥⊥ X2 ⊥⊥ X3 is a union of affine spaces)



Sketch picture

b

b

b

b

b

µ12 = µ13 = 0

µ12 = µ23 = 0

µ13 = µ23 = 0

δ2 = 1η1 = 0η2 = 0

η3 = 0

ΩT

MT



This generalizes

(X1, . . . ,Xn) ∈ 0, 1n represented by leaves of T



The general case

let q ∈MT and Σ = [µij]i,j∈[n] the covariance matrix

the asymptotics of EFN determined by zeros in Σ (marginalindependencies).

b

bc b

b

b

b

b

b

b

b

b

bc

bcbc

bc

bc

bc

bc

T1

T2

T3

S1

S2

S3

1



General formula

Theorem [Z.]

if p−1(q) is a manifold with corners (there are no degree zeronodes) then, as N →∞,

EFN = N S +1 + 2|E| − 2l2

2log N + O(1).

for trivalent trees

EFN = N S +

(1 + 2|E| − 2l2

2− 5l0

4

)log N − c log log N + O(1),

where c is a nonnegative integer. Moreover c = 0 always if eitherboth r is nondegenerate or if r and all its neighbors aredegenerate.



Step 1: Reparametrization

tree cumulants: κ = (κI)I⊆[n], I 6=∅, ω = ((sv)v∈V , (ηe)e∈E)

ΘT

fθω

p// ∆2n−1

fpκ

ΩT

fωθ

OO

ψT // KT

fκp

OO

κI(ω) = 14 (1− s2

r(I))∏

v∈N(I) sdeg(v)−2v

∏e∈E(I) ηe, for I ⊆ [n], |I| ≥ 2.

RLCTΘT (∑

x

(px(θ)− qx)2) = (

n2, 0) + RLCTΩT (

∑|I|≥2

(κI(ω)− κI)2)



Step 2: The main reduction

b

bc b

b

b

b

b

b

b

b

b

bc

bcbc

bc

bc

bc

bc

T1

T2

T3

S1

S2

S3

1

note: if T is trivalent then T1, . . . ,Tk are trivalent



The deepest singularity

let κij = 0 for all i, j ∈ [n] (then κI = 0 for all I ⊆ [n])

RLCTω0(∑

I(κI(ω)− κI)2) = RLCTω0(

∑i,j∈[n] κ

2ij(ω)) for ω0 ∈ Ω.

the q-fiber is given by a union of affine subspaces withnon-empty common intersection denoted by Ωdeep

RLCTω0(K) ≤ RLCTω(K) for every ω ∈ Ω and ω0 ∈ Ωdeep



The monomial case

if ω0 ∈ Ωdeep then

RLCTω0(∑

i,j∈[n] κ2ij(ω)) = RLCT0(

∑i,j∈[n] m2

ij(ω)),

where mij(ω) = sr(ij)∏

e∈E(ij) ηe

e.g. (quartet)

m12 = saηa1ηa2, m13 = saηa1ηabηb3 and m34 = sbηb3ηb4



The Newton diagram method

Theorem

Let f (x) =∑α cαx2α and Γ+ = conv(2α : cα6=0) + Rn

≥0.

Then RLCT0(f ) = ( 1t , c), where and t be the smallest such that

(t, . . . , t) ∈ Γ+ and c is the codimension of the face hit by(t, . . . , t).



rlct from the Newton diagram

coordinates of the ambient space: (xe), (yv)

distinguised facet:∑

e∈Etermxe ≥ 4

show rlct0(∑

ij m2ij) = n

4 by:

note t < 4n then (t, . . . , t) /∈ Γ+

constructing a point P ∈ Γ+ such that P ≤ 4n 1

idea: for n = 41

2

4

5

3

a b

1

2

4

5

3

a b

2 · (2, 0; 2, 2, 0, 0, 0) and 2 · (0, 2; 0, 0, 0, 2, 2) givesP = 1

4 (4, 4; 4, 4, 0, 4, 4) ≤ (1, 1; 1, 1, 1, 1, 1).



rlct from the Newton diagram 2

for n = 5

1

2

4

5

3

a b

v

6

7

1

2

4

5

3

a b

v

6

7

gives 15 (4, 4, 2; 4, 4, 0, 4, 4, 4, 4) ≤ 4

5 1.



Conclusions and remarks

Understanding of q-fibers is essential

In our case the degenerate cases corresponded to graphicalsubmodels

Tree remains important even for the Newton diagram method.

generalization to Gaussian models


Documents

Asymptotic behaviour of the marginal likelihood integral ... · BIC and RLCT q-ﬁbers of GMMs Asymptotics for GMMs The case of zero covariances Asymptotic behaviour of the marginal