Andrea Montanari and Ruediger UrbankeTIFR

Tuesday, January 6th, 2008

Phase Transitions in Coding, Communications, and Inference

Outline

1) Thresholds in coding, the large size limit

 (definition and density evolution characterization)            

2) The inversion of limits (length to infty vs size to infty)                                     3) Phase transitions in measurements                     (compressed sensing versus message passing,  dense versus sparse matrices)

4) Phase transitions in collaborative filtering          (the low-rank matrix model)

Model

Shannon ’48

binary symmetric channel

capacity: R≤1-h(ε)

binary erasures channel

capacity: R≤1-ε

Channel Coding

code

decoding

C={000, 010, 101, 111}

n ... blocklength

xMAP(y)=argmaxX in C p(x | y)

xiMAP(y)=argmaxXi p(xi |y)

Factor Graph Representation of Linear Codes

(7, 4) Hamming code

every linear code

Tanner, Wiberg, Koetter, Loeliger, Frey

parity-check matrix

Low-Density Parity Check Codes

(3, 4)-regular codes

Gallager ‘60

number of edges is linear in n

Ensemble

Variations on the Theme

irregular LDPC ensembleregular RA ensembleirregular MN ensembleirregular RA ensembleARA ensembleturbo code

degree distributions as well as structure

protographirregular LDGM ensemble

(Luby, Mitzenmacher, Shokrollahi, Spielman, and Stehman)Divsalar, Jin, and McEliece Jin, Khandekar, and McEliece Abbasfar, Divsalar, KungBerrou and GlavieuxThorpe, Andrews, DolinarDavey, MacKay

Message-Passing Decoding -- BEC

?

?

00

0

?

?

?

0+?0+? =??

0

0

?

?

?

??

0=00?

?

0

0

0

?0

?

decoded

decoded

0+00+0 =00

Message-Passing Decoding -- BSCGallager Algorithm

Asymptotic Analysis: Computation Graph

probability that computation graphof fixed depth becomes tree

tends to 1 as n tends to infinity

Asymptotic Analysis: Density Evolution -- BEC

x

1-(1-x)r-1

x x

ε (1-(1-x)r-1)l-1

ε

Luby,Mitzenmacher, Shokrollahi,

Spielman, and Steman ‘97

Asymptotic Analysis: Density Evolution -- BEC

ε

phase transition: εBP so that xt → 0 for ε< εBP

xt → x∞>0 for ε> εBP

Asymptotic Analysis: Density Evolution -- BSC, Gallager Algorithm

phase transition: εBP so that xt → 0 for ε< εBP

xt → x∞>0 for ε> εBP

xt =ε (1-p+(xt-1))+(1-ε) p-(xt-1)

p+(x)=((1+(1-2x)r-1)/2) l-1 p-(x)=((1-(1-2x)r-1)/2) l-1

Asymptotic Analysis: Density Evolution -- BP

Inversion of Limits

size versus number of iterations

Density Evolution Limit

Density Evolution Limit

“Practical” Limit

“Practical” Limit

The Two Limits

Easy: (Density Evolution Limit)

Hard(er): (“Practical Limit”)

Binary Erasure Channel

DE Limit

“Practical” Limit

implies

What about “General” Case

expansion

probabilistic methods

Korada and U.

Expansion

Miller and Burshtein: Random element of LDPC(l, r, n) ensemble is expander with

expansion close to 1-1/l with high probability

expansion ~ 1-1/l

Why is Expansion Useful?

Setting: Channel

Setting: Ensemble

Setting: Algorithm

Aim: Show for this setting that ...

DE Limit

“Practical” Limit

implies

Proof Outline

linearize the algorithm combine with density evolution correlation and interaction witness randomizing noise outside the

witness sub-critical birth and death

process

Linearized Decoding Algorithm

Proof Outline

linearize the algorithm combine with density evolution correlation and interaction witness randomizing noise outside the

witness sub-critical birth and death

process

Combine with Density Evolution

Proof Outline

linearize the algorithm combine with density evolution correlation and interaction witness randomizing noise outside the

witness sub-critical birth and death

process

Correlation and Interaction

0 1

1 000Expected growth:

(r-1) 2 ε?< 1

Problem: interaction correlation

(r-1)

2 ε

Correlation and Interaction

Proof Outline

linearize the algorithm combine with density evolution correlation and interaction witness randomizing noise outside the

witness sub-critical birth and death

process

Witness

Witness

Witness

Proof Outline

linearize the algorithm combine with density evolution correlation and interaction witness randomizing noise outside the

witness sub-critical birth and death

process

Monotonicity

Randomizing the Noise Outside

randomizing noise outside the witness increases the probability of error

FKG

→

←⁄

≤

Proof Outline

linearize the algorithm combine with density evolution correlation and interaction witness randomizing noise outside the

witness sub-critical birth and death

process

Expansion

random graph has expansion close to expansion of a treewith high probability

⇒this limits interaction

0 1

1 000

References

For a list of references see:http://ipg.epfl.ch/doku.php?id=en:courses:2007-2208:mct

Results

Open Problems

0.0

0.4

0.3

0.2

0.1

0.2 0.4 0.6 0.8

Pb

channel entropy