Information Theory Basics crandall@cs.unm.edu. What is information theory? A way to quantify...

Information Theory Basics

crandall@cs.unm.edu

What is information theory?

A way to quantify information A lot of the theory comes from two worlds

Channel coding Compression

Useful for lots of other things Claude Shannon, mid- to late- 40's

Requirements

“This data will compress to at most N bits” “This channel will allow us to transmit N bits per

second” “This plaintext will require at least N bans of

ciphertext”

N is a number for the amount of information/uncertainty/entropy of a random variable X, that is, H(X) = N

???

What are the requirements for such a measure?

E.g., Continuity: changing the probabilities a small amount should change the measure by only a small amount.

Maximum

What distribution should be the maximum entropy?

For equiprobable events, what should happen if we increase the number of outcomes.

Maximum

Symmetry

The measure should be unchanged if the outcomes are re-ordered

Additivity

Amount of entropy should be independent of how we divide the process into parts.

Entropy of Discrete RVs

Expected value of the amount of information for an event

Flip a fair coin

(-0.5 lg 0.5) + (-0.5 lg 0.5) = 1.0

Flip three fair coins?

Flip three fair coins

(-0.5 lg 0.5) + (-0.5 lg 0.5) + (-0.5 lg 0.5) + (-0.5 lg 0.5) + (-0.5 lg 0.5) + (-0.5 lg 0.5) = 3.0

(-0.125 lg 0.125)+(-0.125 lg 0.125)+(-0.125 lg 0.125)+(-0.125 lg 0.125)+(-0.125 lg 0.125)+(-0.125 lg 0.125)+(-0.125 lg 0.125)+(-0.125 lg 0.125) = 3.0

Flip biased coin A

60% heads

Biased coin A

(-0.6 lg 0.6) + (-0.4 lg 0.4) = 0.970950594

Biased coin B

95% heads

(-0.95 lg 0.95) + (-0.05 lg 0.05) = 0.286396957

Why is there less information in biased coins?

Information=uncertainty=entropy

Flip A, then flip B

A: (-0.6 lg 0.6) + (-0.4 lg 0.4) = 0.970950594 B: (-0.95 lg 0.95) + (-0.05 lg 0.05) =

0.286396957 ((-0.6 lg 0.6) + (-0.4 lg 0.4)) + ((-0.95 lg 0.95) +

(-0.05 lg 0.05)) = 0.970950594 + 0.286396957 = 1.25734755

(-(0.6*0.95)lg(0.6*0.95))+(-(0.6*0.05)lg(0.6*0.05))+(-(0.4*0.95)lg(0.4*0.95))+(-(0.4*0.05)lg(0.4*0.05)) = 1.25734755

Entropy (summary)

Continuity, maximum, symmetry, additivity

Example: Maximum Entropy

Wikipedia: “Maximum-likelihood estimators can lack asymptotic normality and can be inconsistent if there is a failure of one (or more) of the below regularity conditions... Estimate on boundary, Data boundary parameter-dependent, Nuisance parameters, Increasing information...”

“Subject to known constraints, the probability distribution which best represents the current state of knowledge is the one with largest entropy.” What distribution maximizes entropy?

Beyond Entropy

Flip fair coin for X if heads flip coin A for Y if tails flip coin B for Y

H(X) = 1.0 H(Y) = (-(0.5*0.6+0.5*0.95)lg(0.5*0.6+0.5*0.95))+(-

(0.5*0.4+0.5*0.05)lg(0.5*0.4+0.5*0.05)) = 0.769192829 Joint entropy H(X,Y) = ((-(0.5 * 0.6)) * lg(0.5 * 0.6)) + ((-(0.5 *

0.95)) * lg(0.5 * 0.95)) + ((-(0.5 * 0.4)) * lg(0.5 * 0.4)) + ((-(0.5 * 0.05)) * lg(0.5 * 0.05)) = 1.62867378

Where is the other 1.769192829 – 1.62867378 = 0.140519049 bits of information?

Mutual Information

I(X;Y) = 0.140519049 I(X;Y) = H(X) + H(Y) – H(X,Y) What are H(X|Y) and H(Y|X)?

Example: sufficient statistics

Students asked to flip a coin 100 times and record the result

How to detect the cheaters?

Example: sufficient statistics

f(x) is a family of probability mass functions indexed by θ, X is a sample from a distribution in this family.

T(X) is a statistic Function of the sample, like sample mean, sample

variance, … I(θ;T(X)) ≤ I(θ;X)

Equality only if no information is lost

Kullback-Leibler divergence (a.k.a differential entropy)

Process 1: Flip unbiased coin, if heads flip biased coin A (60% heads), if tails flip biased coin B (95% heads)

Process 2: Roll a fair die. 1, 2, or 3 = (tails, heads). 4 = (heads, heads). 5 = (heads, tails). 6 = (tails, tails).

Process 3: Flip two fair coins, just record the results.

Which, out of 2 and 3, is a better approximate model of 1?

Kullback-Leibler divergence (a.k.a. Differential entropy)

P is true distribution, Q is model Dkl(P1||P2) = 0.48080 Dkl(P1||P3) = 0.37133 Note that Dkl is not symmetric

Dkl(P2||P1)=0.52898, Dkl(P3||P1)=0.61371

Conditional mutual information

I(X;Y|Z) is the expected value of the mutual information between X and Y conditioned on Z

Interaction information

I(X;Y;Z) is the information bound up in a set of variables beyond that which is present in any subset

I(X;Y;Z) = I(X;Y|Z) – I(X;Y) = I(X;Z|Y) – I(X;Z) = I(Y;Z|X) - I(Y;Z)

Negative interaction information: X is rain, Y is dark, Z is clouds

Positive interaction information: X is fuel pump blocked, Y is battery dead, Z is car starts

Other fun things you should look into if you're interested...

Writing on dirty paper Wire-tap channels Algorithmic complexity Chaitin's constant

Goldbach's conjecture, Riemann hypothesis Portfolio theory