Introduction to Information Theory

1

Claude E. Shannon 2

3

A General Communication SystemA General Communication System

4

Shannon’s Information TheoryShannon s Information Theory

• Conceptualization of information & modeling• Conceptualization of information & modeling of information sourcesS di f i f ti th h l• Sending of information across the channel:What are the limits on the amount of information th t b t?that can be sent?What is the effect of noise on this communication. 5

A General Communication SystemA General Communication System

6

A General Communication SystemA General Communication System

CHANNEL

• Information Source• Transmitter• Channel• Receiver• Destination

7

“Th f d l bl f i i“The fundamental problem of communication is that of reproducing at one point either 

l i l l dexactly or approximately a message selected at another point.”

From: A Mathematical Theory of Communication, Shannon, 1948.

8

Examples of Communication SystemsExamples of Communication Systems

9

Motivating Noise…Motivating Noise…

XTransmittedSymbol

YReceivedSymbol

0 0

Symbol Symbol1 1

10

Motivating Noise…Motivating Noise…

XTransmittedSymbol

YReceivedSymbol

0 0

Symbol Symbol1 1

P(Y=1|X=0) = fP(Y=0|X=1) = f

P(Y=0|X=0) = 1 ‐ fP(Y=1|X=1) = 1 ‐ f

11

Motivating Noise…Motivating Noise…

f 0 1 10 000f = 0.1, n = ~10,000

1 f0 0

1 ‐ f

f

1 11 ‐ f

12

QuestionQuestion

How can we achieve perfect communication over anHow can we achieve perfect communication over an imperfect, noisy communication channel?

Use more reliable components; Stabilize the environment; Use larger areas; Use power/cooling to reduce thermal noise.

These are all costly solutions.

13

Alternately…Alternately…

How can we achieve perfect communication over an imperfect, p p ,noisy communication channel?

Accept that there will be noise Accept that there will be noise Add error detection and correction Introduce the concepts of ENCODER/DECODER

Information TheoryTheoretical limitations of such systemsTheoretical limitations of such systems

Coding TheoryCreation of practical encoding/decoding systemsCreation of practical encoding/decoding systems

14

Alternately…Alternately…

How can we achieve perfect communication over an imperfect, p p ,noisy communication channel?

Accept that there will be noise Accept that there will be noise Add error detection and correction Introduce the concepts of ENCODER/DECODER

REDUNDANCY IS KEY!

Information TheoryTheoretical limitations of such systemsTheoretical limitations of such systems

Coding TheoryCreation of practical encoding/decoding systemsCreation of practical encoding/decoding systems

15

Shannon’s Insight:Shannon s Insight:

High Reliability → Low Transmission Rateg e ab ty→ o a s ss o ate

I.e. Perfect reliability → Zero Transmission Ratey

For a given level of noise there is an associated rate gof transmission that can be achieved with arbitrarily good reliability.

e.g. Sending a lone T versus THIS…

16

Meaning? What meaning?

“Frequently the messages havemeaning; that is

g g

Frequently the messages have meaning; that is they refer to or are correlated according to some system with certain physical or conceptual y p y pentities.

These semantic aspects of communication are irrelevant to the engineering problem.”g g p

From: A Mathematical Theory of Communication Shannon 1948From: A Mathematical Theory of Communication, Shannon, 1948.

17

What is information?What is information?

• Just the physical aspects Shannon 1948• Just the physical aspects…Shannon, 1948• The General Definition of Information…Floridi, 2010.

GDI) σ is an instance of information, understood as semantic content, if and only if:

GDI.1) σ consists of n data, for n ≥ 1;GDI.2) the data are well formed;

) h ll f d d f lGDI.3) the well‐formed data are meaningful.

18

What is information?What is information?

• Just the physical aspects Shannon 1948• Just the physical aspects…Shannon, 1948• The General Definition of Information…Floridi, 2010.

GDI) σ is an instance of information, understood as semantic content, if and only if:

GDI.1) σ consists of n data, for n ≥ 1;GDI.2) the data are well formed;

) h ll f d d f lGDI.3) the well‐formed data are meaningful.

19

Shannon Information is…Shannon Information is…

• UncertaintyCan be measured by counting the number of possible messages.

• SurpriseSome messages are more likely than others.

• DifficultWhat is significant is the difficulty of transmitting the message from one point to the other.

E t• EntropyA fundamental measure of information.

20

Understanding EntropyUnderstanding Entropy

l f d !It is sunny in California today!21

Understanding EntropyUnderstanding Entropy Information is quantified using probabilities. Given a finite set of possible messages, associate a probability with 

each message. A message with low probability represents more information than g p y p

one with high probability.

Definition of Information:Definition of Information:

I = log(1/p) ≡ ‐log(p)

Where p is the probability of the messageBase 2 is used for the logarithm so I is measured in bits

22

Example: Information in a coin flipExample: Information in a coin flip

P(HEADS) = ½

I = ‐log(½) = 1 bit

23

Example: Text AnalysisExample: Text Analysis

a 0.06428b 0.01147c 0.02413d 0.03188e 0.10210f 0.01842

0 01543

ab

c

dSPC

g 0.01543h 0.04313i 0.05767j 0.00082k 0.00514l 0.03338m 0.01959

e

fu

vw

x yz

n 0.05761o 0.06179p 0.01571q 0.00084r 0.04973s 0.05199t 0 07327

g

ht

u

t 0.07327u 0.02201v 0.00800w 0.01439x 0.00162y 0.01387z 0.00077

i

j

k

lm

nop

r

s

24

SPC 0.20096o

q

Example Text AnalysisExample Text Analysis

25

Example Text AnalysisExample Text AnalysisLetter          Freq.       I, h(pi)

a 0.06428 3.95951b 0.01147 6.44597c 0.02413 5.37297d 0.03188 4.97116e 0.10210 3.29188f 0.01842 5.76293g 0.01543 6.01840h 0.04313 4.53514i 0.05767 4.11611j 0.00082 10.24909k 0.00514 7.60474l 0.03338 4.90474m 0.01959 5.67385m 0.01959 5.67385n 0.05761 4.11743o 0.06179 4.01654p 0.01571 5.99226q 0.00084 10.21486r 0.04973 4.32981s 0 05199 4 26552s 0.05199 4.26552t 0.07327 3.77056u 0.02201 5.50592v 0.00800 6.96640w 0.01439 6.11899x 0.00162 9.26697

26

y 0.01387 6.17152z 0.00077 10.34877

SPC 0.20096 2.31502

Definition of EntropyDefinition of Entropy

Information (I) is associated with knownInformation (I) is associated with known events/messagesEntropy (H) is the average information w r toEntropy (H) is the average information w.r.to all possible outcomes

27

Example Text AnalysisExample Text AnalysisLetter          Freq.       I, h(pi)

a 0.06428 3.95951b 0.01147 6.44597c 0.02413 5.37297d 0.03188 4.97116e 0.10210 3.29188f 0.01842 5.76293g 0.01543 6.01840h 0.04313 4.53514i 0.05767 4.11611j 0.00082 10.24909k 0.00514 7.60474l 0.03338 4.90474m 0.01959 5.67385n 0.05761 4.11743o 0.06179 4.01654p 0.01571 5.99226q 0.00084 10.21486r 0.04973 4.32981s 0.05199 4.26552t 0.07327 3.77056u 0.02201 5.50592v 0.00800 6.96640w 0.01439 6.11899x 0.00162 9.26697y 0.01387 6.17152

28

y 0.01387 6.17152z 0.00077 10.34877

SPC 0.20096 2.31502

Entropy (2 outcomes)Entropy (2 outcomes)

29

Entropy: PropertiesEntropy: Properties

Entropy is maximized if p is uniform. 30

ReferencesReferences• Eugene Chiu, Jocelyn Lin, Brok Mcferron, Noshirwan Petigara, Satwiksai

Seshasai: Mathematical Theory of Claude Shannon: A study of the style and t t f hi k t th i f i f ti th MIT 6 933J /context of his work up to the genesis of information theory. MIT 6.933J / 

STS.420J The Structure of Engineering Revolutions• Luciano Floridi, 2010: Information: A Very Short Introduction, Oxford 

University Press, 2011.• Luciano Floridi, 2011: The Philosophy of Information, Oxford University Press, 

2011.• James Gleick, 2011: The Information: A History, A Theory, A Flood, Pantheon 

Books 2011Books, 2011.• David Luenberger, 2006: Information Science, Princeton University Press, 

2006.• David J.C. MacKay, 2003: Information Theory, Inference, and Learning 

Algorithms, Cambridge University Press, 2003.• Claude Shannon & Warren Weaver, 1949: The Mathematical Theory of 

Communication, University of Illinois Press, 1949.• W N Francis and H Kucera: Brown University Standard Corpus of Present‐DayW. N. Francis and H. Kucera: Brown University Standard Corpus of Present Day 

American English, Brown University, 1967.

31

32