Information theory Information theory in the in the

Modern Information SocietyModern Information Society

A.J. Han VinckUniversity of Duisburg/Essen

January 2003Vinck@exp-math.uni-essen.de

contentcontent

What is Information theory ?

Why is it important ?

Where do we find it ?

Claude Elwood Shannon: April 30, 1916 - February 24, 2001

Shannon (1948) , Information theory, The Mathematical theory of CommunicationShannon (1948) , Information theory, The Mathematical theory of Communication

What is Information theory What is Information theory about ?about ?

Information:Information: knowledge that can be usedknowledge that can be used

Communication:Communication: exchange of Information exchange of Information

Our goal:Our goal: efficient; reliable; secureefficient; reliable; secure

Express everything in 0 and 1

Discrete ensemble:

a,b,c,d 00, 01, 10, 11

in general: k binary digits specify 2k

messages

Analogue signal:

1) sample and 2) represent sample value binary

11

10

01

00t

v

Output

00, 10, 01, 01, 11

Shannon‘s contributionsShannon‘s contributions

Modeling:Modeling: how to go from analogue to digital• fundamental communication models

Bounds:Bounds: how far can we go?• achievability• impossibility

Constructions:Constructions: constructive communication methods

• with optimum performance

and many more!!!and many more!!!

1011

R

P

efficient:efficient: general problem statement

remove redundancy exact, no errors !!

remove irrelevance distortion !!

Topics: how ? how good ?how fast ? how

complex ?+ + +

efficient: efficient: text

represent every symbol with 8 bit

1 book: 8 * (500 pages) * 1000 symbols = 4 Mbit 1 book

compression possible to 1 Mbit (1:4) 

efficient: efficient: speech

sampling speed 8000 samples/sec; accuracy 8 bits/sample;

speed 64 kBit/s;

45 minutes lecture = 45*60*64k =180Mbit 45 books

     compression possible to 4.8 kBit/s (1:10)

efficient: efficient: CD music

sampling speed 44.1 k samples/sec; accuracy 16 bits/sample

storage capacity for one hour stereo: 5 Gbit 1250 books

compression possible to 4 bits/sample ( 1:4 ) 

efficient: efficient: digital pictures

300 x 400 pixels x 3 colors x 8 bit/sample

        2.9 Mbit/picture; for 25 images/second we need 75 Mb/s

2 hour pictures need 540 Gbit 130.000 books

       compression needed (1:100)

efficient: summaryefficient: summarytext:

1 book storage: = 4 Mbit 1 book

speech:

45 minutes lecture = 45*60*64k =180Mbit 45 books

CD music:

storage capacity for one hour stereo: 5 Gbit 1250 books

 

digital pictures:  

2 hour pictures need 540 Gbit 130.000 books     

efficient: general idea• represent likely symbols with short length binary words

where likely is derived from

- prediction of next symbol in source output

- context between the source symbols

words sounds context in pictures

q qu

q-ue, q-ua, q-ui, q-uo

Morse

efficient: applications

Text: Zip; GIF etc.

Music: MP3

Pictures: JPEG, MPEG

Contributors in data reduction/compression:

Information theorists: A. Lempel and Jacob Ziv

: Huffman a.m.m.

efficient: example JPEG

15.817MB 4.566MB 3.267MB 2.351MB

Secure:Secure: example 1 A wants to send a present to B in a safe way.

USER A USER B send box with open lock to A send locked box to B B has private key

Lock closes without key!

Problem: Is B the owner of the open lock?

Secure:Secure: classical A wants to send a present to B in a safe way.

USER A USER B send key to B send locked box to B B has key

Lock closed

Problem: Is the key present at B?

Secure:Secure: example 2Diffie Hellman key exchange. common paint + + + secret color A + secret color B = = assume that mixture separation is expensive public transport + + + secret color A + secret color B = =

common secret

Reliable:Reliable:

Transmit

0 or 1

Receive

0 or 1

0 0 correct

0 1 in - correct

1 1 correct

1 0 in - correct

What What can can we we do do about about it ?it ?

Reliable:Reliable: 2 examples2 examples

Transmit

A: = 0 0

B: = 1 1

Receive

0 0 or 1 1 OK

0 1 or 1 0 NOK1 error detected!A: = 0 0 0

B: = 1 1 1

000, 001, 010, 100 A

111, 110, 101, 011 B

1 error corrected!

Error Sensitivity: Illustration

Error sensitivity: 0.0001=0.01% Error sensitivity: 0.0005=0.05%

Optical StorageOptical Storage

•DVD's seven-fold increase in data capacity over the CD has been largely achieved by tightening up the tolerances throughout the predecessor system

•The data structure was made more efficient by using a better, more efficient error correction code system.

Errors in networkingErrors in networking

no- commentno- comment

a meshed structure

3 links down

partial

FundamentalFundamental problems to consider

fast re-routing of information

how to include redundancy ?

how much redundancy?

Cost versus reliabilityCost versus reliability

The success story Qualcomm CDMAThe success story Qualcomm CDMA

Founding information theorists:

Irwin Jacobs and Andrew Viterbi

narrow-band and broad-band noise

SOLUTIONSOLUTION: FREQUENCY and TIME DIVISION

1

2

3

time

frequency

PPM Code Example

• 6 code words: 123 231 312 132 213 321

• distance: = 2

1

2

3

Code Division Multiple Access (CDMA)

• 6 users: 123 231 312 132 213 321– 1 2 3 4 5 6

• 2 transmit at the same time

1

2

32/3 2/3 2/3

5/6 5

1/2 2/4 2/4

4/5

2/5 2 2

Why IT at this Why IT at this university?university?

• It is fundamental. The theory is well established

• Based on– Discrete Mathematics; algorithms– Physics

• Applications:– Communications; networking; Computer science– Multi-media; medical imaging; biology; languages– Information retrieval; information control

Other application:

powerline communications

Information TheoryInformation Theory

In 1948, Bell Labs scientist Claude Shannon developed Information Theory, and the world of communications

technology has never been the same.