Information Theory

CH00-1

The instructor

• I’ve been in Taiwan for (only) two semesters• This is the third course that I teach at NCTU• I do research mostly in information theory, but I

also explore other topics

LibreOffice Productivity Suite

3

My Education

2002–2005: undergraduate studiesPolitecnico di Milano, ItalyB.A. in Computer Science

2005–2010: graduate studiesM.S. in Electrical and Computer EngineeringM.S. in StatisticsPh.D. in Electrical and Computer Engineering

Thesis Title:“Cognition and Cooperation in Wireless networks:An information theoretic perspective”

Adviser:Daniela Tuninetti

Co-Adviser:Natasha Devroye

4

2011-2012: postdoctoral student Technical University of Munich (TUM)

2012-2013: postdoctoral student Stanford University

Adviser:Gerhard Kramer

Adviser:Andrea Goldsmith

My Education

In a mapBorn Bachelor

PhD

Post Doc #1Post Doc #2 Assistant

Professor

The course is divided in two parts:

SOURCE CODING Given a source of information, how can we remove the

intrinsic redundancies, so as to obtain the most compact representation

CHANNEL CODING How do we introduce structured redundancy so as to

protect a sequence from random errors？

An overview

When Thursday13:20-16:20

Where room B07

Engineering Building Office Hours

Thursday 4pm-5pm (after class) Or email me at

stefano@nctu.edu.tw

Plus I’m always in the lab, always

The Course

Grading20 % Final Exam20 % Midterm Exam20 % Homework20% Quiz 10% Presentations 　10 % Class participation

C0-8The Course

Final Exam Midterm Exam

Both exams will be take-home exams

The exams will be distributed on Monday morning and will be due on Friday night

The exams will be personalized up to a certain extend to avoid collaboration

The exams will contain a numerical session

CH0-9The Course

Final Exam Midterm Exam

After the exam is submitted, it will be distributed to one of your collogues for comments

I will grade both the midterm and the grades

This should help you understand how hard it is to get into someone`s mind

The Course

Homework

Homework will be assigned every week on Friday and will be due the following Friday

Homework will be posted on the course webpagehttp://stefano.cm.nctu.edu.tw/it.html

Homework will be submitted electronically at tzung.cheng.chi@gmail.comYou can either scan your homework or type it

Exams will be based on the homework exercises

The Course

Quiz

The last 20mins we will have a quiz The quiz will cover the topics of the previous

lectures The quiz should help you memorize some basic

formulas. This will help in the long term (hopefully!)

The Course

Presentations

Homework will be assigned every week on Friday and will be due the following Friday

Homework will be posted on the course webpagehttp://stefano.cm.nctu.edu.tw/it.html

Homework will be submitted electronically at tzung.cheng.chi@gmail.comYou can either scan your homework or type it

Exams will be based on the homework exercises

The Course

Class participation Class participation is very, very important

in the grading Let’s get over the good reasons why you

should be asking questions

CH00

The Course

1. You understand everything so completely that questions aren't necessary. In this case you should not be taking the class

2. You are so utterly lost that you don't even know where to start asking questions. In this case you should not be taking the class

3. You don't want to embarrass yourself or keep the class back with your questions. In this case you are missing an opportunity to

learn for you and your classmates

The Course

I will understand a lot better how her presentations are received. I can then adjust her lectures to the particular audience.

The questioner may receive elucidation of a particular point and benefit from the rest of this particular lesson.

The question might be of benefit to a number of other students who had not thought of it.

I might learn something and look at things in a different light

The teacher may have made a mistake that he can now recognize and recover from.

The Course

I don’t wish to enforce participation with grades or forcing people to ask questions

I’d rather you remember this

You should take control over your learning. You have the power to obtain the information you need to be successful.

…and it will make your leaning more fun!

The Course

How about breaks, professor? The class is very long

This means that you have to digest a lot of material in a single lecture

I propose 2 breaks of 15 mins each so that you can ask questions and refresh your thoughts

I tend to be absorbed, so feel free to remind me

This is the notes that I will follow

The Book

Title Information TheoryLecture notes

AuthorsStefan M. Moser

Online http://moser-isi.ethz.ch/scripts.html

Other books are:

A First Course in Information Theory Raymond W. Yueng, Plenum PubCorp, 2002.

Elements of Information Theory, Thomas M. Cover and Joy A. Thomas,John Wiley and Sons, 1991.

Principles and Practices of Information Theory, Richard E. Blahut,Addison Wesley, 1988.

The Book

How many of you are familiar with Matlab?

Matlab

Each week one exercise will be a numerical exercise to be performed in Matlab Matlab is the industry standard for numerical

simultaions. The earlier you learn to master it, the better

I don’t think we need any particular introduction

You will be given a function template which you have to complete

Matlab

An important note: Whenever you write the code, make sure that it

can be run using the Matlab distribution made available by NCTU

The version in the school computers does not contain any package (communication package, fitting packages and so on) If I can’t run the code, the grade is zero

Matlab

I think that numerical simulations are a key way to test ideas

I will try to have use this approach as often as possible So let’s start with a warm up exercise

Matlab

ASCII table

Morse Code

Do the following exercise Think of the following problem

Given a text of a 5000/10.000 words, develop a Morse encoder then answer the following questions

How many bits are required to save the text in ASCII code and how many bits are needed to save the text in Morse code?

Call the above ratio the compression ration. Take a text in German, Italian and Spanish: how is the compression rate different in different languages?

Plot the compression ration for the different texts

Do the following exercise Now the fun part

Next assume that the bits in the Morse coded message are flipped from zero to one with a certain probability p

Using the above decoder, calculate the number of error in the reconstructed text for flipping probability in the interval [0,1/4]

Repeat the above calculation for the case in which the message is coded using ASCII code

Plot all the error probabilities

Another Exercise

Information Theory is similar to other branches of science of engineering in the following peculiarity: They were bootstrapped by the efforts of a single

individual For this reason you should read the first paper in

information theory

Claude ShannonThe mathematical theory of communication

Available online here: http://cm.bell-labs.com/cm/ms/what/shannonday/shannon1948.pdf

Another ExerciseClaude ShannonThe mathematical theory of communication

Read the paper and answer the following questions:

What kind of problems are defined as communication problem?

What kind of engineering applications did Shannon have in mind?

What kind of mathematics did he developed to study the problem at hand?