Lecture01 Intro Probability Theory (2)

7/23/2019 Lecture01 Intro Probability Theory (2)

1/61

Copyright Syed Ali Khayam 2009

EE800 Stochastic Systems

Welcome and Introduction

Dr. Muhammad Usman IlyasSchool of Electrical Engineering & Computer Science (SEECS)

National University of Sciences & Technology (NUST)


2/61


Course Information

Lecture Timings:

Tuesday: 5:00pm-6:50pm, IAEC CR# 20 Thursday: 6:00pm-7:50pm, IAEC CR# 20

My Office:

Room # A-312

Office Hours

Tuesday, 4:00-4:30pm, or by appointment.

[email protected]

The course will be managed through LMS and facebook

NUST LMS: www.lms.nust.edu.pk

Facebook group: https://www.facebook.com/groups/839067969546757

/2
mailto:[email protected]:[email protected]://www.lms.nust.edu.pk/https://www.facebook.com/groups/839067969546757/https://www.facebook.com/groups/839067969546757/https://www.facebook.com/groups/839067969546757/http://www.lms.nust.edu.pk/mailto:[email protected]


3/61


Timetable - MS EE-6 (Telecom & Comp Networks)

3

MS EE(Digital System and Signal Processing)-7

First Semester (7 Sep 2015 - 15 Jan 2016)

TIME / DAYS Monday Tuesday Wednesday Thursday Friday Saturday

5:00pm-5:50pmAdvanced Digital

System Design

Stochastic

Systems

IAEC (CR#20)Advanced Digital

System Design

Advanced Digital

Communication

Systems

IAEC Lec HallAdvanced Digital Signal

Processing

Library/Make-up

Class/Seminar6:00pm-6:50pm Advanced Digital

Communication

SystemsIAEC Lec Hall

Advanced Digital

Signal Processing Stochastic Systems

IAEC (CR#20)

7:00pm-7:50pmLibrary/Make-up

Class/Seminar

Library/Make-up

Class/Seminar

Library/Make-up

Class/Seminar8:00pm-8:50pm

Library/Make-up

Class/Seminar

Library/Make-up

Class/Seminar

MS EE(Telecommnication & Computer Networks)-7

TIME / DAYS Monday Tuesday Wednesday Thursday Friday Saturday

5:00pm-5:50pmAdv. Computer

Networks

StochasticSystems

IAEC (CR#20) Adv. Computer

Networks

Advanced Digital

CommunicationSystems

IAEC Lec Hall

Library/Make-up

Class/Seminar

Library/Make-up

Class/Seminar

6:00pm-6:50pmAdvanced Digital

Communication

Systems

IAEC Lec Hall

EMC/EMI

RIMMS CR#22

Stochastic Systems

IAEC (CR#20)7:00pm-7:50pm

EMC/EMI

RIMMS CR#22

8:00pm-8:50pmLibrary/Make-up

Class/Seminar

Library/Make-up

Class/Seminar

Library/Make-up

Class/Seminar

Library/Make-up

Class/Seminar


4/61


Textbooks

4

Probability & Random Processes

for Electrical Engineers, 2nd or 3rded.

Albert Leon-Garcia

Introduction to Probability Models,

9th ed.

Sheldon M. Ross

Elements of Information Theory Thomas M. Cover and Joy

A. Thomas

Chaos Theory Tamed Garnett P. Williams


5/61


Course Outline

Syllabus Introduction to Probability Theory

Random Variables

Limits and Inequalities

Central Limit Theorem

Application Area: Information Theory

Stochastic Processes

Prediction and Estimation

Markov Chain

Counting processes, queueing theory (time permitting)

Application Area: Chaos Theory

5


6/61


Grading (subject to change)

Final Exam: 50%

Midterm Exam: 30%

Quizzes: 10%

Homework Assignments: 10%

6


7/61Copyright Syed Ali Khayam 2008

Policies

Quizzes will be unannounced

Late homework submissions will be accepted for up to 24 hourswith a 50% penalty of the total.

I will take strong disciplinary action in cases of plagiarism orcheating in exams, homework or quizzes. There are no secondchances if plagiarism is proven.

Attendance:

Will be taken at any time during class.

The current rules of the school will be followed (min. 75%attendance requirement).

7



Boilerplate Disclaimers

About your 20s (or 30s)

About graduate school

About questions

About cheating and plagiarism

About attendance

About history

8



What will we cover in this lecture?

This lecture is intended to be an introduction to elementary

probability theory

We will cover:

Random Experiments and Random Variables

Axioms of Probability Mutual Exclusivity

Conditional Probability

Independence

Law of Total Probability

Bayes Theorem

9



Definition of Probability

Probability:

1 : the quality or state of being possible

2 : something (as an event or circumstance) that is possible

3 : the ratio of the number of outcomes in an exhaustive set ofequally likely outcomes that produce a given event to thetotal number of possible outcomes, the chance that a given

event will occur

We will revisit these definitions in a little bit

10



Definition of a Random Experiment

A random experiment comprises of:

A procedure

An outcome

Procedure(e.g., flipping a coin)

Outcome

(e.g., the value

observed [head, tail] afterflipping the coin)

Sample Space

(Set of All Possible

Outcomes)

11



Definition of a Random Experiment:Outcomes, Events and the Sample Space

An outcome cannot be further decomposed into other outcomes{s1 = the value 1}, , {s6 = the value 6}

An event is a set of outcomes that are of interest to us

A = {s: such that s is an even number}

The sample space is the set of all possible outcomes, called S

{1,2,3,4,5,6}

12



s1

s2

s3

s4

s5

s6

S

13




Example of a Random Experiment: Experiment:

Roll a fair dice once and record the number of dots

on the top face

Sample space:

{1,2,3,4,5,6} Events:

the outcome is even {2,4,6}

the outcome is greater than 4 {5,6} 14




Axioms of Probability

Probability of any eventA is non-negative:Pr{} 0

The probability that an outcome belongs to the sample

space is 1:Pr{} 1

The probability of the union of mutually exclusive events

is equal to the sum of their probabilities:If, 1 2 ,

Then, Pr{1 2} Pr{1} + Pr{2}

15



Mutual Exclusivity

Are1 and2 mutually exclusive?

For mutually exclusive events1, 2 , we have:

s1

s2

s3

s4

s5

s6

S

A 1

A 2

Find Pr{1 2} and Pr{1} + Pr{2} in the fair dice

example16



Mutual Exclusivity

Discarding the condition of exclusivity, in general, we have:

Pr{1 2} ? ?

s1

s2

s3

s4

s5

s6

S

17



Mutual Exclusivity

Discarding the condition of exclusivity, in general, we have:

Pr{1 2} Pr{1} + Pr{2} Pr{1 2}

s1

s2

s3

s4

s5

s6

S

18




Given that event B has already occurred, what is the probability

that eventA will occur? Given that event B has already occurred, reduces the sample

space ofA

s1

s2

s3

s4

s5

s6

S

s1

s2

s3

s4

s5

s6

Event B has

already occurred

=> s2, s4, s3

cannot occur

S

19




Given that event has already occurred, we define a new

conditional sample space that only contains s outcomes The new event space for is the intersection of and :

Event space |

s1

s2

s3

s4

s5

s6

S

s1

s2

s3

s4

s5

s6

Event B has

already

occurred

S

Whats missing here? {, , , }

| {}20




The probability of an event in the conditional sample space is:

Pr{|}

{}

Pr ={}

{} /

/

s1

s2

s3

s4

s5

s6

S

s1

s2

s3

s4

s5

s6

Event has

already

occurred

S

{1, 5, 6}

21



Independence

Two events are independent if they do not provide any

information about each other:

(|) ()

In other words, the fact that B has already happened does not

affect the probability ofAs outcomes

Implications:

(|) ()

() ()

( ) () ()

22


23/61


Independence: Example

Are events and independent?

Assume that all outcomes are equally likely

s4

s1

s2

s3

s6

s5

S

23


24/61




Check if ( ) () ()

s4

s1

s2

s3

s6

s5

S

24


25/61



Are events A and C independent?

Pr{ } Pr{5} 16

Pr Pr 3

6

2

6

1

6Yes!

s4

s1

s2

s3

s6

s5

S

25


26/61




Assume that all outcomes are equally likely

s4

s1

s2

s3

s6

s5

S

26


27/61




Pr{ } Pr{5} 16

Pr{}Pr{} 3

6

3

6

1

4No!

s4

s1

s2

s3

s6

s5

S

27


28/61


Mutual Exclusivity and Independence

Experiment:

Roll a fair dice twice and record the dots on the top face:

{(1,1), (1,2), (1,3), (1,4), (1,5), (1,6),

(2,1), (2,2), (2,3), (2,4), (2,5), (2,6),

(3,1), (3,2), (3,3), (3,4), (3,5), (3,6),

(4,1), (4,2), (4,3), (4,4), (4,5), (4,6),

(5,1), (5,2), (5,3), (5,4), (5,5), (5,6),(6,1), (6,2), (6,3), (6,4), (6,5), (6,6) }

28


29/61


Define three events:

1 = first roll gives an odd number

2 = second roll gives an odd number = the sum of the two rolls is odd

Find the probability of using probability of1 and2

29



30/61


A 1

A 2

30

S = { (1,1), (1,2), (1,3), (1,4), (1,5), (1,6),

(2,1), (2,2), (2,3), (2,4), (2,5), (2,6),

(3,1), (3,2), (3,3), (3,4), (3,5), (3,6),

(4,1), (4,2), (4,3), (4,4), (4,5), (4,6),

(5,1), (5,2), (5,3), (5,4), (5,5), (5,6),

(6,1), (6,2), (6,3), (6,4), (6,5), (6,6) }



31/61




32/61




33/61




34/61




35/61




36/61




37/61




38/61


Recap

1. Outcomes, events and sample space:

2. For mutually exclusive eventsA1,A2,, AN, we have:

3. In general, we have:

38

R


39/61


4. Conditional probability reduces the sample space:

5. Two eventsA and B are independent only if

6. For independent events:

39

Recap


40/61


Four Rules of Thumb

1. Whenever you see two events which have an OR relationship (i.e., eventA or

event B), their joint event will be their union, { }Example: On a binary channel, find the probability of error?

An error occurs when

A: a 0 is transmitted and a 1 is received OR

B: a 1 is transmitted and a 0 is received

Thus probability of error is: Pr{ }

T0

T1

R0

R1

Pr{R0|T0}

Pr{R1|T1}

40


41/61


2. Whenever you see two events which have an AND relationship (i.e., both

eventA and event B), their joint event will be their intersection, { }

Example: On a binary channel, find the probability that a 0 is transmitted and a

1 is received?


A: a 0 is transmitted AND

B: a 1 is receivedThus probability of above event is: Pr{ }

T0

T1

R0

R1

Pr{R0|T0}

Pr{R1|T1}

41

Four Rules of Thumb


42/61


3. Whenever you see two events which have an OR relationship (i.e., ), check if theyare mutually exclusive. If so, set

Example: On a binary channel, find the probability of error?


A: a 0 is transmitted and a 1 is received OR

B: a 1 is transmitted and a 0 is received

Thus probability of error is:AreA and B are mutually exclusive?

YES!

A and B are mutually exclusive; transmission of a 0 precludes the possibility oftransmission of a 1, and vice versa. Therefore, we can set

T0

T1

R0

R1

Pr{R0|T0}

Pr{R1|T1}

42

Four Rules of Thumb


43/61


4. Whenever you see two events which have an AND relationship (i.e., ), check if

they are independent. If so, set Pr{ } Pr{}Pr{}

Example: On a binary channel, find the probability that a 0 is transmitted and a 1 is

received?

A: a 0 is transmitted AND

B: a 1 is received

Probability of above event is: Pr{ }

AreA and B independent?

No.

Pr Pr Pr 1

2

2

Pr Pr 1

2

1

2

1

4

T0

T1

R0

R1

Pr{R0|T0}

Pr{R1|T1}

43

Four Rules of Thumb


44/61


Total Probability

1, 2, , form a partition of a sample space wehave: 1 2

,

B1

B2

B3B

4s2 s4

s6

s1 s5

s3

44


45/61


Total Probability

If 1, 2, , form a mutually exclusive partition:

What does this imply?

B1

B2

B3

B4A

s2 s4

s6

s1 s5

s3

A

45


46/61


Total Probability


What does this imply? 1 2 and 1 2

B1

B2

B3

B4A

s2 s4

s6

s1 s5

s3

A

46


47/61


Total Probability

If B1, B2,, BN form a mutually exclusive partition:

What does this imply? 1 2 . . and 1 2

How to express A in term of Bi?

B1

B2

B3

B4A

s2 s4

s6

s1 s5

s3

A

47


48/61


Total Probability



How to express A in term of Bi? ( 1) ( 2) ( )

B1

B2

B3

B4A

s2 s4

s6

s1 s5

s3

A

48


49/61


Total Probability




What is the probability of A?

B1

B2

B3

B4A

s2 s4

s6

s1 s5

s3

A

49


50/61


Total Probability




What is the probability of A? Pr{} Pr{ 1} + Pr{ 2} + +

Pr{ }

B1

B2

B3

B4A

s2 s4

s6

s1 s5

s3

A

50


51/61


Total Probability

Using the definition of conditional probability:

Pr } Pr{ } / Pr{}

> Pr Pr } Pr{}

B1

B2

B3

B4A s2 s4 s6

s1 s5

s3

A

51


52/61


The Law of Total Probability

The Law of Total Probability states:

B1

B2

B3

B4

As2

s4

s6

s1 s5

s3

A

If 1, 2, , form a partition then for any event

Pr{} Pr{|1} Pr{1} + Pr{|2} Pr{2} + + Pr{|} Pr{}

52


53/61


Based on the Law of Total Probability, Thomas Bayes decided to

look at the probability of a partition given a particular event, theso-called inverse probability.

Bayes Theorem

B1

B2

B3

B4

A

s2

s4 s6

s1 s5

s3

A

53

h


54/61


Bayes Theorem

Based on the Law of Total Probability, Thomas Bayes decided to

look at the probability of a partition given a particular eventPr

Pr

Pr Pr Pr Pr

Pr Pr Pr

Pr

B1

B2

B3

B4A s2 s4

s6

s1 s5

s3

A

54

h


55/61


Bayes Theorem

Pr

Pr Pr

Pr{}From the Law of Total Probability, we have:

Pr{} Pr{|1} Pr{1} + Pr{|2} Pr{2} + + Pr{|} Pr{}

B1

B2

B3

B4A

s2 s4

s6

s1 s5

s3

A

Bayes Rule

55

B Th


56/61


Bayes Theorem

Think of the event A as making a certain observation or taking a

certain value of a measurement.

B1

B2

B3

B4A

s2 s4

s6

s1 s5

s3

A

56

Bayes Theorem


57/61


Bayes Theorem

Pr Pr | Pr

=

Pr | Pr

Pr{|} posterior / aposterioriprobability, i.e. the probabilityyou know aftermaking an observation or taking ameasurement

Pr{|} likelihood probability

Pr{} prior / a prioriprobability, i.e. the probability you knewbefore taking into consideration any observations ormeasurements

Pr{} evidence probability, i.e. the probability of obtaining acertain observation or measurement

57

A Fi h P bl


58/61


Asad is a fisherman.

Asad is an educated person.Asad builds a fishing robot that will do his work for him.

A Fishy Problem

58

B4

B2

B1

B3

A Fi h P bl


59/61


Question: If Asads robot catches a fish that is detected red, what

species is it,

,

,

or

?

Answer: It could be any of four species in the sea.

A Fishy Problem

59

B4

B2

B1

B3

A Fi h P bl


60/61


Lets change the question:

What is the probability that a red fish is a species B1, B2,B3 and B4?

A Fishy Problem

60

B4

B2

B1

B3

A Fi h P bl


61/61

A Fishy Problem

B4

B2

B1

B3

Documents

Lecture01 Intro Probability Theory (2)