01 Probability review

Unit 1Probability ReviewWang, Yuan-Kai, 王元凱

[email protected]://www.ykwang.tw

Department of Electrical Engineering, Fu Jen Univ.輔仁大學電機工程系

2006~2011

Bayesian Networks

Fu Jen University Department of Electronic Engineering Yuan-Kai Wang Copyright

Reference this document as: Wang, Yuan-Kai, “Probability Review," Lecture Notes of Wang, Yuan-Kai,

Fu Jen University, Taiwan, 2008.

王元凱 Unit - Probability Review p.

Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai Copyright

Goal of this Unit Review basic concepts of

probability in terms of Image processing terminologies

2



Related Units Next units Statistics Review Uncertainty Inference (Discrete) Uncertainty Inference (Continuous)

3



Self-Study Probability Theory textbook Artificial Intelligence: a modern

approach Russell & Norvig, 2nd, Prentice Hall,

2003. pp.462~474, Chapter 13, Sec. 13.1~13.3

4



55

Contents1. Probability ......................................... 62. Random Variable .............................. 93. Probability Distribution .................... 194. Joint Distribution .............................. 265. Conditional Probability .................... 466. Bayes Theorem ................................. 597. Summary …………………………….. 648. References ………………………….. 67



6

1. ProbabilityWe write P(A) as “the fraction of

possible worlds in which A is true”Event space of all possible worlds

Its area is 1Worlds in which A is False

Worlds in which A is true

P(A) = Area ofreddish oval



7

The Axioms of Probability0 P(A) 1P(True) = 1P(False) = 0P(A B) = P(A) + P(B) - P(A B)

Where do these axioms come from?

Were they “discovered”? Answers coming up later.



8

Theorems from the Axioms0 P(A) 1,

P(True) = 1, P(False) = 0

P(A B) = P(A) + P(B) - P(A B)From these we can prove:P(not A) = P(~A) = 1-P(A)P(A) = P(A B) + P(A ~B)

How?



9

2. Random VariableA variable with randomnessprobability, degree of belief

An exampleRain : a random variableIts possible values: true, falseIts randomness

P(Rain=true)=0.8, P(Rain=false)=0.2



10

Types of Random VariableBoolean random variableRain : true, false

Discrete random variable(Multivalued R.V.)Rain: cloudy, sunny, drizzle,

drenchContinuous random variableRain: rainfall in millimeter



11

Discrete R.V. (1/4)Let A=Rain A can take on more than 2 valuesA is a random variable with arity k

if it can take on exactly one value out of {v1,v2, .. vk}Thus

jivAvAP ji if 0)(1)( 21 kvAvAvAP



12

Discrete R.V. (2/4)

Using the axioms of probability…0 P(A) 1, P(True) = 1, P(False) = 0P(A B) = P(A) + P(B) - P(A B)

And assume that A obeysjivAvAP ji if 0)(

1)( 21 kvAvAvAP

)()(1

21

i

jji vAPvAvAvAP

Prove it?



13

Discrete R.V. (3/4)

Using the axioms of probability…0 P(A) 1, P(True) = 1, P(False) = 0P(A or B) = P(A) + P(B) - P(A and B)

And assuming that A obeys…jivAvAP ji if 0)(

1)( 21 kvAvAvAP

)()(1

21

i

jji vAPvAvAvAP

1)(1

k

jjvAP



14

Discrete R.V. (4/4)

Using the axioms of probability…0 P(A) 1, P(True) = 1, P(False) = 0P(A or B) = P(A) + P(B) - P(A and B)

And assuming that A obeys…

)(])[(1

21

i

jji vABPvAvAvABP

jivAvAP ji if 0)(1)( 21 kvAvAvAP

)()(1

k

jjvABPBP



15

Random Vector A random vector is a vector of

random variables X = (X1, X2, ..., XN) X1 ... XN are random variables



16

Example in Image Processing For a gray-level image,

random variable is “gray level”• Random variable X

(Gray level) has npossible values {x1, x2, ..., xn}, n=256

• N random data x1, x2, .., xN of X, N=Width*Height

Discrete v.s. continuous ?



17

Example in Image Processing For a color image,

random vector is (r,g,b)



18

Example in Computer Vision Random vector is a vector of

“features” Face recognition



19

3. Probability Distribution Probability distribution is a set of

probabilities Boolean R.V. P(A): P(A=true)=0.2, P(A=false)=0.8

Discrete R.V. P(A): P(A=v1)=0.2, P(A=v2)=0.4,

P(A=v3)=0.4, P(A=v4)=0.1, Continuous R.V.

Usually we plot the probability distribution as a function(Probability Distribution Function, pdf)



20

Probability Distribution (1/3)Continuous R.V.

Rain (mm)0 100 200 300 400 500 600

0.80.60.40.2

0

• P(Rain) is a probability density function (pdf)

• P(Rain=200) is a probability



21

Probability Distribution(2/3)Boolean R.V.P(Rain) is a probability distributionP(Rain=false) is a probability

Rain

P(Rain)

false true

0.8

0.2



22

Probability Distribution(3/3)Discrete R.V.P(Rain) is a probability distributionP(Rain=drizzle) is a probability

Rain

P(Rain)

0.8

0.2

clou

dy

sunn

y

driz

zle

dren

ch



23

Common Probability Distributions

Normal distribution (Gaussian)Parameters: , Discussed in Section 3

Uniform distributionExponential distributionPoisson distribution...



24

The Uniform Distribution

-w/2 0 w/2

1/w

0][ XE12

]Var[2wX

2||if0

2||if1

)( wx

wxwxp

X

P(x)



25

Example in Image Processing

• Random variable X (Gray level) has n possible values {x1, x2, ..., xn}, n=256

• Distribution P(xi) of the image•Is not a usual p.d.f.



26

4. Joint Distribution When there are more than one

random variables: X, Y We want to know "the probabilities

of both random variables" Joint probability distribution P(X Y) P(X Y ...)



27

An ExampleA set of random variables for the

typhoon worldRain, Wind, Speed, ...Every random variable Describes one facet of typhoonHas a probability distribution

A typhoon is a situation of joint probabilityP(Rain=200mm Wind=South

Speed=100km/hr) = 0.4



28

P(X,Y) with X Y are Discrete P(X,Y) : (M x N) entries

P(X=x1,Y=y1) P(X=x2,Y=y1) P(X=x3,Y=y1)

P(X=x1,Y=y2) P(X=x2,Y=y2) P(X=x3,Y=y2)

X

Y

x1 x2 x3

y1

y2

0.2 0.1 0.10.1 0.2 0.3

X

Y

x1 x2 x3

y1

y2



29

Atomic EventAtomic event is aCombination of values of all random

variablesSituation (state) of typhoon

(statistical world)For the typhoon example(Rain=200mm Wind=South

Speed=100km/hr) is an atomic event(Rain=200mm Wind=South) is not



30

Joint Probability DistributionP(Rain=200mm Wind=South

Speed=100km/hr) is a joint probabilityP(Rain Wind Speed) is a probability

of all joint probabilitiesJoint probability function

A set of random variables can have mixed typesEx: Rain: Boolean, Wind: Discrete,

Speed: ContinuousWe will consider all-discrete case



31

Prior and Joint Probability (Math)

Let {X1,...,Xn} be a set of random variablesP(Xi) is a probability functionP(Xi=xi) is a prior probabilityP(X1 X2 Xn) is a joint

probability functionP(X1=x1 X2=x2 Xn=xn) is

a joint probability



32

For Boolean & Discrete R.V.Let X be a single R.V.P(X) is a vector

Boolean R.V.Rain: true, falseP(Rain) = <0.72, 0.28>

Discrete R.V.Rain: cloudy, sunny, drizzle, drenchP(Rain) = <0.72, 0.1, 0.08, 0.1>Normalized, i.e. sums to 1



33

Full Joint Distribution (1/3)

For a set of random variables {X1, X2 , , Xn}

X1 X2 Xn are atomic eventsP(X1 X2 Xn) is A full joint probability distribution

(FJD)A table of all joint prob. of all atomic

events, if {X1, , Xn} are discrete



34

Full Joint Distribution (2/3)Ex: X1: Rain, X2: WindX1: drizzle, drench, cloudy, X2: strong, weakX1 X2 are atomic eventsP(X1 X2) is a 3x2 matrix of values

RainWind Drizzle Drench Cloudy

Strong 0.15 0.12 0.06Weak 0.55 0.08 0.04

P(X1=drizzle X2=strong) = 0.15, ...



35

Full Joint Distribution (3/3)

All questions about probability of joint events can be answered by the tableP(Wind=Strong),

P(Rain=Drizzle Wind=Strong), ...



36

An Example with 3 Boolean R.V. (1/2)

1. Make a truth table listing all combinations of values of your variables• if there are M

Boolean variables then the table will have 2M rows

Boolean variables A, B, CA B C0 0 00 0 10 1 00 1 11 0 01 0 11 1 01 1 1



37

An Example with 3 Boolean R.V. (2/2)

2. For each combination of values, say how probable it is

A B C Prob0 0 0 0.300 0 1 0.050 1 0 0.100 1 1 0.051 0 0 0.051 0 1 0.101 1 0 0.251 1 1 0.10

Sum 1.0

A

B

C0.050.25

0.10 0.050.05

0.10

0.100.30



38

Using the FJD • Once you have the FJD • You can ask for the probability of

any logical expression: Inference

E

PEP matching rows

)row()(



39

Examples of Inference (1/3)

P(Poor Male) = 0.4654 E

PEP matching rows

)row()(



40


P(Poor) = 0.7604 E

PEP matching rows

)row()(



41


2

2 1

matching rows

and matching rows

2

2121 )row(

)row(

)()()|(

E

EE

P

P

EPEEPEEP

P(Male | Poor) = 0.4654 / 0.7604 = 0.612



42

Inference is a Big DealI’ve got this evidenceWhat’s the chance that this

conclusion is true?I’ve got a sore neck: how likely am

I to have meningitis?I see my lights are out and it’s

9pm. What’s the chance my spouse is already asleep?



43

Where do Full Joint Distributions Come from?

(1/2)Idea One: Expert HumansIdea Two: Learn them from data!



44

Where do FJD Come from? (2/2)

Build a JD table for your attributes in which the probabilities are unspecified

Then fill in each row with

records ofnumber totalrow matching records)row(ˆ P

A B C Prob0 0 0 ?0 0 1 ?0 1 0 ?0 1 1 ?1 0 0 ?1 0 1 ?1 1 0 ?1 1 1 ?

A B C Prob0 0 0 0.300 0 1 0.050 1 0 0.100 1 1 0.051 0 0 0.051 0 1 0.101 1 0 0.25

1 1 1 0.10Fraction of all records in whichA and B are True but C is False



45

Example of Learning a FJDThis Joint was obtained by

learning from three attributes in the UCI “Adult” Census Database [Kohavi 1995]



46

5. Conditional ProbabilityP(A|B) = Fraction of worlds in which B

is true that also have A true

F

H

H = “Have a headache”F = “Coming down with Flu”

P(H) = 1/10P(F) = 1/40P(H|F) = 1/2

“Headaches are rare and flu is rarer, but if you’re coming down with ‘flu there’s a 50-50 chance you’ll have a headache”



47

Formula

F

H

H = “Have a headache”F = “Coming down with Flu”

P(H) = 1/10P(F) = 1/40P(H|F) = 1/2

)()()|(

FPFHPFHP

P(H|F) v.s. P(HF)



48

Conditional, Joint, PriorRelationship among conditional,

joint, and prior probabilities

P(X1) P(X1 X2)P(X2)

)()()|(

2

2121 XP

XXPXXP

)()|()( 22121 XPXXPXXP



49

Posterior v.s. Prior Probabilities

P(Cavity|Toothache)Conditional probabilityPosterior probability

(after the fact/evidence)P(Cavity)Prior probability (the fact)



50

An Example (1/2)For a dental diagnosisLet {Cavity,Toothache} be a set

of Boolean random variablesDenotations for Boolean R.V.P(Cavity=true) = P(cavity)P(Cavity=false) = P(cavity)



51

An Example (2/2)The full joint probability

distribution P(Toothache Cavity)

cavity 0.04 0.06cavity 0.01 0.89

toothache toothache

11.006.001.004.0)( toothachecavityP

80.001.004.0

04.0)(

)()|(

toothacheP

toothachecavityPtoothachecavityP



52

Conditional Probability (Math)P(X1=x1i| X2 =x2j) is a conditional

probabilityP(X1 | X2) is a conditional

distribution functionAll P(X1=x1i| X2 =x2j) for all possible i, j

For Boolean & discrete R.V.s, conditional distribution function is a conditional probability tableConditional distribution of continuous

R.V. will not used in our discussions



53

Conditional Probability Table (CPT)

For the dental diagnosis problem,Toothache & Cavity are Boolean R.V.sP(Toothache|Cavity) is a CPT

Cavity P(toothache|Cavity)T 0.90F 0.05

Cavity P(toothache|Cavity) P(toothache|Cavity)T 0.90 0.1F 0.05 0.95



54

CPT v.s. FJD

Cavity P(toothache|Cavity) P(toothache|Cavity)T 0.90 0.1F 0.05 0.95

cavity 0.04 0.06cavity 0.01 0.89

toothache toothache

Sum of all atomic events = 1

Sum of a row = 1



55

More Than 2 Random Variables in

Conditional ProbabilityJoint conditional probability

)()(

)|(

43

4321

4321

XXPXXXXP

XXXXP

)|()|( 4324321 XXXPXXXXP



56

Chain Rule

)()()|(

2

2121 XP

XXPXXP

)()|()|()|()()|(

))(()(

4434324321

4324321

4321

4321

XPXXPXXXPXXXXPXXXPXXXXP

XXXXPXXXXP

)|()( 11

21 nii

n

in XXXPXXXP

• Joint probability can be calculated from a chain of conditional probability

)()|()( 22121 XPXXPXXP



57

Useful Easy-to-prove Facts

1)|()|( BAPBAP

1)|(1

An

kk BvAP



58

Where Are We?We have recalled the fundamentals

of probabilityWe know what JDs are and how to

use themNext two sectionsBayes ruleGaussian distribution



59

6. Bayes Theorem

)()|()()|( APABPBPBAP

)()|()( BPBAPBAP

)()|()( APABPABP

Bayes, Thomas (1763) An essay towards solving a problem in the doctrine of chances. Philosophical Transactions of the Royal Society of London, 53:370-418

)()(

)|()|(

BPAP

ABPBAP



60

Useful Forms of Bayes Rule

)()()|()|(

BPAPABPBAP

)()(

BPBAP

)()()|()|(

CBPCAPCABPCBAP

)|()|(),|(

),(),(),|(),|(

CBPCAPCABP

CBPCAPCABPCBAP



61

More General Forms of Bayes Rule

AA n

kkk

iin

kk

ii

vAPvABP

vAPvABP

BvAP

BvAPBvAP

11)()|(

)()|(

)(

)()|(

)()()(

)()()|(

BAPBAPBAP

BPBAPBAP

If A is a Boolean R.V.

If A is a discrete R.V.



62

Example –Bayesian Detection (1/2)

2-class recognition Ex.: Skin color detection

Let A be skin color pixel A be non-skin color pixel c be the color (R,G,B) of a pixel

We can get : P(c|A), P(c|A) For a pixel in a new image, is it a skin

color pixel?

)()()|()|(

cPAPAcPcAP

)()()|()|(

cPAPAcPcAP

>



63

Example –Bayesian Detection (2/2)

2-class recognition Ex.: Background detection

Let A be background pixel A be non-background pixel c be the color (R,G,B) of a pixel

We can get : P(c|A), P(c|A) For a pixel in a new image, is it a

background pixel?

)()()|()|(

cPAPAcPcAP

)()()|()|(

cPAPAcPcAP

>



64

7. SummaryContinuous variable

Scalar* Scalar

Function of two variables MxN matrix

Function of two variables MxN matrix

Function of one variable M vector

Function of one variable N vector

Scalar* Scalar

Discrete variable

Function of one variable M vector

P(X=x)

P(X,Y)

P(X|Y)

P(X|Y=y)

P(X=x|Y)

P(X=x|Y=y)

P(X)(X has M values, Y has N values)



65

Ex (1/2)



66

Ex (2/2)



67

8. Reference Artificial Intelligence: a modern

approach Russell & Norvig, 2nd, Prentice Hall, 2003. Sections 13.1~13.3, pp.462~474.

統計學的世界墨爾著，鄭惟厚譯天下文化，2002

深入淺出統計學 D. Grifiths, 楊仁和譯，2009 O’ Reilly

Education

01 Probability review