01 Probability review

Unit 1Probability ReviewWang, Yuan-Kai, 王元凱

ykwang@mails.fju.edu.twhttp://www.ykwang.tw

Department of Electrical Engineering, Fu Jen Univ.輔仁大學電機工程系

2006~2011

Bayesian Networks

Fu Jen University Department of Electronic Engineering Yuan-Kai Wang Copyright

Reference this document as: Wang, Yuan-Kai, “Probability Review," Lecture Notes of Wang, Yuan-Kai,

Fu Jen University, Taiwan, 2008.

王元凱 Unit - Probability Review p.

Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai Copyright

Goal of this Unit Review basic concepts of

probability in terms of Image processing terminologies

Related Units Next units Statistics Review Uncertainty Inference (Discrete) Uncertainty Inference (Continuous)

Self-Study Probability Theory textbook Artificial Intelligence: a modern

approach Russell & Norvig, 2nd, Prentice Hall,

2003. pp.462~474, Chapter 13, Sec. 13.1~13.3

Contents1. Probability ......................................... 62. Random Variable .............................. 93. Probability Distribution .................... 194. Joint Distribution .............................. 265. Conditional Probability .................... 466. Bayes Theorem ................................. 597. Summary …………………………….. 648. References ………………………….. 67

1. ProbabilityWe write P(A) as “the fraction of

possible worlds in which A is true”Event space of all possible worlds

Its area is 1Worlds in which A is False

Worlds in which A is true

P(A) = Area ofreddish oval

The Axioms of Probability0 P(A) 1P(True) = 1P(False) = 0P(A B) = P(A) + P(B) - P(A B)

Where do these axioms come from?

Were they “discovered”? Answers coming up later.

Theorems from the Axioms0 P(A) 1,

P(True) = 1, P(False) = 0

P(A B) = P(A) + P(B) - P(A B)From these we can prove:P(not A) = P(~A) = 1-P(A)P(A) = P(A B) + P(A ~B)

2. Random VariableA variable with randomnessprobability, degree of belief

An exampleRain : a random variableIts possible values: true, falseIts randomness

P(Rain=true)=0.8, P(Rain=false)=0.2

Types of Random VariableBoolean random variableRain : true, false

Discrete random variable(Multivalued R.V.)Rain: cloudy, sunny, drizzle,

drenchContinuous random variableRain: rainfall in millimeter

Discrete R.V. (1/4)Let A=Rain A can take on more than 2 valuesA is a random variable with arity k

if it can take on exactly one value out of {v1,v2, .. vk}Thus

jivAvAP ji if 0)(1)( 21 kvAvAvAP

Discrete R.V. (2/4)

Using the axioms of probability…0 P(A) 1, P(True) = 1, P(False) = 0P(A B) = P(A) + P(B) - P(A B)

And assume that A obeysjivAvAP ji if 0)(

1)( 21 kvAvAvAP

jji vAPvAvAvAP

Prove it?

Discrete R.V. (3/4)

Using the axioms of probability…0 P(A) 1, P(True) = 1, P(False) = 0P(A or B) = P(A) + P(B) - P(A and B)

And assuming that A obeys…jivAvAP ji if 0)(

1)( 21 kvAvAvAP

jji vAPvAvAvAP

Discrete R.V. (4/4)

Using the axioms of probability…0 P(A) 1, P(True) = 1, P(False) = 0P(A or B) = P(A) + P(B) - P(A and B)

And assuming that A obeys…

)(])[(1

jji vABPvAvAvABP

jivAvAP ji if 0)(1)( 21 kvAvAvAP

jjvABPBP

Random Vector A random vector is a vector of

random variables X = (X1, X2, ..., XN) X1 ... XN are random variables

Example in Image Processing For a gray-level image,

random variable is “gray level”• Random variable X

(Gray level) has npossible values {x1, x2, ..., xn}, n=256

• N random data x1, x2, .., xN of X, N=Width*Height

Discrete v.s. continuous ?

Example in Image Processing For a color image,

random vector is (r,g,b)

Example in Computer Vision Random vector is a vector of

“features” Face recognition

3. Probability Distribution Probability distribution is a set of

probabilities Boolean R.V. P(A): P(A=true)=0.2, P(A=false)=0.8

Discrete R.V. P(A): P(A=v1)=0.2, P(A=v2)=0.4,

P(A=v3)=0.4, P(A=v4)=0.1, Continuous R.V.

Usually we plot the probability distribution as a function(Probability Distribution Function, pdf)

Probability Distribution (1/3)Continuous R.V.

Rain (mm)0 100 200 300 400 500 600

0.80.60.40.2

• P(Rain) is a probability density function (pdf)

• P(Rain=200) is a probability

Probability Distribution(2/3)Boolean R.V.P(Rain) is a probability distributionP(Rain=false) is a probability

P(Rain)

false true

Probability Distribution(3/3)Discrete R.V.P(Rain) is a probability distributionP(Rain=drizzle) is a probability

P(Rain)

Common Probability Distributions

Normal distribution (Gaussian)Parameters: , Discussed in Section 3

Uniform distributionExponential distributionPoisson distribution...

The Uniform Distribution

-w/2 0 w/2

0][ XE12

]Var[2wX

2||if0

2||if1

Example in Image Processing

• Random variable X (Gray level) has n possible values {x1, x2, ..., xn}, n=256

• Distribution P(xi) of the image•Is not a usual p.d.f.

4. Joint Distribution When there are more than one

random variables: X, Y We want to know "the probabilities

of both random variables" Joint probability distribution P(X Y) P(X Y ...)

An ExampleA set of random variables for the

typhoon worldRain, Wind, Speed, ...Every random variable Describes one facet of typhoonHas a probability distribution

A typhoon is a situation of joint probabilityP(Rain=200mm Wind=South

Speed=100km/hr) = 0.4

P(X,Y) with X Y are Discrete P(X,Y) : (M x N) entries

P(X=x1,Y=y1) P(X=x2,Y=y1) P(X=x3,Y=y1)

P(X=x1,Y=y2) P(X=x2,Y=y2) P(X=x3,Y=y2)

x1 x2 x3

0.2 0.1 0.10.1 0.2 0.3

x1 x2 x3

Atomic EventAtomic event is aCombination of values of all random

variablesSituation (state) of typhoon

(statistical world)For the typhoon example(Rain=200mm Wind=South

Speed=100km/hr) is an atomic event(Rain=200mm Wind=South) is not

Joint Probability DistributionP(Rain=200mm Wind=South

Speed=100km/hr) is a joint probabilityP(Rain Wind Speed) is a probability

of all joint probabilitiesJoint probability function

A set of random variables can have mixed typesEx: Rain: Boolean, Wind: Discrete,

Speed: ContinuousWe will consider all-discrete case

Prior and Joint Probability (Math)

Let {X1,...,Xn} be a set of random variablesP(Xi) is a probability functionP(Xi=xi) is a prior probabilityP(X1 X2 Xn) is a joint

probability functionP(X1=x1 X2=x2 Xn=xn) is

a joint probability

For Boolean & Discrete R.V.Let X be a single R.V.P(X) is a vector

Boolean R.V.Rain: true, falseP(Rain) = <0.72, 0.28>

Discrete R.V.Rain: cloudy, sunny, drizzle, drenchP(Rain) = <0.72, 0.1, 0.08, 0.1>Normalized, i.e. sums to 1

Full Joint Distribution (1/3)

For a set of random variables {X1, X2 , , Xn}

X1 X2 Xn are atomic eventsP(X1 X2 Xn) is A full joint probability distribution

(FJD)A table of all joint prob. of all atomic

events, if {X1, , Xn} are discrete

Full Joint Distribution (2/3)Ex: X1: Rain, X2: WindX1: drizzle, drench, cloudy, X2: strong, weakX1 X2 are atomic eventsP(X1 X2) is a 3x2 matrix of values

RainWind Drizzle Drench Cloudy

Strong 0.15 0.12 0.06Weak 0.55 0.08 0.04

P(X1=drizzle X2=strong) = 0.15, ...

Full Joint Distribution (3/3)

All questions about probability of joint events can be answered by the tableP(Wind=Strong),

P(Rain=Drizzle Wind=Strong), ...

An Example with 3 Boolean R.V. (1/2)

1. Make a truth table listing all combinations of values of your variables• if there are M

Boolean variables then the table will have 2M rows

Boolean variables A, B, CA B C0 0 00 0 10 1 00 1 11 0 01 0 11 1 01 1 1

An Example with 3 Boolean R.V. (2/2)

2. For each combination of values, say how probable it is

A B C Prob0 0 0 0.300 0 1 0.050 1 0 0.100 1 1 0.051 0 0 0.051 0 1 0.101 1 0 0.251 1 1 0.10

Sum 1.0

C0.050.25

0.10 0.050.05

0.100.30

Using the FJD • Once you have the FJD • You can ask for the probability of

any logical expression: Inference

PEP matching rows

)row()(

Examples of Inference (1/3)

P(Poor Male) = 0.4654 E

PEP matching rows

)row()(

P(Poor) = 0.7604 E

PEP matching rows

)row()(

matching rows

and matching rows

2121 )row(

)()()|(

EPEEPEEP

P(Male | Poor) = 0.4654 / 0.7604 = 0.612

Inference is a Big DealI’ve got this evidenceWhat’s the chance that this

conclusion is true?I’ve got a sore neck: how likely am

I to have meningitis?I see my lights are out and it’s

9pm. What’s the chance my spouse is already asleep?

Where do Full Joint Distributions Come from?

(1/2)Idea One: Expert HumansIdea Two: Learn them from data!

Where do FJD Come from? (2/2)

Build a JD table for your attributes in which the probabilities are unspecified

Then fill in each row with

records ofnumber totalrow matching records)row(ˆ P

A B C Prob0 0 0 ?0 0 1 ?0 1 0 ?0 1 1 ?1 0 0 ?1 0 1 ?1 1 0 ?1 1 1 ?

A B C Prob0 0 0 0.300 0 1 0.050 1 0 0.100 1 1 0.051 0 0 0.051 0 1 0.101 1 0 0.25

1 1 1 0.10Fraction of all records in whichA and B are True but C is False

Example of Learning a FJDThis Joint was obtained by

learning from three attributes in the UCI “Adult” Census Database [Kohavi 1995]

5. Conditional ProbabilityP(A|B) = Fraction of worlds in which B

is true that also have A true

H = “Have a headache”F = “Coming down with Flu”

P(H) = 1/10P(F) = 1/40P(H|F) = 1/2

“Headaches are rare and flu is rarer, but if you’re coming down with ‘flu there’s a 50-50 chance you’ll have a headache”

Formula

H = “Have a headache”F = “Coming down with Flu”

P(H) = 1/10P(F) = 1/40P(H|F) = 1/2

)()()|(

FPFHPFHP

P(H|F) v.s. P(HF)

Conditional, Joint, PriorRelationship among conditional,

joint, and prior probabilities

P(X1) P(X1 X2)P(X2)

)()()|(

2121 XP

XXPXXP

)()|()( 22121 XPXXPXXP

Posterior v.s. Prior Probabilities

P(Cavity|Toothache)Conditional probabilityPosterior probability

(after the fact/evidence)P(Cavity)Prior probability (the fact)

An Example (1/2)For a dental diagnosisLet {Cavity,Toothache} be a set

of Boolean random variablesDenotations for Boolean R.V.P(Cavity=true) = P(cavity)P(Cavity=false) = P(cavity)

An Example (2/2)The full joint probability

distribution P(Toothache Cavity)

cavity 0.04 0.06cavity 0.01 0.89

toothache toothache

11.006.001.004.0)( toothachecavityP

80.001.004.0

04.0)(

toothacheP

toothachecavityPtoothachecavityP

Conditional Probability (Math)P(X1=x1i| X2 =x2j) is a conditional

probabilityP(X1 | X2) is a conditional

distribution functionAll P(X1=x1i| X2 =x2j) for all possible i, j

For Boolean & discrete R.V.s, conditional distribution function is a conditional probability tableConditional distribution of continuous

R.V. will not used in our discussions

Conditional Probability Table (CPT)

For the dental diagnosis problem,Toothache & Cavity are Boolean R.V.sP(Toothache|Cavity) is a CPT

Cavity P(toothache|Cavity)T 0.90F 0.05

Cavity P(toothache|Cavity) P(toothache|Cavity)T 0.90 0.1F 0.05 0.95

CPT v.s. FJD

Cavity P(toothache|Cavity) P(toothache|Cavity)T 0.90 0.1F 0.05 0.95

cavity 0.04 0.06cavity 0.01 0.89

toothache toothache

Sum of all atomic events = 1

Sum of a row = 1

More Than 2 Random Variables in

Conditional ProbabilityJoint conditional probability

XXPXXXXP

)|()|( 4324321 XXXPXXXXP

Chain Rule

)()()|(

2121 XP

XXPXXP

)()|()|()|()()|(

))(()(

4434324321

4324321

XPXXPXXXPXXXXPXXXPXXXXP

XXXXPXXXXP

)|()( 11

21 nii

in XXXPXXXP

• Joint probability can be calculated from a chain of conditional probability

)()|()( 22121 XPXXPXXP

Useful Easy-to-prove Facts

1)|()|( BAPBAP

kk BvAP

Where Are We?We have recalled the fundamentals

of probabilityWe know what JDs are and how to

use themNext two sectionsBayes ruleGaussian distribution

6. Bayes Theorem

)()|()()|( APABPBPBAP

)()|()( BPBAPBAP

)()|()( APABPABP

Bayes, Thomas (1763) An essay towards solving a problem in the doctrine of chances. Philosophical Transactions of the Royal Society of London, 53:370-418

)|()|(

ABPBAP

Useful Forms of Bayes Rule

)()()|()|(

BPAPABPBAP

)()()|()|(

CBPCAPCABPCBAP

)|()|(),|(

),(),(),|(),|(

CBPCAPCABP

CBPCAPCABPCBAP

More General Forms of Bayes Rule

vAPvABP

BvAPBvAP

11)()|(

)()()(

)()()|(

BAPBAPBAP

BPBAPBAP

If A is a Boolean R.V.

If A is a discrete R.V.

Example –Bayesian Detection (1/2)

2-class recognition Ex.: Skin color detection

Let A be skin color pixel A be non-skin color pixel c be the color (R,G,B) of a pixel

We can get : P(c|A), P(c|A) For a pixel in a new image, is it a skin

color pixel?

)()()|()|(

cPAPAcPcAP

)()()|()|(

cPAPAcPcAP

Example –Bayesian Detection (2/2)

2-class recognition Ex.: Background detection

Let A be background pixel A be non-background pixel c be the color (R,G,B) of a pixel

We can get : P(c|A), P(c|A) For a pixel in a new image, is it a

background pixel?

)()()|()|(

cPAPAcPcAP

)()()|()|(

cPAPAcPcAP

7. SummaryContinuous variable

Scalar* Scalar

Function of two variables MxN matrix

Function of one variable M vector

Function of one variable N vector

Scalar* Scalar

Discrete variable

Function of one variable M vector

P(X=x)

P(X,Y)

P(X|Y)

P(X|Y=y)

P(X=x|Y)

P(X=x|Y=y)

P(X)(X has M values, Y has N values)

Ex (1/2)

Ex (2/2)

8. Reference Artificial Intelligence: a modern

approach Russell & Norvig, 2nd, Prentice Hall, 2003. Sections 13.1~13.3, pp.462~474.

統計學的世界墨爾著，鄭惟厚譯天下文化，2002

深入淺出統計學 D. Grifiths, 楊仁和譯，2009 O’ Reilly

01 Probability review

Education

Probability Review Answers

Probability: Review

Review (probability & linear algebra)

Probability Review before Quiz - Honors Math 2hm2ghhs.weebly.com/.../unit_6_day_6_prob_review_before_quiz_spr1… · Probability Review before Quiz Unit 6 Day 6 Probability. Warm-up:

Probability and Statistics Review

Lecture 02: probability review + Bayes filterspublish.illinois.edu/.../01/sp-2020_02-lecture.pdf · Lecture 02: probability review + filtering Katie DC Jan. 23, 2019 Notes from Probabilistic

CS229 - Probability Theory Review

Lecture 2 Review Probability

Probability Jeopardy Review

Probability and Statistics Review

Review of Probability Theory

Appendix A Review of Probability Distribution - CERN · Appendix A Review of Probability Distribution In this appendix we review some fundamental concepts of probability theory. After

Brief Review Probability and Statistics. Probability distributions Continuous distributions

Probability Test Review KEY

Review of Probability

Lecture 3 Review Probability

Ch2 (Review of Probability)

Final Review Probability

01 Probability and Probability Distributions

1610 probability review