CS325ArtiﬁcialIntelligence Ch.15,20

CS325 Artificial IntelligenceCh. 15,20 – Hidden Markov Models and Particle Filtering

Cengiz Günay, Emory Univ.

Spring 2013

Günay Ch. 15,20 – Hidden Markov Models and Particle FilteringSpring 2013 1 / 21

http://news.sciencemag.org/sciencenow/2010/11/can-google-predict-the-stock.html

Get Rich Fast!

Or go bankrupt?

So, how can we predict time-series data?


http://magiclex.wordpress.com/

Get Rich Fast!

Or go bankrupt?




Get Rich Fast!

Or go bankrupt?




Hidden Markov Models


Hidden Markov Models


Entry/Exit Surveys

Exit survey: Reinforcement LearningWhat’s the difference between MDPs and Reinforcement Learning?What is the dilemma between exploration and exploitation?

Entry survey: Hidden Markov Models (0.25 points of final grade)What previous algorithm would you use for time series prediction?What time series do you wish you could predict?


Time Series Prediction?

Have we done this before?

Belief states with action schemas?

Not for continuous variablesGoal-based

MDPs and RL?

Goal-basedNo time sequence






MDPs and RL?







MDPs and RL?







MDPs and RL?







MDPs and RL?



Time Series Prediction with Hidden Markov Models (HMMs)

Dr. Thrun is very happy – HMMs are his specialty.HMMs:

analyze & predict time series datacan deal with noisy sensors

Example domains:finance (get rich fast!)roboticsmedicalspeech and language

Alternatives:Recurrent neural networks (not probabilistic)


Time Series Prediction with Hidden Markov Models (HMMs)

Dr. Thrun is very happy – HMMs are his specialty.HMMs:

analyze & predict time series datacan deal with noisy sensors

Example domains:finance (get rich fast!)roboticsmedicalspeech and language

Alternatives:Recurrent neural networks (not probabilistic)


What are HMMs?

Markov chain:

Hidden states : S1 → S2 → · · · → Sn

↓ ↓Measurements : Z1 · · · Zn

It’s essentially a Bayes Net!

Implementations:Kalman Filter (see Ch. 15)Particle Filter


What are HMMs?

Markov chain:






What are HMMs?

Markov chain:






Video: Lost Robots, Speech Recognition

https://www.youtube.com/watch?v=S_Lm8aN-la0&feature=player_embedded

Future Prediction with Markov Chains

Is tomorrow going to be Rainy or Sunny?

R S

0.4

0.2

0.80.6

Start with “today is rainy”: P(R0) = 1, then P(S0) = 0What’s

P(S1) = ?

P(S2) = ?

P(S3) = ?

P(St+1) = 0.4× P(Rt) + 0.8× P(St)




R S

0.4

0.2

0.80.6

Start with “today is rainy”: P(R0) = 1, then P(S0) = 0

What’sP(S1) = ?

P(S2) = ?

P(S3) = ?

P(St+1) = 0.4× P(Rt) + 0.8× P(St)




R S

0.4

0.2

0.80.6


P(S1) = ?

P(S2) = ?

P(S3) = ?

P(St+1) = 0.4× P(Rt) + 0.8× P(St)




R S

0.4

0.2

0.80.6


P(S1) = 0.4

P(S2) = ?

P(S3) = ?

P(St+1) = 0.4× P(Rt) + 0.8× P(St)




R S

0.4

0.2

0.80.6


P(S1) = 0.4

P(S2) = 0.56

P(S3) = ?

P(St+1) = 0.4× P(Rt) + 0.8× P(St)




R S

0.4

0.2

0.80.6


P(S1) = 0.4

P(S2) = 0.56

P(S3) = 0.624

P(St+1) = 0.4× P(Rt) + 0.8× P(St)




R S

0.4

0.2

0.80.6


P(S1) = 0.4

P(S2) = 0.56

P(S3) = 0.624

P(St+1) = 0.4× P(Rt) + 0.8× P(St)


Back to the Future?

How far can we see into the future? P(A∞) =?

Until it reaches a stationary state (or limit cycle)

Use calculus:lim

t→∞P(At+1) = P(At)

R S

0.4

0.2

0.80.6

P(S∞) = ?

limt→∞

P(St+1) = 0.4× P(Rt) + 0.8× P(St),

subst. x = P(St+1) = P(St) = 1− P(Rt) = · · ·


Back to the Future?


Until it reaches a stationary state (or limit cycle)Use calculus:

limt→∞

P(At+1) = P(At)

R S

0.4

0.2

0.80.6

P(S∞) = ?

limt→∞

P(St+1) = 0.4× P(Rt) + 0.8× P(St),



Back to the Future?



limt→∞

P(At+1) = P(At)

R S

0.4

0.2

0.80.6

P(S∞) = ?

limt→∞

P(St+1) = 0.4× P(Rt) + 0.8× P(St),



Back to the Future?



limt→∞

P(At+1) = P(At)

R S

0.4

0.2

0.80.6

P(S∞) = 2/3

limt→∞

P(St+1) = 0.4× P(Rt) + 0.8× P(St),



Back to the Future?



limt→∞

P(At+1) = P(At)

R S

0.4

0.2

0.80.6

P(S∞) = 2/3

limt→∞

P(St+1) = 0.4× P(Rt) + 0.8× P(St),



And How Do We Get The Transition Probabilities?

R S

?

?

??

Observed sequence in Atlanta ,: RRSRRRSRUse Maximum Likelihood

P(S |S) = ?

P(R|S) = 2/2, P(S |R) = 2/5, P(R|R) = 3/5

Edge effects? P(S |S) = 0?



R S

?

?

??


P(S |S) = ?

P(R|S) = 2/2, P(S |R) = 2/5, P(R|R) = 3/5




R S

?

?

??


P(S |S) = observed transitionstotal transitions from S

=02

P(R|S) = 2/2, P(S |R) = 2/5, P(R|R) = 3/5




R S

?

?

??



=02

P(R|S) = 2/2, P(S |R) = 2/5, P(R|R) = 3/5




R S

?

?

??



=02

P(R|S) = 2/2, P(S |R) = 2/5, P(R|R) = 3/5



Overcoming Overfitting: Remember Laplacian Smoothing?

Observed sequence in Atlanta ,: RRSRRRSR

Laplacian smoothing K = 1


K ,N selected such that 0 ≤ P ≤ 1.





P(S |S) = observed transitions + Ktotal transitions from S + N






P(S |S) = observed transitions + Ktotal transitions from S + N

=0+ 12+ 2



Where is Markov Hidden?

R

H G

S

H G

0.4

0.2

0.80.6

0.4 0.6 0.9 0.1

Hidden:rainy or sunny

Observe:happy or grumpy

Initial conditions

P(R0) = 1/2, P(S0) = 1/2

P(S1|H1) = ?

Bayes rule!



R

H G

S

H G

0.4

0.2

0.80.6

0.4 0.6 0.9 0.1



Initial conditions

P(R0) = 1/2, P(S0) = 1/2

P(S1|H1) = ?

Bayes rule!



R

H G

S

H G

0.4

0.2

0.80.6

0.4 0.6 0.9 0.1



Initial conditions

P(R0) = 1/2, P(S0) = 1/2

P(S1|H1) =P(H1|S1)P(S1)

P(H1)

Bayes rule!Günay Ch. 15,20 – Hidden Markov Models and Particle FilteringSpring 2013 13 / 21

Congrats, Done with Prediction and State Estimation

What else can we do with HMMs?Localization of the lost robotBlindfolded person

Video: Robot localization


https://www.youtube.com/watch?v=Nc9-iLy_rgY&feature=player_embedded

Congrats, Done with Prediction and State Estimation

What else can we do with HMMs?Localization of the lost robotBlindfolded person

Video: Robot localization


https://www.youtube.com/watch?v=Nc9-iLy_rgY&feature=player_embedded

HMMs, Formally



Question:P(S1|S2) ⊥ P(Sn|S2) ?

HMM equations:State estimation: P(S1|Z1) = αP(Z1|S1)P(S1)

Prediction: P(S2) =∑

S1 P(S2|S1)P(S1)


HMMs, Formally



Question:P(S1|S2) ⊥ P(Sn|S2) Yes! Past and future are independent.



S1 P(S2|S1)P(S1)


HMMs, Formally






S1 P(S2|S1)P(S1)


HMMs, Formally






S1 P(S2|S1)P(S1)


HMMs for Localization Example

Robot knows map, but not location: use multiplication and convolution

















Particle Filters: For Clean Water?

Nope, but same idea.

Video: Robot localization with particle filters

Belief representationPoints are hypothesesParticles survive if consistent with measurementsEasy implementation!


https://www.youtube.com/watch?v=H0G1yslM5rc&feature=player_embedded













Localization with Particle Filters

Particle filtering: weights show likelihood; pick particles, shift, and repeat

Continuous space!Computational resources used efficiently!






















Particle Filter Algorithm

S: Particle set {< x ,w >, . . .},U: Control vector (e.g., map),Z: Measure vector

S ′ = Ø, η = 0For i=1. . . nsample j ∼ {w} w/ replacementx ′ ∼ P(x ′|U, Sj)w ′ = P(Z |x ′)η = η + w ′

S ′ = S ′ ∪ {< x ′,w ′ >}EndFor i=1. . . n // Normalization step

wi =1ηwi

End


Particle Filter Pros & Cons

In general works well! Stanley uses it for navigation.

Pros:Easy to implementEfficientComplex and changing environments in robotics

Cons:Dimensionality problem: need many particlesProblems with degenerate conditions (adding noise may help)


Time Series Prediction Conclusion

Particle filtering:Most widely used algorithm!Can handle time series and uncertainty

Other application areas:Financial predictionWeather

Alternative methods:Kalman filtersRecurrent neural nets


Documents

CS325ArtiﬁcialIntelligence Ch.15,20