Dieter Fox University of Washington Department of Computer
Science & Engineering
Slide 2
Joint work with Students: Brian Ferris, Julie Letchner, Lin
Liao, Don Patterson, Alvin Raj, Amarnag Subramanya, Danny Wyatt UW
faculty: Jeff Bilmes, Gaetano Borriello, Henry Kautz Intel Research
Seattle: Tanzeem Choudhury, Matthai Philipose Funded by NSF, DARPA,
MSR, Toyota Dieter Fox: Activity Recognition From Wearable
Sensors2573 AU-08
Slide 3
U.S. by 2050: 80 million people over age 65 and 13-16 million
Alzheimer's patients Goal: Develop technology to Support
independent living by people with cognitive disabilities At home At
work Throughout community Improve health care Long term monitoring
of activities of daily living (ADLs) Intervention before a health
crisis 573 AU-08Dieter Fox: Activity Recognition From Wearable
Sensors3
Slide 4
4 Transportation routines via Dynamic Bayes Nets Significant
places via Conditional Random Fields Activity recognition projects
Discussion 573 AU-08
Slide 5
Given data stream from a wearable GPS unit Infer the users
location and mode of transportation (foot, car, bus, bike,...)
Predict where user will go Detect novel behavior / user errors 573
AU-08Dieter Fox: Activity Recognition From Wearable Sensors5
[Liao-Fox-Kautz: AAAI-04, AIJ-07]
Slide 6
Dead and semi-dead zones near buildings, trees, etc. Sparse
measurements inside vehicles, especially bus Multi-path propagation
Inaccurate street map 573 AU-08Dieter Fox: Activity Recognition
From Wearable Sensors6
Slide 7
Map is directed graph Location: Edge e Distance d from start of
edge Prediction: Move along edges according to velocity model
Correction: Update estimate based on GPS reading 573 AU-08Dieter
Fox: Activity Recognition From Wearable Sensors7
Slide 8
573 AU-08Dieter Fox: Activity Recognition From Wearable
Sensors8 GPS reading z k-1 zkzk Edge, velocity, position x k-1 xkxk
Time k-1 Time k Task: Estimate posterior over hidden states
Slide 9
Slide 10
Problem: Predicted location is multi-modal 573 AU-08Dieter Fox:
Activity Recognition From Wearable Sensors10 e 1 e 2 e 3 k-1 x e 0
? ?
Slide 11
Problem: GPS reading is not on the graph 573 AU-08Dieter Fox:
Activity Recognition From Wearable Sensors11 e 1 e 2 e 3 z k k
x
Slide 12
573 AU-08Dieter Fox: Activity Recognition From Wearable
Sensors12 e 1 e 2 e 3 z k k x if =e 1 k x Problem: GPS reading is
not on the graph
Slide 13
573 AU-08Dieter Fox: Activity Recognition From Wearable
Sensors13 e 1 e 2 e 3 z k k x if =e 2 k x Problem: GPS reading is
not on the graph
Slide 14
573 AU-08Dieter Fox: Activity Recognition From Wearable
Sensors14 GPS reading z k-1 zkzk Edge, velocity, position x k-1
xkxk Time k-1 Time k Task: Estimate posterior over all hidden
states k-1 k GPS association e k-1 ekek Edge transition
Slide 15
STREET MAP Source: Tiger / Line data BUS ROUTES / STOPS Source:
Metro GIS 573 AU-08Dieter Fox: Activity Recognition From Wearable
Sensors15 RESTAURANTS / STORES Source: MS MapPoint
Slide 16
Represents posterior by sets of weighted particles: Basic idea:
Sample some variables of state space and solve other variables
analytically conditioned on samples Here: Each particle contains
Kalman filter for location: Update sets by sampling procedure
(SISR) 573 AU-08Dieter Fox: Activity Recognition From Wearable
Sensors16 Edge transitions, velocities, edge associations KF for
position
573 AU-08Dieter Fox: Activity Recognition From Wearable
Sensors18 GPS reading z k-1 zkzk Edge, velocity, position x k-1
xkxk Time k-1 Time k m k-1 mkmk Transportation mode Particles:
Slide 19
Encode prior knowledge into the model Modes have different
velocity distributions Buses run on bus routes Get on/off the bus
near bus stops Switch to car near car location 573 AU-08Dieter Fox:
Activity Recognition From Wearable Sensors19
Slide 20
573 AU-08Dieter Fox: Activity Recognition From Wearable
Sensors20 Measurements Projections Bus mode Car mode Foot mode
Green Red Blue
Slide 21
Goal (destination): workplace (home, friends, restaurant,...)
Trip segments: Home to Bus stop A on Foot Bus stop A to Bus stop B
on Bus Bus stop B to workplace on Foot 573 AU-08Dieter Fox:
Activity Recognition From Wearable Sensors21 BAWorkplace
Slide 22
573 AU-08Dieter Fox: Activity Recognition From Wearable
Sensors22 GPS reading z k-1 zkzk Edge, velocity, position x k-1
xkxk Time k-1 Time k m k-1 mkmk Transportation mode t k-1 tktk Trip
segment g k-1 gkgk Goal Particles: [Bui-Venkatesh-West: JAIR-02]
[Liao-Fox-Kautz: AAAI-04, AIJ-07]
Slide 23
Customized model for each user Key to goal / path prediction
and error detection Use low level model to learn variable domains
Goals: locations where user stays for long time Transition points:
locations with high mode transition probability Trip segments:
connecting transition points or goals Use full model to learn
parameters Transition probabilities (goals, trips, edges) via EM
573 AU-08Dieter Fox: Activity Recognition From Wearable
Sensors23
573 AU-08Dieter Fox: Activity Recognition From Wearable
Sensors27 GPS reading z k-1 zkzk Edge, velocity, position x k-1
xkxk Time k-1 Time k m k-1 mkmk Transportation mode t k-1 tktk Trip
segment g k-1 gkgk Goal b k-1 bkbk Behavior mode normal / unknown /
error [Patterson-Liao-etAl: Ubicomp-04]
Slide 28
573 AU-08Dieter Fox: Activity Recognition From Wearable
Sensors28
Slide 29
573 AU-08Dieter Fox: Activity Recognition From Wearable
Sensors29 Untrained flat model Trained model Instantiated
model
Slide 30
573 AU-08Dieter Fox: Activity Recognition From Wearable
Sensors30
Slide 31
Dieter Fox: Activity Recognition From Wearable Sensors31
Transportation routines via Dynamic Bayes Nets Significant places
via Conditional Random Fields Activity recognition projects
Discussion 573 AU-08
Slide 32
So far No distinction between different types of goals Model
not transferable to other person Goals learned solely based on
duration 573 AU-08Dieter Fox: Activity Recognition From Wearable
Sensors32 [Liao-Fox-Kautz: NIPS-05,IJCAI-05,IJRR-07]
Slide 33
573 AU-08Dieter Fox: Activity Recognition From Wearable
Sensors33 Store Home Bus stop Parking Friend Restaurant Work
Slide 34
Undirected graphical model Introduced for labeling sequence
data [Lafferty-et al: ICML-01] No independence assumption on
observations! Trained discriminatively from labeled data Applied
successfully to Natural language processing [McCallum-Li:
CoNLL-03], [Roth-Yih: ICML-05] Computer vision [Kumar-Hebert:
NIPS-04], [Quattoni-Collins-Darrel: NIPS-05] Robotics
[Limketkai-Liao-Fox: IJCAI-05], [Friedman-Pasula-Fox: IJCAI-07]
Dieter Fox: Activity Recognition From Wearable Sensors34573
AU-08
Slide 35
Posterior factorizes into log-linear clique potentials Local
potentials link states to observations / features Neighborhood
potentials link states to neighboring states Dieter Fox: Activity
Recognition From Wearable Sensors35 Hidden states x Observations z
573 AU-08 Conditional Random Fields
Slide 36
Posterior factorizes into log-linear clique potentials Local
potentials link states to observations / features Neighborhood
potentials link states to neighboring states Dieter Fox: Activity
Recognition From Wearable Sensors36 Hidden states x Observations z
573 AU-08 Conditional Random Fields
Slide 37
Posterior factorizes into log-linear clique potentials Local
potentials link states to observations / features Neighborhood
potentials link states to neighboring states Dieter Fox: Activity
Recognition From Wearable Sensors37 Hidden states x Observations z
573 AU-08 Conditional Random Fields Partition function
WeightsFeature functions
Slide 38
Conditional distribution parameterized via weights w: Maximize
conditional log-likelihood with shrinkage prior: Maximization via
stochastic / conjugate gradient, L-BFGS Alternative: maximize
pseudo log-likelihood Dieter Fox: Activity Recognition From
Wearable Sensors38573 AU-08 [Besag: 1975]
Slide 39
BP computes posteriors via local message passing Sum-product
for posterior Max-product for MAP Exact if network has no loops
Otherwise, run loopy belief propagation and hope it works
Alternatives: ICM, graph-cut, MCMC, Dieter Fox: Activity
Recognition From Wearable Sensors39573 AU-08....
Slide 40
Temporal pattern: duration, time of day, etc. Geographic
evidence: is there a restaurant / store / bus stop nearby?
Transition relation: adjacent activities Spatial feature: relation
between place and activity Summation constraint: number of places
labeled home or workplace 573 AU-08Dieter Fox: Activity Recognition
From Wearable Sensors40
Slide 41
GPS trace 573 AU-08Dieter Fox: Activity Recognition From
Wearable Sensors41 Activity sequence walk, drive, ride bus, work,
visit, sleep, pickup, get on/off bus a1a1 a2a2 a3a3 a4a4 a5a5 a N-2
a N-1 aNaN [Liao-Fox-Kautz: NIPS-05]
Slide 42
573 AU-08Dieter Fox: Activity Recognition From Wearable
Sensors42 GPS trace Two weeks data (> 50,000 gps points)
Inference: loopy BP (10 minutes); Learning: Pseudo-likelihood (3
minutes) Activity sequence walk, drive, ride bus, work, visit,
sleep, pickup, get on/off bus a1a1 a2a2 a3a3 a4a4 a5a5 a N-2 a N-1
aNaN Significant places home, work, bus stop, parking lot, friends
home p1p1 p2p2 p3p3 pKpK Global, soft constraints # homes,
workplaces wh [Liao-Fox-Kautz: NIPS-05]
Slide 43
Generalize from people with labeled data to others without
labeled data Relational Markov Networks to specify CRFs and
parameter sharing via templates 573 AU-08Dieter Fox: Activity
Recognition From Wearable Sensors43 Training data User1 User2 User3
Generic model Other users Supervised learning Shared parameters
[Taskar-Abbeel-Koller: UAI-02]
Slide 44
573 AU-08Dieter Fox: Activity Recognition From Wearable
Sensors44
Slide 45
573 AU-08Dieter Fox: Activity Recognition From Wearable
Sensors45
Slide 46
573 AU-08Dieter Fox: Activity Recognition From Wearable
Sensors46
Slide 47
573 AU-08Dieter Fox: Activity Recognition From Wearable
Sensors47 Cross-validation using data from 4 persons
Slide 48
573 AU-08Dieter Fox: Activity Recognition From Wearable
Sensors48
Slide 49
Dieter Fox: Activity Recognition From Wearable Sensors49
Transportation routines via Dynamic Bayes Nets Significant places
via Conditional Random Fields Activity recognition projects
Discussion 573 AU-08
Slide 50
Dieter Fox: Activity Recognition From Wearable Sensors50
[Ferris-Haehnel-Fox: RSS-06]
Slide 51
GP regression nonparametric smooth interpolation uncertainty
estimates 573 AU-08 Dieter Fox: Activity Recognition From Wearable
Sensors 51 Mean Variance
Slide 52
573 AU-08Dieter Fox: Activity Recognition From Wearable
Sensors52
Slide 53
573 AU-08Dieter Fox: Activity Recognition From Wearable
Sensors53 Battery Camera (on ribbon cable) GPS receiver iMote2 +
two sensor boards MicrophoneCamera Light sensors 2 GB SD card
Indicator LEDs Records 4 hours of audio, images (1/sec), GPS, and
sensor data (accelerometer, barometric pressure, light intensity,
gyroscope, magnetometer)
Slide 54
573 AU-08Dieter Fox: Activity Recognition From Wearable
Sensors54 Courtesy of G. Borriello
573 AU-08Dieter Fox: Activity Recognition From Wearable
Sensors58
Slide 59
59573 AU-08Dieter Fox: Activity Recognition From Wearable
Sensors
Slide 60
Performs object aggregation and smoothing over object types
Accuracy: 88% SED: 6.4 60573 AU-08Dieter Fox: Activity Recognition
From Wearable Sensors
Slide 61
61 Transportation routines via Dynamic Bayes Nets Significant
places via Conditional Random Fields Activity recognition projects
Discussion 573 AU-08
Slide 62
Graph-based representation well suited to compactly represent
and learn motion patterns Hierarchical graphical models (DBN, CRF)
powerful tools for bridging gap between continuous sensor data and
abstract states Conditional Random Fields can handle high-
dimensional / dependent feature vectors Sensor-based activity
recognition is an exciting research area with potentially huge
impact! 573 AU-08Dieter Fox: Activity Recognition From Wearable
Sensors62
![22-gp-473 - courses.cs.washington.edu · 2015. 6. 1. · Pictures-from--[Bishop:-PRML,-2006]-Dieter Fox CSE-473: Introduction to AI a 4 p(x a |x b)= ... Dieter Fox CSE-473: Introduction](https://img.pdfslide.net/doc/110x75/6132425edfd10f4dd73a55a7/22-gp-473-2015-6-1-pictures-from-bishop-prml-2006-dieter-fox-cse-473.jpg)
