Recognising Situations in context aware systemsusing Dempster-Shafer Theory

Dr. Susan McKeeverNov 4th 2013

Context Aware systems – e.g. Smart home

• Sensors in a smart home• Situation tracking – what is the user doing? What

activity are they undertaking?• E.g Monitoring elderly

Context Aware systems• Pervasive /ubiquitious /ambient systems – embedded

in the environment s• E.g. intelligent homes, location tracking system

• They understand their own “context”.• Context-awareness is the ability to track the state of

the environment in order to identify situations

• Situations are human understandable representations of the environment, derived from sensor data

Research focus: e.g .Gator Tech Smart home

Van Kasteren sensored smart home

14 digital sensors For a month:

7 Situations:Preparing breakfast, dinner, drink, leave house, use toilet, take shower, go to bed

Abstracting sensor data to situations

Location sensor reading(X,Y,Z, ID239, 12:30:04)

Sensor 1, 2, 3

Abstracted Context

Situations

John located in Kitchen @ time 12:30

John is ‘preparing meal’

Is abstracted to

Is evidence of

Sensor 1, 2, 3Sensor 1, 2, 3

Application e.g. elderly alert system

Sensor dataSituation

Recognition

Situation(s) occurring at time, t

12:53 preparing breafast

(12:53, 0)(2.15,5.04,3.16, 12:34)

Situation Recognition

Knowledge• Expert? Past data?

• Situation recognition is a critical, continuous, dynamic process – often required in real time.

• The recognition process is difficult and uncertain – no single approach suitable for all

Situation Recognition - ScenarioScenario“The person is in the kitchen. It is morning time. They carry out a series of tasks, such as taking cereal out of the groceries cupboard, using the kettle, opening the fridge, and using the toaster”

Human Observer: “preparing breakfast”

Why?

•Individual tasks may not confirm that breakfast is in progress, but together, indicate the ’preparing breakfast’ situation.

•Morning time

•Informative sensors e.g. toaster

Recognising situations – Automated

Sensor overlap - Kettle and fridge: ’preparing drink?

Different people “prepare breakfast” in different ways.. Individual efinitions?

Gaps of seconds or minutes occuring with no sensor activity – classify?

Sensors can breakdown and have error rate – toaster sensor doesn’t fire?

As more tasks are done, system is more certain of ‘preparing breakfast situation’ – Temporal aspect

The person does not prepare breakfast in the same way every day.

The tasks are not necessarily performed in any particular order.

Co-occurring situations? (’on telephone’); Cannot o-occur (’user asleep’)? -Valid combinations of situations.

A second occupant now enters the kitchen – how to distinguish?

Recognising situations – Some approaches

• Machine learning techniques, inc.• Bayesian networks• Decision trees• Hidden Markhov modelsreliant on training data

• Specification based approaches, inc.• Logic approaches• Fuzzy logic• Temporal logic

Problems to be solved (not exhaustive)How to recognise situations in pervasive

environments, allowing for particular challenges:

1. Uncertainty (sensor data, situation definitions, context fuzziness)

2. Difficulties in obtaining training data

My solution: Use and enhance evidence theory (Dempster Shafer theory)

Why Dempster Shafer theoryDevised in 1970s

Mathematical theory for combining separate pieces of information (evidence) to calculate the belief in an event.

Applied in military applications, cartography, image processing, expert systems, risk management, robotics and medical diagnosis

Key features:(1) its ability to specifically quantify and preserve

uncertainty (2) its facility for assigning evidence to combinations

Various researchers applying in pervasive systems

Approach

• Apply Dempster Shafer (evidence) theory to situation recognition• Create a network structure to propagate

evidence from sensors

• Extend the theory to allow for:• New operations needed support evidence

processing of situation• Temporal features of situation• Rich (static and dynamic) sensor quality

Dempster Shafer theory: ExampleTwo sensors are used to detect user location in an office.

The locations of interest are: (1) Cafe, (2) the user’s desk, (3) the meeting room and (4)

‘lobby’ in the building.

Meeting roomCafé User’s desk Lobby

Sensor 1 Sensor 2

Any uncertainty is assigned to ‘ignorance’ hypthesis 𝞱– {desk ^ cafe ^ meetingRoom ^ lobby}

Frame of Discernment‘hypotheses’

(allows combinations)

Each sensors assigns belief as a ‘mass function’ which totals per sensor to 1

Evidence sources

Dempster Shafer theory: Example

Sensor 1Detects the user’s location in the cafe. The sensor is 70% reliable, so its belief is assigned across the frame as {cafe 0:7; 0:3 𝞱)Sensor 2

The second sensor has conflicting evidence, assigning{meetingRoom 0:2, desk ^cafe^lobby 0:6, 0:2 𝞱}To combine evidence source:Use dempster combination rule

mass functions

Dempster Shafer theory: Combination rule

M12 (A) is the combination of two evidence sources or mass functions for a hypotheses A.

Denominator is a normalisation factor 1-K where K = conflicting evidence

Evidence sources must sum to 1:

Dempster Shafer theory: example

Conflict (K ) = 0.14 ;

All evidence is normalised by 1-K giving:

Café 0.65; meeting 0.07; desk/café/lobby 0.21, uncertainty 0.07

Sensor 1

Sensor 2Combined evidence

Dempster Shafer theory: problems

Zadeh’s paradox

Conflicting sensor: Appear to agree completely if any agreement – not intuitive

Dempster Shafer theory: problems

Single sensor dominance

A single sensor can overrule a majority of agreeing sensors if it disagrees:

e.G .if 5 sensors determine a user location in a house, a single “categorical” (certain) sensor that assigns all its belief to a contradictory option will negate the evidence from the remaining 4.

Sensor 1 Sensor 2 Sensor 3 Sensor5Sensor 4

Kitchen 0.7

Kitchen 0.6

Kitchen 0.8

Kitchen 0.9

Sitting room

1

Dempster Shafer theory: gapsNo support for evidence spread over time.Assumes evidence is all co-occuring but in reality evidencemay be spread over time.

e.g. detecting “prepare dinner” situation detected by sensors on cupboards and fridges.

GroceriesCupboardAccessed

FridgeAccessed

Freezer Accessed

PansCupboardAccessed

PlatesCupboardAccessed

Prepare Dinner Timeline

40 minutes

Dempster Shafer theory: gapsOnly deals with fusing evidence: no “theory” for propogating evidence across other rules in order to recognise situations Limited to just combining n “sources”: Need a set of additional mathemtical operations for propogating evidence

Sensor 1, 2, 3

Abstracted Context

Situations

Sensor 1, 2, 3Sensor 1,

2, 3Sensor 1,

2, 3

Abstracted Context

Situations

Sensor 1, 2, 3Sensor 1,

2, 3Sensor 1,

2, 3

Abstracted Context

Situations

Sensor 1, 2, 3Sensor 1,

2, 3Location sensor reading(X,Y,Z, ID239, 12:30:04)

John located in Kitchen @ time 12:30

John is ‘preparing meal’

Is abstracted to

Is evidence of

sensor Sensor

ContextValue

situation SituationSituation

Sensor

ContextValue

ContextValue

ContextValue

ContextValue

ContextValue Context

Value

Certainty0.n

Certainty0.n

Certainty0.n

Sensor Level

Abstracted Context

Situations

sensor sensor sensor

situation situation

Dempster Shafer theory: gapsOnly deals with fusing evidence: no “theory” for propogating evidence across other rules in order to recognise situations (and a way to capture all this knowledge)

Recognising situations – Using Dempster Shafer theory

• Want an approach that reduces or eliminates reliance on training data. OK (provided we can define mass functions to say what sensor readings mean)

• That allows for “uncertainty” OK • That allows temporal information to be included To be added• That allows sensors belief to be propogated (distributed) up into

situation hierachies based on “knowledge” rules To be added• That addresses the issue of Zadeh’s paradox and dominant

sensors To be added• Ultimately: Develop a full decision making architecture for real

time situation recognition (overleaf) To be added

Needed to extend Dempster Shafer theory

Knowlege

Sensor Readings

Belief Distribution

DecisionStage

RecognisedSituations

Valid situationcombinations

At time t

Applicati-ons

Develop a full decision making architecture for real time situation recognition using extended DS theory

Extended DS theory

Prep Breakfast 0.3,Take a shower 0.6

Knowledge: an interconnected hierarchy of sensor and situations

sensor Sensor

ContextValue

situation SituationSituation

Sensor

ContextValue

ContextValue

ContextValue

ContextValue

ContextValue Context

Value

Certainty0.n

Certainty0.n

Certainty0.n

Sensor Level

Abstracted Context

Situations

sensor sensor sensor

situation situation

PlatesUsed

CupUsed

FridgeUsed

GroceriesUsed

MicrowaveUsed

Pans Used

FreezerUsed

GetDrink

PrepareBreakfast

PrepareDinner

<2> <15> <62>

0.80.2

0.8 0.8

0.4

0.8

Morning

PlatesCupboard

Cup Fridge GroceriesCupboard

Microwave Pans CupboadrFreezer TimeMoning

Nighttime

VanKasteren e.g. 3 of the situations

First : Define a notation for knowledge capture : denoting sensor evidence /context/ situations –

Situation DAG

sensor Sensor

Situation Situation

Situation

ContextValue

Certainty0.n

Certainty0.n

Certainty0.n

Discount0.n

< 5> > 10 >

ContextValue

ContextValue

ContextValue

ContextValue

ContextValue

Belief distribution

Situations

Sensors

Context Values

Belief distribution

First : Define a notation for denoting sensor evidence /context/ situations – Situation DAG i.e to capture the knowledge of what sensors indicate what situation

is a type of

is evidence of

< duration> Duration of situation, evidence not in sequenceDuration of situation, evidence in sequence

>duration > Sensor, context value or situation

Discount 0.n Discount factor applied to a sensor: 0< n <1

Certainty 0.n Certainty applied to an inference rule: 0 < n < 1

Second: Create evidence propogation rules to distribute/propogate belief up to situation level

sensor Sensor

ContextValue

situation SituationSituation

Sensor

ContextValue

ContextValue

ContextValue

ContextValue

ContextValue Context

Value

Certainty0.n

Certainty0.n

Certainty0.n

Sensor Level

Abstracted Context

Situations

sensor sensor sensor

situation situation

TranslateSensor readings into beliefs here ..

Up to situation certainties here

Second: Create evidence propogation rules to distribute/propogate belief up to situation level

sensor Sensor

ContextValue

situation SituationSituation

Sensor

ContextValue

ContextValue

ContextValue

ContextValue

ContextValue Context

Value

Certainty0.n

Certainty0.n

Certainty0.n

Sensor Level

Abstracted Context

Situations

sensor sensor sensor

situation situation

Is a type of:

e.g. Situation X is occuring if either Situation Y OR Z is occuringOccupant is “resting” if they are “watching TV” or “in bed”

Second: Create evidence propogation rules to distribute/propogate belief up to situation level:

Examples

Distributing combined belief across single situations

Second: Create evidence propogation rules to distribute/propogate belief up to situation level:

Examples: Sensor QualitySome sensors are inherently lower quality as an evidence source

e.g. Calendar sensor is indicative of real calendar owner’s location 70% of the time – Discount (d) evidence from the sensor

Third: Include temporal evidence:

GroceriesCupboardAccessed

GroceryCupboardaccessed

Freezer Accessed

PlatesCupboardAccessed

FridgeAccessed

Prepare Dinner Timeline40 minutes

Different Sensors fire intermittently – no single sensor sufficient for situation recognition

(1) Use absolute time as evidence(2) Find a way to combine transitory evidence

GroceriesCupboardAccessed

FridgeAccessed

Freezer Accessed

PansCupboardAccessed

PlatesCupboardAccessed

Prepare Dinner: Time Extended Evidence Time

FridgeExtended

FridgeExtended

FridgeExtended

FridgeExtended

FridgeExtended

GroceriesCupboardExtended

GroceriesCupboardExtended

GroceriesCupboardExtended

GroceriesCupboardExtended

PlatesCupboardExtended

PlatesCupboardExtended

PlatesCupboardExtended

Freezer Extended

Freezer ExtendedPansCupboardExtended

Prepare DinnerStarts

Prepare BreakfastEndsSituation

Duration

Third: extend evidence for duration of situation

Fusing time extended evidence:

Adjust Dempster Shafer fusion rules to allow for time extension of evidence

Two transitory extended mass functions for hypothesis h with duration t dur, a t time t +t rem

Fourth: Allow for Zadeh’s and Single sensor dominance

Use an alternative combination rule (Murphy’s) which averages out the evidence BEFORE fusing

Use a simpler averaging rule to fuse evidenceLacks convergenceRemoves Zadeh’s problem

Two options:

Fifth: Combine all this and apply to real world data for situation recogntion

Knowlege

Sensor Readings

Belief Distribution

Decision

StageRecognisedSituations

Valid situation

combinations

At time t

Applicati-ons

Extended DS theory

Prep Breakfast 0.3,Take a shower 0.6

Test our approach using annotated datasets of sensor readings

ExperimentsData set (1) “Van Kasteren”Heavily used by other researchers - compare results on situation recognition 7 situation annotated, 14 sensors

Data set (2)“CASL”Office data set: 3 situations annotated, •Location sensors, •Calendar sensor, •Keyboard sensor

Question Data set1 How accuracy is our DS

approach for situation recognition?

Both

2 Do DS temporal extensions improve situation recognition?

Van Kasteren

3 Do DS quality extensions improve situation recognition?

CASL

Evaluation

Various sub questions also addressed: comparison with published results, comparison of DS fusion rules, impact of quality on situation transitions, quality parameter sensitivity, static versus dynamic quality

Evaluation1. 2 annotated published real world datasets –

VanKasteren (Smart home) and CASL (office-based)

2. Situation DAGs created for both datasets

3. Situation recognition accuracy measured using f-measure of timesliced data sets;

4. Recognition accuracy using temporal and quality extensions evaluated

5. J45 Decision Tree and Naive Bayes used for comparison , and published results ; Cross validation used.

leave house

use toilet take shower

go to bed prepare breakfast

prepare dinner

get drink0.00

0.20

0.40

0.60

0.80

1.00

No time Absolute time Time Extended

Use of DS theory with temporal extensions for situation recognition

F-Measure for each situation using DS theory – (1) no time, (2) absolute time, (3) time extended (VanKasteren dataset )

Temporal DS theory compared to two other approches: Naïve Bayes, J48 decision tree.

leave house

use toilet take shower

go to bed prepare breakfast

prepare dinner

get drink0

0.2

0.4

0.6

0.8

1

No time EDN Temporal EDN Naïve Bayes J48

Situations

Our approach compared to the three available published results

Same experimental measures

* Excludes timeslices with no sensors firing which are harder to infer – ‘inactive’ Timeslices harder to infer

*

Use of DS theory with temporal extensions • Use of temporal extensions significantly

improves situation accuracy (over baseline DS theory alone)

• Performs better than J45, Naive Bayes (particularly with limited training data). This improvement narrows when more training data used (LODO)

• Achieves 69% class accuracy in comparison to VanKasteren (49.2%) and Ye*(88.3%)

Use of DS theory with quality extensions

0.00

0.20

0.40

0.60

0.80

1.00

No Quality With Quality

F-Measure for each situation using DS theory – with and without quality

• Use of quality parameters significantly improves situation recognition accuracy (over baseline)

• Performance close to Naive Bayes (4%) and J48 (2%) -

• Each individual sensor’s quality contributes to improvement

• Sensitivity analysis of quality parameters indicates the relative quality of sensors may be important

• Time based dynamic quality parameters impact situation transitions – application dependant

Use of DS theory with quality extensions

Our DS theory is a viable approach to situation recognition:• Not reliant on training data• Incorporates domain knowledge• Caters for uncertainty• Encoding temporal and quality knowledge improves

performance over basic DS approachBUT

• Knowledge must be available• Different fusion rules appropriate in different scenarios –

requires expert “evidence theory” knowledge• Environment changes – no feedback loop for drift• Potentially high computation effort can be reduced

Conclusions

Contributions1. A situation recognition approach based on DS

theory 2. Selection of existing and creation of new

evidential operations and algorithms to create evidence decision networks

3. Temporal and quality extensions to DS theory 4. Diagramming technique to capture structure of

evidence for an environment (Situation DAG)5. A thorough application, evaluation and analysis

of the extended DS theory approach6. An analysis of alternative fusion rules

Related Publications• Journal

1. Journal of Pervasive and Mobile Computing2. JAISE Volume 2, Number 2 2010

• International Conferences1. EuroSSC Smart Sensing UK 20092. ICITST Pervasive Services Italy 2008

• International (Peer viewed) Workshops1. Pervasive 2010, Helsinki, Finland2. CHI 2009 Boston, US3. QualConn 2009, Stuttgart, Germany4. Pervasive 2009, Sydney, Australia,

Questions?

ExperimentsEstablish situation DAG for each dataset

SystemDevelopers

-Users-Application

experts

Sensors

ContextValues

Situations