d912f50edb2226e921

8/12/2019 d912f50edb2226e921

1/9

Accurate Activity Recognition in a Home Setting

Tim van Kasteren, Athanasios Noulas, Gwenn Englebienne and Ben KroseIntelligent Systems Lab Amsterdam

University of Amsterdam

Kruislaan 403, 1098 SJ, Amsterdam , The [email protected]

ABSTRACT

A sensor system capable of automatically recognizing activ-ities would allow many potential ubiquitous applications. Inthis paper, we present an easy to install sensor network andan accurate but inexpensive annotation method. A recordeddataset consisting of 28 days of sensor data and its anno-tation is described and made available to the community.Through a number of experiments we show how the hid-den Markov model and conditional random fields performin recognizing activities. We achieve a timeslice accuracy of95.6% and a class accuracy of 79.4%.

Author Keywords

Activity Recognition, Probabilistic Models, Dataset, Anno-tation, Sensor Networks

ACM Classification Keywords

I.2.1 - Applications and Expert Systems, I.2.6 - Learning

INTRODUCTION

Activities are a very important piece of information for ubiq-uitous applications [3]. A sensor system capable of auto-matically recognizing activities would allow many potential

applications in areas such as health care, comfort and secu-rity. For example, in elderly care, activities of daily living(ADLs) are used to assess the cognitive and physical capa-bilities of an elderly person [10]. An activity recognitionsystem allows us to automatically monitor their decline overtime and detect anomalies [21].

For activity recognition research, we need a large variety ofdatasets. Many datasets for activity recognition were recordedin houses especially built for research purposes [2, 7, 13].Sensors are installed during construction and people live thereonly for the duration of an experiment. Since people are un-familiar with the house the resulting data is not very repre-sentative. Also, recording datasets in the same house over

and over again makes research sensitive to the architectureof that particular house.

Permission to make digital or hard copies of all or part of this work forpersonal or classroom use is granted without fee provided that copies arenot made or distributed for profit or commercial advantage and that copiesbear this notice and the full citation on the first page. To copy otherwise, orrepublish, to post on servers or to redistribute to lists, requires prior specificpermission and/or a fee.UbiComp08, September 21-24, 2008, Seoul, Korea.

Copyright 2008 ACM 978-1-60558-136-1/08/09...$5.00.

Furthermore, accurate annotation of activities is also impor-tant. It is required for evaluating the performance of recog-nition models and allows the use of supervised models frommachine learning. Current annotation methods are eithervery expensive [13] or have limited accuracy [18].

Our paper contributes on the following topics. First, wepresent a sensor network setup that can be easily installedand used in different houses. Second, we present an inex-pensive and accurate method for annotation and provide thesoftware for doing so. Third, we offer our sensor data andits annotation online, so that it can be shared by the researchcommunity. Fourth, we run a number of experiments on oursensor data showing how to effectively use a probabilisticmodel for recognizing activities.

The remainder of this paper is organized as follows. In thenext section we discuss related work. After that, we give adescription of our sensor and annotation system, continuedby a description of our recorded dataset. We then presentthe probabilistic models we used for activity recognition anddiscuss the experiments and results. Finally, we conclude bysumming up our findings.

RELATED WORK

The sensors we can use range from various wall-mountedsensors (e.g. reed switches [18, 20], motion detectors [1],cameras [5]) to all sorts of wearables (e.g. accelerometers[12], wrist worn RFID reader [14]). The various technolo-gies differ from each other in terms of price, intrusiveness,ease to install and the type of data they output [16].

Annotation has been performed in many different ways. Theleast interfering method while inhabitants are performingtheir activities is using cameras [13]. The downside of thismethod is that processing video data is very costly. Lesscostly alternatives typically come in the form of self-reporting.The downside here is that the inhabitants are continuouslyaware of the annotation process, which may lead to biasedor unrealistic data. Examples of self-reporting methods arekeeping an activity diary on paper or using a PDA whichtriggers self-report entries either based on the sensor outputor at a constant time interval [8].

Models used for recognizing activities can be probabilis-tic based [5, 14, 20], logic based [11] or hand-crafted [6].Probabilistic models are popular because sensor readings arenoisy and activities are typically performed in a non-deterministicfashion. Different models have been used in different set-

8/12/2019 d912f50edb2226e921

2/9

tings, dynamic Bayesian networks were used to recognizethe activities of multiple people in a house [20]. HiddenMarkov models were used to perform activity recognition atan object usage level using a wrist worn RFID reader [14].Finally, hierarchical hidden Markov models were used toperform activity recognition from video [5].

In previous work we showed how the use of temporal prob-

abilistic models increases the performance in activity recog-nition. However, results strongly suffered from the limitedaccuracy of the annotation in the dataset used [19]. Sincethe annotation is used for training the models, it is very im-portant to get as accurate training examples as possible. Inthis work we will show how accurate annotation results invery good performance results using temporal probabilisticmodels for activity recognition.

SYSTEM

Our system was built upon two main lines: ease of installa-tion and minimal intrusion. Furthermore, we coupled it withan efficient, accurate and inexpensive annotation method.

SensorsOur sensor network consists of wireless network nodes towhich simple off-the-shelf sensors can be attached. Afterconsidering different commercially available wireless net-work kits, we selected the RFM DM 1810 (fig. 1). The RFMkit comes with a very rich and well documented API and thestandard firmware includes an energy efficient network pro-tocol. The kit comes with a base station which is attached toa PC using USB. A new sensor node can be easily added tothe network by a simple pairing procedure, which involvespressing a button on both the base station and the new node.

Figure 1. Wireless sensor node used: RFM DM 1810.

The RFM wireless network node has an analog and digitalinput. It sends an event when the state of the digital inputchanges or when some threshold of the analog input is vio-lated. This allows the nodes to be used with a large varietyof sensors, ranging from simple contact switches to temper-ature or humidity sensors. Special low-energy consumingradio technology, together with an energy saving sleepingmode result in a long battery life. The node can reach a datatransmission rate of 4.8 kb/s, which is enough for the binarysensor data that we need to collect.

Nodes are easily installed using some tape and do not requireany drilling. Because they run on batteries they can be leftrunning for several months and there is no need to connect itto the powernet. Furthermore, because sensors are mountedon walls or inside cupboards, as much out of sight as possi-ble, the intrusion of the sensor system is minimal.

Annotation

Annotation was performed using a bluetooth headset com-bined with speech recognition software. The starting andend point of an activity were annotated out of a predefinedset of commands.

We used the Jabra BT250v bluetooth headset (fig. 2) for an-notation. It has a range up to 10 meters and battery powerfor 300 hours standby or 10 hours active talking. This ismore than enough for a full day of annotation, the headsetwas recharged during sleep. The headset contains a buttonwhich we used to trigger the software to add a new annota-tion entry.

Figure 2. Jabra BT250v bluetooth headset.

The software for storing the annotation was custom made byour research team. It was written in C and combines ele-ments of the bluetooth API with the Microsoft Speech API 1.The bluetooth API was needed to catch the event of the head-set button being pressed and should work with any bluetoothdongle and headset that uses the Widcomm2 bluetooth stack.

The Microsoft Speech API provided an easy way to use bothspeech recognition and text to speech. When the headsetbutton is pressed the speech recognition engine starts listen-ing for commands it can recognize. We created our ownspeech grammar, which contains possible combinations ofcommands the recognition engine could expect. By usingvery distinctive commands such as begin use toilet and be-

gin take shower, the recognition engine had multiple wordsby which it could distinguish different commands. This re-sulted in near perfect recognition results during annotation.The recognized sentence is outputted using the text-to-speechengine. Any errors that do occur can be immediately cor-rected using a correct last command.

1For details about the Microsoft Speech API see:http://www.microsoft.com/speech/2For details about the Widcomm stack see:http://www.broadcom.com/products/bluetooth sdk.php

8/12/2019 d912f50edb2226e921

3/9

Because annotation is provided by the user on the spot thismethod is very efficient and accurate. The use of a headsettogether with speech recognition results in very little inter-ference while performing activities. This results in annota-tion data which requires very little post processing, makingthis a very inexpensive method.

Our custom made annotation software is available for down-

load from:http://www.science.uva.nl/

tlmkaste

ANNOTATED DATA SET

Using our system, we recorded a dataset in the house of a26-year-old man. He lives alone in a three-room apartmentwhere 14 state-change sensors were installed. We only useddigital sensors for this dataset, although the use of analogsensors is possible. Locations of sensors include doors, cup-boards, refrigerator and a toilet flush sensor (see fig. 3). Sen-sors were left unattended, collecting data for 28 days in theapartment. This resulted in 2120 sensor events and 245 ac-tivity instances.

Figure 3. Floorplan of the house, red rectangle boxes indicate sensornodes. (created using: http://www.floorplanner.com/)

Activities were annotated by the subject himself using a blue-tooth headset as described above. Seven different activitieswere annotated, namely: Leave house, Toileting, Show-ering, Sleeping, Preparing breakfast,Preparing dinnerand Preparing a beverage. These activities were chosenbased on the Katz ADL index, a commonly used tool inhealthcare to assess cognitive and physical capabilities of anelderly person [10]. Times at which no activity is annotatedis referred to as Idle. Table 1 shows the number of separateinstances of activities and the percentage of time each activ-ity takes up in the data set. This table clearly shows howsome activities occur very frequently (e.g. toileting), whileothers that occur less frequently have a longer duration andtherefore take up more time (e.g. leaving and sleeping).

The dataset together with its annotations is available for down-

Number Percentageof instances of time

Idle - 11.5%Leaving 34 56.4%Toileting 114 1.0%Showering 23 0.7%Sleeping 24 29.0%Breakfast 20 0.3%

Dinner 10 0.9%Drink 20 0.2%

Table 1. Number of instances and percentage of time activities occur inthe dataset.

load from: http://www.science.uva.nl/tlmkaste

TEMPORAL PROBABILISTIC MODEL

Our objective is to recognize activities from sensor readingsin a house. This is a formidable task, since the label in ques-tion (activity performed) cannot be estimated directly fromour data (sensor readings). Furthermore, the temporal pat-

terns of our data contain valuable information regarding theperformed activity. A suitable framework for this task istemporal probabilistic models.

In order to work with these models, we divide our time se-ries data in time slices of constant length and label the ac-tivity for each slice. We denote the duration of a time slicewitht. We denote a sensor reading for time t as xit, in-dicating whether sensor i fired at least once between timet and time t + t, with xit {0, 1}. In a house withN sensors installed, we define a binary observation vectorxt = (x

1t , x

2t , . . . , x

Nt )

T. The activity at time slice t is de-noted withyt and so formally our task is to find a mappingbetween a sequence of observations x = {x1, x2, . . . , xT}

and a sequence of labels y = {y1, y2, . . . , yT}for a total ofTtime steps (fig. 4).

Figure 4. Showing the relation between sensor readings xi and timeintervalst.

We utilize the framework of probabilistic models to capturethis mapping, which requires us to do two things: First,we have to learn the parameters of our probabilistic modelfrom training data. Second, we have to infer the labels for

8/12/2019 d912f50edb2226e921

4/9

Figure 5. The graphical representation of a HMM. The shaded nodesrepresent observable variables, while the white nodes represent hiddenones.

novel observation sequences. In this section we will presenthow these two steps work for both hidden Markov models(HMMs) and conditional random fields (CRFs).

Hidden Markov Model

The Hidden Markov Model (HMM) is a generative prob-abilistic model consisting of a hidden variable and an ob-servable variable at each time step (fig. 5). In our case the

hidden variable is the activity performed, and the observ-able variable is the vector of sensor readings. There are twodependency assumptions that define this model, representedwith the directed arrows in the figure.

The hidden variable at time t, namely yt, depends onlyon the previous hidden variableyt1(Markov assumption[15]).

The observable variable at time t, namely xt, dependsonly on the hidden variable yt at that time slice.

With these assumptions we can specify an HMM using threeprobability distributions: the distribution over initial statesp(y1); the transition distributionp (yt|yt1)representing the

probability of going from one state to the next; and the ob-servation distributionp (xt|yt) indicating the probability thatthe stateytwould generate observationxt.

Learning the parameters of these distributions correspondsto maximizing the joint probabilityp(x, y)of the paired ob-servation and label sequences in the training data. We canfactorize the joint distribution in terms of the three distribu-tions described above as follows [17]:

p (x, y) =

T

t=1p (yt | yt1)p (xt | yt) (1)

in which we write the distribution over initial states p(y1)asp(y1 | y0), to simplify notation.

The parameters that maximize this joint probability are foundby frequency counting. Because in our case we are dealingwith discrete data, we can simply count the number of oc-currences of transitions, observations and states [15].

Inferring which sequence of labels best explains a new se-quence of observations can be performed efficiently using

Figure 6. The graphical representation of a linear-chain CRF. Theshaded nodes represent observable variables, while the white nodesrepresent hidden ones.

the Viterbi algorithm. This dynamic programming techniqueis commonly used in combination with Hidden Markov mod-els [15].

Conditional Random Fields

A Conditional Random Field (CRF) is a discriminative prob-abilistic model that can come in many different forms. Theform that most closely resembles the HMM is known as a

linear-chain CRF and is the model we use in this paper (fig.6). As the figure shows, the model still consists of a hid-den variable and an observable variable at each time step.However, the arrowheads of the edges between the variousnodes have disappeared, making this an undirected graphicalmodel. This means that two connected nodes no longer rep-resent a conditional distribution (e.g. a given b), but insteadwe speak of the potential between two connected nodes. Un-like a probability, potentials are not restricted to a value be-tween 0 and 1 [4].

The potential functions that specify the linear-chain CRF are(yt, yt1) and (yt, xt). In this work, we adopt the no-tation [17, 4] which allows different forms of CRFs to be

expressed using a common formula. Therefore we define:(yt = i, yt1 = j) = ijkfijk(yt, yt1, xt) in whichthe ijk is the parameter value (the actual potential) andfijk(yt, yt1, xt) is a feature function that in the simplestcase returns 1 when yt = i and yt1 = j, and 0 other-wise. The second potential function is defined similarly:(yt = i, xt = k) = ijkfijk(yt, yt1, xt), where ijkis the parameter value and the feature function now returns1whenyt = i and xt = k , and0 otherwise. The indexijkis typically replaced by a one-dimensional index, so we caneasily represent the summation over all the different poten-tial functions [17].

In comparison with HMM, the conditional probabilitiesp (xt|yt)

andp (yt|yt1)have been replaced by the corresponding po-tentials. The essential difference lies in the way we learnthe model parameters. In the case of HMMs the parametersare learned by maximizing the jointprobability distributionp(x, y). The parameters of a CRF are learned by maximiz-ing the conditionalprobability distribution p(y | x) whichbelongs to the family of exponential distributions as [17]:

p (y|x) = 1

Z(x)exp

Kk=1

kfk(yt, yt1, xt)

(2)

8/12/2019 d912f50edb2226e921

5/9

whereZ(x)is the normalization function

Z(x) =

y

exp

Kk=1

kfk(yt, yt1, xt)

(3)

notice that the formula sums over all the different potentialfunctions. The feature functionfk(yt, yt1, xt) will returna0 or 1 depending on the values of the input variables and

therefore determines whether a potential should be includedin the calculation. Finally, the sum of potentials is divided bythe normalization termZ(x)which guarantees that the out-come is a probability. One of the main consequences of thischoice, is that while learning the parameters of a CRF weavoid modeling the distribution of the observations, p (x).As a result, we can only use CRFs to perform inference (andnot to generate data), which is a characteristic of the discrim-inative models. In activity recognition, the only thing weare interested in is classification and therefore CRFs fit ourpurpose perfectly. Model parameters can be learned usingan iterative gradient method. Particularly successful havebeen quasi-Newton methods such as BFGS, because theytake into account the curvature of the likelihood function.

Inference can be performed using a slightly altered versionof the viterbi algorithm [17].

EXPERIMENTS

In this section we present the experimental results acquiredin this work. We start describing our experimental setup, andthen describe the objective of our experiments. This sectionconcludes with a presentation of the acquired results.

Setup

In our experiments the sensor readings are divided in datasegments of lengtht = 60seconds. This time slice dura-tion is long enough to be discriminative and short enough toprovide high accuracy labeling results. Unless stated other-

wise, we separate our data into a test and training set using aleave one day out approach. In this approach, one full dayof sensor readings is used for testing and the remaining daysare used for training. In this way, we get inferred labels forthe whole dataset by concatenating the results acquired foreach day.

a)

b)

c)

Figure 7. Example of sensor firing showing the a) raw, b) change pointand c)lastobservation representation.

Our raw sensor representation gives a1 when the sensor isfiring and a0 otherwise (fig. 7a). Next to this raw data asobservations we experiment with a change pointrepresenta-tion. In this representation, the sensor gives a1to timesliceswhere the sensor reading changes (fig. 7b). Finally, we ex-periment with alastsensor fired representation, in which the

last sensor that changed state continues to give 1 and changesto0when a different sensor changes state (fig. 7c).

Because our dataset contains some classes that appear muchmore frequent than other classes, classes are considered tobe imbalanced [9]. We therefore evaluate the performanceof our models by two measures, the time slice accuracy andthe class accuracy. The timeslice accuracy represents the

percentage of correctly classified timeslices, while the classaccuracy represents the average percentage of correctly clas-sified timeslices per class. The measures are calculated asfollows:

Time slice:

N

n=1[inferred(n)=true(n)]

N

Class: 1C

Cc=1

Nc

n=1[inferredc(n)=truec(n)]

Nc

in which[a= b]is a binary indicator giving1when true and0when false. Nis the total number of time slices, Cis thenumber of classes andNcthe total number of time slices forclassc.

Measuring the time-slice accuracy is a typical way of evalu-ating time-series analysis. However, we also report the classaverage accuracy, which is a common technique in datasetswith a dominant class. In these cases classifying all the testdata as the dominant class yields good time-slice accuracy,but no useful output. The class average though would remainlow, and therefore be representative of the actual model per-formance.

Objective

Using the models and the recorded dataset discussed in theprevious sections, we ran three experiments.

The first experiment compares the accuracy of all the possi-

ble model-representation combinations. We test the HMMor CRF model coupled with one of the raw, change point,lastand a concatenation ofchange point and lastdata repre-sentation.

The second experiment explores the relationship of the sizeof the available training data with the performance of theacquired parameters. This is a good indication of the size oftraining data our framework would need, when installed ona novel real-world situation.

The third experiment shows the difference in performancein using offline inference and online inference. Offline in-ference means that all the data is available a priori for our

calculations. In processing each time slice, this allows usto use sensor data from both before and after the point ofinference. Online inference, on the other hand, implies wecan only incorporate data collected up to the point of infer-ence. Online inference is significantly harder, however it isnecessary for specific applications.

Experiment 1: Model comparison

In this experiment we compare which sensor representationcombined with which model performs best on our recorded

8/12/2019 d912f50edb2226e921

6/9

Idle

Leaving

Toileting

Showering

Sleeping

Breakfast

Dinner

Drink

Idle 60.9 6.4 0.7 9.8 4.6 0.4 16.3 0.9Leaving 0.8 98.4 0.2 0.3 0.0 0.1 0.2 0.0Toileting 6.3 2.9 83.9 3.2 2.6 0.3 0.3 0.5Showering 7.2 0.0 3.8 89.1 0.0 0.0 0.0 0.0

Sleeping 0.1 0.0 0.4 0.1 99.4 0.0 0.0 0.0Breakfast 22.0 0.0 0.9 0.0 0.9 52.3 16.5 7.3Dinner 12.4 0.3 0.3 0.0 0.0 6.0 78.4 2.6Drink 11.9 1.7 1.7 0.0 0.0 3.4 13.6 67.8

Table 2. Confusion Matrix for HMM using changepoint+last represen-tation andofflineinference. The values are percentages.

dataset. Experiments were performed using offline infer-ence.

Table 3 shows the accuracies for all model-representationcombinations. We see that for both models the changepoint+lastrepresentation gives the best results. Furthermore, we see

that the CRF achieves a higher timeslice accuracy for all thesensor representations. The HMM achieves the overall high-est class accuracy using the changepoint+last representation.

Timeslice Class

HMM

raw 51.5% 49.2%changepoint 80.0% 67.2%last 91.8% 71.2%changepoint+last 94.5% 79.4%

CRF

raw 69.4% 44.6%changepoint 89.4% 61.7%last 95.1% 63.2%changepoint+last 95.6% 70.8%

Table 3. Timeslice and class accuracies for HMM and CRF using vari-ous sensor representations.

The confusion matrix for HMM using the changepoint + lastrepresentation can be found in table 2. The activities Leav-ing, Toileting, Showering and Sleeping give the high-est accuracy. Activities Idle and Breakfast perform worst.Most confusion takes place in the Idle activity and the threekitchen activities Breakfast, Dinner and Drink.

The confusion matrix for CRF using the changepoint + lastrepresentation can be found in table 4. The activities Idle,Leaving and Sleeping give the highest accuracy. Activi-

ties Breakfast, Dinner and Drink perform worst. Mostconfusion takes place in the Idle activity and the three kitchenactivities Breakfast, Dinner and Drink.

Experiment 2: Minimum amount of training data

In this experiment we show how the number of training daysaffects the accuracy of the classifier. We use one day fortesting, the previousndays for training and cycle over all thedays in the dataset. If the previousn days are not available,we continue from the back of the dataset.

Idle

Leaving

Toileting

Showering

Sleeping

Breakfast

Dinner

Drink



Table 4. C onfusion Matrix for CRF using changepoint+last represen-tation andofflineinference. The values are percentages.

Figure 8 shows the timeslice and class accuracy for bothHMM and CRF as function of the number of days used fortraining. The figure shows that CRFs continuously outper-form HMMs in terms of timeslice accuracy, while HMMscontinuously outperform CRFs in terms of class accuracy.The timeslice accuracy shows a horizontal trend for 3 train-

ing days or higher, the class accuracy shows a horizontaltrend for 12 training days or higher.

5 10 15 20 250.4

0.5

0.6

0.7

0.8

0.9

1

Number of Training Days

Accuracy

CRF timeslice accuracy

HMM timeslice accuracy

CRF class accuracy

HMM class accuracy

Figure 8. Timeslice and class accuracy as a function of the number ofdays used for training.

Experiment 3: Offline vs online inference

In this experiment we compare offline and online inference,we used the changepoint + last representation for both mod-els and inference methods.

Table 6 shows the accuracies for both HMMs and CRFs us-ing offline and online inference. We see that for both mod-els offline inference performance better, the difference be-tween offline and online inference is bigger for CRFs thanfor HMMs.

The confusion matrix for HMM using offline inference canbe found in table 2 and online inference in table 5. Com-paring the online and offline confusion matrices we see theaccuracy of activities Toileting, Showering, Dinner andDrink are lower for online, while the activity Breakfast isslightly higher.

8/12/2019 d912f50edb2226e921

7/9

Idle

Leaving

Toileting

Showering

Sleeping

Breakfast

Dinner

Drink



Table 5. Confusion Matrix for HMM using changepoint+last represen-tation andonlineinference. The values are percentages.

The confusion matrix for CRF using offline inference can befound in table 4 and online inference in table 7. Comparingthe online and offline confusion matrices we see the accuracyof all activities except for Breakfast are lower, the accuracyfor Breakfast is slightly higher.

Comparing the two confusion matrices we see that the Idleactivity has significantly higher accuracy with CRFs. WhileToileting, Showering, Dinner and Drink have signifi-cantly higher accuracies with HMMs.

Timeslice Class

HMM online 94.4% 73.3%

offline 94.5% 79.4%

CRF online 88.2% 61.0%

offline 95.6% 70.8%

Table 6. Timesliceand class accuracies for HMMand CRF using offlineand online inference.

Discussion

The first experiment shows that our proposed framework givesstate of the art results in recognizing activities in a homesetting. Out of the different sensor data representations theraw representation gave the worst results. This was ex-pected, because in the dataset many sensors continue to firebecause a door is left open, while the activity involved al-ready ended. The changepoint representation solves thisissue by only representing the point in time at which a sen-sor was used. Such a representation allows for a much bet-ter discrimination and therefore leads to much better results.The last representation, in which the last used sensor con-tinues to fire until another sensor is used, gave even betterresults. This representation is especially useful for door sen-sors. When somebody is inside a room and closes the door,the only sensors that can fire are within that room. Sincepeople tend to typically close doors behind them for a num-ber of activities (e.g. sleeping, showering and leaving) thissensor representation works surprisingly well. By using boththe changepoint and last representation in a concatenatedfeature matrix, we were able to use best of both worlds andachieved the highest accuracy for both HMMs and CRFs.

With respect to comparing the two probabilisitic models we

Idle

Leaving

Toileting

Showering

Sleeping

Breakfast

Dinner

Drink



Table 7. C onfusion Matrix for CRF using changepoint+last represen-tation andonlineinference. The values are percentages.

see that CRFs outperform HMMs in all the cases with re-spect to timeslice accuracy, but HMMs achieve the over-all highest class accuracy. This is a result of the way bothmodels maximize their parameters. HMMs make use of aBayesian framework in which a separate model p(x | y) islearned for each class. Using Bayes rule we can calculatethe posterior probability p(y | x) for a novel point. In thecase of CRFs we use a single model for all classes, by cal-culatingp(y | x) directly. Parameters are learned by max-imizing the conditional likelihood p(y | x). Because CRFsuse a single model for all classes, classes compete duringmaximization. In a dataset with a dominant class, modellingeverything as the dominant class might yield a higher like-lihood than including the minor classes and misclassifyingparts of the dominant class. This view is confirmed by thetwo confusion matrices for HMM (table 2) and CRF (table4). The accuracy of the Idle activity is much higher inthe case of CRFs than HMMs, but the accuracy of Toilet-ing, Showering, Dinner and Drink is much higher inthe case of HMMs. The Idle activity takes up 11.5%of thedataset, while the other activities each take up 1%at most.

The second experiment shows that for this dataset and thesemodels a minimum of 12 days is needed for accurate param-eter estimation. The timeslice accuracy seems to indicatethat after 3 training days the accuracy already converged.However, this is because some classes appear very rarely inthe dataset. Using too few training days, leads to bad param-eter estimations for these classes. The increase in timesliceaccuracy is so little, because they only take up a few times-lices.

The third experiment shows that HMMs still give good re-sults when using online inference. This is important, be-cause when using offline inference, activities could only beinferred when a full day has passed. Using online inference,we can infer activities the minute after they happen. Someapplications, such as sounding an alarm after an anomalydetection need to be able to respond this quickly.

We end this discussion with a note that the dataset describedin this paper only uses digital state change sensors, however,our framework also allows the use of analog sensors. First,the network nodes we use also have an analog input. Second,

8/12/2019 d912f50edb2226e921

8/9

by setting a threshold value, the nodes only send out an eventwhen the threshold is violated. Since most activity relatedthings we wish to sense have an on and off state (e.g. shower,stove), we could easily measure their status using an analogsensor (e.g. humidity, temperature). This would result inbinary data, just like with the use of digital sensors. Third,even though the sensors might need some time to pick upthe change in state (e.g. room slowly becomes more humid),

the probabilistic nature of our models is ideal for modellingsuch correlations.

CONCLUSIONS

This paper introduces a sensor and annotation system forperforming activity recognition in a house setting. We ex-perimented with probabilistic models that can utilize the datacollected from this system to learn the parameters of activ-ities in order to detect them in future sensor readings. Anumber of sensor reading representations were proposed andtheir effectiveness in activity recognition was compared. Westrongly believe that other researchers can use these repre-sentations and select the optimal one on their field. Fur-thermore, our recorded dataset is available on the authors

website to motivate the comparison of other models with theones we proposed. Additionally, we proposed two measuresfor model accuracy, which we believe capture the potentialvalue of a model in the field of activity recognition. As longas it concerns our choice of models, we performed a seriesof experiments that reveal the potential of discriminative andgenerative modeling in activity recognition.

Acknowledgements

This work is part of the Context Awareness in Residence forElders (CARE) project. The CARE project is partly fundedby the Centre for Intelligent Observation Systems (CIOS)which is a collaboration between UvA and TNO, and partlyby the EU Integrated Project COGNIRON (The Cognitive

Robot Companion). Athanasios Noulas was funded by Mul-timediaN.

REFERENCES

1. Living independently - quietcare system.www.livingindependently.com.

2. G. Abowd, A. Bobick, I. Essa, E. Mynatt, andW. Rogers. The aware home: Developing technologiesfor successful aging. In Proceedings of AAAI Workshopand Automation as a Care Giver, 2002.

3. Gregory D. Abowd, Anind K. Dey, Peter J. Brown,Nigel Davies, Mark Smith, and Pete Steggles. Towardsa better understanding of context and

context-awareness. InHUC 99: Proceedings of the 1stinternational symposium on Handheld and UbiquitousComputing, pages 304307, London, UK, 1999.Springer-Verlag.

4. Christopher M. Bishop.Pattern Recognition andMachine Learning (Information Science and Statistics).Springer, August 2006.

5. Thi V. Duong, Hung H. Bui, Dinh Q. Phung, andSvetha Venkatesh. Activity recognition and

abnormality detection with the switching hiddensemi-markov model. InCVPR 05: Proceedings of the2005 IEEE Computer Society Conference on ComputerVision and Pattern Recognition (CVPR05) - Volume 1,pages 838845, Washington, DC, USA, 2005. IEEEComputer Society.

6. James Fogarty, Carolyn Au, and Scott E. Hudson.Sensing from the basement: a feasibility study ofunobtrusive and low-cost home activity recognition. InUIST 06: Proceedings of the 19th annual ACMsymposium on User interface software and technology,pages 91100, New York, NY, USA, 2006. ACM Press.

7. Sumi Helal, William Mann, Hicham El-Zabadani,Jeffrey King, Youssef Kaddoura, and Erwin Jansen.The gator tech smart house: A programmable pervasivespace.Computer, 38(3):5060, 2005.

8. Stephen S. Intille, Emmanuel Munguia Tapia, JohnRondoni, Jennifer Beaudin, Chuck Kukla, SitijAgarwal, Ling Bao, and Kent Larson. Tools forstudying behavior and technology in natural settings. In

Ubicomp, pages 157174, 2003.

9. Nathalie Japkowicz and Shaju Stephen. The classimbalance problem: A systematic study. Intell. DataAnal, 6(5):429449, 2002.

10. S. Katz, T.D. Down, H.R. Cash, and et al. Progress inthe development of the index of adl. Gerontologist,10:2030, 1970.

11. Niels Landwehr, Bernd Gutmann, Ingo Thon, MatthaiPhilipose, and Luc De Raedt. Relationaltransformation-based tagging for human activityrecognition. In Donato Malerba, Annalisa Appice, andMichelangelo Ceci, editors,Proceedings of the 6thInternational Workshop on Multi-relational DataMining (MRDM07), pages 8192, Warsaw, Poland,September 2007.

12. Jonathan Lester, Tanzeem Choudhury, Nicky Kern,Gaetano Borriello, and Blake Hannaford. A hybriddiscriminative/generative approach for modelinghuman activities. InIJCAI, pages 766772, 2005.

13. Beth Logan, Jennifer Healey, Matthai Philipose,Emmanuel Munguia Tapia, and Stephen S. Intille. Along-term evaluation of sensing modalities for activityrecognition. InUbicomp, pages 483500, 2007.

14. Donald J. Patterson, Dieter Fox, Henry A. Kautz, andMatthai Philipose. Fine-grained activity recognition byaggregating abstract object usage. In ISWC, pages4451. IEEE Computer Society, 2005.

15. L. R. Rabiner. A tutorial on hidden markov models andselected applications in speech recognition.Proceedings of the IEEE, 77(2):257286, 1989.

16. Albrecht Schmidt.Ubiquitous Computing - Computingin Context. PhD thesis, Lancaster University, 2002.

8/12/2019 d912f50edb2226e921

9/9

17. Charles Sutton and Andrew McCallum.Introduction toStatistical Relational Learning, chapter 1: Anintroduction to Conditional Random Fields forRelational Learning, page (Available online). MITPress, 2006.

18. Emmanuel Munguia Tapia, Stephen S. Intille, and KentLarson. Activity recognition in the home using simpleand ubiquitous sensors. In Pervasive Computing,

Second International Conference, PERVASIVE 2004,pages 158175, Vienna, Austria, April 2004.

19. T. L. M. van Kasteren and B. J. A. Krose. Bayesianactivity recognition in residence for elders. InIntelligent Environments, 2007. IE 07. 3rd IETInternational Conference, pages 209212, Ulm,Germany, 2007.

20. Daniel Wilson and Chris Atkeson. Simultaneoustracking and activity recognition (star) using manyanonymous binary sensors. InPervasive Computing,Third International Conference, PERVASIVE 2005,pages 6279, Munich , Germany, 2005.

21. Daniel H. Wilson.Assistive Intelligent Environmentsfor Automatic Health Monitoring. PhD thesis, CarnegieMellon University, 2005.

Documents

d912f50edb2226e921