From Context-awareness to Human Behavior Patterns

From Context-Awareness to Human Behavior Patterns Detection of Daily Routines using Smartphones

Ville Antila (Research Scientist, M.Sc.)

VTT Technical Research Centre of Finland, Oulu, FinlandPhilips Research, Eindhoven, The Netherlands (Visiting researcher)

Background – Smarcos project

• Smarcos creates solutions to allow devices and services to exchange context information, user actions, and semantic data

• One important part of the work has been to investigate the practical usage of context information and to develop models that can be dynamic and adaptive as well as applicable to different applications

• www.smarcos-project.eu

Outline of the talk

• Introduction

•From context logging to routine detection• Continuos, low-power “life-logging”

• Interpreting the data (what’s the meaning of it)

• Using domain knowledge to reason about what we don’t know

•Example applications

•Discussion and summary

•Video?

Introduction - context awareness

• Idea that computers can both sense and react to their environment• “any information that can be used to characterize the situation of an entity” [Dey et al., 2001]

• Human factors: information about the user, social environment, user’s task

• Physical factors: Location (absolute, relative, co-location), infrastructure, physical conditions

• Context aware systems should be able to gather context (sense the situation), abstract and understand it and adapt application behavior based on the context

• Some classical use cases:

• adapt user interfaces

• tailor application-relevant data (e.g. filtering information)

• increase precision of information retrieval

• make the user interaction implicit

• discover services & build smart environments

Introduction - context awareness

• Smartphone is the epitome of a sensing platform for context-awareness

• Personal and mobile (almost always with the user)

• Comes with a lot of in-built sensors and communication capabilities

• Used everywhere, for multiple tasks (on-the-go)

Introduction - challenges & opportunities

• The notion of ‘context’ can not be objectively defined (a prior) by settings, actions and actors

• Rather, context is the meaning that the actions and actors acquire at any given time from the subjective perspective [Mancini et al., 2009]

• Context information can have multiple purposes (from user’s perspective)• “Declaring one’s position is perhaps as much about deixis (pointing at and referencing features of the

environment) as it is about telling someone exactly where you are” [Benford et al., 2004]

• When going from individuals to larger groups of people it’s possible to extract patterns from the data [Eagle and Pentland, 2006]

• social patterns (e.g. urban heat maps)

• identification significant locations

• model organizational rhythms

From Context Logging to Routine Detection

• Use the gathered knowledge to form decisions about system behavior in different contexts/situations

Context Interpreter

Support for decision making

Reasoning Domain knowledge

Data Layer

Applications

• Infer new knowledge from information

• Capture raw data from device sensors

• Transform raw data into information about the user’s context

• Example applications...

Continuous Life Logging

• Context-awareness

• Using smartphones as sensors for human activities (e.g. important locations, mobility patterns)

• Low-power context logging software

• Semantic location detection using cell-id (low power, always available)

• Device usage detection (algorithms for mining location relative to smartphone application usage)

• Lo-fi physical activity detection (e.g. is the user moving currently)

• Scanning Bluetooth snapshots to determine indoor environment (e.g. is the user at his office desk)

Data Layer

Context Components For real-time user behavior detection

• Collection of software components for enabling

• continuous context logging

• development of context-based adaptation for variety of applications

• Implementations available for several platforms (some of which are becoming obsolete)

• Android, Symbian, Maemo/Meego/Linux, BlackBerry

Interpreting the data...

• Estimation of life patterns such as the semantic location of the user (e.g. “home”, “office”)

• Detection of device usage in different locations

• Detection of physical activity in different situations

• Detection of changes in routines

Context Interpreter

Context CaptureContext-based awareness cues in information sharing

• We explored the usage of contextual information cues in informal information sharing

• The study focused on practices of ‘abstraction’ when publicly sharing contextual information

• Field test for our backend

Smartphones are used almost everywhere...Moreover there is an “app” for almost anything...

An opportunity (to use smartphone apps as sensors for situations...)?

Image sources: http://adage.com/article/digital/placing-ads-underestimate-mobile/230853/

http://www.resultrix.com/blog/index.php/tag/tablet-marketinghttp://www.google.com/googleblogs/pdfs/mobile_understanding_smartphone_users.pdf

• An approach to detect day-to-day ‘routines’ by logging the smartphone application usage and locations where they are used

• Analysis of logged usage data into identifiable patterns (clustering based on location and time of use)

• Implemented an experimental smartphone application with a functionality to create automated ‘tasks’ out of the identified patterns

• Conducted a two-week user study to analyze the approach and to gather user feedback

Routine Maker End-user automation of smartphone routines

Extending the knowledge...

• By modeling domain knowledge we can reason about the consequences of what the derived context information means in the particular application scenario

• For example, if know (to a certain degree of accuracy) that the user is cycling, then we can reason that:

• available devices are mobile devices (phone & activity monitor)

• availability for receiving messages is low

Domain knowledge Reasoning

Where would this information be useful?

• Determine the devices that surround the user

• e.g. at work, the user has access to his personal computer

• Time and adapt system feedback based on the situation

• e.g. time-shift notification to where user is more receptive

• Log important events and use those to automate tasks

• e.g. migrating task sessions automatically between personal devices

Example application:

Context-adaptive Feedback

• Goal: increase effectiveness and decrease interruptions

• How: adaptive selection of device, modality and timing of feedback

• Example of adaptive feedback delivery:

• IF the situation is suitable, THEN send the message (as it is)

• IF the situation is not suitable AND the message is not urgent, it should be time-shifted

• IF the situation is not suitable AND the message is urgent, then the content should be adapted to the situation

Applications

Example application:

Context-based User Interface Migration

• Migration concept• Interface moving from source device to target device

• Interaction state persistence

• Interaction continuity

Applications

Example application:

Context-based User Interface Migration

• Taking advantage of the known properties of the environment in any given time (e.g. Bluetooth, GPS, device stability and orientation) we can automate tasks such as UI migration triggering

Applications

Discussion

• Just like the smartphone sensor APIs have matured (GPS, acc, gyro, proximity), the basic context abstractions will also be served as OS-level services in the future (e.g. walking/running/still, home/office/school/on-the-go)

• Better optimizations regarding battery, CPU and memory usage etc. (resolves the iOS background processing challenge)

• Cross application usage

• Fusion with other available data on the device and “in the cloud”

• Going further we can also foresee taking inputs from the environment (e.g. WSNs) as well as “negotiating with” other smart devices while trying to reach a better approximation of the situation

Discussion - challenges

• Quality/accuracy of detection (have to be started off with simple cases), provenance of quality measures into the application level

• User interaction/awareness of application behavior

• Designing for applications that adapt to situations... Patterns, guidelines, best practices?

• Prototyping context-adaptive applications (from early interactive prototypes to functional prototypes)

• Testing

• Functional testing, performance/quality testing

• User testing (‘in the wild’)

Thank you!Questions?

Ville Antila ville.antila@vtt.fi

From Context-Awareness to Human Behavior PatternsDetection of Daily Routines Using Smartphones