VUB AI-II Situated Computing Module 1: Context Awareness

VUB AI-II2001-2002 WVdV

Situated ComputingContext Awareness - 1

VUB AI-II

Situated Computing

Module 1: Context Awareness

Walter Van de Velde

[email protected]



What is Context Awareness?

• History: Shilit et al. Context-Aware Computing Applications (1994) was primarily location based and use active badge technology(but user profiles in ITS go back a long way)

• A rough definition: “The ability of a device or program to sense, react or adapt to its environment of use.” (Pascoe et al., HUC 99)



Personal Augmentation (early 80’s)

ActivePassive Demand on User’s Attention Resources

Situational

Awareness

Low

High

Pagers

Watch

Eye-glassesPen-paper

WalkmanDesktop PCs

Airplane Cockpit

Automobiles Source: N

itin Saw

hney, MIT

Media L

ab




Situational

Awareness

Low

High

Pagers

Mobile-Phones

Watch


Walkman

HMD Wearables

Network PCs

Automobiles

Airplane Cockpit

Newton

Virtual Reality

Personal Organizers

Personal Augmentation (early 90’s)S

ource: Nitin S

awhney, M

IT M

edia Lab




Situational

Awareness

Low

High

Pagers

Smart-Phones

Watch

Palm-Pilot


Nomadic Radio

Walkman

HMD Wearables

COMRIS

Audio-Aura

Desktop PCs

Automobiles

Airplane Cockpit

Telematics

Newton Mobile Audio

Virtual Reality

Speech Wear

Personal Augmentation (1997-future)S

ource: Nitin S

awhney, M

IT M

edia Lab



Why Context Awareness?• Improving interaction with a device

– better defaults, automated choices,...

• Improving quality of existing services– communication, information,...

• Extended sensing and monitoring– security, health and safety,…

• Enabling of new services– tourist services– health services ...



Example artefacts

• Active and smart badges• e-notes (and other GPS-based services)• Media Cups• Aware GSM• Memory aid camera• COMRIS• Intelligent Jogging Suite• ….



Example Technology: TEA

• TEA: Technology for Enabling Awareness

• 2 year EU project, ending July 2000– Starlab (B)– Omega Generation (I)– NOKIA (S)– TecO (Univesity of Karlsruhe) (G)



TEA Objective

“To produce an affordable add-on component to existing portable communication and

computation devices that adds awareness of location and activity, and thus enables

context sensitive device control.”




Situational

Awareness

Low

High

Pagers

Smart-Phones

Watch

Palm-Pilot


Nomadic Radio

Walkman

HMD Wearables

COMRIS

Audio-Aura

Desktop PCs

Automobiles

Airplane Cockpit

Newton Mobile Audio

Virtual Reality

Speech Wear

Personal Augmentation (1997-future)

Source: N

itin Saw

hney, MIT

Media L

ab

TE

A



TEA Objective

GSM

PDA

Potential Specialized Plug-ins

TEA Device



The Essence of MobilityEnvironment

Activity



Context and location

Context is much more than location



The Context Space



TEA Scenarios

• GSM for a mobile journalist

• GSM for school child

• GSM for construction worker

• PDA for business executive

• PDA for university student

• PDA for field engineer



On the Nature of Context-Aware Applications

• There are Many Classes of Context-Aware Applications

• Context-Aware applications tend to be resource hungry

• Context-Awareness has a high development cost

• The computing environments are diverse• The greater the context, the greater the

application• I.e., arguments for context services



Sensors in Different Contexts



Tracking context change



The Kohonen Self-Organising Map (KSOM)

• Intuition: mapping an n-dimensional data space to an m-dimensional one (m < n), with preservation of nearness

• E.g., mapping a 8 dimensional data space to a 2-dimensional one.



KSOM

Output layer

Input vector xk

wk

Wi+1 = Wi + A * F(Wi - Xi)

Learning rate Neighboorhood function



From Sensors to Scripts

Sensors

Cues

Contexts

Scripts



Machine Learning Architecture

• Sensors• Cues• Clusters • Classes• Contexts



Comparative Study Clustering Algorithms

• Focusing on scalability, on-line learning and stability:– Kohonen Self-Organizing Map

– on-line K-Means

– Sequential Leader Clustering

– Neural Gas

– Neural Gas with Hebbian Learning

– Growing Neural Gas

– Stable Growing Neural Gas

– Recurrent Self-Organizing Map

– ...

with different metrics, parameters and adaptation rules

• Results are accessible on internet



Layered Model for

Sensor based Context Awareness



Sensor Layer• physical sensors

– acceleration, light, temperature, audio, etc.

– domain dependent

– sensor placement matters

– trade off, e.g power, price vs. added value• logical sensors

– time, current application, phone number, memory available, etc.

• modeled as time variant function



Cue Layer

• abstraction of raw sensor data– summary over time– extraction of features– reduce amount of data– one sensor can generate one

or more cues

• time window– sensor and cue dependent

raw acceleration data

amplitude standard deviation mean...



Fusion - Context Layer

• description of the current situation on an abstract level

• calculated from all cues available

• predefined domain determined by application

• context types– exclusive:

in the office vs. at home– non-exclusive:

walking, running, talking



Applications and Scripting

• application use contexts

• supported by scripting primitives

– enter a context– leave a context– while in a context

if enter(v, p, n) then perform action(i)

if leave(v, p, n) then perform action(i)

if in(v, p, m) then perform action(i)



Why makes it sense to use little processing power?

• What is the vision?

– Within a few years the context-awareness technologies we research here is applicable to nearly every artifact in our environment

• Power will be one of the most critical issues for mobile artifacts!



low processing power and little memory

• Ubiquitous computing

• cheap artifacts with computing power

• very little power (power harvesting, ...)

• what we can do today on microcontroller (~16Euro and 40x20mm) will be feasible in 9 year at 0.25 Euro and at a size 1x1mm …

• Power is going down similar

• -->make a pen context aware ....



Aware GSM

• Hardware Overview

– 35 x 90 x 0.6 mm

– Tailored for 6100

– 8 sensors + Processing

– Serial Communication



Aware GSM

• Achievements

– Integrated TEA component for GSM

– Automatic profile changing

• “In Hand”

• “On Table

• “In Pocket”

• “Worn”



Aware GSM

• Problems

– Battery Integration

• Nokia Proprietary circuit

• Extra Sensitive

• Random Battery Failure

– Software Integratrion

• Phone Communication



SMS sending

• TEA SMS sending– Obsolete w/ 6100’s

• Main Reasons– Phone is outdated– Intransigent Hardware vs. Flexible Software

• Phone specific changes– Feasibility tests

• Successful yet difficult• hard-coded changes - limited message-size



SMS sending

• Current Technology makes SMS possible

– Phones now more flexible software than electronic equipment

– AT Standard being widely used

• old modem standard adapted for mobile phones

• 10 bytes serially compared to kilobytes hard-coded



WAP

• Platform for new phone applications

• we assumed that we have context knowledge (e.g. TEA, GPS, user)

• can we enhance telephony?



Communication in Context I

• initiating face-to-face communication

– social skill

– trained from early childhood on

• context matters - manly implicitly

– how important is it for me?

– how convenient seams it for the other person?

– relation between the communication partners?

– what type of conversation will it be?

– is it socially acceptable (topic/situation)?



Communication in Context II

• initiating remote (tele)-communication

– little knowledge about the called end

– best guess approach

• knowledge available

– important to caller and called party

– identity of communication partner

– land line location situation

– time location/situation/activity



Context-Call

• novel phone interface

• phone users can selectively share context

– information about the situation

– information about availability

– …

• caller can decided

– knows her own constraints

– has some information about the other side

– can judge if the call will be appropriate



Context-Call - Implementation I

• WAP application

– WML, WTAI, Apache, CGI

• context-call application

• context-selection page

• server-backend



Context-Call - Implementation II

• context-call application• novel phone interface



Context-Call - Implementation III• context-selection page• tell the system your context (similar to profiles)• using TEA for automation



Hardware integration

PIC

Sensors

PDA/GSM

PIC

Sensors

CPU PDA/GSM



Hybrid Algorithms.3

.6.3

.1

Raw Sensor Data

Sensor Cues

Supervision

Clu

ster

ing



Basic Hardware (March 00)



Training and Testing (March 00)



Contexts

standing running

sittingwalking

talking

outside

kitchen

stairs

NN LabHardw Lab

Activity

Location



Context as a service

• Context Toolkit (Georgia Tech)

• SitComp (Device/person centric)

• Technology for Enabling Awareness (Starlab et al).



Context Awareness



Enabling Technologies Overview

InputDevices

OutputDevices

Power Networks SoftwareEng.

Textual Textual Batteries Low-orbitsatellites

Jini fromSun

Audio Audio HumanBody

Bluetooth JavaSpacesfrom Sun

Video Video Low-powerarch.

IEEE 802.15PersonalArea Nets

InfernofromLucent

GPS/GIS Tactile T-Spacesfrom IBM

Context

Affectiveinputs



Video input• Computer vision

– requires a lot of computing power

– A lot can be inferred from cameras that see what we see in terms of immediate context

– State of the art vision research systems can recognize location, and sequences of actions such as crossing the street, in the supermarket, in a conversation, etc.

– Real-time face recognition could be an interesting input modality

– Statistical object recognition could be used to post notes on objects around us (‘virtual post-it’)



GPS/GIS• Location based context has immediate applications such as

the tourist guide systems

• -: GPS doesn’t work indoors

• Example prototype system from Columbia University:

Prototype tour guide Here’s what the user sees



Affective inputs

• There is a one-to-many mapping between our physiological state and our psychological state. Given that we can make some measurements about our physiological state (heart rate, skin conductance, pupil dilation, etc.), can we infer our psychological state?

• Such information can be useful in early detection of diseases, evaluation of machine performance, learning preferences over time



Affective Computing

• Affective Computing was coined as a term by Rosalind Picard (MIT Media Lab, 1996) to indicate computing that has features for recognising or synthesising emotions.



Emotions and Intelligence

• Emotions and Rationality?

• Can intelligence exist without emotions?

• Emotional intelligence (Goleman)

• Neuroscience and Emotions (Damasio, LeDoux)



Emotions and the Brain

Simplified brain structure (MacLean 1970)• Cortex (‘grey matter’)• Limbic system (includes Amygdala)• Inner Brain (brain stem)

Assumed main functions:• Cortex: rationality, perception• Limbic: emotions, attention, memory• Inner brain: basic functions



Emotions and the Brain

There is a variety of evidence that emotions and rationality are tightly linked:

• Vast neural connectivity

• Brain activity maps

• Somatic markers to ‘prune’ huge searches

• Learned behavior based on ‘emotional’ feedback



Can we control emotions?

• If it were useful to control our emotions we would have evolved ways of doing so.

• There must be an evolutionary advantage in having emotions (or in having emotional reactions)



Components of Emotion System (I)

1. Emotional Behavior

2. Fast Primary Emotions

3. Cognitive Emotions

4. Emotional Experience• Cognitive Awareness• Physiological Awareness• Subjective Feelings



Components of Emotion System (II)

5. Body-Mind Interaction– Memory– Perception– Decision Making– Learning– Concerns, goals, motivations– Attention, Interest– Prioritizing– Planning– Sentic Modulation– Immune system functions– Regulatory Mechanisms



Sentic Modulation

• The emotional character is expressed by a specific subtle modulation of the motor action involved which corresponds precisely to the demands of the sentic state. (Clynes 1977)



Sentic Modulation

• Facial expression

• Voice intonation

• Gestures, movement

• Posture

• Pupilary dilation



Sentic Modulation

• Facial expression• Voice intonation• Gestures, movement• Posture• Pupilary dilation

• Respiration• Heart rate, pulse• Temperature• Electrodermal

response, perspiration• Muscle action

potentials• Blood pressure



Emotional Intelligence (Salovey and Mayer 1990)

• Recognition and expression of emotions– In self– In others

• Regulating emotions– In self– In others

• Utilizing emotions– In self– In others



Key sensors and cues for emotion detection

• EMG

• GSR

• Blood flow

• Respiration

• Facial expression

• Voice



The future of awareness

• More ubiquity

• Smaller, better and cheaper sensors

• Better context description

• Better context analysis

• Coverage of the whole awareness space

• Context services and modularisation

• Context standardisation (TEA Foundation)

Documents

VUB AI-II Situated Computing Module 1: Context Awareness