Mobile Phone Sensing

CMPE 58C – Wireless Sensor Networks

• Why Mobile Phone Sensing? – Enabling Factors

• Sensors and Potentials

• Applications

• Mobile Phone Sensing Architecture – Sense

– Learn

– Share, Inform

• Open Topics/Challenges

• Inter-disciplinary collaboration potentials

Contents

1. Nicholas D. Lane, Emiliano Miluzzo, Hong Lu, Daniel Peebles, Tanzeem Choudhury, Andrew T. Campbell. "A Survey of Mobile Phone Sensing", In IEEE Communications Magazine, September, 2010.

2. Andrew T. Campbell, Shane B. Eisenman, Nicholas D. Lane, Emiliano Miluzzo, Ronald Peterson, Hong Lu, Xiao Zheng, Mirco Musolesi, Kristof Fodor, Gahng-Seop Ahn,"The Rise of People-Centric Sensing", In IEEE Internet Computing Special Issue on Sensor Networks, 2008.

3. Deborah Estrin, “Participatory sensing: applications and architecture”, in Proceedings of MobiSys 2010.

References

what is the future of sensing?

Evolution of Sensing Prototype

accelerometer

digital compass

microphone

WiFi/bluetooth GPS

….

light sensor/camera

sensing

gyroscope

density

programmability

- free SDK

- multitasking

application distribution

collect huge amount of data for research

purposes

cloud infrastructure

cloud - backend support

store and crunch big data (fusion)

run machine learning algorithms (learning)

- sensing

- run machine learning algorithms locally (feature extraction + inference)

societal scale sensing

global mobile sensor network

• Acceloremeter: Orientation/User Experience – Characterizing the Physical Movements of the User:

Running, Walking, Standing (CenceMe Application)

• Microphone – Classifying distinct sounds: Being in a conversation,

using ATM, listening to music, driving (SoundSense Application), identifying a speaker

• Camera – Photo Blogging, Traffic monitoring, Pollution

monitoring

Sensors and Potentials

E. Miluzzo et al., “Sensing meets Mobile Social Networks: The Design, Implementation,

and Evaluation of the CenceMe Application,” Proc. 6th ACM SenSys, 2008, pp. 337–50.

H. Lu et al., “Sound-Sense: Scalable Sound Sensing for People-Centric Applications

on Mobile Phones,” Proc. 7th ACM MobiSys, 2009, pp. 165–78.

• GPS: – Tracking users

– +acceloremeter: mode of transportation (walking, biking, using a vehicle)

• Bluetooth: Identifying Social Interactions (Reality Mining Project at MIT)

• GSM, WiFi: Using traces for localization, even tracking of mode of transportation

Sensors and Potentials

Reddyet al., “Using mobile phones to determine transportation modes” , ACM

Transactions on Sensor Networks, 2010.

Sohn et al., “Mobility Detection Using Everyday GSM Traces” , Ubicomp, 2006.

Reality Mining Project: Machine Perception and Learning of Complex Social Systems,

http://reality.media.mit.edu/

• Transportation – Large-scale network to provide fine-grained

traffic information (e.g., accurate travel time estimation)

• UC Berkeley/Nokia/NAVTEQ, “Mobile Millennium”; http://traffic.berkeley.edu/

• A. Thiagarajan et al., “VTrack: Accurate, Energy-Aware Traffic Delay Estimation Using Mobile Phones,” Proc. 7th ACM SenSys, 2009.

• Social Networking – Sharing activity and presence information through Social

Networks • CenceMe: Available on AppStore and compatible with Facebook

Application Domains

• Environmental Monitoring – Environmental Impact Report, pollution:

• UCLA PEIR Project: M. Mun et al., “Peir, the Personal Environmental Impact Report, as a Platform for Participatory Sensing Systems Research,” Proc. 7th ACM MobiSys, 2009.

– Sound, Air Pollution, etc. (Can be used for city planning) • UCLA Urban Sensing Project (Weather, air quality) • EarPhone: An End-to-End Participatory Urban Noise

Mapping System • Health and Well-Being

– Physical Activity Monitoring • UbiFit Garden: S. Consolvo et al., “Activity Sensing in the Wild: A

Field Trial of Ubifit Garden,” Proc. 26th Annual ACM SIGCHI Conf. Human Factors Comp. Sys., 2008

• Mobile Habits, Towards a Physical Diary • BikeNet: Sharing biking experience

– Spread of a Disease in Urban Areas, Chronic Disease Management – Nutrition, Medication, Interaction Monitoring

Application Domains

Sensing Scale

"A Survey of Mobile Phone Sensing", In IEEE Communications Magazine, September, 2010.

• Challenge – Impact of scaling sensing applications from

personal to population scale is unknown • Information sharing (trust)

• Privacy

• Data Mining on large scale data

• Low commitment of users

Sensing Scale

• Opportunistic Sensing: Automated Data Collection, no user involvement – Lower burden on the user

– Good for community sensing

– Phone Context Problem

• Participatory Sensing: An individual actively participates in the data collection – Higher burden on the user

– Depends on the enthusiasm of the user

Sensing Paradigms

•Develop models to best understand the usability and performance issues

of these schemes

•Hybrid sensing schemes?

An Example Architecture

•Sense •Interpret/Learn •Inform/Share/Persuasion

"A Survey of Mobile Phone Sensing", In IEEE Communications

Magazine, September, 2010.

• Programmability: – Open, Offer SDK’s, existing libraries (machine

learning – Weka) can be cross-compiled. – Vendors did not think that 3rd parties would use

continuous sensing to develop applications – Unpredictable, variable sampling: Acceloremeter

sampling on Nokia Symbian Phones range 25-38 Hz (depending on CPU load)

– GPS using a varying amount of power – Vendors: Will/Should improve as sensing

applications become popular – Research Topics: Propose sensing abstractions

and API’s that can be standardized and adopted by vendors

Sense: The Mobile Phone as a Sensor

• Continuous Sensing: – Multi-tasking not supported by most mobile

phones (except Android and Nokia Maemo)

– Still a resource-limited device for complex signal processing and inference

– Still battery limited for real-time applications

– Open topics:

Sense: The Mobile Phone as a Sensor

study the trade-off of continuous sensing with the goal of

minimizing the energy cost while offering sufficient accuracy

and real-time responsiveness

• Phone Context – Different use of mobile phones (on the go,

etc) by different people, background environment or actions (phone in the pocket), etc.

– Affects the success of machine learning algorithms to interpret data (dynamic & unexpected conditions)

– Open Topics:

Sense: The Mobile Phone as a Sensor

Sharing learning models between phones

Model Evolution: Semi-supervised learning, active learning

Collaborative Sensing

• Training Models – Supervised Learning requires labeling of

training data: Feasible for small scale sensing but cannot handle wide range of behaviors and context

– Open Topics: • Use of unsupervised, semi-supervised learning?

Learn: Interpreting Sensor Data

"A Survey of Mobile Phone Sensing", In IEEE Communications Magazine, September, 2010.

• Multi-modal Sensing – Better recognition of user activities with multiple

sensor streams (GPS+microphone = more than location, instead of at home -> in a conversation at home)

– Open topics: Using sensor fusion techniques for multi-modal sensing

• Scaling Models – Different people with different usage of phones? – Open Topics:

• Incorporate people in the process using active learning and community guided learning?

• Adapting models on the fly? • Common machine learning toolkit that allows researchers to

build and share models

Learn: Interpreting Sensor Data

• Sharing – Visualization using a web portal

– Community awareness: Social networks

– Personalized Sensing: Human Phone Interfacing

• Persuasion: Persuade users to make positive behavioral changes – Largely unexplored: Which metaphors and

feedback are most effective? (requires inter-disciplinary research)

• Privacy

Inform/Share/Persuasion

• Machine Learning: Application/Development of Inference/Classification Models

• Security/Privacy/Trust: Most fundamental requirement of a sensing system

• Wireless Networking: Opportunistic Networking, Vertical Handover Decisions to collect data, Localization/Tracking with Multi-modal sensing

• Data Mining: Large-scale information processing?

• Social Networking: How to share/inform data?

• HCI and Persuasive Computing: How to design feedback systems, user studies?

Potential Inter-Disciplinary Research

• Mobile Phone Sensing is an emerging field • Challenges/Open Issues:

– Privacy and Resource-Sensitive Reasoning with Noisy Data and Noisy Fields

– Adapt machine learning algorithms into the field – How to collect ground-truth data to assess the

accuracy of algorithms that interpret sensor data?

– How to validate experiments? – How to deal with the potentially massive amount

of data made available? – How to run such large-scale studies?

To Wrap Up

Mobile Phone Sensing @ CMPE

• Activity Recognition on Mobile Phones – How to recognize the activities of a user using the

accelerometers available on the phones?

• Efficient Feature Extraction on Mobile Phones – Which features are more efficient in order to

recognize the activities of a user?

• Movement Based Group Sensing – Can we identify groups of people by analyzing their

acceleration signals?

27

• Offline; Daily routine of a person

• Online; “what the user is currently doing”

Recruitment for

Participatory

Sensing

Applications

Offline vs. Online Sensing

Collect sensor data from the users that are “currently walking” in downtown area

Fittness coaching, rehabilitation

Why Online Training?

Drawback of Training Phase

• Costly training phase vs. Limited memory on phones

Hot topic

• Activity Recognition systems with no training data or requires limited training by end user

Target • Reduce the training burden on the user

• What is the performance of classifiers when executed online with online training in smartphones? – Will smartphones replace the PCs?

• Limited Training set which is collected in few minutes

• Application tested with different Android Phones instead of focusing on one model

Motivation

Activity Logger & Classifier Application

• Activity Logger;

– Collecting raw data from accelerometer

– Logging collected data

Activity Classifier; ◦ Classification with selected classifier ◦ Extracts features from training sets ◦ System parameters;

◦ Window size ◦ Sampling rate ◦ Order interval

• Naive Bayes

• Min Distance

• Clustered KNN;

– Decreased computation burden

– Preprocessing

– K sets for max, min, average, and std deviation per activity

– Classification process

Classifiers

Read all raw data from training log file

Calculate average of magnitudes for each

data value

Select K nearest points to the average magnitude by comparing their euclidean

distances

Store set of K points in memory to be used at

classification step

Collect raw sensor data during a

window duration

Calculate average of magnitudes for the data points in the

window

Compare average mag. with each point in

previously generated 4*K sets in terms of euclidean

distances

Select K nearest neighbor

Check activity label of each data in the

final K set

Label the window for the average feature with the

activity which is voted the most

• 5 Test subjects

– Average age 27, 1 female & 4 male

• Phone located in front pocket of pants

• Phone models

• System Parameters

– Window size

– Sampling Rate

– K Value

Test Design – System Parameters

Test Design – System Parameter

•Test cycle (60 sec) •Labelled Data

•Voice commands •Performance metrics

•Precision •Recall •Accuracy •F-measure

Running

Walking

Sitting

Standing

Results

Clustered KNN Naïve Bayes

Accuracy 92.13% 47.61%

Precision 92.45% 51.15%

Recall 92.09% 42.30%

F-measure 92.27% 46.19%

Classification Running Walking Standing Sitting

Tru

th Running 1020 46 14 7

Walking 178 729 146 7 Standing 121 12 1063 18 Sitting 18 2 30 1084

Comparison of Clustered KNN and Naive Bayes

Overall Confusion Matrix

Results

Window size (sec)

0.5 1 2

Sampling Interval (msec)

10 50 100 10 50 100 10 50 100

K

10 87.9 87.8 87.7 88.4 90.3 89.5 88.6 87.8 89.3

50 91.1 90.0 91.4 91.9 92.1 90.8 88.9 89.4 91.0

Average Accuracy Rates (%)

• Clustered KNN Results Based on System Parameters

Results

Activity Classifier CPU Usage Memory Usage

KNN Clustered 29% 21.9 MB Naïve Bayes 42% 12.6 MB Min Distance 27% 19.6 MB TTS Service 13% 12.4 MB

Benchmark Applications CPU Usage Memory Usage

System 10% 28.8 MB Norton Mobile 4% 19.4 MB

Internet 2% 37.0 MB Google Maps 1% 31.2 MB

CPU & Memory Usage for Activity Classifier & Other Applications on Samsung Galaxy Gio

Paper Discussion

• K. K. Rachuri, C. Mascolo, M. Musolesi, and P. J. Rentfrow, SociableSense: Exploring the Trade-offs of Adaptive Sampling and Computation Offloading for Social Sensing, ACM MobiCom '11, Las Vegas, USA, 2011.

• Emiliano Miluzzo, Nicholas D. Lane, Kristóf Fodor, Ronald A. Peterson,Hong Lu, Mirco Musolesi, Shane. B. Eisenman, Xiao Zheng, Andrew T. Campbell, "Sensing Meets Mobile Social Networks: The Design, Implementation and Evaluation of the CenceMe Application", In Proc. of 6th ACM Conference on Embedded Networked Sensor Systems (SenSys '08), Rale igh, NC, USA, Nov. 5-7, 2008.

38