SenseDroid

Sensemaking from Distributed and Mobile Sensing Data: A Middleware

Perspec;ve

S.Sarma, N. Venkatasubramanian, N. DuA

1

Overview

•  Introduc0on to Crowdsensing and Sensemaking

•  A Middleware Perspec0ve •  Example Middleware Pla?orms and techniques

•  Research Direc0ons

2

Mobile Phone Trends

•  Mobile subscrip;on 5.96 billion 2011 es;mate

•  Smartphones (487.7 million) exceeding PCs (414.6 million)

•  More Mobile Internet Users Than Wireline Users in the U.S. by 2015

•  Smartphone and bandwidth cost reduces

•  Smart devices contribute to more than 90% of mobile data traffic

3

Sensors In Mobile Phones

•  MEMS & sensors for cell phones, expanding from $ 3.5 bn in 2009 to $7.9 bn in 2015 [Yole Developpement]

•  Smartphone sensors to be $ 6 bn business by 2016 [Juniper Research] •  44 % of the mobile phones will be smartphones in 2015 •  7x increase in mobile health apps from 2010 to 2011 •  mo;on sensor in smartphones and tablets will expand to $ US 2.1 billion in

2015 with a 25.3 % CAGR, up from $1.19 billion in 2011 (IHS iSuppli) 4

Mobile Sensors Trends

Source: IHS Consumer & Mobile MEMS Market Tracker, April 2014. 5

Mobile Data Delivery Everywhere

6

Smart devices contribute to more than 90% of mobile data traffic

The exploding number of apps is driven by a huge up;ck in the number of smart devices

~55%

Cisco’s report 2014

Crowdsourcing and CrowdSensing

7

Pushing toward more interven0on

Power of the Crowd

•  Using mobile crowdsensing to –  Leverage already deployed smartphones

–  Extend the ranges of exis0ng in-‐situ sensors

–  Send mobile users to specific loca0ons

•  Crowdsensing broad use cases –  Disaster and emergency response –  Personal health monitoring and wellness

–  Smart spaces and their effec0ve u0liza0on

8 [YKL11] M. Yuen, I. King, and K. Leung. A survey of crowdsourcing systems. In Proc. of IEEE Interna0onal Conference on Social Compu0ng (SocialCom’11), pages 766–773, Boston, MA, October 2011.

• Earthquakes • Hurricanes • Tornadoes • Energy/utility outages • Fire hazards • Hazardous materials releases • Terrorism/

Emergency Use Cases

9

Emergency Response

During Fire accidents can cause electric power failure. Mobile broadcast can be used to provide direc;ons to the users about rescue opera;ons.

10

Emergency situa;on Automa;c Altering can be used to inform family, rescue teams, or nearby cars / passengers in case of accidents.

Emergency Response

11

Sensing -‐> Sensemaking

Alert System

Severity

Personal Sensing to indicate Fall detec0ons, injury severity, alerts in old age people to

provide scalable health care 12

Sensing -‐> Sensemaking Radia0on field near Fukushima

Crisis Map Showing Latest Informa0on

Hazardous gas in campus

Spa0al Field Sensing With Mobile Sensors

13

Sensing -‐> Sensemaking

•  Avoiding congested streets in a city •  Finding the most popular booth in a fair •  Searching for the ride with shortest lineup in an amusement park

14

SenseMaking : Purpose & Goals

u Simple and Easy-‐to-‐Use Framework for Sensing, Actua0on and Collabora0on using mobile phone

u Powerful addi0onal sensing abili0es and features for community of users by community of users

u Understand user and group context efficiently u Building energy-‐efficient collabora0on apps

over exis0ng mobile pla?orms u Supported and empowered by community of

users for community of user

15

The Problem – A cross layer, end to end issue

§  Several barriers and huge investment of 0me to build collabora0ve smart applica0ons

§  Lack of a framework to ease and speed the development of applica0ons

§  Non-‐Scalable, Ad-‐hoc, non-‐standardized API §  Unsupported network infrastructure, and

configura0ons

16

Solu0on to the Problem – Middleware Approach, Hierarchy for Scale…

•  Design and Develop and Open source distributed middleware framework suppor0ng collabora0ve mobile sensing

•  Provide API and libraries to perform: – Collabora0on – Virtual Sensing and Compressive Context Determina0on

– Computa0onal Offloading – Cloud interface for scalability

17

Middleware Pla?orms and Techniques for Sensemaking

•  On phone, on broker (SenseDroid, SATWARE) •  Techniques implemented in middleware

– Compressive and Collabora0ve Sensing – Virtual Sensing for Sensemaking – Seman0cs Driven Sensing and Actua0on

•  Combining In-‐situ Sensors with Mobile Crowdsensing

18

Combining In-‐situ Sensors with Mobile Crowdsensing

Pushing toward more interven0on

•  For sensing tasks not covered by any in-‐situ sensors –  Try opportunis0c and par0cipatory sensing using nearby mobile users

•  What if there are no nearby mobile users •  Pushing toward even more interven0on à Crowdsourcing

19

Explosion of Contextual Data Delivery

20

Emergency response

Transporta0on

~2.5 M mobile apps

Entertainment Mobile social networks

Healthcare

Shopping

Apps have various performance needs (reliability, ;meliness, quality…)


21


22

SenseDroid Architecture

…

Mobile Users

…

…

Internet /Public Cloud

Middleware Broker

Wi-‐Fi AP

3G AP

Query/ Response

Cloud Users

•  Use compressive sensing with computa0onal offloading for energy-‐efficiency

•  Use collabora0on for addi0onal and efficient sensing abili0es

•  Leverage reconstruc0on abili0es of compressive sensing to improve robustness and reliability

23

Mul0-‐0ered Hierarchical Architecture

24

SenseDROID Distributed Middleware

APPS$1$

Communica.on$

Sensing$&$Sampling$

Context$Processing$&$Fusion$$

Query$+$Storage$

Manager$

Privacy$&$se>ngs$

Communica.on$

Sensing$&$Sampling$

Context$Processing$&$Fusion$

Query$+$Storage$

Manager$

Privacy$&$se>ngs$

Query$&$$Response$

Analysis$&$Processing$

Query$+$Storage$

Communica.on$

Collabora.on$

Data$Collec.on&$Comp.$Sampling$

Infrastructure$Sensing$$

Manager$

S1$ S2$ Sm$…….$

Query$ Response$

…….$

Query$&$$Response$

Infrastructure$Sensors$

Mobile$Node$

Broker$

Mobile$Node$

APPS$2$

APPS$N$

Cloud

AP

S1$

Sn$

S1$

Sn$

25

Sensemaking Using Compressed Sensing

•  A random sampling technique that can represent Sparse signal with few random measurements

•  Represents a Sparse Signal with few salient coefficients in a transformed domain

•  Integrates sensing, compression, processing based on new uncertainty principles

26

Collabora0ve Compressive Sensing

Sink Node(Broker) Mobile Node Sampled Mobile Sensor Legend No#of#Measurements##

Reco

nstruc

tion##Error#(M

SE)#

Number of Measurement Accuracy of Sensemaking Number of Measurement Energy Consumed in Sensing Accuracy of Sensemaking Scalability and Coverage

Traded-‐off

27

Sensemaking using Virtual Sensing

Ambient Light

3D Magnetometer

3D Accelerometer

Barometer

Processing( CompressedSensing andCalibration)

SensorFusion

3D Gyroscope

Ambient Light

Barometer

Thermometer

Accelerometer

Gyrometer

Inclinometer

Orientation

Compass

Physical Devices

IsDriving

IsRunning

IsWalking

IsSitting

AtHome

InOffice

IsIndoor

IsAlone

hasFallen

IsHappy

Virtual SensingProcessing

Sampling &Data

Collection(Compressive

Sampling,Adaptive

Sampling)

Location Contexts

Activity Contexts

Context Processing

Social Contexts

Emotional Contexts

EnvironmentalContexts

Health Contexts

28

Research Direc0ons

•  Energy Efficiency –  Exploit collabora0ve & compressive sensing for energy efficiency

•  Incen0ve Mechanisms – Device incep0ves for par0cipa0on and collabora0on

•  Privacy Regula0on –  Facilitate privacy preserving incen0ves

•  Heterogeneity in Mobile Cloud – Use and exploit heterogeneity of sensors and devices

29

30

RELATED WORK REVIEW •  Energy-Efficient Smart Spaces - Smartphone

Augmented Infrastructure Sensing

•  Optimizing Event Detection on Smartphones

•  Spatial-temporal Information Gathering using Smartphones

Smart Spaces

•  Difference scales of intelligent systems: such as ci0es, stadiums, airports, building, and roads

•  Ci0zens of a smart space are not observers but ac0vely help the officials to make the space berer, e.g., –  Safer – More entertaining – More energy efficient – More situa0on-‐aware

•  Similar to smart home, but across mul0ple users

31

Pla?orm for Public Smart Spaces

•  Goal: develop a pla?orm to provide safety with sustainability for smart spaces

•  Detec0ng many events in an energy-‐efficient way – Security related events: fights riots, protests, and demonstra0ons

– Hazardous events: fires, chemical leaks, and stampedes

– High crowd levels for poten0al conflicts

London School of Economics’ app that monitors crowd safety at events 32

Limita0on of Current Approach

State-‐of-‐the-‐art: Infrastructure sensing using in-‐situ sensors –  High installa0on and maintenance cost –  Insufficient node coverage ß limited budget –  Does not scale! ß for crowded events

33

Usage Scenario #1

•  Task: Sensing temperature at CS building •  What if there is no working thermometer at the CS building?

–  Infer the temperature by nearby buildings –  Infer the temperature provided by 3G/4G smartphone users walking by the CS building

34

Usage Scenario #2 •  Task: Traffic surveillance for safety applica;ons •  What if the fixed surveillance videos are insufficient ?

–  Leverage videos from nearby in-‐situ cameras –  Leverage videos captured by police officers, fire fighters, and EMTs

–  Leverage large volume of user-‐generated, geo-‐tagged videos captured by ci0zens

35

Dashboard hrp://info.theomegagroup.com/blog/bid/134307

36

System Architecture

37

Current Prototype

38

Challenges

•  How to efficiently carry out the sensing requests? •  How does the broker assign the requests to workers? •  How to guide workers to the correct sensing loca0on? •  How to efficiently process the raw sensory data? •  Where to process the raw sensory data? •  Can we leverage mul0ple close-‐by sensors for higher accuracy?

39

40





Event Detec0on on Smartphones

•  Each event may be detected by mul0ple subsets of sensors ß subop0mal sensor subsets? –  E.g., traffic jam may be detected by GPS, accelerometer, or GPS + accelerometer

•  Mul0ple events may be (par0ally) detected by the same sensors ß uncoordinated sensor usage leads to redundant sensor ac0va0on –  E.g., earthquake may also be detected by accelerometer

•  Problem: how to select efficient sensing strategies

41

Context-‐aware Mobile Applica0ons

•  Increasingly more context-‐aware apps leverage the smartphone sensors for berer user experience

•  What is context-‐aware? –  Essen0ally inferred from sensor readings!

42

An Equivalent Research Problem

•  Context-‐aware apps may –  Infer the same context using various combina0ons (sets) of sensors

–  Impose diverse accuracy requirements •  How to select efficient sensing strategy?

–  Sa0sfy all apps’ requirements – Minimize energy consump0on

•  Proposal: OSM (Op0mal Sensor Management) middleware

43

OSM Middleware

OSM Middleware

•  It sits between apps and hardware •  Apps may register or unregister requests through an API at any 0me.

•  Our middleware is response to – Maintain a database of ac0ve requests – Determine what sensors to ac0vate at what 0me

44

System Architecture

45

API: 1.  Register()/Unregister() 2.  Feedback()

Request Manager 1.  Manages a Request

Queue 2.  Preprocess the contexts

Context Analyzer 1.  Context Updater 2.  Model Trainer

Resource Manager 1.  Barery Monitor 2.   Scheduling Algorithm

System Model •  Combina0on/Accuracy/

Energy

• Coordinated and efficient sensor usage! • Avoid redundant energy waste!

How to Op0mally Schedule Sensor Ac0va0ons?

•  Tradeoff between accuracy and energy consump0on

•  Our scheduling algorithms have to pick the best combina0on for all requests

•  The already-‐on sensors have to be considered

46

What if WiFi is already on?

Our Proposed Scheduling Problems

Two op0miza0on criteria: – Energy Minimiza;on (EM) Schedule with the lowest energy to sa0sfy all the apps’ requirements – Accuracy Maximiza;on (AM) Schedule with the highest overall accuracy under an energy budget

47

Energy Minimiza0on (EM) Formula0on

48

Minimize energy

Sa0sfy accuracy requirements

Within energy budget

Maximize accuracy

Accuracy Maximiza0on (AM) Formula0on

49

Proposed Scheduling Algorithms

•  Energy Minimiza;on Algorithm (EMA) Accuracy Maximiza;on Algorithm (AMA) •  Good performance •  Suitable for smaller problems due to high complexity

•  Efficient Energy Minimiza;on Algorithm (EEMA) Efficient Accuracy Maximiza;on Algorithm (EAMA) •  Shorter running 0me •  More suitable for smartphones •  Inspired by two approxima0on algorithms for the weighted set cover and 0/1 knapsack problems ß But the approxima0on factor proofs do not work in our problems

50

Our Simulator

•  We developed an event-‐driven simulator in Java •  Baseline algorithm

–  Selects the sensors for the highest accuracy of each context

•  We compare the scheduling algorithms: –  Op0mal : EMA/AMA –  Heuris0c : EEMA/EAMA –  Baseline

•  Collect running apps in Android ac0vity stack from 5 users for three weeks

•  Measure power consump0on on a Samsung Galaxy S

8

Energy Saving

•  Save at least 40%, compared to the baseline •  EEMA achieves a small gap of ∼2% than EMA •  EMA terminates in 50ms and EEMA terminates in 1ms

9

Accuracy Improvement

•  Increase accuracy by up to 39.06% than the baseline •  EAMA achieves a gap of ~1% than AMA •  AMA terminates in 5000ms and EAMA terminates in 1ms 53

More Restricted Environments Lead to Higher Gains

54

Lower Accuracy Requirement Less Energy Budget

Save More Energy Higher Accuracy Boost

Larger Problems Result in Higher Gains

55

Save More Energy Higher Accuracy Boost

Real Prototype System

•  Implement two heuris0c algorithms and the proposed OSM on Android

•  EEMA – Prolongs barery life two 0mes – Achieves accuracy : 93.94%

•  EAMA – Prolongs barery life 1.5 0me – Achieves accuracy : 94.85%

56

Summary

•  We propose an Op0mal Sensor Management middleware

•  Four algorithms with different op0mal criteria and complexity levels for sensor scheduling

•  EEMA (EAMA) saves energy (boost accuracy) in real-‐0me •  Real implementa0on on smartphone

•  Designed for a single smartphone, but the same sensor management mechanisms may be used for event detec0on in smart spaces

57

58





Geospa0al Informa0on Gathering

•  A new class of crowdsourcing systems •  Requesters: companies and organiza0ons

•  Submit geospa0al and temporal-‐dependent tasks (specific 0me and loca0on)

•  Task: capturing videos/pictures or collec0ng sensor readings

•  Workers: smartphone users •  Report their des0na0on and deadline •  They wouldn’t mind to take some detour routes for small rewards

59

Detour Planning Problem

•  Sample scenario: A smartphone user who needs to get to the Chia-‐Yi HSR Sta,on at 7 p.m. may have a few hours to spare. Why not making some money? –  But it’s hard for a person to come up with the detour path

•  Our problem: How to find the best detour path for each worker –  to maximize the profit (= rewards – costs) – while guaranteeing on-‐0me arrival at the des0na0on

60

System Architecture

61

Feasible Spots

62

Problem Formula0on

Maximize overall profits

Start and end points

No rep. feasible spots

Arrive des0na0on in 0me

Visit each request once Start 0me of each request

Finish 0me of each request

63

Orienteering Problem with Time Window

•  A similar problem –  Goal: maximize the score –  Game: players go to specific spots, and finish the predetermined job for a reward –  Not exactly the same: (1) mul0ple feasible spots and (2) travel cost (gas and car deprecia0on)

•  We enhanced a dynamic programming based OPTW algorithm [GS09] for an op0mal Detour Planning (DP) algorithm – Runs in polynomial 0me: O( N3Z3 )

64

[RS09] Decremental state space relaxa0on strategies and ini0aliza0on heuris0cs for solving the orienteering problem with 0me windows with dynamic programming. Computers and Opera0ons Research, 36(4):1191–1203, April 2009.

Collec0ng Feasible Spots

•  Find 25 landmarks in Taipei (hrp://taipeitravel.net) and Vancouver (hrp://hotels.com)

•  Use Flickr API to download the pictures tagged with each landmark, and retrieve the longitude/la0tude

•  Use hierarchical clustering algorithm to group these photos at the granularity of blocks (~100 m) ß gives us the feasible spots

•  Employ Google map to compute the distance between any two feasible spots

65

Simulator Implementa0on

•  We implement a trace-‐driven simulator in C++ •  It supports five algorithms

– The proposed DP algorithm – Four heuris0c algorithms

•  Highest-‐Reward (HR) ß mimic human behavior •  Closest-‐Request (CR) ß mimic human behavior •  Highest-‐Reward with On0me (HROT) •  Closest-‐Request with On0me (CROT)

66

Simula0on Design

•  Parameters – N: number of requests: {5, 10, 15, 20, 25} – T: deadline: {1, 2, 4, 8, 16} (hr) – C: travel cost: {0, 0.06, 0.12, 0.24, 0.48} ($/km)

•  Metrics – Total rewards – Running-‐0me – On0me-‐ra0o

67

On0me Ra0o

HR and CR (mimicing humans) à low on0me ra0os! 68

Total Profits

•  Although HROT and CROT guarantee on0me arrival, they suffer from low profits

•  Compared to HROT and CROT, DP doubles the rewards with 25 requests – More requests à larger gap! 69

DP is Efficient

•  Terminates in less than 60 ms •  Slower for Vancouver (right) ß more feasible spots

70

Implica0on of Travel Cost

•  Higher profits when per-‐km cost is lower

71

Summary

•  Studies a new class of crowdsourcing problems –  Geospa0al informa0on gathering

•  Proposes an op0mal detour planning algorithm based on an OPTW algorithm

•  Simula0on results are encouraging •  Poten0al Extensions

–  Implemen0ng a working prototype –  Guide the workers to shoot photos using augmented reality –  Quality assurance and cheat detec0on mechanisms

•  Designed for collec0ng spa0al-‐temporal mul0media informa0on, but can be extended for event detec0on

72

Ques0ons?

73

Challenges to realize smart spaces • How to efficiently carry out the sensing requests? • How does the broker assign the requests to workers? • How to guide workers to the correct sensing loca0on? • How to efficiently process the raw sensory data? • Where to process the raw sensory data? • Can we leverage mul0ple close-‐by sensors for higher accuracy?

Documents

SenseDroid