Darwin Phones: the Evolution of Sensing and Inference on Mobile Phones

Darwin Phones: the Evolution of Sensing and Inference on

Mobile Phones

Emiliano Miluzzo*, Cory T. Cornelius*, Ashwin Ramaswamy*, Tanzeem Choudhury*, Zhigang Liu**,

Andrew T. Campbell*

* CS Department – Dartmouth College** Nokia Research Center – Palo Alto

miluzzo@cs.dartmouth.eduEmiliano Miluzzo

evolution of sensing and inferenceon mobile phones

Emiliano Miluzzo

PR time

miluzzo@cs.dartmouth.edu

ok… so what ??

density

accelerometer

digital compass

microphone

WiFi/bluetooth GPS

light sensor/camera

sensing

accelerometer

digital compass

microphone

WiFi/bluetooth GPS

light sensor/camera

gyroscope

air quality /pollution sensor

sensing….

- free SDK- multitasking

programmability

- 600 MHz CPU

- up to 1GB application memory

hardware

computation capability is increasing

application distribution

deploy apps onto millions of phones at

the blink of an eye

collect huge amount of data for research

purposes

deploy apps onto millions of phones at

the blink of an eye

cloud infrastructurecloud - backend support

we want to push intelligence to the

preserve the phone user experience

(battery lifetime, ability to make calls, etc.)

- sensing- run machine learning

algorithms locally (feature extraction +

inference)

run machine learningalgorithms (learning)

store and crunch big data(fusion)

inference)

3 to 5 years from now our phones will be as powerful as a - sensing

- run machine learning algorithms locally

(feature extraction + inference)

- Sensing

(feature extraction + learning + inference)

3 to 5 years from now our phones will be as powerful as a

sensingprogrammability

cloud infrastructure

sensingprogrammability

cloud infrastructure

societal scale sensing

global mobile sensor network

reality mining using mobile phones

will play a big role in the future

end of PR – now darwin

Emiliano Miluzzo miluzzo@cs.dartmouth.edu

a small building block towards the big vision

Emiliano Miluzzo miluzzo@cs.dartmouth.edu

from motes to mobile phones

darwin

- classification model evolution

- classification model pooling

- collaborative inference

microphone

camera

GPS/WiFi/cellular

air quality pollution

sensing apps

social context

audio / pollution / RF fingerprinting

image / video manipulation

darwin applies distributed computing and collaborative inference concepts to

mobile sensing systems

darwin

- classification model evolution

- classification model pooling

- collaborative inference

why darwin?

mobile phone sensing today

why darwin?

train classification model X in the lab

why darwin?

deploy classifier X

why darwin?

train classification model X in the lab deploy classifier X

train classification model X’ in the lab

why darwin?

deploy classifier X

deploy classifier X’

why darwin?

train classification model X in the lab deploy classifier X

deploy classifier X’

a fully supervised approach doesn’t

scale!

why darwin? a same classifier does not scale to multiple

environments (e.g., quiet and noisy env)

darwin creates new classification models transparently from the user

(classification model evolution)

why darwin?

ability for an application torapidly scale to many devices

why darwin?

ability for an application torapidly scale to many devices

darwin re-uses classification models when possible

(classification model pooling)

why darwin?

leverage the large ensemble of in-situ resources

why darwin?

leverage the large ensemble of in-situ resources

darwin exploits spatial diversity and co-operate to alleviate the “sensing context”

problem(collaborative inference)

darwin design

speaker recognition (subject to audio noise, sensing context, etc.)

darwin phases

initial training (derive model seed)supervised

darwin phases

initial training (derive model seed)

classification model evolution

supervised

unsupervised

darwin phases

classification model pooling

supervised

unsupervised

darwin phases

collaborative inference

supervised

unsupervised

classification model training

sensed event

filtering (silence suppression +

voicing)

sensed event

voicing)

featureextraction(MFCC)

voicing)

featureextraction(MFCC)

modeltraining(GMM)

baseline

sensed event

send model + baseline back to phone

send MFCC tobackend to train the model

backend

phone: feature extraction(low

computation)

backend

backend: model training (high

computation)

phone: determines when to evolve

training

training sampled

match?

do not evolve

match?

evolve(train new model using

backend as before)

new speaker voice model

training

Speaker A’s modelPhone A Phone B

Phone C

Speaker C’s model

Speaker B’s modelSpeaker B’s model

Speaker C’s model

Phone C

Speaker C’s model

we have two options

1. train a new classifier for each speaker (costly for power, inference delay)

Phone C

Speaker C’s model

we have two options

2. re-use already available classifiers

Phone C

Speaker C’s model

we have two options

2. re-use already available classifiers

Phone C

Speaker C’s model

Phone C

Speaker B’s modelSpeaker C’s model

Speaker C’s model

Speaker B’s model

Speaker A’s model

Speaker C’s model

Phone C

Speaker C’s model

Speaker B’s model

Speaker A’s modelSpeaker C’s model

Speaker A’s model

Speaker B’s model

Phone C

Speaker C’s model

Speaker A’s modelSpeaker B’s model

Speaker B’s model

Phone C

Speaker C’s model

Speaker A’s modelSpeaker B’s model

Speaker B’s model

ready to run the collaborative inference algorithm

- local inference first- final inference later

two phases

1. local inference (running independently in parallel on each mobile phone)

two phases

1. local inference (running independently in parallel on each mobile phone)

two phases

2. final inference (after collecting Local Inference results, to get better confidence about the final classification result)

local inference (LI)

Phone A Phone B

Phone C

Phone A Phone B

Phone C

speaker A speaking!!!local inference (LI)

Phone A Phone B

Phone C

A’s LI results:Prob(A speaking) = 0.65Prob(B speaking) = 0.25Prob(C speaking) = 0.10

C’s LI results:Prob(A speaking) = 0.30Prob(B speaking) = 0.67Prob(C speaking) = 0.03

B’s LI results:Prob(A speaking) = 0.79Prob(B speaking) = 0.11Prob(C speaking) = 0.10

Phone A Phone B

Phone C

Phone A Phone B

Phone C

speaker A speaking!!!

Phone A Phone B

Phone C

speaker A speaking!!!

Phone A Phone B

Phone C

Phone A Phone B

Phone C

individual classification can be misleading!

final inference (FI)

Phone A Phone B

Phone C

each phone gathers LI results

A’s LI results

C’s LI results

B’s LI results

A’s LI results A’s LI results

C’s LI results C’s LI results

B’s LI resultsB’s LI results

collaborative inferenceon each phone

FI results (normalized):Confidence (A speaking) = 1 Confidence (B speaking) =

0.12Confidence (C speaking) =

=FI results (normalized):Confidence (A speaking) = 1 Confidence (B speaking) =

collaborative inference compensates the inaccuracies of individual

inferences

FI results (normalized):Confidence (A speaking) = 1 Confidence (B speaking) =

evaluation

C/C++ &

implemented on Nokia N97 andiPhone in support of a speaker

recognition app

evaluation

C/C++ &

unix server

recognition app

evaluation

C/C++ &

unix server

lightweight reliable protocol to transfer models from the server

and between phones

recognition app

evaluation

C/C++ &

UDP multicast protocol to distribute

local inference results between phones

recognition app

experimental scenarios

up to eight people in conversation in three different scenarios (quiet indoor, down the

street, in a restaurant)

some numerical results

need for evolution

train indoor, evaluate outdoor

need for evolution

accuracy improvement after evolution

accuracy

indoor quiet scenario

8 people talking around a table

collaborative inference + classification model evolution

boost the performance of a mobile sensing app

impact of the number of mobile phones

the larger the number of mobile phones collaborating, the better the final inference result

battery lifetime Vs inference responsiveness

high responsiveness

short battery life

longer battery duration

low responsiveness

smart duty-cycling techniques and machine learning algorithms with better performance in

terms of energy usage on mobile phones need to be identified

a quick recap

smartphone’s are everywhere, let’s exploit their collective sensing

and computation capabilities

a quick recapsmartphone’s are everywhere – let’s exploit their collective sensing and computation capabilities

smartphone sensing opens up new frontiers: applications can be spread and

big data collected at unprecedented scale enabling endless research opportunities

continuous sensing is still challenging; efficient mobile sensing requires to

preserve the phone user experience (need for energy efficient ML algorithms

and smart duty-cycling techniques)

smartphone sensing opens up new frontiers: applications can be spread and big data collected at unprecedented

scale enabling endless research opportunities

continuous sensing is still challenging; efficient mobile sensing requires to preserve the phone user experience (need for energy efficient ML algorithms and smart duty-

cycling techniques)

ML algorithms should perform reliably in the wild

scale enabling endless research opportunities

cycling techniques)

scale enabling endless research opportunitiesok I think I’m done…

cycling techniques)

scale enabling endless research opportunitiesbut please bear in mind…

Mobile Phone Sensing is the Next Big Thing!

Thank you!!

Mobile Sensing Grouphttp://sensorlab.cs.dartmouth.edu

Darwin Phones: the Evolution of Sensing and Inference on Mobile Phones

Documents

Darwin Phones: The Evolution of Sensing and Inference …pac.cs.cornell.edu/pubs/mobisys2010_darwin.pdf · Darwin Phones: the Evolution of Sensing and Inference on Mobile Phones Emiliano

DARWIN HARBOUR · 4 SKYCITY Darwin 5 Cullen Bay Holiday Apartments 6 Marina View Serviced Apartments 7 Rydges Darwin Airport Resort 8 Rydges Darwin Airport Hotel 9 Best Western Darwin

Charles Darwin William Paley. Variation Competition Wallace Darwin

Smart phones vs feature phones

DARWIN 4. La Ciencia en Tiempos de Darwin

Darwin Phones: the Evolution of Sensing and Inference on ...sensorlab.cs.dartmouth.edu/pubs/mobi040-miluzzo.pdfDarwin Phones: the Evolution of Sensing and Inference on Mobile Phones

Attachment A - Welcome to Darwin Port | Darwin Port

Darwin Core Ratified in the Year of Darwin

DARWIN, DARWINISME, NEODARWINISME · 2016. 5. 8. · DARWIN, DARWINISME, NEODARWINISME ~ per ANTONI BENNASSAR ROIG DARWIN, 1809-82 DARWIN Charles Darwin va néixer l'any 1809 en el

Con Darwin, oltre Darwin

evolutionary theory worksheet - home - Evolution of life€¦ · Development of evolutionary theory Darwin 1838 Darwin 1837 Darwin 1837 Darwin 1859 Bates 1862 Haeckel 1866 Darwin

Entwicklung der Evolutionstheorie - evolution-of-life.com · Entwicklung der Evolutionstheorie Darwin 1838 Darwin 1837 Darwin 1837 Darwin 1859 Bates 1862 Haeckel 1866 Darwin 1871

Darwin 6. Darwin Vuelve A Casa

Lecture 5 Fuzzy expert systems: Fuzzy inference Mamdani fuzzy inference Mamdani fuzzy inference Sugeno fuzzy inference Sugeno fuzzy inference Case study

Darwin anti darwin muy bueno

Darwin 1 Y 2 IntroducciÓn & La Familia Darwin

Darwin Si Dupa Darwin 2-1

Darwin Phones: The Evolution of Sensing and Inference on …campbell/papers/darwin.pdf · ization of the person’s microcosmos could be seen as a new form of communication? We believe

Darwin 3. El Joven Darwin

Darwin Phones: The Evolution of Sensing and Inference on Mobile Phones Presented By: Brandon Ochs