Context Aware: TNBT, … Thing?

The Big Bet of (CaC) is that The Next Big Thing is:Dramatic Increase of Human Productivity… using

computational skills provided by computers as well as …sensing details system that will

augment people experience in interaction with reality

Gordon Bellwww.MyLifeBits.com www.TotalRecallBook.com

8 March 2010

CA: Context Aware Computing… Assume it is TNBT; then is it?

• New capabilities based on these platforms or technologies? a. All the bits in your e-Memory (Bell-Gemmell PoV)b. “Personal_state(t)”: Loc/pt. x,y,z,ρ,θ,φ; HR,BP, stress; T, Δ’sc. Social Networkingd. Cloud enabling new client platformse. All the above

• New product(s)? New Service(s)? Features? Featurettes?– Revenue? Business model?

• CA-thing is very specific as the intersection of ≥2 “dimensions”• CA-ness can enable a function or a “point” product/service• What problem(s) does having a/more context solve?

– How?– Why?... More context, more opportunities for action?– When in doubt, create a taxonomy!

CA and CAO: GB thoughts re. definitionMS makes $s as an opportunity enabler

• Context Awareness … the intersection of two or more “state dimensions” providing an opportunity for some kind of action e.g. meeting, insight, path to take/avoid, something to buy… – Time and place– Place and place (e.g.meet friends, shopping)

• CA is the detection & creation of CAO’s… SEARCH!• More state => more opportunities for action• More public state => interpersonal opportunities• More mobility (physical or cyberspace) => opportunity• Availability of entire life => opportunity

What is Context? (ISO)

User• Goals• Tasks

SocialPhysical

Infrastructure• Hardware• Software• Materials

9/10/2008 from Context Vision

Users, their goals and tasks, infrastructure (hardware, software, materials), physical and social environment in which the product is used.*

* As defined by ISO 9421-11

Scope: User 9/2008

9/10/2008 from Context Vision Microsoft Confidential 5

Goals & Tasks• Activities

Health• Health History• Current Health

Traditions• Person• Family• Social Groups• Nation

Social Relations

Psychological Type & Current

Emotions

Life Experience &

Personal Interests

ContactIdentity Data

• Name, addresses,

emails, phones, etc.• Biometric

“Context combines users, their goals and tasks, infrastructure (hardware, software, materials), physical and social environment in which the product is used.”

My context = My e-Memory + my current state

eMemory =ground truthbioMemory = metadata...

Memory centric view: Context = eMemory + bioMemory (aka State of mind) + environment

Web Web• Mail, dcs, books, web, journals, podcasts… Vibe

• Social: iM, facebook, … calls, conversations…

• Financial: $ transaction• Health: PHR, activity,

diet • iLife {photo/movie TV

music}, trip trajectories• Logs … for all “lives” …for all activities

Memex aka MyLife Bitsaka eMemory

“Personal_state(t)Loc/pt. x,y,z,ρ,θ,φHR, BP, stressTemp, Δ’s… ???s

“Lives ...”

Realizing Memex… Digital Capture, Storage, and Utilization of All Personal Information

www.MyLifeBits.com

MyLifeBits Platform c2005

MyLifeBits store

database

Voice annotation tool

Telephone capture tool

TV capture tool

TV EPG download tool

Radio capture & EPG

PocketPC transfer tool

PocketRadio player

Import files

MyLifeBits Shell

Browser tool

Internet

IM capture

GPS import & Map display

SenseCam

Screen saver

Text annotation tool

MAPI interface

Legacy email client

Outlookinterface

files

Legacy applications

VIBElogging

RoomCapture

Using an e-Memory to augment b-memory(how did we get the bits for context?)

• 1998-2001 > Capturing the high order past & present bits– Instigated by Raj Reddy’s Million Book Project– Stimulated by telepresence, Billg “someday you can recall everything…”, idea of

tidiness, design for CyberMuseum (Computer History Museum)– “MyLifeBits is about converting everything to bits”

• 2001-2005 > Getting bits from e-Memory… as many bits as possible– Stimulated by Bush, Gray (database) … – MyLifeBits Project moves to SQL. Acquire “context” from everything we can– “MyLifeBits is a transaction processing system for a life”

• 2005: Memex Project. Redmond := Gemmell and Lueder; Project := 0• 2009: Total Recall := Book_write_&_publish (Bell_and_Gemmell)

– “Mylifebits is an e-memory… to aid bio-memory for life &afterlife”• Right Period to create “Memex” based on technologies

– We can encode it: Scanning and recognition problem (worked on e-Paper)– We can store it: Terabyte (we thought that would store a digital life)– We can find it: Search came into its prime

NOT LIFE-BLOGGING… We think life-Blogging is nuts! LIFE-BLOGGING =>

GREATER CAO

Work: E

mail

Work: O

ffice Contacts

Work: O

ffice Calendar

Work: O

ffice docu

ments

Home Office

: Financia

l

Work: L

egacy docu

ments

Media: P

hotos

Learn

ing: Books,

journals,

etc.

Health: P

HR

>>Health: D

iet & Exerci

se

Home Office

: Legal d

ocuments

Work: W

eb pages

>Life: c

hatbot

Life: M

emorabilia, e

phemera

Media: M

usic (C

Ds, ca

ssettes,

reco

rds)

Media: H

ome video (Film

and VHS)

Life: S

enseCam Days

>Work: C

onversations

>Work: T

&M (V

IBE)

>Life: T

racked Days

>Work: T

elephone conversa

tions

0

25

50

75

100

Current

NEED

Current Experiment NEED

Capturing every step.If I know where I am, what good does it do me or anyone?

HEALTH RECORDSBY THE INDIVIDUAL, FOR THE INDIVIDUAL

Hospital, GP, specialist, lab, dentist, chiropractor, pharmacist…You need to have control of your records

Health: Context matters! When you are about to …

Cell Phone as a Platform for HealthcareTechFest 2009 SCS

Where was I,what was I doing,who was I with,(what meeting was I in)when I had a high HR or high stress indicator ?

The killer app. Health“those spots look a bit better, don’t they?”

NavigenicsReport

1942 Mayo Clinic visit

BIO-METRIC LIFE-LOGGING

EKG just before pacemaker 040629

A flaky connection between chambers!

Capturing every heartbeat

• 72.6 beats/min; 38.16 Million beats/year• 3.13 billion beats year• The important number is 4-4.5 years, or ETS• Battery life: the expected trip to surgery!

Bypass Feb 83

Tracking Test Results…….

It’s all context… effecting mental state

Philips in-home devices

Bodybugg

Smart materials

In-body health sensing

Nanobot in the bloodstreampillcam

EndoSure Wireless Pressure Sensor in an aneurysm sac

Health Monitoring:“Your husband just died, … here’s his black box”

Context awareness actions are based on f(which lives you are in right now,what activity are you engaged in,

your accessible digital life)

Cont

acts

Offi

ce D

ocum

ents

Co

mm

unic

ation

em

ail

Prof

essi

on s

peci

fic

Fina

nce

& le

gal

Hea

lth re

cord

sO

n bo

dy re

cord

ing

Book

sIn

stru

ction

al M

at.

Mus

icPh

otos

Vide

os

Loca

tion

Ever

ythi

ng I

see

Ever

ythi

ng I

hear

FamilyParents, Siblings, ChildrenFriends

Clubs and Organizations

Educational Institutions Work organizations… Projects

Office Health Learning Ambience PersonHome Job

Who

: Liv

es…

Life

time

perio

dsW

hich

live

sWhat: Degree “being digital”. Your “digital life”

Where: “Location” of your digita

l life

PC … Home … Cellphone … Cloud… Public Cloud

Degree of privacy

“Lives” aka “Lifebrowsing”“Personal Role Management”“Autobiographical Memory”for structuring life into organizational groups; general events; and event specific knowledge time periods

Who: which LifeWhat: activityWhere: you are

Ches

ter

Bell

(189

7)Lo

la B

ell (

1899

)Ch

este

r G

ordo

n Be

llSh

aron

(Sm

ith)

Kirk

svill

e, M

OM

.I.T.

(Bos

ton)

U.N

SW (A

ustr

alia

)

Gw

en D

ruyo

r Be

llBr

igha

m (s

on)

Fion

a Be

llBr

idge

t Be

llLa

ura

(dau

ghte

r)Ko

lbe

Schu

ltz

Stry

ker

Schu

ltz

Sher

idan

& L

ogan

F...

M.I.

T. S

peec

h La

bD

igit

al (M

ayna

rd)

CMU

(Pitt

sbur

gh)

Com

pute

r St

ruct

ures

Dig

ital

(May

nard

)En

core

(Bos

ton)

Ang

el (1

00 S

V St

art.

..N

SF (w

ashi

ngto

n)A

rden

t (S

ilico

n Va

lley)

Bell-

Mas

onBe

ll (M

e Lt

d.)

Mic

roso

ft R

es. (

SF)

MyL

ifeBi

ts, D

igit

al L

...H

PC, 4

th P

arad

igm

, …Co

mpu

ter

Mus

eum

Vang

uard

, Dia

mon

X

ACM

, IEE

E, N

AE,

NA

SG

ordo

n Be

ll Pr

ize

Med

al o

f Tec

hnol

ogy

AM

ACA

DN

at. A

cad.

of S

ci.

Aus

t. A

cad

of T

ec S

ci

1910

1920

1930

1940

1950

1960

1970

1980

1990

2000

2010

Personal LifeLog Applications

Conservator

Baby Book

Companion Caretaker

Babysitter

AdvisorMentorTutor

Autobiography

Photo Album

PersonalAssistant

Diary/Journal

Biography

Medical Manager

ExecutorObituary

Others Self

Assistant for Elderly

Application controlled by:

Oth

ers

Sel

fA

pp

licat

ion

use

d b

y:

Personal Proxy

Parole OfficerPersonal

Flight Recorder

Meeting Prep

Captain’s Log

Trustee

Financial Manager

Doug Gage, ARPA LifeLog

My Bits… then (c2000) and now (c2010)• Single system… • Where are they?• How owns them?• Who can access them?• How do I keep them

private?• …

Bush’s Memex c1945

Spaces

Hotmail

Abby’s Home PC

Toby’s PC

Abby’s Laptop

Abby’s Camera

SkyDrive UX

Microsoft Storage

Toby’s Pocket

PC

Office Live

Phone

Memex Store

From Susan Dumais c2005(I think)

My Bits 2010Dist’d state &Context

Where are all of your bits and meta-data?Islands of content & pointers

1. Folder/file hierarchy 2. Outlook (msg, contact, calendar, task); Journal & VIBE3. OneNote; EverNote4. Social/professional: iM, Skype, Connection Manager,

LinkedIn, Facebook, Twitter, etc.5. Money… financial institutions (bank, broker, insurance)6. HealthVault … BodyBugg, … medical providers7. Music: iTunes, Media Player, Media Center, …Zune8. Photos: Live photo gallery, Picassa, Flicker9. Media Center10. iLife={iPhoto, iMovie, iTunes, iWeb, iBook}

Outlook_CalendarItems2

PK,FK1 item_id

Subject Start End Description Location Creation Time Modified

location_data2

PK,FK1,I1,I2 item_id

U1,I1,I2 TimeI1 LatitudeI2 Longitude ElevationU1 Source

_EventTypes2

PK event_type_id

description

ntfs_dirs2

PK,FK1,I2 item_id

I1 ntfs_root_id volume_serial_number frn parent_frn CreatedI2 Modified Accessed Attributes IsRoot SyncOnStartup

MSMSGS_logs2

PK,FK1 item_id

Handle Logon Name Buddy Handle Buddy Logon Begin Time End Time

_FullTextColumns2

PK,FK1 subtype_idPK table_namePK column_name

Outlook_Contacts2

PK,FK1 item_id

Title Display Name First Name Middle Name Last Name Suffix Email Email2 Email3 IM Address Company Department Job Title Business Street Business City Business State Business Postal Code Business Country Home Street Home City Home State Home Postal Code Home Country Other Street Other City Other State Other Postal Code Other Country Assistant's Phone Business Fax Business Phone Business Phone 2 Callback Car Phone Company Main Phone Home Fax Home Phone Home Phone 2 ISDN Mobile Phone Other Fax Other Phone Pager Primary PHone Radio Phone TTY/TDD Phone Telex Account Anniversary Assistant's Name Birthday Location Manager's Name Notes Office Location Profession Sensitivity Spouse Web Page Creation Time Modified

TAPI_PhoneCalls2

PK,FK1,I1 item_id

Phone Call Type CID CID Name CID #I1 Begin End Seconds Connected Ended Roaming Trimmed Recorded Transcript

IE_Web_Pages2

PK,FK1,I1,I2 item_id

Domain Page UrlI1,I2 TitleI1,I2 VisitedI2 Top Level Secure IndexedContent big_thumbnail

Music_Songs2

PK,FK1 item_id

Title Author Album Genre Duration Bitrate

Outlook_Tasks2

PK,FK1 item_id

Subject Start Date Due Date Percent Complete Status Priority Notes Creation Time Modified

ntfs_files2

PK,FK1,I2,I3 item_id

I1 ntfs_root_id volume_serial_number frn parent_frn Extension LengthI3 CreatedI2 Modified Accessed Attributes IndexedContent

_ItemSubTypes2

PK subtype_id

name openwith small_icon_id large_icon_id delete_stored_procedure

_Item-Events2

PK,I1 event_id

FK2,I1 item_idFK1,I1 event_type_idI1 occurred

IMG_Images2

PK,FK1,I1,I2,I3 item_id

Width HeightI1 Date Taken Camera Make Camera ModelI2 LatitudeI3 Longitude Elevation

_Item-ItemSubTypes2

PK,FK1,I1 item_idPK,FK2,I1 subtype_id

_Links2

PK,FK3,I1,I2 link_type_idPK,FK1,I1,I2 source_item_idPK,FK2,I1,I2 target_item_id

_Items2

PK,I1,I2,I3,I4,I5 item_id

I5 nameI1 creation_dateI4 last_modified_dateI2 interval_begin interval_endI3 is_collection is_annotationI3 is_hidden thumbnail thumbnail_width thumbnail_height collection_cnt annotation_cnt descendants_cnt annotates_cnt child_cnt large_icon_id small_icon_id rank

TV_Recordings2

PK,FK1,I1 item_id

Title Genre Year Rating DescriptionI1 Date Recorded Duration Channel Channel Description Bytes unc_path cached_local cached_path

Radio_Recordings2

PK,FK1 item_id

Title Date Recorded Duration Station unc_path cached_local cached_path

Outlook_MailItems2

PK,FK1 item_id

From Name SubjectI1 Received Sent Body From Address From Type To Cc Bcc Attachments Read Modified

_LinkTypes2

PK link_type_id

name

BLOB_Data2

PK item_id

blob blob_file_extension blob_size

Outlook_StickyNotes2

PK,FK1 item_id

Body Creation Time Modified

_log1

timestamp nestlevel sender description time numrecords

NTFS_Promotion_Handlers1

PK Extension

Assembly

_Agent_Work_Items1

PK work_item_id

I1 sequence_noI2 item_idI3 work_item_class work_item_type submit_time completion_time priority status message retry_step retry_next retry_counter

ntfs_roots1

PK item_id

path crawl_needed

IE_Popup_Log1

FK1 site_id date url allowed

docsim_doclen1

PK item_id

doclen

cardswipe_data1

PK,I2 item_id

I2,I1 Read Time Booth Number Demo Title Full Name Email Title Reports To Function Cost Center

docsim_idf1

PK term_idPK idf

SenseCam_Raw1

I2,I3 item_idI3 sample_time

sample_no hours mins tilt_x tilt_y lightI2 tempI2,I3 trigger image_no red green blue

ntfs_sync1

ntfs_sync_id

ntfs_root_id file_id volume_serial_number frn parent_frn name extension length attributes createdI2 modified accessed is_dir is_new is_modified

IE_Popup_Rules1

PK site_id

I1 domain rule

_msgs1

timestamp msg source

docsim_item-terms1

PK item_idPK,I1 term_id

tf Shell_SavedViewState1

view_state_id view_state

Outlook_CrawlInfo1

PK store_idPK entry_id

crawl_date is_root should_crawl last_full_crawl last_quick_sync last_slow_sync quick_sync_n quick_sync_units slow_sync_n slow_sync_units is_syncing

_promotion_agent_handlers1

PK extension

file_id description

_hashValues1

item_id hashvalue

VIBE_Activity1

Time HWND Title Foreground Document Keyboard Mouse Application item_id

docsim_item-terms_prox1

PK item_idPK term_idPK position

_Icons1

PK icon_id

U1 class image width height

classif_aliases1

PK topic_idPK alias

_QueryResults1

PK,FK1,I1 query_result_idPK,I1 item_idPK instance

num

docsim_noise1

PK term

_config1

PK key

value

docsim_query1

PK query_idPK term

classif_facets1

PK facet_id

name xmlns

_custom_queries1

PK item_id

Description create_results_table query query_where_clause override_select_sql default_order_by

docsim_terms1

PK term_id

U1 term occurrences

_QueryResults_Info1

PK query_result_id

created accessed delete_req sql

classif_topics1

PK topic_id

name description facet_id parent_topic_id

VIBE_KMTimeline1

Time Keyboard Mouse

NTFS_ChangeLog1

PK entry_id

I1 ntfs_root_idI1 sequence_no priorityI2 URI Type Submitted Updated Status Comments Arg1 ErrArg1

Outlook_Ids1

I1,I2 store_idI1,I2 entry_idI1 delete_processor_is_valid

I1,I2 item_id

ntfs_volume_info1

PK volume_serial_number

drive_letter usn_journal_id next_usn

docsim_tfidf1

PK item_idPK,I1 term_id

tf_idf

_QueryResults_Schema1

PK query_result_idPK table_or_view

number

_event_queries1

PK item_id

event_type_id topn column_name sort_table_or_view sort_column sort_dir description

Wmp_EventArguments1

PK event_id

current_position old_position new_position

cliplets_raw1

PK,I1,I2 item_id

Media Start Media StopI1 Record BeginI2 Record End video_item_id

BLOB_Data1

PK item_id

blob blob_file_extension blob_size

SenseCam_Raw21

I2,I3 item_idI3 sample_time

sample_no hours mins tilt_x tilt_y lightI2 tempI2,I3 trigger image_no red green blue

Work: E

mail

Work: O

ffice Contacts

Work: O

ffice Calendar

Work: O

ffice docu

ments

Home Office

: Financia

l

Work: L

egacy docu

ments

Media: P

hotos

Learn

ing: Books,

journals,

etc.

Health: P

HR

>>Health: D

iet & Exerci

se

Home Office

: Legal d

ocuments

Work: W

eb pages

>Life: c

hatbot

Life: M

emorabilia, e

phemera

Media: M

usic (C

Ds, ca

ssettes,

reco

rds)

Media: H

ome video (Film

and VHS)

Life: S

enseCam Days

>Work: C

onversations

>Work: T

&M (V

IBE)

>Life: T

racked Days

>Work: T

elephone conversa

tions

0

25

50

75

100

Current

NEED

Current Experiment NEED

MyLifeBits Project aka MemexInfluencing products and research

• Products & Research Efforts– DARPA LifeLog; DARPA Assist; DARPA CALO and SRI spinoff SIRI– EverNote as a OneNote competitor but focused on e-Memory, – reQall a voice reminder and memory (knows about time and space) – http://www.lifebio.com/ Capture your story for immortality; www.famento.com - your

family history– Stanford SALT project to Archive the faculty– IBM Pensieve (IBM Research Personal Memory Organizer) – Korean and Japanese Research Insitutes both Dedicated to MyLifeBits– UK Memories for Life Project… – Caprica TV prequel to Battleship Galactica – Aimee Baldridge, Organize Your Digital Life: How to Store Your Photographs, Music,

Videos, and Personal Documents in a Digital World– MSR Global Heritage Project

• Conferences: – ACM Workshop on Continuous Archival and Retrieval of Personal Experiences (CARPE)– Society for Imaging Science &Technology Conference on Archiving, April 2004– First Digital Lives Conference, British Libraries, Feb 2009– National Conference on Archive, West Midlands, UK, November 2009– Saving Our Present for the Future: Personal Archiving 2010, archive.org, Feb 2010– Cultural Heritage Track for ACM Multimedia

A couple of recent Google slides…Comment by Marc Benioff

“Enterprise software had traditionally been built by engineers who worked in corporate office parks with little or no interaction with the customers until a pre-determined “five-year plan” was complete. This isolationist existence has seriously stymied innovation in the technology industry. Companies were not moving fast enough for customers.”-Marc Benioff, Behind The Cloud

Cloud-client computing: research role?

• The PC is a less relevant environment in the face of the cloud and especially IF/When office apps go there.

• Many users would be happy to have apps run in the cloud and have their data there permanently…

• CS Research needs to be working and training students for the cloud environments

• Bing Product cycle: 4 months; Win/Office: >36 mos.– Idea to use is 9X faster;

a published idea ships immediately• R > AD …. > Product groups > Product (3-5 years at best)• Transfer is fundamentally flawed.

Product groups can just look at the competitors

Memex: A Supported and Active Research Platform for MSR’s Research*Ken Wood, Roger Lueder, Jim Gemmell, Mary Czerwinski, Gordon Bell

• Supported Platform created and based on: MyLifeBits, VIBE, SenseCam, Digital Geographics, and MLB University Research Program

• The narcissistic effect: “By having a common platform in use and built on, we can expect exponential increases in functionality through cross-app use and common schema, plus incremental improvements to performance and reliability. This is the nature of all such systems.”

• The Memex program has four components or goals: 1. continued research on the platform that includes validating WinFS

efficacy --especially for the PC for personal data, real time, and a person’s transactions;

2. research coming from Redmond, Cambridge, product efforts and universities;

3. a pre-WinFS platform for building and testing new database-based apps in a real use environment that allows us to”eat our dogfood”; and

4. live use that creates stability, understanding, and new research ideas, problems, and attraction for more apps. *Proposal 6/15/2005

End

The Agenda for the Lifetime e-Memory1. Guarantee that data will live forever! “dear appy” problem2. Data Aggregation from the myriad of data places: cloud, work, home, social sites3. More content: Cheap, easy, and data-rich (e.g. time, place) capture of:

GPS and time everywherePaper capture has to be as easy as discarding (scanner/shredder)Personal meeting capture...Telephony and audio capture with indexing– speech to textE-book…e-magazines & journals need to have critical mass! Media Center compatible for entertainment (photos, video, TV, radio)Connection to other sensor nets in home, auto, and elsewhere as they slowly come into existence

4. Annotations/meta-information add every-increasing valueEasy annotation for aiding search and it becomes the content

5. Automatic classification and organization into taxonomies, facets, etc.6. Content analysis (critical for photo & video!)7. Information control: privacy, security, expunge/deniability,… 8. Having to be schizophrenic or have a lobotomy when leaving a “life”9. One dbase for everything (articles, books, conversations, ... financial transactions) …vs.

long-term use of hierarchical files. Is dbase intuitive?10. The “killer apps”: Alzheimer, immortality, surrogate memory?11. Clean living, cloaking, versioning, structure to contacts (family, org charts)12. GUI’s to improve use (e.g. time to learn, use, retention)

HMI 2009: 1.3 Thr;3.6 Zby; 10.8 PwordsReport on US Consumers, Bohn and Short, UCSD

Per User / Per Day

Hours Megabytes KWordsTraditonal TV 29.2 82033 261Gaming incl. computer 3.9 189899 11TV, Games+Movies+DVD 33.3 275987 275Books, News, Magazines 1.1 6.6 16Radio & MP3 5.3 382 24Internet, offline programs 2.6 131 36Telephony: Pots, mobile 1.1 22.8 8

From HMI Report on Info Sources in the US