29
Eyesight Sharing in Blind Grocery Shopping: Remote P2P Caregiving through Cloud Computing Vladimir Kulyukin, Tanwir Zaman Abhishek Andhavarapu, Aliasgar Kutiyanawala Department of Computer Science Utah State University Logan, UT, USA

Eyesight Sharing in Blind Grocery Shopping: Remote P2P Caregiving through Cloud Computing

Embed Size (px)

Citation preview

Page 1: Eyesight Sharing in Blind Grocery Shopping: Remote P2P Caregiving through Cloud Computing

Eyesight Sharing in Blind Grocery Shopping: Remote P2P Caregiving

through Cloud Computing

Vladimir Kulyukin, Tanwir Zaman Abhishek Andhavarapu, Aliasgar Kutiyanawala

Department of Computer ScienceUtah State University

Logan, UT, USA

Page 2: Eyesight Sharing in Blind Grocery Shopping: Remote P2P Caregiving through Cloud Computing

Big Picture

Independent Shopping for Visually Impaired (VI) and Blind Individuals

Page 3: Eyesight Sharing in Blind Grocery Shopping: Remote P2P Caregiving through Cloud Computing

Why Is Blind Shopping Difficult?A modern supermarket has a median area of 4 300 m2, stocks an average of 38 718 products, and services approximately 600 customers per hour (6 000 to 7 000 per day)

Page 4: Eyesight Sharing in Blind Grocery Shopping: Remote P2P Caregiving through Cloud Computing

Hardware Evolution2003-2005

2006-2008

2008-2010

2010 -Now

RoboCart ShopTalk ShopMobile

Page 5: Eyesight Sharing in Blind Grocery Shopping: Remote P2P Caregiving through Cloud Computing

ShopMobileComputer Vision + Camera Alignment

= Independent Eyes-Free Store Browsing on Smartphones

Page 6: Eyesight Sharing in Blind Grocery Shopping: Remote P2P Caregiving through Cloud Computing

Parallel Threads of Computation

Barcode Scanning Camera Alignment

Page 7: Eyesight Sharing in Blind Grocery Shopping: Remote P2P Caregiving through Cloud Computing

Automated Approaches: Problems & Alternatives

● Two problems with automated approaches:– Error-prone: false positives and false

negatives– Greedy power consumption

● Two alternatives: – Remote caregiving – Crowdsourcing

● Both can be used to augment automated approaches

Page 8: Eyesight Sharing in Blind Grocery Shopping: Remote P2P Caregiving through Cloud Computing

Remote Caregiving vs. Crowdsourcing● Crowdsourcing may not be suitable for time

sensitive tasks and tasks that require mutual trust● Time sensitivity can be addressed through

increased helper volume (unlikely to materialize for smaller disabled populations)

● Trust may be a more serious issue for crowdsourcing

● Remote caregiving addresses trust but requires dedicated caregivers

Page 9: Eyesight Sharing in Blind Grocery Shopping: Remote P2P Caregiving through Cloud Computing

Eyesight Sharing & Cloud Computing ● Cloud computing infrastructures and real-time

video streaming protocols make it possible for sighted individuals to share their sight with their VI friends remotely

● Hearing can also be shared remotely● Amount of caregiving can be dynamically adjusted

at run time

Page 10: Eyesight Sharing in Blind Grocery Shopping: Remote P2P Caregiving through Cloud Computing

TeleShop

Blind Shopper

Caregiver

Caregiver

Wi-Fi/3G/4G

Blind Shopper

Page 11: Eyesight Sharing in Blind Grocery Shopping: Remote P2P Caregiving through Cloud Computing

TeleShop Architecture

Caregiver Blind Shopper

Page 12: Eyesight Sharing in Blind Grocery Shopping: Remote P2P Caregiving through Cloud Computing

A Cloud-Based Caregiving Architecture

Client Client

Page 13: Eyesight Sharing in Blind Grocery Shopping: Remote P2P Caregiving through Cloud Computing

A Cloud-Based Caregiving Architecture

Client Client

Caregiver

Page 14: Eyesight Sharing in Blind Grocery Shopping: Remote P2P Caregiving through Cloud Computing

A Cloud-Based Caregiving Architecture

Client

Amazon EC2Elastic Computing

Service

Client

Caregiver

Amazon EC2Elastic Computing

Service

Page 15: Eyesight Sharing in Blind Grocery Shopping: Remote P2P Caregiving through Cloud Computing

A Cloud-Based Caregiving Architecture

Client

Amazon EC2Elastic Computing

Service

Client

Caregiver

Amazon EC2Elastic Computing

Service

Android C2DMCloud 2 Device

Messaging

Page 16: Eyesight Sharing in Blind Grocery Shopping: Remote P2P Caregiving through Cloud Computing

A Cloud-Based Caregiving Architecture

Request Client

Amazon EC2Elastic Computing

Service

Client

Caregiver

Amazon EC2Elastic Computing

Service

Android C2DMCloud 2 Device

Messaging

Page 17: Eyesight Sharing in Blind Grocery Shopping: Remote P2P Caregiving through Cloud Computing

A Cloud-Based Caregiving Architecture

Request

Notification

Client

Amazon EC2Elastic Computing

Service

Client

Caregiver

Amazon EC2Elastic Computing

Service

Android C2DMCloud 2 Device

Messaging

Page 18: Eyesight Sharing in Blind Grocery Shopping: Remote P2P Caregiving through Cloud Computing

A Cloud-Based Caregiving Architecture

Request

Notification

Notification

Client

Amazon EC2Elastic Computing

Service

Client

Caregiver

Amazon EC2Elastic Computing

Service

Android C2DMCloud 2 Device

Messaging

Page 19: Eyesight Sharing in Blind Grocery Shopping: Remote P2P Caregiving through Cloud Computing

A Cloud-Based Caregiving Architecture

Request

Notification

Notification

Help

Client

Amazon EC2Elastic Computing

Service

Client

Caregiver

Amazon EC2Elastic Computing

Service

Android C2DMCloud 2 Device

Messaging

Page 20: Eyesight Sharing in Blind Grocery Shopping: Remote P2P Caregiving through Cloud Computing

A Cloud-Based Caregiving Architecture

Request

Notification

Notification

Help

Help

Client

Amazon EC2Elastic Computing

Service

Client

Caregiver

Amazon EC2Elastic Computing

Service

Android C2DMCloud 2 Device

Messaging

Page 21: Eyesight Sharing in Blind Grocery Shopping: Remote P2P Caregiving through Cloud Computing

A Cloud-Based Caregiving Architecture

Request

Notification

Notification

Help

Help

Client

Amazon EC2Elastic Computing

Service

Client

Caregiver

Amazon EC2Elastic Computing

Service

Android C2DMCloud 2 Device

Messaging

Client

Client

Client

Client

Caregiver

Caregiver

Caregiver

Caregiver

Page 22: Eyesight Sharing in Blind Grocery Shopping: Remote P2P Caregiving through Cloud Computing

Clients & Caregivers

● Clients send product images (video streams are possible but consume data plans fast)

● Caregivers look at images, speak product names (type if SR does not work), send text back to clients

● When caregivers cannot identify products from received images, they can request a new image

Page 23: Eyesight Sharing in Blind Grocery Shopping: Remote P2P Caregiving through Cloud Computing

Cloud-Based Image Matching

● SURF ran in the cloud was used as a black box image matching algorithm

● SURF was used to speed up caregiver's product recognition

● SURF returned top n (n=5 in our case) images and caregiver would verify the correct product name, if it is in top n, or speak/type the correct name, if it is not

Page 24: Eyesight Sharing in Blind Grocery Shopping: Remote P2P Caregiving through Cloud Computing

Three Experiments

● A laboratory study with a blindfolded subject● Two field studies with a blindfolded subject

at Fresh Market, a supermarket in Logan, UT● A field study with a blind subject was

completed at Fresh Market, a supermarket in Logan, UT, after the paper was accepted at ICCHP

Page 25: Eyesight Sharing in Blind Grocery Shopping: Remote P2P Caregiving through Cloud Computing

Lab Study ● 20 products (boxes, bottles, cans)● SURF trained on 100 images (5 images per

product)● Blindfolded subject was in a lab; caregiver was in a

different room● Subject given one product at at time and asked to

recognize each● Subject and caregiver used Google Nexus One with

Android 2.3.3● Data link was Wi-Fi

Page 26: Eyesight Sharing in Blind Grocery Shopping: Remote P2P Caregiving through Cloud Computing

Supermarket Experiments ● Setting was at Fresh Market, a supermarket in

Logan, UT● 45 products (boxes, bottles, cans) from 9 aisles● SURF was trained on 370 images● Blindfolded subject used Galaxy S2 with Android

2.3.6 was at Fresh Market● Caregiver used Google Nexus One with Android

2.3.6 was in a different building, a mile away from the supermarket

● Data line was 4G

Page 27: Eyesight Sharing in Blind Grocery Shopping: Remote P2P Caregiving through Cloud Computing

Supermarket Experiments

● In experiment 1, subject was given 16 products by assistant, one product at a time

● In experiment 1, SURF was turned on● In experiment 2, subject was given 17 products by

assistant, one product at a time● In experiment 2, SURF was turned off, because of

poor performance in experiment 1

Page 28: Eyesight Sharing in Blind Grocery Shopping: Remote P2P Caregiving through Cloud Computing

Experimental Results

Environment # Products Mean Time STD TOP 5 Mean SR SR Fails

Lab Study 16 40 .00021 8 1.1 0

Store 1 16 60 .00033 0 1.2 2

Store 2 17 60 .00081 0 1.1 3

Page 29: Eyesight Sharing in Blind Grocery Shopping: Remote P2P Caregiving through Cloud Computing

Conclusions ● SR appears to be a viable option for

product naming and tagging on Android phones

● Poor SURF performance was probably due to our limited understanding of its parameters

● Basic trade-off: battery life vs. data plan consumption