Upload
vladimir-kulyukin
View
874
Download
16
Embed Size (px)
Citation preview
Eyesight Sharing in Blind Grocery Shopping: Remote P2P Caregiving
through Cloud Computing
Vladimir Kulyukin, Tanwir Zaman Abhishek Andhavarapu, Aliasgar Kutiyanawala
Department of Computer ScienceUtah State University
Logan, UT, USA
Big Picture
Independent Shopping for Visually Impaired (VI) and Blind Individuals
Why Is Blind Shopping Difficult?A modern supermarket has a median area of 4 300 m2, stocks an average of 38 718 products, and services approximately 600 customers per hour (6 000 to 7 000 per day)
Hardware Evolution2003-2005
2006-2008
2008-2010
2010 -Now
RoboCart ShopTalk ShopMobile
ShopMobileComputer Vision + Camera Alignment
= Independent Eyes-Free Store Browsing on Smartphones
Parallel Threads of Computation
Barcode Scanning Camera Alignment
Automated Approaches: Problems & Alternatives
● Two problems with automated approaches:– Error-prone: false positives and false
negatives– Greedy power consumption
● Two alternatives: – Remote caregiving – Crowdsourcing
● Both can be used to augment automated approaches
Remote Caregiving vs. Crowdsourcing● Crowdsourcing may not be suitable for time
sensitive tasks and tasks that require mutual trust● Time sensitivity can be addressed through
increased helper volume (unlikely to materialize for smaller disabled populations)
● Trust may be a more serious issue for crowdsourcing
● Remote caregiving addresses trust but requires dedicated caregivers
Eyesight Sharing & Cloud Computing ● Cloud computing infrastructures and real-time
video streaming protocols make it possible for sighted individuals to share their sight with their VI friends remotely
● Hearing can also be shared remotely● Amount of caregiving can be dynamically adjusted
at run time
TeleShop
Blind Shopper
Caregiver
Caregiver
Wi-Fi/3G/4G
Blind Shopper
TeleShop Architecture
Caregiver Blind Shopper
A Cloud-Based Caregiving Architecture
Client Client
A Cloud-Based Caregiving Architecture
Client Client
Caregiver
A Cloud-Based Caregiving Architecture
Client
Amazon EC2Elastic Computing
Service
Client
Caregiver
Amazon EC2Elastic Computing
Service
A Cloud-Based Caregiving Architecture
Client
Amazon EC2Elastic Computing
Service
Client
Caregiver
Amazon EC2Elastic Computing
Service
Android C2DMCloud 2 Device
Messaging
A Cloud-Based Caregiving Architecture
Request Client
Amazon EC2Elastic Computing
Service
Client
Caregiver
Amazon EC2Elastic Computing
Service
Android C2DMCloud 2 Device
Messaging
A Cloud-Based Caregiving Architecture
Request
Notification
Client
Amazon EC2Elastic Computing
Service
Client
Caregiver
Amazon EC2Elastic Computing
Service
Android C2DMCloud 2 Device
Messaging
A Cloud-Based Caregiving Architecture
Request
Notification
Notification
Client
Amazon EC2Elastic Computing
Service
Client
Caregiver
Amazon EC2Elastic Computing
Service
Android C2DMCloud 2 Device
Messaging
A Cloud-Based Caregiving Architecture
Request
Notification
Notification
Help
Client
Amazon EC2Elastic Computing
Service
Client
Caregiver
Amazon EC2Elastic Computing
Service
Android C2DMCloud 2 Device
Messaging
A Cloud-Based Caregiving Architecture
Request
Notification
Notification
Help
Help
Client
Amazon EC2Elastic Computing
Service
Client
Caregiver
Amazon EC2Elastic Computing
Service
Android C2DMCloud 2 Device
Messaging
A Cloud-Based Caregiving Architecture
Request
Notification
Notification
Help
Help
Client
Amazon EC2Elastic Computing
Service
Client
Caregiver
Amazon EC2Elastic Computing
Service
Android C2DMCloud 2 Device
Messaging
Client
Client
Client
Client
Caregiver
Caregiver
Caregiver
Caregiver
Clients & Caregivers
● Clients send product images (video streams are possible but consume data plans fast)
● Caregivers look at images, speak product names (type if SR does not work), send text back to clients
● When caregivers cannot identify products from received images, they can request a new image
Cloud-Based Image Matching
● SURF ran in the cloud was used as a black box image matching algorithm
● SURF was used to speed up caregiver's product recognition
● SURF returned top n (n=5 in our case) images and caregiver would verify the correct product name, if it is in top n, or speak/type the correct name, if it is not
Three Experiments
● A laboratory study with a blindfolded subject● Two field studies with a blindfolded subject
at Fresh Market, a supermarket in Logan, UT● A field study with a blind subject was
completed at Fresh Market, a supermarket in Logan, UT, after the paper was accepted at ICCHP
Lab Study ● 20 products (boxes, bottles, cans)● SURF trained on 100 images (5 images per
product)● Blindfolded subject was in a lab; caregiver was in a
different room● Subject given one product at at time and asked to
recognize each● Subject and caregiver used Google Nexus One with
Android 2.3.3● Data link was Wi-Fi
Supermarket Experiments ● Setting was at Fresh Market, a supermarket in
Logan, UT● 45 products (boxes, bottles, cans) from 9 aisles● SURF was trained on 370 images● Blindfolded subject used Galaxy S2 with Android
2.3.6 was at Fresh Market● Caregiver used Google Nexus One with Android
2.3.6 was in a different building, a mile away from the supermarket
● Data line was 4G
Supermarket Experiments
● In experiment 1, subject was given 16 products by assistant, one product at a time
● In experiment 1, SURF was turned on● In experiment 2, subject was given 17 products by
assistant, one product at a time● In experiment 2, SURF was turned off, because of
poor performance in experiment 1
Experimental Results
Environment # Products Mean Time STD TOP 5 Mean SR SR Fails
Lab Study 16 40 .00021 8 1.1 0
Store 1 16 60 .00033 0 1.2 2
Store 2 17 60 .00081 0 1.1 3
Conclusions ● SR appears to be a viable option for
product naming and tagging on Android phones
● Poor SURF performance was probably due to our limited understanding of its parameters
● Basic trade-off: battery life vs. data plan consumption