ApplianceReader: A Wearable, Crowdsourced, Vision-based System to Make Appliances AccessibleAnhong Guo, Xiang ‘Anthony’ Chen, Jeffrey BighamHuman-Computer Interaction Institute, Carnegie Mellon University

INTRODUCTION

Visual impairment affects almost every activity of daily life, such as getting to where you want to go, reading information and recognizing objects around you, and participating in social interactions with other people. Even simple and essential tasks like heating up food with the microwave oven is difficult when the functions of the buttons cannot be read and are not tactually differentiable. Our initial user research obtained several key insights that inform our subsequent system design. We are developing a system that leverages the increasing availability of wearable devices and brings together the strengths of computer vision and crowdsourcing to complementarily and collaboratively improve the accessibility of home appliances.

SYSTEM DESIGN

Crowdsourced LabelingWhen a visually impaired person encounters an appliance for the first time, his smart eyewear captures a picture of the device and sends it to the crowd. This picture then becomes a reference image: crowd workers will visually label the overall layout and the control panel of the appliance, and annotate the specific functions of each control button.

Recognition and ControlLater, when the visually impaired person tries to use the appliance, he simply turns on the smart eyewear camera, looks at the control and hovers a finger over it. Combining computer vision techniques and the reference image, our algorithm detects which control the user intends to use and its functionality. With audio feedback, the user can continuously search and locate buttons.

Coordination and CombinationWhen image quality or orientation affects the recognition, the captured image will be sent to the crowd to manually inform the user what the particular control is. Such crowd-labeled images will then be collected by the system and added to a library of reference images to increase the robustness of future recognition. When geo-tagged, the images can also be used to locate the nearest appliance.

Less Tactile Feedback Social Burden When Braille Fails

Reheat  

Ask  crowd?  

“Google Glass” symbol by Luis Prado, “Braille” symbol by Jakob Vogel, “Picture” symbol by B. Agustín Amenábar Larraín, “Question” symbol by Jardson A., “Tag” symbol by Vassilis Terzopoulos, “Arrow” symbol by Juan Garces, “Microwave Oven” symbol by Darrin Higgins, Microwave” symbol by Hedie Assadi Joulaee, “Idea” symbol by Stefano Vetere, “Crowd” symbol by Alex Kwa, “Quote” symbol by Rohith M S, “Timeline” symbol by Qing Li, “Map” symbol by Simple Icons, “Touch” symbol by Jacob Lowe, “Button” symbol by Dan Hetteix, "Social Network" symbol by Matthew Hawdon, "Right" by José Campos from thenounproject.com collection.

Designed by Yueying Tang and Anhong Guo.

Input    Photo  

Labeled    Image  

Reference    Images  

Geo-­‐tagged  Images