AISN

http://www.cs.bgu.ac.il/~royif/AISN

Auditory Imaging for Sightless Navigation

ARD PresentationDecember, 2010

Academic Advisor:Dr. Yuval Elovici

Technical Advisor:Dr. Rami Puzis

Team Members:Yakir DahanRoyi Freifeld

Vitali Sepetnitsky

Project Team

2

Introduction

3

The Problem Domain

Most of our navigation in the everyday life heavily depends on visual feedback that we get from our environment

When the ability to see the surroundings is missing due to visual impairments, the ability to navigate is also damaged

4

Existing Solutions

Physical sense:› White Cane› Guide Dog

Sensory substitution:› Warning of obstacles (e.g.

Optical Radar)› Sonar-like images scanning (e.g. The

vOICe)

5

Sightless navigation by sensory

substitution

› Development of an application that allows a

person to navigate, relying primarily on the

sense of hearing

Integration with a spatial auditory

environment

Providing a flexible environment for future

research

Vision and Main Goals

6

Our SolutionA Combination of visual information processing and 3D sound creation and positioning: Taking a stream of frames from a

web-camera Processing the frames and retrieving

visual information relevant to the user Creating appropriate sounds according

the recognized information Performing an auditory spatialization of

the sound and informing the user about the locations of the detected information 7

Our Solution (cont.)

8

External Interfaces Hardware and Software

OpenCV

OpenAL

MATLAB engine library

9

System Users End Users

› Visually impaired (or even blind) people who use the system for the purpose of hearing their physical environment

Configuration Users› The system installation and initial tuning,

such as user profiles creation, will be done by configuration users having the ability to see the operations they perform

Researchers› Cognitive science researchers who wish

to conduct experiments regarding 3D sound

10

Functional Requirements

11

Functional Requirements Core Functionality For all users (especially the

researcher):1. Support several types of computer vision

and image processing algorithms for extraction of the following information:

Feature points (points of interest) Contours BLOBS (regions that are either darker or brighter

than the surrounding)

2. Provide a utility to add new implementations of the above algorithms according to a predefined API

3. Support specific configurability options for each algorithm type 12

Functional Requirements (cont.) Core Functionality (cont.) For all users:

4. Create appropriate sounds according to the following features:

Location Brightness Color

5. Support sound spatialization using OpenAL API implementations and HRTF datasets conforming with a predefined format

6. Allow to install new HRTF datasets and OpenAL implementations for improving the quality of sound localization and research purposes 13

Functional Requirements Operational Functionality For the configuration user:

1. Ability to install the system along with all the peripheral software and initial set of HRTF datasets

2. User profiles managing: Support creation of user profiles, which store

the system settings optimized to the user preferences

Support the ability to view the settings stored in a user profile

Support the ability to modify and delete profiles Supply a set of predefined (default) profiles

used for initial system configuring Ability to initialize the system according to a

given user profile and switch between profiles14

Functional Requirements Operational Functionality (cont.) For the blind user:

1. Support an extensive training mechanism for:

3D sound perception Environment understanding

2. Support the following training types: Visualizing random shapes Visualizing pre-defined image files Fully immersive use of the system by emphasis

of some feature

For the researcher: Support defining a training experiment task Support recording of the task results and

retrieve them later 15

Non FunctionalRequirements

16

Non Functional Requirements Performance Constraints (partial) Speed requirements:

1. Response time:The system will produce a 3D sound according to a frame taken by the camera within 0.1 seconds at most (we will strive to 0.03 seconds – 30 fps)

2. Training Speed:• A simple training in order to reach 50% accuracy of

recognition should take no more than 30 minutes for a blind user .

• A blind user should pass at least 80% of the accuracy tests after 2 days of extensive system usage.

• A regular user should pass at least 80% of the accuracy tests after 3 days of extensive system usage.

17

Portability requirements:1. Currently the system is designed to be

deployed on Microsoft Windows (XP / Vista / 7 and later) operating systems only

2. The system will be compatible with 32 / 64 bit machines having web-camera and audio drivers installed

Capacity requirements:1. The system should work on machines with

at least 1 GB of RAM2. The system will support many different

OpenAL implementations and HRTF datasets, the limit is the hard disk capacity only

Non Functional Requirements (cont.) Performance Constraints (cont.)

18

Non Functional Requirements (cont.) Look, Feel and Use Constraints

User Interface requirements:1. The UI should be easy to use even for

users that are not well familiar to the computers technology

2. User interface will be in English

Documentation and Help:1. A extensive documentation will be

supported along with an installation guide

2. Operations will be implemented as wizards

3. Error messages heard via headphones19

Non Functional Requirements (cont.) SE Project + Platform Constraints

1. The application core and the UI will be written in C++ language using .NET 3.5 Framework and Visual Studio 10.0 IDE.

2. MATLAB will be used as a computational engine

3. During the development stage of the system a home-made simple device will be used (a PC web-camera strapped to a top of headphones)

4. For the demo and testing purposes, a real device will be supplied by DT labs which are spy-sunglasses (sunglasses with a tiny camera hidden in the nose bridge of glasses) 20

Usage Scenarios

21

Usage Scenarios Use Cases Diagram

22

Usage Scenarios (cont.) Use Cases: UC-1: Visualize Environment

A blind user starts the visualization process

23

Usage Scenarios (cont.) Use Cases: UC-2: Train

A blind user performs a training process

24

Usage Scenarios (cont.) Use Cases: UC-3: Choose a user profile

A blind user chooses an existing user profile for the purpose of performing a training or in order to use the system

25

Usage Scenarios (cont.) Use Cases: UC-4: Visualize Image

The core of the visualization process

26

Usage Scenarios (cont.) UC-4: Visualize Image

27

QUESTIONS?THANK YOU

!

28