View
221
Download
2
Category
Preview:
Citation preview
Design and Implementation of SEE-Phone in
SEES (Smart Environment Explorer Stick)
M. Yusro1,3, K.M. Hou1, E. Pissaloux2, K. Ramli3, D. Sudiana3, Z. Lizhong1 and H.L. Shi1 1Laboratoire d'Informatique de Modélisation et d'Optimisation des Systèmes, Université Blaise
Pascal, France, yusro@isima.fr, kun-mean.hou@isima.fr and hongling.shi@isima.fr 2Institut des Systèmes Intelligents et de Robotique, Université Pierre et Marie Curie
(UPMC, PARIS 6, Paris-Sorbonne), France, Edwige.Pissaloux@upmc.fr 3Department of Electrical Engineering, Faculty of Engineering, Universitas Indonesia, Indonesia,
k.ramli@ui.ac.id and dodi.sudiana@ui.ac.id
Abstract – The mobility is a complex, yet not well
understood, human cognitive process; it subtends several
different tasks such as walking and orientation. The visually
impaired people (VIP) mobility heavily depends on the white
stick (or white cane) which however does not provide an
efficient support for walking and orientation. Indeed, these
two later functions implementation requires the usage of high
technology which should be conveniently integrated to the
white stick in order to obtain the smart stick. This paper
proposes to fill in this gap, and outlines the SEES (Smart
Environment Explorer Stick). This paper focuses on SEE-
phone as a part from whole SEES system. The first prototype
of SEE-phone, a part of the SEES system, is presented.
Keywords: smart phone, GPS, Google map, web service,
SEE-Phone, smart-stick, ubiquitous SEES, Visually Impaired
People (VIP), assistive device
I. INTRODUCTION
Blindness and visual impairment are a major hindrance
in daily life for accessing information, mobility, finding a
way, interaction with the environment, communities and
the others activities. Both of them became a very serious
problem in health and social security. In addition, the other
important issue for VIP is independent ability to move
outdoor. Usually, they are able to move independently
only along the routes which they have already learned
together with a sighted guide [1]. On the other hand, they
have a strong need to walk around by themselves, even
when they are in new location or accidentally lost their
way.
In general, navigation problems in the blind population
have risen regarding mobility and orientation [2-3].
Mobility is the ability to travel safely, comfortably,
gracefully, and independently [2]. Mobility of the visually
impaired can also be interpreted as an activity that
combines earlier defined capabilities (space perception,
orientation, way-finding, navigation and obstacle
avoidance). The human autonomous mobility is a
capability to reach a place without assistance of any other
person.
The below definition raises several problems which
should be cognitively solved [4-5]; two of them are self
orientation and space self awareness. The concept of self
orientation means the knowledge of the relative spatial
position between our current position and targeted spatial
location. Two basic questions which subtend the
orientation are: “where I am?” and “how to reach the
destination from my current position?”. Consequently, the
orientation allows to planning a specific route (path) to
reach the targeted location from the current point.
Space awareness is the capability to know about urban
and social data in our peri-personal space (“space around
our body”), and our navigational space [6]. Some
questions which subtend the space awareness might be:
“what are the nearest streets located one with respect to
the other and to me ? what is the traffic light at the street I
like to cross ? where is the library I like to get in ? “
This paper proposes a support for orientation and space
awareness using the high level technologies, which should
be integrated in the VIP mobility assistive device. More
precisely, the SEES system (or ‘Smart Environment
Explorer Stick’) is defined. The SEES system integrates: a
global remote server (iSEE), an embedded local server
(SEE-phone) and a smart stick (SEE-stick). In this paper,
we focus on SEE-phone and its useful features such as
GPS, camera, TTS (Text-to-Speech), voice command, etc.
Therefore, the paper is organized as follows: Section 2
provides a state-of-the-art on the existing commercial and
academic solutions for VIP navigation and focus on their
supports of orientation and space awareness concepts.
Section 3 defines the SEES concept and outlines its
implementation in the first prototype. Section 4 proposes
the SEE-Phone concept and design. Section 5 addresses
the results of the first SEE-Phone prototype experimental
evaluation. The final section 6 indicates some future works
toward the fully operational of SEES prototype.
II. STATE OF THE ART
Two complementary tracks are investigated for
navigation assistance for VIPs design are GPS-based and
vision-based solutions.
A. GPS-Based Navigation Technology for VIP
Since 1960 various assistive devices for the navigation
system dedicated to VIP have been developed, ranging
from the low to high technology devices. In 1991,
Golledge and team were persons who first proposed for
using of GIS, GPS, speech and the sound sensor (sonic
sensors) to navigate the blind [7]. A sample of navigation
system for VIP is Personal Guidance System (PGS) and an
early model was developed in 1993 [8].
In general, it is possible to materialize the concept of a
navigation system for VIP as a set of the following
modules (figure 1): a) GPS for determination the
traveller’s orientation and position, b) Geographic
Information System (GIS), a software for route planning,
c) The spatial database for relating the traveller’s
orientation and GPS coordinates to the surrounding
environment and d) the user interface.
Fig. 1. Functional Components of PGS System [8]
The system usually provides a range of functions which
generally contains simulation of a journey. It displays the
route (path), searching in the database from a starting point
to targeted (end) point. The provided in real-time audio
instructions enable users to follow the path.
The PGS system above consists of three modules [9].
The first module is used for user location and orientation.
A GPS receiver with differential GPS receiver (DGPS) is
used to determine location and to trail the path of travel.
The DGPS configuration also gives absolute positional
accuracy on the order of 1m root-mean-squared error
given good satellite availability. The second module is the
GIS components that consist of a digitized base map and
software design to trail the traveller’s path, select routes,
recommend the traveller about local features and
landmarks, correct for signal error or loss, and control
dynamic access to the database. The module also contains
the commands needed to navigation aid. The spatial
attributes might concern some criterion of system
centrality, feature or population density, connectivity,
nearest neighbours, distances to other features and etc. The
third module is the user interfaces that provides for two-
way communication between the GIS and the user. In this
part, there are two display options for GIS to user
communication, namely a virtual acoustic display using
binaural earphones and conventional speech display with
an earphone or speaker.
There are some devices of GPS based navigation
systems has been designed or modified, namely: Trekker
of Human Ware, BrailleNote GPS, MoBIC, Loadstone
GPS, Wayfinder Access and Mobile Geo [10]. The system
provides a range of functions which generally contains
simulation. It could display the route on the way, the
search of database from a certain point and real-time
instruction (audio) which enables users to follow the track
to a specific destination.
Trekker of Human Ware is a personal digital assistant
(PDA) or pocket PC adapted for the blind and visually
impaired with talking menus, talking maps and GPS
information. This device has a good functional
possibilities and built-in TTS (Text to Speech). It can
expand to accommodate new hardware platforms and
more detailed geographic information. This device does
not provide a route remote tracking for monitoring the user
location and it cannot detect the obstacle.
BrailleNote GPS is a portable tool with speech and
braille output. It is like a combination of a personal digital
assistant, map-quest software and a mechanical voice. A
BrailleNote has a GPS receiver, maps, and points of
interest database that provide spoken and/or braille access
to location information in any outdoor environment. This
device does not have a route remote tracking for
monitoring the user location and it cannot detect the
obstacle.
MoBIC (Mobility for Blind and elderly people
Interacting with Computers) is designed to allow a blind
person access to information from many sources. The
resulting system has two parts: the pre-journey system,
and the outdoor system. The output system is in the form
of spoken messages. The planning system helps blind
people to study and plan their routes in advance, indoors.
This device does not provide a route remote tracking for
monitoring the user position and it does not have function
to the obstacles detection.
The Loadstone GPS project develops open source
software for satellite navigation for blind and visually
impaired people. The device uses a smart phone, operated
with screen reader, limited possibilities and unstable
operation. Like with the previously device, the Loadstone
GPS does not provide a route remote tracking for
monitoring the user location and it cannot be used to
detect the obstacle.
Wayfinder Access is a navigation system for mobile
phones specially developed for visually impaired. This
application is only for symbian phones, operated by screen
reader and requiring a GPS maps. This device does not
have a route remote tracking for monitoring the user
location and it cannot detect the obstacle.
Mobile Geo is a product designed to convey most of the
information displayed on commercial GPS receivers and
location databases to people with visual disabilities. This
device is used only for smart phones, PDA or Pocket PC
with MS Window Mobile. It is operated with screen reader
and requires GPS maps.
In general as conclusion, several systems above do not
provide assistance for VIP space awareness for
independent navigation. Those systems also do not have a
route remote tracking for monitoring the user location and
do not have the ability for detecting an obstacle (e.g. pole,
traffic lights, building, etc.).
B. Navigation System Based on Vision System for VIP
It is important in independent mobility to provide an
ability to detect a physical obstacle. The ability to know
the presence of a physical object completely can be
performed by vision system. Some of researches on
navigation systems for VIP have been developed the new
devices by combining the orientation function (GPS and
GIS) and vision system. The vision-based mobility
systems an image is captured then conveyed to the VIP via
different sensory channels such as sound or touch.
Concept of using video camera as vision sensors was
introduced through portables systems by Peter Meijer,
namely vOICE. The vOICE system [11
mounted single video camera and translates the image into
sounds (headset). The scene in front of the user is
via head movements, from up to down, and
data are cognitively integrated and interpreted
user. The system does not provide nor orientation neither
space awareness assistance to the VIP.
The NAVIG system [12] attempts to combine
artificial vision and visual geo-located landmark
allow completion of the path using the GPS
user current position. The NAVIG system has
modules: the objects location is used to allocate the
necessary objects for the user and the u
function is used to detect visual objects that are not
displayed to the user, but it is used to improve GPS
positioning. The system used Spike-Net NAVIG
that provides a fast algorithm based on visual research
human.
Jie Xu, et.al [13] proposed a system -
which integrates GPS and vision on a PDA devic
system is made to provide contextual cues for
navigate safely in dynamic situations. The
captures the dynamic information environment and
information landmarks along the route of the map data
using a Visual Studio eSuperMAp.
Do-Hoon Kim and Heung-Gyoon Ryu [
navigation system for VIP which combines the
camera and ultrasonic sensors. This system gets
information about the environment using GPS and
distance data using ultrasonic sensors, and then inform the
user. The camera is used to assess the situatio
users as a color. In the case of short-range obstacles,
vibrators transmit sound and vibration to the user. This
system used a bluetooth communication for tra
data.
Conclusions from several systems above are
systems have combined the orientation function and vision
function, the systems use camera and ultrasonic to detect
an obstacles, but the systems do not provide
monitoring for tracing the user location.
III. THE SEES SYSTEM
A. SEES Concept.
The SEES system aims to provide assistance for VIP
mobility. This paper addresses the SEES support for VIP
orientation and space awareness as defined in Section 1.
The main innovation of SEES is internet connection
which enables the VIP to get help and be
monitored [15], therefore it is possible to consider
ubiquitous smart stick.
The SEES system (figure 2) contains three main
components: a global remote server (iSEE), an embedded
local server (SEE-phone) and a smart stick (
stick). iSEE is a global server providing the web services
for the VIP such as remote real-time hint and help and
remote monitoring (trace the VIP location).
The SEE-phone is based on a commercial smart phone.
It is used as an embedded local server and provides the
local services for the SEE stick such as route vector and
video camera as vision sensors was
introduced through portables systems by Peter Meijer,
] uses a head
translates the image into
sounds (headset). The scene in front of the user is scanned
and all auditory
rpreted by the end
The system does not provide nor orientation neither
to combine the
located landmarks that will
the GPS provided end-
The NAVIG system has two main
is used to allocate the
ser positioning
function is used to detect visual objects that are not
sed to improve GPS
NAVIG library
that provides a fast algorithm based on visual research
- audio guide -
a PDA device. The
contextual cues for VIP to
The PDA camera
dynamic information environment and
along the route of the map database
Gyoon Ryu [14] proposed a
navigation system for VIP which combines the GPS,
. This system gets
information about the environment using GPS and
d then inform the
. The camera is used to assess the situation in front of
range obstacles,
transmit sound and vibration to the user. This
system used a bluetooth communication for transmitting
several systems above are these
systems have combined the orientation function and vision
the systems use camera and ultrasonic to detect
an obstacles, but the systems do not provide a route remote
The SEES system aims to provide assistance for VIP
mobility. This paper addresses the SEES support for VIP
orientation and space awareness as defined in Section 1.
The main innovation of SEES is internet connection
t help and be internet
t is possible to consider it as an
contains three main
components: a global remote server (iSEE), an embedded
phone) and a smart stick (wheeled SEE-
stick). iSEE is a global server providing the web services
time hint and help and
remote monitoring (trace the VIP location).
phone is based on a commercial smart phone.
rver and provides the
local services for the SEE stick such as route vector and
internet access to iSEE. SEE-phone is always connected to
SEE-stick through Wi-Fi.
Each sub-system provides independently
walking and to orientation of the VIP
data for walking, while SEE-phone is the key device for
orientation.
The system provides two channel feedback to the VIP
tactile (point-wise) and audio. The
elaborated by fusing several working in parallel
Fig. 2. Concept of SEES System
Notice that 6LoWPAN (IPv6 over Low Power Personal
Area Network and) and RPL routing protocol are adopted
to implement the SEE-stick thus according to the context
the SEE-stick can connect P2P with the iSEE and
transportation system [16-17]. Therefore,
can be considered as the implementation of ITS
‘Intelligent Transportation System’ concept
this system can be used in the near future
urban transportation system such as car, bus and train.
The mobility cues collected by SEES sensors will be
transformed into high level knowledge (passed to the
via voice or tactile feedbacks) in order to: 1) estimate or
predict the status on an object (for example:
traffic light); 2) obtain the accurate location data (what
allows to track the user and check if (s
correct way); 3) obtain a correct
environment conditions such as obstacles, status of the
traffic light and status of the walking surface; and 4) send
quickly error/alerts messages to VIP
mistake occurs.
B. Considered Mobility Cues and SEES
SEES targets to assist some walking
functions. The considered walking sub
following: obstacle detection, distance to obstacle
estimation, obstacle shape recognition,
estimation, surface roughness estimation
(position and status) detection.
The considered orientation sub-functions are
estimation to the targeted location and end
location/position estimation.
The assisted functions have a direct impact on selected
sensors. The SEES available sensors are:
sensor, camera, wheel encoder, accelerometer
compass.
phone is always connected to
independently assistance, to
of the VIP. SEE-stick collects
phone is the key device for
two channel feedback to the VIP:
These feedbacks are
working in parallel sensors.
Concept of SEES System
IPv6 over Low Power Personal
Area Network and) and RPL routing protocol are adopted
stick thus according to the context
stick can connect P2P with the iSEE and future
Therefore, the SEES system
implementation of ITS
concept. It means that
in the near future by VIP to access
em such as car, bus and train.
SEES sensors will be
transformed into high level knowledge (passed to the VIP
via voice or tactile feedbacks) in order to: 1) estimate or
tatus on an object (for example: status of the
traffic light); 2) obtain the accurate location data (what
allows to track the user and check if (s)he is on his/her
information about
environment conditions such as obstacles, status of the
the walking surface; and 4) send
VIP when orientation
SEES Sensors.
walking and orientation
sub-functions are the
distance to obstacle
obstacle shape recognition, walked distance
estimation, surface roughness estimation and traffic light
functions are: direction
and end-user current
a direct impact on selected
SEES available sensors are: ultrasonic
, camera, wheel encoder, accelerometer and
The ultrasonic sensor is used to detect the obstacles
front of the VIP in very near distance (such as
etc.) with provide the precision.
The camera in SEE-stick is used to identify and
recognize the type of road junctions (e.g.
corner-road, side-road, etc.). In addition, the camera is also
used to read the environmental conditions,
illumination (day and night) of traffic light status
Status of the traffic lights (figure 3) is detected
the smart phone camera (in SEE-phone). The
function dedicated to detect traffic lights status
VIP gets the information from SEE-stick about
lights position.
Fig. 3. The Traffic Lights Model
The wheel encoder sensor is mounted on
wheels, and is used to estimate VIP travel distance
the starting place.
In association with camera information,
data is used to estimate walking surface roughness
the special road pad for VIP (figure 4).
Fig. 4. The Special Road Pad for Blind People
The compass sensor is used to detect a moving direction
of the VIP. Data from the compass will be integrated with
the GPS data and wheel encoder data to enhance the
precision of VIP location and distance estimation
The orientation function will use two main
sensors: camera and GPS (includes a Google
The GPS is used to define the current location
VIP, while the Google map allows to plans the user
journey in order to reach his/her spatial target.
the GPS data will be sent to the database server for
monitoring the VIP, displaying and checking
itinerary in real-time.
Table 1 summarizes the different sensor
mobility functions supported by the SEES system
the obstacles in
(such as tree, wall,
to identify and
(e.g. crossroad,
In addition, the camera is also
used to read the environmental conditions, such as
of traffic light status.
is detected also with
phone). The camera has a
status once the
stick about traffic
The Traffic Lights Model
mounted on the SEE-stick
VIP travel distance from
In association with camera information, accelerometer
walking surface roughness, such as
The Special Road Pad for Blind People
moving direction
compass will be integrated with
the GPS data and wheel encoder data to enhance the
location and distance estimation.
will use two main SEE-phone
Google map).
the current location of the
map allows to plans the user
spatial target. Periodically
server for remote
, displaying and checking his/her
sensors and the
the SEES system.
TABLE 1: THE TYPE AND FUNCTION OF SENSORS
Name of Sensor
SEES System
SEE-
stick
SEE-
phone
Ultrasonic
Wheel Encoder
Accelerometer Compass
Camera
GPS
V
V
V V
V
V
V
V
V
Data collected from different SEES sensors will
combined as follows in order to provide more reliable
outputs (mobility cues): 1) Location data from GPS
phone and SEE-stick will be combined
current accurate location. Location
with data from encoder sensor (distance measurement); 2)
Image data (color and obstacle) from camera
and SEE-stick will be combined in accurate
recognition and object detection
accelerometer data will also be fused
information; 3) Direction data from SEE
stick (compass sensor) will be combined
direction. In case of disagreement a message may be sen
to get help from the iSEE global server; and
data from ultrasonic sensor, distance data
sensor, and surface roughness from accelerometer sensor
will be used as raw data for further processing
The mobility cues will be the inputs to the SEES layer
which will transform them into high level knowledge
useful for mobility; this knowledge which can be
conveyed to the VIP. Figure 5 shows the model of
interconnection in SEES system.
Fig. 5. Sensors Interconnection Model
C. SEES Running Models (Yusro 2013)
SEES is a modular system at the physical level
transmission between the SEE-phone
performed via Wi-Fi protocol. Table
SEES modes. Each mode can be selected
(s)he will go to a targeted location.
TABLE 2: THE SEES MODES
Mode SEES System
SEE-stick SEE-phone iSEE
Mode 0
Mode 1
Mode 2
Mode 3
V
V
V
V
V
V
V
V
THE TYPE AND FUNCTION OF SENSORS
Output Data
Obstacle/object
Travel distance
Surface roughness Direction
Color/obstacle
Location/position
collected from different SEES sensors will be
as follows in order to provide more reliable
Location data from GPS SEE-
combined into the end-user
data communicates
r sensor (distance measurement); 2)
) from camera SEE-phone
accurate result color
recognition and object detection. Ultrasonic and
be fused with camera
SEE-phone and SEE-
s sensor) will be combined into an accurate
In case of disagreement a message may be sent
elp from the iSEE global server; and 4) Obstacle
, distance data from encoder
from accelerometer sensor
further processing.
The mobility cues will be the inputs to the SEES layer
nto high level knowledge
which can be directly
shows the model of sensors
Model in SEES System
(Yusro 2013).
at the physical level. Data
phone and SEE-stick are
Table 2 shows all detailed
selected by the VIP when
THE SEES MODES
Description iSEE
V
V
Basic mode
Phone mode
Local mode con
Complete mode
The SEES interface to VIP offers four assistance
1) Mode 0 (always active, SEES minimal
only the SEE-stick is active). This mode will
the VIP to walk without a smart phone.
2) Mode 1 or phone mode (the SEES work
SEE-phone and iSEE). This mode will allow
stay moving without a smart stick. The navigation
be done using smart phone sensors (GPS, camera and
compass) and iSEE server.
3) Mode 2 or local mode connexion. In this mode
VIP can get a remote help from person
the server as the SEES provide the
specific hints and helps;
4) Mode 3 or the complete mode. In this mode, the VIP
can get a remote help as the SEES provide
selected travel specific hints and helps.
IV. SEE-PHONE
The SEE-Phone is a smart phone that be
two main tasks are orientation and walking
true independent mobility. SEE-phone is the key device
for orientation; indeed, the SEE-phone communicates with
the GPS, through the web server accessing to the map
database and with the others mobile devices.
A. SEE-Phone as Orientation Function
As a function of orientation, the SEE-phone is used to
give direction to destination location, create a new route or
guide the way of the route that has been created previously
(route remote tracking). To implement the navigation
function, SEE-phone uses GPS on smart phone.
In general, the GPS on smart phone
(World Geodetic System 1984) as a reference system.
WGS84 regards the earth as a spheroid object is
determined based on the satellites observation satellites in
earth orbit. The WGS84 major axis is 6,378,137.0 meters
and a minor axis is 6,356,752.3 meters [18].
The tracking of a VIP during his/her travel can be
decomposed in the two following elementary operations:
1) Detection the VIP’s current position and
2) VIP’s current position real-time tracking with
and Google map.
The implementation of these tasks have
follows: 1) to develop an application on Android
phone for detecting the VIP location; 2) to develop an
application for tracking the VIP position in real time with
GPS; 3) to connect the developed application to the server
in the others place (campus), 4) to monitor VIP position
from map display on monitor PC/notebook server; and 5)
to display the VIP position by server.
Figure 6 shows the access model from GPS receiver (in
smart phone) to database server. The VIP current location
data is obtained from smart phone GPS and sent to the
web server to be stored in database server. The last, the
data can be shown on the monitor as line track/route.
assistance levels:
, SEES minimal configuration,
his mode will allow
.
the SEES work by using the
his mode will allow a user to
stick. The navigation will
(GPS, camera and
In this mode, the
persons who monitor
selected travel
or the complete mode. In this mode, the VIP
the SEES provides the
selected travel specific hints and helps.
that be functioned to
walking functions for a
phone is the key device
phone communicates with
the GPS, through the web server accessing to the map
others mobile devices.
phone is used to
give direction to destination location, create a new route or
been created previously
To implement the navigation
phone.
uses WGS84
) as a reference system.
WGS84 regards the earth as a spheroid object is
determined based on the satellites observation satellites in
earth orbit. The WGS84 major axis is 6,378,137.0 meters
].
The tracking of a VIP during his/her travel can be
ing elementary operations:
the VIP’s current position and
time tracking with GPS
s have goals as
: 1) to develop an application on Android smart
or detecting the VIP location; 2) to develop an
application for tracking the VIP position in real time with
GPS; 3) to connect the developed application to the server
in the others place (campus), 4) to monitor VIP position
otebook server; and 5)
shows the access model from GPS receiver (in
) to database server. The VIP current location
GPS and sent to the
database server. The last, the
data can be shown on the monitor as line track/route.
Fig. 6. The Global Concept of SEE
B. SEE-Phone for Color Detection
The implementation of the traffic light status detection
with SEE-phone requires the followin
operations:
1) Traffic lights detection,
2) Traffic light (color) status recognition
3) Status data feedback to the VIP
Figure 7 shows a possible SEES
namely: the camera on smart phone will detect
lights (through two ways detection models, namely:
traffic lights position is detected by the camera sensor
the stick (SEE-stick) and the traffic
detected based on GIS data in databa
the image data from the traffic lights status; the Android
smart phone will process the image; finally, the
phone will produce the audio/voice output to VIP.
Fig. 7. SEE-Phone for Traffic Lights
The whole process on Android system
steps (figure 8): image acquisition, image pre
color filtering and thresholding.
Fig. 8. The Image Processing on Android
The thresholding function will return a binary
which can be easily “translated” into audio data for the
VIP.
C. User Interfaces on SEE-Phone
Navigation guidance using verbal description and
instruction is considered as efficient way
VIPs [19]. Two smart phone applications will be
TTS and STT.
The mobile TTS (Text-to-Speech)
smart phone screen message reading and
for the VIP [20]. Figure 9 shows an overview of a typical
TTS System.
. The Global Concept of SEE-Phone
of the traffic light status detection
requires the following elementary
(color) status recognition; and
VIP.
a possible SEES implementation,
will detect the traffic
detection models, namely: the
detected by the camera sensor in
traffic lights position is
database server), and take
the image data from the traffic lights status; the Android
will process the image; finally, the smart
output to VIP.
Lights Detection
on Android system requires four
image acquisition, image pre-processing,
. The Image Processing on Android
return a binary image
which can be easily “translated” into audio data for the
Navigation guidance using verbal description and
efficient way to pass data to
phone applications will be used:
) is an application for
and audio translation
shows an overview of a typical
Fig. 9. Overview of a typical TTS System
The speech recognition application (STT,
Text) allows a machine or program to receive and interpret
dictation, or to understand and carry out spoken
commands [20]. In practice, the size of a voice
program's effective vocabulary is directly related to the
random access memory capacity of the computer in which
it is installed.
In SEE-Phone, the instruction inputs or commands will
be given in voice. The VIP will speak to run a specific
application with his/her voice (STA, Speech
The SEE-phone output also produces the voice or sound
to be heard by VIP. Figure 10 shows the scheme of voice
command on SEE-Phone implementation.
Fig. 10. The Voice Command on SEE-Phone
The TTS application on SEE-Phone will be
to give the information to VIP about the real
situation captured by camera smart phone;
TTS will translate the text of traffic lights status to
voice/sound. Figure 11 shows the scheme of TTS on SEE
Phone implementation.
Fig. 11. The TTS Application on SEE-
V. EXPERIMENTAL EVALUATION OF SEE
Two or three functions have been implemented in
Phone which are useful for assistance of mobility
color detection using smart phone (for traffic l
detection) and remote route tracking (for continuous check
of the VIP itinerary). Experiments have been realized
Android emulator system (running on personal
and on smart phone. Their goal was to evaluate the
technical performance of the SEE-phone.
. Overview of a typical TTS System
STT, Speech-to-
a machine or program to receive and interpret
dictation, or to understand and carry out spoken
In practice, the size of a voice-recognition
ctly related to the
random access memory capacity of the computer in which
Phone, the instruction inputs or commands will
be given in voice. The VIP will speak to run a specific
Speech-To-Action).
phone output also produces the voice or sound
shows the scheme of voice
Phone (STA).
will be developed
to give the information to VIP about the real-condition or
; therefore, the
of traffic lights status to
shows the scheme of TTS on SEE-
-Phone
EXPERIMENTAL EVALUATION OF SEE-PHONE
have been implemented in SEE-
assistance of mobility, namely:
(for traffic light status
detection) and remote route tracking (for continuous check
have been realized on
ersonal computer)
Their goal was to evaluate the
A. Traffic Lights Status Detection (Yusro 2013)
In this experiment, a camera of smart phone
SIII Model GT-I9300 has been used to capture the color
image. This process uses simple application program to
produce HSV level of color image. Table
HSV level from three colors image (red, yellow
in this experiment.
The average HSV is a reference data for the image
captured by the camera (SEE-phone). When the value of
the colors (traffic lights) is captured by the SEE
camera that was in the range of reference data, the
program will call the voice application. In this experiment,
the voice function used TTS (Text To Spe
The TTS converted the result of images detection as voice,
(“red stop here”, “yellow slowly run” and “
run away”).
TABLE 3: SAMPLE DATA OF HSV LEVEL
Color Images HSV min
Red1
Red2
Red3
Red4
175, 236,169
175, 234,201
175, 242,242
175, 240,246
Average of Red 175, 238, 215
Yellow1
Yellow2 Yellow3
Yellow4
29, 240, 242
26, 245, 243 27, 242, 244
30, 247, 252
Average of Yellow 28, 244, 245
Green1
Green2
Green3
Green4
89, 244, 238
89, 228, 239
88, 233, 229
90, 218, 249
Average of Green 89, 231, 239
The results of this experiment are displayed in figure
12. The upper images (left to right)
lights status for pedestrian. The lower images (left to right)
show results of color detection by SEE
experiment, android application program
colors: red, yellow and green.
Fig. 12. The Color Detection on Android Emulator
B. Route Remote Tracking with Database
2013)
Figure 13 shows map view from the
application program (AndroidTrack) tested on
emulator and (b) Android smart phone
(Yusro 2013)
mart phone Samsung
I9300 has been used to capture the color
application program to
of color image. Table 3 shows data of
red, yellow and green)
The average HSV is a reference data for the image
phone). When the value of
the colors (traffic lights) is captured by the SEE-phone
camera that was in the range of reference data, the
program will call the voice application. In this experiment,
Text To Speech) concept.
The TTS converted the result of images detection as voice,
yellow slowly run” and “green please
OF HSV LEVEL
HSV max
179, 255, 255
179, 255, 255
177, 255, 255
177, 255, 255
178, 255, 255
32, 255, 255
32, 255, 255 32, 255, 255
32, 255, 255
32, 255, 255
91, 255, 255
91, 255, 255
90, 255, 255
91, 255, 255
91, 255, 255
periment are displayed in figure
The upper images (left to right) are signs of traffic
The lower images (left to right)
of color detection by SEE-phone. In this
experiment, android application program can detect three
on Android Emulator
Database Server (Yusro
shows map view from the Android
tested on (a) Android
smart phone.
In this experiment, Android Emulator Google APIs,
Platform 4.1.2, Level 16 and Smart phone Samsung SIII
Model GT-I9300, Android Version 4.1.1 were used.
Figure 14 shows a map of sample route in this
experiment. The user walks carrying a smart phone (SEE-
phone) from G point (start-point) to D point (end-point).
When the user walks, the Android application will run and
the GPS receiver is in “on” position. A GPS receiver of
the smart phone will receive data from GPS satellite and
SEE-phone will send the data to database server via Wi-Fi
network.
(a) (b) Fig. 13. Map View on Android Emulator and on SEE phone.
Fig. 14. The Sample of Route Map
Data locations from SEE-phone collected by database
server are shown in Table 4.
TABLE 4: LOCATION DATA FROM SEE-PHONE
Host:localhost Database : android
Generated by : phpMyAdmin 3.5.2.2/MySQL 5.5.27
No Latitude Longitude Time
1 2
3
4 5
6
7
45.75892 45.75858
45.75850
45.75846 45.75846
45.75847
45.75850
3.11157 3.11185
3.11197
3.11212 3.11231
3.11248
3.11262
17:25:16 17:25:56
17:26:10
17:26:16 17:26:26
17:26:36
17:26:49
Figure 15 shows results of tracking route on monitor
server (display). Red line on figure is route track from user
with 7 points of data location.
Fig. 15. Display of Track Route on Monitor
C. TTS and Voice Command on SEE-Phone
In this experiment, we have tested the Google voice
commands to run some functions. The experiments have
run on smart phone Samsung SIII Model GT-I9300.
The system starts to work once the user said ‘on’, until
to ‘off’ word recognition.
The ‘new’ command will create the new route when
user moves and sends the track data to database server.
The ‘stop’, finishes the application.
In the system SEES current version 2 menus (two
classes of functions) are available:
1) ‘new route’ menu, for making the new route on
database server, and
2) ‘follow route’ menu, for guiding the user
displacements based on data that already existing in
the database server.
Figure 16 below shows a display on database server
when user walks on the wrong way (track in ‘green’ line).
The database server will send the alert message to SEE-
phone through voice (audio) that be heard by user.
Fig. 16. User Walks in the Wrong Way
VI. CONCLUSION
This paper has introduced the design and
implementation of the SEE-Phone which is a part of SEE-
system, an assistive device for the VIP to improve their
orientation and walking skills.
The SEE-Phone has been technically evaluated in two
preliminary conditions, namely: 1) detection of the color
status of traffic lights; 2) conversion of traffic light data
into speech for an audio feedback to VIP, and 3)
development of the remote tracking with database server
for remote monitoring the user during its walk.
The original concept of the SEES system integrates
three main devices, iSEE, SEE-phone and SEE-stick
which complements each other. The SEES system will be
built as an Open Source platform and will integrates
several sensors. Finally, the system will be validated with
end users for its pertinence to targeted assistance and
appropriation. Its technical performances will be evaluated
and compared to these existing systems.
ACKNOWLEDGMENT
This work has been sponsored by the Indonesian and French governments, and French government research program "Investissements d'avenir" through the IMobS3 Laboratory of Excellence (ANR-10-LABX-16-01), by the
European Union through the program Regional Competitiveness and Employment 2007-2013 (ERDF–Auvergne region), and by the Auvergne region.
REFERENCES
[1]. Kamiński Ł, Stepnowski A and Demkowicz J (2011) Wearable
System Supporting Navigation of the Blind. International Journal of
Geology, Isssue 2, Volume 5, 2011, pp. 34-40.
[2]. Strumillo P (2010) Electronics Interface Aiding the Visually
Impaired in Enviromental Acces, Mobility and Navigation. The 3rd
International Conference on Human System Interaction (HSI) 2010,
May 13-15, 2010, pp. 17-24.
[3]. Brilhault A, et al (2011) Fusion and Artificial Vision and GPS to
Improve Blind Pedestrian Positioning, New Technologies, Mobility
and Security (NTMS). The 4th IFIP International Conference,
February 7-10, 2011, pp. 1-5. [4]. Pissaloux E (2009) Assistive Technologies to Support Independent
Mobility of Visually Impaired. Conference and Workshop on
Assistive Technologies for People with Vision and Hearing
Impairments, Past Successes and Futures Challenges, CVHI-2009,
M.A. Hersh (ed.) (tutorial). http://cwst.icchp.org/files/tutorialbook_tut3_part2.pdf
[5]. Pissaloux E (2011) Mobilité de Déficients Visuels : Concepts,
Modèles Computationnels et Assistances Technologiques. The 2rd Int. Conf. Sur l'Accessibilité et les Systèmes de Suppléance aux
personnes en situations de Handicap, Paris, CNRS, 17-18 January
2011, pp. 306-315
[6]. Tversky B (2005) Functional Significance of Visuo-spatial
Representations, in Shah, P., Miyake, A., "Handbook of higher-
level visuospatial thinking", Cambridge: Cambridge University
Press, 2005.
[7]. Golledge R.G., et al (1991) Designing a Personal Guidance System
to Aid Navigation without Sight: Progress on The GIS Component.
International Journal of Geographical Information Systems, Taylor
and Francis Ltd, London.
[8]. Loomis J.M., Golledge R.G. and Klatzky R.L. (1998) Navigation System for the Blind: Auditory Display Modes and Guidance.
Precense, Vol.7, No.2 April 1998, pp. 193-203.
[9]. Golledge R.G., et al (1998) A Geographical Information System for a GPS based Personal Guidance System. International Journal of
Geographical Information Science, 1998, Vol.12, No.7, pp. 727-
749.
[10]. Hersh M.A. and Johnson M.A. (2010) A Robotic Guide for Blind
People part 1: A Multi-National Survey of the Attitudes,
Requirements and Preferences of Potential End-User. Dept of
Electronics and Electrical Engineering, University of Glasgow,
Scotland.
[11]. Balakrishnan G, et al (2006) A Stereo Image Processing System for
Visually Impaired. International Journal of Information and
Communication Engineering, 2:3, 2006, pp. 136-145.
[12]. Katz B.F.G., et al (2009) NAVIG: Navigation Assisted by Artificial Vision and GNSS. The Project was supported by the French
National Research Agency (ANR).
[13]. Xu J, et al (2010) An Outdoor Navigation Aid System for the Visually Impaired. International Conference on Industrial
Engineering and Engineering Management (IEEM), December 7-
10, 2010, Macao.
[14]. Kim D.H and Ryu H.G (2011) Obstacles Recognition System using
Ultrasonic Sensor and DRFC (Duplex Radio-Frequency Camera)
for the Visually Impaired Person. International Conference on
Advanced Communication Technology (ICACT), February 13-16,
2011.
[15]. Yusro M, et al (2013) SEES: Concept and Design of a Smart
Environment Explorer Stick. The 6th International Conference on
Human System Interaction (HSI) 2013, June 6-8, 2013, Sopot,
Poland. [16]. Yibo C, et al (2011) 6LoWPAN Stacks: a Survey. The 7th Int'l
Conf. on Wireless Communications, Networking and Mobile Computing WiCOM -2011, September 23-25, 2011 Wuhan, China.
[17]. Shi HL, et al (2011) Energy Efficient and Fault Tolerant Multicore
Wireless Sensor Network: E2MWSN. The 7th Int'l Conf. on Wireless Communications, Networking and Mobile Computing
WiCOM-2011, September 23-25, 2011 Wuhan, China.
[18]. Modern GPS coordinate systems, about WGS84.
http://www.ordnancesurvey.co.uk/oswebsite/gps/information/coordi
natesystemsinfo/guidecontents/guide4.html. Accessed in January 2013.
[19]. Liao C.F, Rakauskas M and Rayankula A., Development of Mobile
Accesible Pedestrian Signals (MAPS) for Blind Pedestrians at
Signalized Intersections, Center for Transportation Studies,
Department of Civil Engineering, Minnesota Traffic Observatory,
University of Minnesota.
[20]. TTS (Text-to-Speech) Application on Smart phone.
http://searchmobilecomputing.techtarget.com/definition/text-to-
speech. Accessed in March 2013.
Recommended