Blind Navigation with a WearableRange Camera and Vibrotactile Helmet

Steve Mann, Jason Huang, Ryan Janzen, Raymond Lo, Valmiki Rampersad,Alexander Chen, Taqveer Doha

University of Toronto, Department of Electrical and Computer Engineeringhttp://wearcam.org/blindvision.htm

ABSTRACTWe present a way finding system that uses a range cameraand an array of vibrotactile elements we built into a helmet.

The range camera is a Kinect 3D sensor from Microsoftthat is meant to be kept stationary, and used to watch theuser (i.e., to detect the person’s gestures). Rather than usingthe camera to look at the user, we reverse the situation, byputting the Kinect range camera on a helmet for being wornby the user. In our case, the Kinect is in motion rather thanstationary.

Whereas stationary cameras have previously been usedfor gesture recognition, which the Kinect does very well, inour new modality, we take advantage of the Kinect’s re-silience against rapidly changing background scenery, wherethe background in our case is now in motion (i.e., a con-ventional wearable camera would be presented with a con-stantly changing background that is difficult to manage bymere background subtraction).

The goal of our project is collision avoidance for blindor visually impaired individuals, and for workers in harshenvironments such as industrial environments with signifi-cant 3-dimensional obstacles, as well as for use in low-lightenvironments.

Categories and Subject DescriptorsI.4.8 [Image Processing and Computer Vision]: SceneAnalysis—Depth cues, Range data, Motion; I.4.9 [ImageProcessing and Computer Vision]: Applications; H.1.2[Models and Principles]: User/Machine Systems—Hu-man factors

General TermsDesign, Experimentation, Human Factors

Keywordspersonal safety devices, blind navigation, depth sensor,Microsoft Kinect, vibrotactile helmet, collision avoidance

*Area Chair: Pal Halvorsen

Permission to make digital or hard copies of all or part of this work forpersonal or classroom use is granted without fee provided that copies arenot made or distributed for profit or commercial advantage and that copiesbear this notice and the full citation on the first page. To copy otherwise, torepublish, to post on servers or to redistribute to lists, requires prior specificpermission and/or a fee.All figures copyrighted by the authorsMM’11 November 28–December 1, 2011, Scottsdale, Arizona, USA.Copyright 2011 ACM 978-1-4503-0616-4/11/11 ...$10.00.

1. INTRODUCTION

1.1 Microsoft KinectThe Microsoft Kinect was designed for use with Microsoft’s

Xbox 360 gaming console. It allows the gamer to interactwith games without the need for physical controls. It accom-plishes this by tracking the gamer’s movements and positionin three-dimensional space, with respect to itself, in real-time. In normal use, the Kinect sits stationary and observesthe gamer as he/she moves.

We propose the use of the Kinect in a different manner,where the Kinect moves with the user, so that it observesthe world in a similar fashion as the user observes (or wouldhave observed, in the case of a blind individual). Ratherthan having the Kinect watch the user, the user uses it towatch their environments.

Microsoft’s Kinect employs PrimeSense’s 3-D sensing tech-nology. PrimeSense’s 3-D sensor uses light coding to codethe scene volume, using active IR (infrared) illumination [2,3, 4]. The sensor then uses a CMOS image sensor to readthe coded light back from the scene. The coded light is pro-cessed by PrimeSense’s SoC chip [5], contained in the 3-Dsensor, to give the depth information.

In our implementation, the Kinect is used to extract the3D depth information of the environment being observed bythe user. This depth information is passed to the user inthe form of tactile feedback, using an array of vibrotactileactuators as shown in Figure 2.

1.2 Other head-mounted navigational aidsMost previous head-mounted navigational aids have used

standard camera systems, to present tactile information tothe user. One such example is called“seeing with the tongue”[8].

Standard camera systems work well for gesture recogni-tion because the stationary background can be subtractedfrom the image, so that people can be clearly seen with sim-ple computer image processing. However, when the camerais wearable, the background is constantly changing, mak-ing it difficult to separate distant background clutter fromnearby objects.

Some specialized blind collision avoidance aids such asthe VibraVest [1] provided 3D range information but re-quired expensive special-purpose hardware such as a minia-ture radar system.

The Kinect is a new widely deployed commercial off-the-shelf technology that provides a low-cost solution to theproblems associated with sensitivity to distant background

1325

KINECT

SKIN

MECHANORECEPTORS MAP

VIBRATION INTENSITY TO DEPTH

VIBRATION PROFILE

ACTUATOR PWM DATA

11−BIT DISPARITY DATA

DISPARITY DATA PROCESSING

OUTPUTS DISPARITY VALUES

EXTRACTS DEPTH INFORMATION

CALCULATES ACTUATOR INTENSITY

CREATES DEPTH PROFILE

CONVERTS PWM DATA INTO

ACTUATOR DRIVE VOLTAGE

ACTUATOR CONTROLLER

Figure 1: System signal flow path overview

clutter. Since background clutter is especially prevalent in awearable camera situation, the technology used in the Kinectshows great promise in wearable vision systems.

2. PHYSICAL SETUPFigure 1 shows the signal flow path in our system archi-

tecture. Our goal is to convert depth information obtainedusing Kinect into haptic feedback so that users can perceivedepth within a range that matters most for collision avoid-ance, while not being overwhelmed by distant backgroundclutter.

An array of six vibrating actuators mounted inside a hel-met are controlled using the depth values using an algo-rithm that calculates the vibration intensity profile for eachof these actuators. The intensity profile is transmitted to anArduino microcontroller which drives each of the actuatorsusing Pulse-Width Modulation (PWM). PWM allows volt-age on the actuators to be regulated for varying degrees ofvibration.

This sense of depth moves with the head in a naturalmanner. Thus, the user can scan the head back and forthto get a natural understanding of subject matter in theirenvironment. The general layout of our helmet is depictedin Fig 2.

We mounted the Kinect securely on top of a welding hel-met. An array of 6 vibration actuators were positioned alongthe head strap of the helmet. The helmet is placed on thehead as shown in Fig 2.

For testing by sighted users, a dark welding shade wasused, which could either entirely stop light from passingthrough, or under computer program control, vary the amountof light passing through. In this way, the device could func-tion as a switchable blindfold for testing purposes.

2.1 Vibrotactile actuators, and motor controllersTo provide haptic feedback, we used a set of vibrating

actuator motors. The vision processing algorithm controlsthe motors through a serial connection to an Arduino. Thesevalues correspond directly to PWM output from pins 2 to 7on the microcontroller. Each output pin is used as controlsignal in motor driver circuit which determines the actuator

KINECT

AR3 AR2 AR1 AL1 AL2 AL3

KINECT

AR3

AR2

AR1AL1

AL2

AL3

FRONT VIEW

FRONT

BACK

KINECT

SIDE VIEW

TOP VIEW

AR1AR2AR3 AR2

MOUNTING HELMET

Figure 2: Wearable sensor and actuator configura-tion on a helmet, showing placement of the motoractuators around the forehead.

vibration response for AL3 to AR3 as shown in Fig 2.For our setup, we used 10x3.4mm shaftless vibration mo-

tor for each of the actuators. The motor is rated to bedriven at a maximum voltage of 3.6V. Depending on thePWM value from each of the Arduino pins, the correspond-ing actuator can be driven at voltages calculated as:

Vactuator = PWM/255 ∗ 3.6, PWM ∈ [0, 255] (1)

The actuator Voltage and Current response was tested tobe linear. Based on this, we determined that the vibratingactuator also had a linear response, when driven betweenvoltages from 0 to 3.6V.

3. REAL-TIME IMAGE PROCESSING TOCONTROL VIBRATION ACTUATORS

3.1 Distance map recovered from KinectWe accessed the Kinect data in real-time with a wearable

laptop computer on a backpack. The Kinect provides datain a proprietary data format called “disparity” values.

To recover the actual distance from the raw depth datain the proprietary format, we used the following conversionequation

distance = R =1

α · disparity + β(2)

The parameters have been empirically found in [7] to be:α = −0.0030711016 and β = 3.3309495161. As a result, therange extremities become:

disparity distanceMIN distance detectable 0 0.30 mMAX distance detectable 1030 5.96 m

At distances closer than 30cm, the Kinect is not able tomeasure disparity. The disparity values are calibrated suchthat the device is able to read values up to 6m withoutsignificant loss of resolution within acceptable error marginwhile operating indoors. We found this range of 0.3 to 6metres to be useful for collision avoidance at typical walkingspeeds.

This range is also greater than the range of a typicalcane used by visually impared users, and permits a warn-ing sooner in advance of collision, at a given walking speed.It is the relatively near range (just beyond an arm’s length)where the greatest sensitivity is needed.

1326

3.2 Partitioning the distance mapThe Kinect operates with horizontal field of view of 57◦

horizontally and 43◦ vertically. A distance map across thisfield (akin to a 2D video feed) was computed in real-timebased on the disparity values.

In our setup, the depth sensing region was divided intosix smaller zones, three on the left (SL1, SL2, and SL3)and three on the right (SR1, SR2, SR3). Each of the zonescorresponds to the vibration in one actuator.

3.3 Controlling a 1-dimensionalarray of actuators

Fig 2 and 3 show the layout of the actuators and viewablearea of sensing regions respectively. Our setup allows theuser to scan their head back and forth and feel various ob-jects as if they were pressing against their forehead. Whilenot providing high resolution imaging, we found that it waspossible to navigate a hallway and find various doors, door-ways, etc., and also avoid collision with other people in acrowded hallway.

3.4 Transfer function with 1-to-1 mapping fromeach sensing region to each actuator

We desired objects in the visual field to cause the vi-brations to become stronger as the objects get closer. Inthis way the system creates the sensation of objects press-ing against the forehead at-a-distance, i.e. before collisionoccurs. The sensation increases in strength as collision be-comes more imminent.

This can be accomplished by making the vibration (assensed) be inversely proportional to distance, i.e. V ∝ 1/R.

Alternately we can make the sensed vibration vary asV ∝ 1/R2, as with a force field such as a magnetic field.For example, when holding two magnets close to each other,the magnetic repulsion (or attraction) is inversely propor-tional to the separation distance squared. Thus we canmimic nature, in this regard, in order to make the effectcomprehensible and intuitive.

We now partition the depth map for each of the differentvibration actuators. For an inverse-square law, we make thetotal vibration a weighted integral across the sensing region:

vn =1

SFOV

∫SFOV

1

R2(θ, φ)aθ,n(θ)aφ,n(φ)dS (3)

for actuator n. aθ,n and aφ,n are aperture functions, weight-ings which vary depending on the horizontal and verticallocations, respectively. They are different for each sensingregion n. S is sensing surface in steradians (sr).

We found empirically that the background noise from Eqn. 3,coming from objects in the background behind the subjectmatter, was distracting. Our second implementation simplyused the closest object in zone, still weighted by the aperturefunctions:

vn = minSFOV

1

R2(θ, φ)aθ,n(θ)aφ,n(φ) n ∈ 1...N (4)

We also experimented with the following 1/R law, whichgave an improved advance warning of faraway objects ap-proaching (> 3m).

vn = minSFOV

1

R(θ, φ)aθ,n(θ)aφ,n(φ) n ∈ 1...N (5)

The result is a center-weighted mapping, as illustrated inFig 3.

HORIZONTAL

APERTURE FUNCTION

VERTICAL

APERTURE FUNCTION

R

dS

dS

DEPTH MAP SENSOR

(FOCAL POINT)

dS

dS

ONE SENSING ZONE

(WITH FUZZY BOUNDARIES)

WITHIN TOTAL FIELD OF VIEW

Figure 3: The viewable area of the depth sensor wasdivided into zones, each of which corresponded toone of the actuator motors. The sensitivity profilewas center-weighted as illustrated.

3.5 Fuzzy zone boundariesIt is also possible to go beyond a simple 1-to-1 mapping

from each spatial sensing region to each actuator. For ex-ample, we experimented with making fuzzy boundaries oneach sensing region, using a horizontal aperture functionthat extended beyond the boundaries of each sensing re-gion (see horizontal aperture function in Fig 3). As a result,each actuator was slightly responsive to neighbouring sens-ing regions. The overlap and center-weighting, combinedwith natural exploratory motions of the user’s head, gavesome sub-pixel accuracy that allowed the user to sense somedegree of fine detail.

3.6 Compensating for non-linear behavior ofmotors and human perception

One challenge of our design is to convert the depth mapvalues to another sensory mode such as tactile feedback us-ing an actuator.

It is clear to see that using a linear model for mappingraw depth value to the actuator is inadequate for severalreasons. For many of the sensory modalities, our sensoryperceptions are non-linear and have a highly compressivenature. For example, humans perceive loudness of sound ina logarithmic scale. This logarithmic scale recurs often inthe human senses and comes from Weber’s analysis of “just-noticeable differences” [6]. A perceived sensation P resultsfrom fractional changes in a physical quantity I as in:

∆P ∝ ∆I

I(6)

After setting P = 0 at the minimum perceptible physicalquantity I0, the solution becomes:

P = k log

(I

I0

)(7)

Weber found this logarithmic law to exist for the sense oftouch [6]. Therefore, we needed to modify the vibrationsoutlined earlier, into an exaggerated actuator stimulus that

1327

Microsoft Kinect Draw depth data

[2D]

Convert proprietaryformat to actual distance

[2D]

Spatial Aperture Function andDistance to Vibration Mapping

[6 channels] PCompensation for

non-linearity inmotors andelectronics

desired sensation MCompensation for

non-linearity inhuman perception

Physicalvibration

[6 actuators]

distance (m) A-1 -1

PMMotors andelectronics

Human Perception

Figure 4: This figure shows the conversion of the depth map data to the actual physical vibration of the 6actuators in the helmet. The underlying non-linear relationships among the raw depth sensor, motor andelectronics, and human perceptions are estimated empirically. By aggregating the M−1 and P−1 functions, wecan determine the mapping of the vibrating intensity to the sensitivity range of human senses experimentally.

Figure 5: The MindMeshTM BCI (Brain Com-puter Interface) FPGA mesh-computing architec-ture has 64 electrodes, so that various items suchas the Kinect or more conventional cameras can be“plugged into the brain”.

would properly be perceived by the user as the given 1/Rlaw or 1/R2 law, as desired.

An additional compensation was necessary because theraw depth data collected from the Kinect is not a directmeasurement of the actual distance in the real world.

Since the non-linearities and the underlying interactionsbetween the actuator and human perception are difficult torecover, we estimated these relationships experimentally byperforming trials on different users. We have found thatusing an exponential decade function as follows provides ad-equate results, and also conforms with the non-linear re-sponse of the human senses that we desired to overcome:

w = (0.978)255d (8)

where d is the measured distance normalized with 1 indi-cating the maximum range, and w the PWM value whichcontrols the electrical actuator.

Figure 4 shows the conversions and compensation we haveintroduced in the signal flow path. Notice that our systemhas aggregated the inverse of the non-linear responses ofthe motor and electronics as well as human perception forsimplicity. With the proper calibration and compensationfor non-linearity and sensory thresholds, users were able tolearn the relationship between the distance and the vibrationintensity after several minutes of training with the system.

4. FUTURE WORKIn further variations of the system, we implemented vari-

ous 2-dimensional arrays, such as a 3 by 4 array of 12 actu-ators, and a 2 by 6 array of 12 actuators (the Arduino has12 PWM outputs). In further explorations, we also exper-imented with larger arrays using multiple microcontrollers.However, we found that a small number of actuators wasoften sufficient.

We are currently completing a BCI (Brain Computer In-terface) in place of the vibrotactile devices. Our system,which we call the MindMeshTM , is depicted in Figure 5.

5. CONCLUSIONWe have proposed a novel way of using the Microsoft

Kinect 3-D camera, for navigation which we hope will some-day assist the visually impaired. Rather than having theKinect observe a user, we put the Kinect on the user toobserve the user’s environment. We found that the typicaloperating range of the Kinect (30cm to 6m) was well suitedto indoor navigation in typical crowded corridors, and thelike. This preliminary work suggests that eyeglasses could bemade using PrimeSense’s 3-D sensing technology, for poten-tial use by the visually impaired. Videos of the helmet in ac-tion can be viewed at: http://wearcam.org/blindvision/

6. REFERENCES[1] S. Mann. Intelligent Image Processing. John Wiley and

Sons, November 2 2001. ISBN: 0-471-40637-6.

[2] A. Shpunt. DEPTH MAPPING USING MULTI-BEAMILLUMINATION, Jan. 21 2008. US Patent App.12/522,172.

[3] A. Shpunt and B. Pesach. OPTICAL PATTERNPROJECTION, July 21 2010. US Patent App.12/840,312.

[4] A. Shpunt and Z. Zalevsky. Depth-varying light fieldsfor three dimensional sensing, Mar. 13 2007. US PatentApp. 11/724,068.

[5] E. Spektor, Z. Mor, and D. Rais. INTEGRATEDPROCESSOR FOR 3D MAPPING, Mar. 4 2009. USPatent App. 12/397,362.

[6] E. Weber, H. Ross, and D. Murray. EH Weber on thetactile senses. Psychology Press, 1996.

[7] Wikipedia. Ros (robot operating system) — wikipedia,the free encyclopedia, 2011. [Online; accessed12-April-2011].

[8] P. B. y Rita, M. E. Tyler, and K. A. Kaczmarek. Seeingwith the Brain, volume 15(2), pages 285–295. Hillsdale,NJ: Lawrence Erlbaum Associates Inc, 2003.

1328