24 PERVASIVE computing Published by the IEEE CS Q 1536-1268/12/$31.00 © 2012 IEEE

P E R V A S I V E I N T E R A C T I O N

Ultrasonic Doppler Sensing in HCI

S ensors are the eyes, ears, and skin of computing interfaces. Whether they’re simple buttons or sophisticated vision systems, we’re em powered by their capabilities and constrained

by their shortcomings. A tremendous amount of HCI research has focused on maximizing the effectiveness and use of these channels. These developments, in concert with signi!-cant advances in electronics, have enabled us to bring the power of computation to a wider audi-

ence and into more aspects of our lives.

Researchers and practi-tioners can now draw upon a large suite of sensing tech-nologies for their work. Re-lying on thermal, chemical, electro magnetic, optical, acoustic, mechanical, and other means, these sensors can detect faces, hand ges-tures, temperatures, humid-

ity, blood pressure, pupil dilation, accelera-tion, proximities, and many other aspects of our state and environment.1,2 Here, we present an overview of our work on an ultrasonic Dop-pler sensor. This technique has unique qualities that we believe make it a valuable addition to the suite of sensing approaches HCI research-ers and practitioners should consider in their applications.

Doppler-Based Motion CharacterizationInstead of operating like sonar, which sends out “pings” of ultrasonic sound (for example, ultrasonic range !nders and proximity sensors), our sensor emits a single, continuous tone that bounces off objects in the detection !eld. The re"ected signals that return to the sensor are captured for analysis. In a static environment, the return signal will be at the same frequency, but with a different phase and amplitude. However, if an object is moving, the echoes are Doppler shifted, creating components at other frequencies, proportional to their velocity rel-ative to the sensor (see Figure 1). This prop-erty separates our sensor from other sensing techniques.

The Doppler shift caused by the re"ection from a moving target is approximately fd " 2vf/(c � v), where fd is the observed frequency shift, v is the target’s velocity (in the direction of the sensor), f is the emitted frequency, and c is the speed of sound.

If multiple objects are moving with different velocities, the re"ected signal will contain mul-tiple frequencies, one for each object. In other words, the return signal’s power spectrum in-cludes the velocities (in the direction of the sen-sor) of all objects in its !eld of view.

This is the same principle of operation as the radar guns police use to catch speeding drivers. However, in addition to using an acoustic signal

Several properties differentiate ultrasonic Doppler sensing from other sensing techniques—high frame rate, low computational overhead, instantaneous velocity readings, and range independence. Also, because it isn’t vision-based, it might open doors to sensing in once taboo locations.

Bhiksha RajCarnegie Mellon University

Kaustubh KalgaonkarGeorgia Institute of Technology

Chris HarrisonCarnegie Mellon University

Paul DietzMicrosoft

PC-11-02-Har.indd 24 3/21/12 11:12 AM

APRIL–JUNE 2012 PERVASIVE computing 25

instead of RF, our sensor aims to observe the full velocity pro!le in the !eld of view. Police radars, in contrast, only at-tempt to detect a single object’s velocity.

Our Doppler SensorOur Doppler sensor comprises an ultra-sound emitter and one or more receiv-ers that capture re"ections. We used an

off-the-shelf MA40A3S ultrasound emitter with a resonant frequency of 40 kilohertz. This frequency is far above what the human ear can detect, so the

T he largest use of ultrasound has been for medical (http://aium.org) and structural diagnostics (http://en.wikipedia.

org/wiki/ultrasonic_testing). Outside of diagnostic purposes, ultrasound has largely been used for ranging (see, for example, http://arduino.cc/en/Tutorial/Ping). The few other interactive applications that exist have generally centered around the prin-ciples used in diagnostic imaging1 or ranging.

Doppler radars, in addition to their well-known uses in avia-tion, meteorology, and speed guns, have also been widely used in commercial and domestic settings for motion detection, in-truder alarms (see www.protechusa.com), and light control.

In the HCI area, their use has largely been directed toward the characterization and analysis of human gait,2–5 and, to a much smaller extent, as a sensing mechanism for recognizing gestures.6–9 However, despite the development of micro-Doppler sensors,3 Doppler radar remains a relatively expensive option, requiring either complicated hardware design9 or specialized hardware.3

REFERENCES

1. F. Vogt et al., “Tongue ‘n’ Groove: An Ultrasound Based Music Con-troller,” Proc. 2002 Conf. New Interfaces for Musical Expression (NIME 02), Nat’l Univ. of Singapore, 2002; www.nime.org/2004/NIME02/vogt.pdf.

2. T. Yardibi et al., “Gait Characterization via Pulse-Doppler Radar,” Proc. 2011 IEEE Pervasive Computing and Communications Workshops (Percom 11 Workshops), IEEE CS Press, 2011, pp. 662–667.

3. D. Tahmoush and J. Silvious, “Radar Micro-Doppler for Long Range Front-View Gait Recognition,” Proc. 3rd Int’l Conf. Biometrics: Theory, Applications, and Systems (BTAS 09), IEEE Press, 2009, pp. 1–6.

4. F.H.C. Tivivie, A. Bouzerdoum, and M.G. Amin, “A Human Gait Classi!cation Method Based on Radar Doppler Spectrograms,” EURASIP J. Advances in Signal Processing, vol. 2010, 2010, article ID 389716.

5. C. Hornsteiner and J. Detlefsen, “Characterisation of Human Gait Us-ing a Continuous-Wave Radar at 24GHz,” Advances in Radio Science, vol. 6, 2008, pp. 67–70; www.adv-radio-sci.net/6/67/2008/ars-6-67-2008.pdf.

6. J.L. Geisheimer, W.S. Marshall, and E. Greneker, “A Continuous-Wave (CW) Radar for Gait Analysis,” Proc. 35th Asilomar Conf. Signals, Sys-tems and Computers, IEEE Press, 2001, pp. 834–838.

7. G.-W. Raes, “Microwave Gesture Sensing,” tech. memo, Ghent Uni-versity College and Logos Foundation, 2009; http://logosfoundation.org/ii/dopplerFMradar.html.

8. G.-W. Raes, “Bumblebee,” tech. memo, Ghent Univ. College and Logos Foundation, 2009; http://logosfoundation.org/ii/bumblebee.html.

9. J. Paradiso, N. Yu, and C.K. Leo, “Gesture-Sensing Radars Project,” MIT Media Lab, 1997; http://resenv.media.mit.edu/Radar/index.html.

10. C.K. Leo, “Contact and Free-Gesture Tracking for Large Interactive Surfaces,” masters thesis, School of Architecture and Planning, Massachusetts Inst. of Technology, 2002.

Related Work in Ultrasound and Doppler Sensing

Figure 1. The velocity of proximate objects alters the frequency of the re!ected sound. This is captured by the sensor, which performs a fast Fourier transform. A simpli"ed version of this output shows that the frequency can indicate whether the object (a) is still, (b) moving closer, or (c) moving farther away and (d) how fast the object is moving.

Sensor Object Sensor Object Sensor Object Sensor Object

(a) (b) (c) (d)

PC-11-02-Har.indd 25 3/21/12 11:12 AM

26 PERVASIVE computing www.computer.org/pervasive

PERVASIVE INTERACTION

device operates silently. The emitter is driven by a PIC microcontroller pro-grammed to produce a 40-kHz square wave with a duty cycle of 50 percent. The emitter is highly resonant, so it produces a nearly perfect sinusoid, even when excited by a square wave. The re-ceiver is an MA40A3R ultrasound sen-sor that can sense frequencies in a nar-row frequency band around 40 kHz, with a three-decibel bandwidth of less than 3,000 Hz. The sensor’s relatively narrow bandwidth ensures that it doesn’t pick up out-of-band noise.

To ease the processing burden, the sensor exploits the signal and receiver’s limited bandwidth. At one meter per second, we’d expect to see a frequency shift of only 242 Hz. The Nyquist theo-rem states that a signal must be sam-pled at greater than twice its bandwidth (which can be much smaller than its highest frequency) to allow reconstruc-tion. So, we chose to sample at a few kHz, which is suf!cient for capturing human motion. In some experiments, we used higher-!delity equipment that could sample at much higher rates and with far greater precision. In these cases, the digitized signal was hetero-dyned to shift the carrier frequency of 40 kHz down to 4 kHz and resampled to 16,000 samples per second.

Figure 2 shows various prototypes of our sensor. In Figure 2a, the trans-mitter, driving circuit, and receiver are all on the same board. For voice- processing applications, we separated the transmitter and receiver from the board and strapped them onto a micro-phone (see Figure 2b), so that all three can be collocated. For gesture-recognition applications, we employed multiple re-ceivers, once again separating the trans-mitter and receivers from the board (see Figure 2c). The transmitter in Figure 2c is driven by the same circuit used in Figure 2a, but the signals from the three receivers were jointly captured by a multi channel A/D. In Figure 2a, the de-vice’s power consumption is on the or-der of tens of milliwatts, although the exact value varies with con!guration.

The captured signal is always spec-trographically analyzed to derive fea-tures; the analysis frame size and rate vary with the application. In our work, for gesture-recognition tasks, the hand movements were relatively fast, so we employed an analysis window of 32 milliseconds, with a 50 percent overlap between adjacent windows, re-sulting in a frame rate of 66.66 frames per second. For other tasks, such as gait recognition and speaker iden-tification, we employed a relatively larger analysis window of 40 to 64 ms,

with a 50 percent overlap between analysis frames.

We derived a sequence of cepstral vectors3 from the sequence of analy-sis frames as follows: each analy-sis frame was Hamming windowed, and a 513-point power spectrum was computed from each sequence using a 1,024-point Fourier transform. We logarithmically compressed the power spectrum and applied a discrete cosine transform (DCT) to it. We retained the !rst 40 DCT coef!cients to obtain a 40-dimensional cepstral vector. Then, we further augmented each cepstral vector using the difference between the cepstral vectors derived from adjacent frames to obtain a !nal 80-dimensional feature vector representing each frame. The entire feature computation is light-weight and easily performed on a digi-tal signal processor (DSP). We classi!ed the resulting sequence of feature vectors into one of a desired set of classes using a Gaussian Mixture classi!er.4

The Doppler sensor can operate over a range of up to 10 meters for applica-tions such as gait recognition, where the movements to be characterized include large swings in position and velocity. For finer-grained movements, such as !nger gestures, the range is much smaller, on the order of one meter. The ultrasound sensor has an angle of 50 degrees. The signal’s attenua-tion at greater distances also naturally provides robustness to spurious back-ground activity (outside the sensor’s range).

PropertiesWe employed Doppler sensing in sev-eral distinct manners to assess its capa-bilities and shortcomings—especially as they relate to uses in the HCI !eld. Here, we review eight notable proper-ties, comparing them against camera-based approaches, which are currently popular and similar in capability.

First, Doppler-based measurements capture snapshots of the instantaneous velocities of one or more moving ob-jects (in the direction of the sensor).

Figure 2. The prototype ultrasonic Doppler sensor in various applications: (a) to recognize objects, the transmitter, driving circuit, and receiver are all on the same board; (b) for speech applications, a central microphone is augmented with an ultrasonic transmitter and a receiver; and (c) for gesture recognition, a single transmitter is in the center and three receivers are on the three peripheral arms.

(a)

(b)

(c)

PC-11-02-Har.indd 26 3/21/12 11:12 AM

APRIL–JUNE 2012 PERVASIVE computing 27

This is in contrast to cameras, as well as optical and ultrasonic range !nd-ers, which capture a series of static dis-tances or images, so the velocities must be estimated through differentiation.

Second, Doppler-based systems are active—they generate the signal they observe, so they can operate in the dark. Vision-based systems depend on the presence of light.

A third difference is that camera-based systems work best when the motion to be characterized is in a plane perpendicular to the vector from the object to the camera. Conversely, Doppler-based systems are most ef-fective when the motion is toward or away from the sensor (velocities per-pendicular to the vector to the sensor are undetectable). In other words, Dop-pler systems measure motion in an axis orthogonal to cameras (and could offer interesting opportunities if combined with cameras).

Next, images observed by cameras shrink as the distance from the camera increases. Doppler sensors detect fre-quency spectra independent of the sig-nal’s amplitude and thus independent of the target object’s distance (as long as the signal is above noise).

Vision-based algorithms are highly dependent on the ability to accurately extract and track the subject’s silhou-ette. Even when the only moving ob-ject in the video is the subject, various phenomena (such as shadows or the background layout) can affect accurate tracking. So the !fth difference is that Doppler-based devices are relatively insensitive to constant background ef-fects. Additionally, because the emis-sion frequency of the ultrasonic trans-ducer is known (40 kHz, for example), segmenting motion from the environ-ment is straightforward (simply look for the presence of other frequencies). Thus, no calibration or training is re-quired, making it robust and easily deployable.

The sixth notable property is that Doppler sensors have far less data to deal with than vision-based systems

(because they don’t need to store pre-vious frames or training data), so less advanced hardware is needed for com-putation and the power requirements are lower. Vision processing on mobile phones is only now becoming feasible on the highest-end devices, and it places a tremendous strain on the processors and batteries. On the other hand, our sensor of"oads all signal processing (FFT, for example) to a dedicated and highly ef!cient DSP chip, which could be integrated easily into a small mobile device. Classi!cation is also fairly light-weight and could be moved to embed-ded hardware.

Also, our Doppler-based approach is currently less expensive than compa-rable vision-based systems.

Finally, people are particularly sensi-tive to cameras when it comes to their privacy.5 They’re often uncomfortable with the notion of something “viewing” them, even if the collected data will only be processed locally. This suspi-cion of vision-based sensing has slowed its adoption in places such as the home, classroom, and of!ce. Doppler-based sensing, however, requires no lenses or other vision components and thus might escape the stigma entirely (simi-lar to motion detectors).

ApplicationsOur Doppler-based approach is highly versatile and can be used largely un-changed in a variety of applications. Other sensing mechanisms, such as video, require application-customized processing. Here, we describe our ex-perience with several applications that demonstrate this versatility.

SpeechWe can use Doppler sensors to provide secondary measurements for speech. For example, as Figure 2b showed, we mounted a transmitter and receiver alongside a conventional microphone to capture both sounds and minute facial movements. With this setup, you could also dispense with the receiver alto-gether and use a broadband microphone

to simultaneously capture both the speech and ultrasound re"ections, be-cause they occur in widely separated frequency bands.

We experimented with this technique in a speaker-identi!cation application.6 We recruited 50 participants, who sat approximately one meter away from a microphone/Doppler setup. We in-structed the participants to speak nor-mally, facing the microphone’s general direction. Each participant recorded 75 sentences from the Timit corpus,7 with an average length of about three seconds. We used one third of these re-cordings to train a Gaussian mixture distribution for each speaker and used the rest as test data. Surprisingly, us-ing only the Doppler recordings (that is, no audio) of a single sentence, we achieved an identi!cation accuracy of 90 percent.

In another application, we used the Doppler sensor to detect speech activ-ity.8 For most speech-driven interfaces, the user presses a button to control the interaction, but it would be bet-ter if such interfaces could automati-cally detect the onset of speech. Using a Doppler sensor, we detected the on-set of speech more than 90 percent of the time, including under conditions in which conventional methods (based only on speech) detected speech on-set only approximately 10 percent of the time.

Furthermore, many speech-based interfaces can’t detect when a subject is addressing the system as opposed to merely speaking in the vicinity of the sensor, so cameras typically must determine whether the subject is fac-ing the system. The ultrasound sensor performed this task by detecting facial movements typical of a subject address-ing the sensor. Deployed on an of!ce "oor, the sensor achieved false alarm rates that were 1/30th of that obtained by a speech-only system.

Biological MotionThe human body is an articulated ob-ject, comprising numerous rigid bones

PC-11-02-Har.indd 27 3/21/12 11:12 AM

28 PERVASIVE computing www.computer.org/pervasive

PERVASIVE INTERACTION

connected by joints. During move-ments such as walking, the body moves in characteristic patterns. Each body part’s velocity part depends on the following: its distance from the hinge around which it moves, the velocity with which the hinge itself might move around other hinges, and the body’s overall movement. This often forms complex, cyclic pattern of velocities characteristic of particular motions.

Doppler sensing is well suited to cap-ture this information. Figure 3 shows a spectrographic view of a 40-kHz tone reflected by a walking person. The limbs’ cyclic motion is clearly apparent

in the repeated spectrographic patterns. Moreover, the shape and detail of these patterns are characteristic of the man-ner in which the walker’s limbs move—and of the walker themselves.

Sensing in this domain has many potential applications. For example, it could make smart homes more intelli-gent by determining the rough age of the occupants in any given space. The biomechanics of walking are radically different from that of crawling, and limb length is highly correlated with age, so sensors could roughly classify occupants into one of three groups: infant, child, or adult. You could also exploit ultrasonic Doppler sensing to wirelessly achieve activity recognition and determine whether a user is walk-ing or running, for example, without requiring on-body sensors.9 Finally, the biomechanics of quadrupeds produces a unique movement signature and could, for example, lead to automatic unlocking of the dog "ap versus door based on who approaches.

As a proof of concept, we developed an application for identifying people based primarily on their gait.10 In pre-liminary experiments, we asked 30 subjects to walk five meters toward and away from a sensor mounted approximately hip high on a desk

(see Figure 4). We used half of the data from each subject to train a Gaussian mixture classi!er for that subject and used the other half to evaluate the sys-tem. The system identi!ed the walker with 91.7 percent accuracy and the walker’s direction with 96.3 percent accuracy. Additionally, we determined the participant’s gender with greater than 80 percent accuracy.

GesturesGestures are a convenient and intuitive mechanism for communicating with interactive systems, and the application that originally spurred our interest in Doppler sensors in this area was an in-teractive theme park show. The show required guests to raise their hands to answer questions, but none of the sys-tems investigated to detect this gesture were suf!ciently accurate. For exam-ple, a vision system was attempted, but spurious background activity made it unreliable.

The solution was to place a 40-kHz emitter/receiver pair in the ceiling above each guest participating in the show.11 The signal processing was done in the analog domain, looking for power in a band around 40.4 kHz. Although there’s vertical motion in normal walk-ing, it doesn’t produce velocities of the same magnitude as an intentional hand raise. Similarly, standing up is a much slower motion. The system could be confused by unintended gestures, but these were of such a nature that even a human observer might well catego-rize them as intentional hand raising. Ultimately, the Doppler-based solution dramatically out-performed the vision systems under consideration, and at a tiny fraction of their cost.

A second effort looked at recogniz-ing more complex gestures.12 We em-ployed a single ultrasonic transmitter and multiple receivers (located in front and to the right and left of the user, as shown in Figure 2c). The gesture sen-sor can recognize a number of different gestures, including forward and back-ward motion, left and right motion,

Figure 3. Spectrogram of ultrasonic re!ections from a subject walking toward a sensor. The cyclic motion of the limbs is clearly apparent in the repeated spectrographic patterns. Moreover, the shape and detail of these patterns can help characterize the walker.

48

44

40

36

500 2,0001,5001,000Time (milliseconds)

Freq

uenc

y (k

Hz)

Figure 4. Mounting the ultrasonic sensor for gait recognition. The system identi"ed the walker with 91.7 percent accuracy and the walker’s direction with 96.3 percent accuracy.

PC-11-02-Har.indd 28 3/21/12 11:12 AM

APRIL–JUNE 2012 PERVASIVE computing 29

up and down motion, and clockwise or anticlockwise hand rotation. In a preliminary experiment, we collected data from 10 participants performing eight exemplar gestures. We told par-ticipants how to perform each gesture in front of the setup. Following this ini-tial training, participants performed each gesture 10 times. This proce-dure produced 100 instances of each gesture—60 of which we used for train-ing. We reserved the rest for testing. We employed a Gaussian mixture classi!er, which jointly classi!ed the ensemble of reflections captured by the three sensors. Our proof-of-concept setup yielded a classification accuracy of 88.4 percent.

C onventional sensors such as video, although applicable in all of these scenarios, would require significant

application-specific customization. The Doppler sensor, on the other hand, can use essentially the same processing mechanism for all of these scenarios—simple spectral characterization of the signal followed by Bayesian classi!ca-tion. The only variation required is the physical layout of the sensor itself.

Our review of several preliminary uses of the technology highlights the sensors’ effectiveness in several ex-ample HCI domains. Moreover, the Doppler sensor could augment other sensing modalities at minimal cost. Overall, the Doppler ultrasound sen-sor should make a useful addition to the suite of techniques HCI researchers and practitioners might consider in their applications.

REFERENCES 1. K. Hinckley et al., “Sensing Techniques

for Mobile Interaction,” Proc. 13th Ann. ACM Symp. User Interface Software and Technology (UIST 00), ACM, 2000, pp. 91–100.

2. J.A. Paradiso et al., “Sensor Systems for Interactive Surfaces,” IBM Systems J., vol. 39, nos. 3–4, 2000, pp. 892–914.

3. A. Oppenheim and R.W. Schafer, Digital Signal Processing, Prentice Hall, 1974.

4. R.O. Duda, P.E. Hart, and D.G. Stork, Pattern Classification, 2nd ed., John Wiley & Sons, 2001.

5. S.E. Hudson and I. Smith, “Techniques for Addressing Fundamental Privacy and Disruption Tradeoffs in Awareness Support Systems,” Proc. ACM Conf. Computer Supported Cooperative Work 1996, ACM, 1996, pp. 248–257.

6. K. Kalgaonkar and B. Raj, “Recogniz-ing Talking Faces from Acoustic Dop-pler Re"ections,” Proc. IEEE Int’l Conf. Automatic Face and Gesture Recogni-tion, IEEE Press, 2008, pp. 1–6.

7. J.S. Garofolo et al., TIMIT Acoustic-Phonetic Continuous Speech Corpus, Linguistic Data Consortium, LDC cata-log no. LDC93S1, 1993.

8. K. Kalgaonkar and B. Raj, “Ultrasonic Doppler Sensor for Voice Activity Detec-tion,” IEEE Signal Processing Letters, vol. 14, no. 10, 2007, pp. 754–757.

9. S. Consolvo et al., “Activity Sensing in the Wild: A Field Trial of Ubi!t Garden,” Proc. 26th Ann. SIGCHI Conf. Human Factors in Computing Systems, (CHI 08), ACM, 2008, pp. 1797–1806.

10. K. Kalgaonkar and B. Raj, “Acoustic Doppler Sonar for Gait Recogination,” Proc. 2007 IEEE Conf. Advanced Video and Signal Based Surveillance (AVSS 07), IEEE CS Press, 2007, pp. 27–32.

11. P.H. Dietz, Apparatus for Detecting Guest Interactions and Method There-fore, US patent no. 6,307,952, Patent and Trademark Of!ce, 2001.

12. K. Kalgaonkar and B. Raj, “One-Handed Gesture Recognition Using Ultrasonic Doppler Sonar,” Proc. 2009 IEEE Int’l Conf. Acoustics, Speech and Signal Pro-cessing, (ICASSP 09), IEEE CS Press, 2009, pp. 1889–1892.

the AUTHORSBhiksha Raj is an associate professor in the Language Technologies Institute at Carnegie Mellon University. He also holds af!liate positions in the Electrical Engineering and Machine Learning departments at CMU. His research interests include automatic speech recognition, machine learning, sound and multi-media processing, discourse modeling, and sensing technologies for biometrics and human-computer interfaces. Raj received his PhD in electrical engineering from Carnegie Mellon University. Contact him at bhiksha@cs.cmu.edu.

Kaustubh Kalgaonkar was a member of the Center for Signal and Image Processing at the Georgia Institute of Technology until 2011. His research in-terests include speech enhancement, speech recognition in adverse acoustical environments, acoustic modeling, and machine learning for signal processing. Kalgaonkar received his PhD in electrical engineering from the Georgia Insti-tute of Technology. Contact him at kaustubh.kalgaonkar@gatech.edu.

Chris Harrison is a PhD student in the Human-Computer Interaction Institute at Carnegie Mellon University. He’s investigating how to “interact with small devices in big ways” using novel sensing technologies and interaction tech-niques. Harrison received his BA in computer science from New York University. He’s a recipient of a Microsoft Research PhD Fellowship and is editor in chief of XRDS magazine. Contact him at chris.harrison@cs.cmu.edu.

Paul Dietz is an assistant director of research in Microsoft’s Applied Sciences Group. His best-known projects include DiamondTouch (a platform for multi-touch research); Pal Mickey (a location-interactive plush toy sold in Walt Disney World); and iGlassware (the RFID, level-sensing drinking glass that automati-cally calls for re!lls). Dietz received his PhD in electrical engineering from Carnegie Mellon University. Contact him at paul.dietz@microsoft.com.

Selected CS articles and columns are also available for free at http://ComputingNow.computer.org.

PC-11-02-Har.indd 29 3/21/12 11:12 AM