Real-Time Dual-Band HapticMusic Player for Mobile Devices

Inwook Hwang, Student Member, IEEE, Hyeseon Lee, and Seungmoon Choi, Member, IEEE

Abstract—We introduce a novel dual-band haptic music player for real-time simultaneous vibrotactile playback with music in mobile

devices. Our haptic music player features a new miniature dual-mode actuator that can produce vibrations consisting of two principal

frequencies and a real-time vibration generation algorithm that can extract vibration commands from a music file for dual-band

playback (bass and treble). The algorithm uses a “haptic equalizer” and provides plausible sound-to-touch modality conversion based

on human perceptual data. In addition, we present a user study carried out to evaluate the subjective performance (precision,

harmony, fun, and preference) of the haptic music player, in comparison with the current practice of bass-band-only vibrotactile

playback via a single-frequency voice-coil actuator. The evaluation results indicated that the new dual-band playback outperforms the

bass-only rendering, also providing several insights for further improvements. The developed system and experimental findings have

implications for improving the multimedia experience with mobile devices.

Index Terms—Haptic I/O, vibration, music, real time, dual band, dual-mode actuator

Ç

1 INTRODUCTION

MULTIMODAL sensory displays have great potential forimproving user interfaces and task performance.

Visual and auditory displays are standard in the majorityof consumer electronic devices. Haptic displays are alsoapplied in an increasing number of applications, such asuser-interface (UI) components of mobile devices, specialeffects for entertainment, and information delivery invehicles. Haptic feedback is generally regarded as particu-larly effective in environments under sensory overload [1]and as ambient interfaces [2].

This paper is in line with the recent research thrusts

aiming to improve entertainment applications of mobile

devices with haptic feedback. We present a haptic music

player for “vibrotactile music,” which features the use of a

miniature vibration actuator with dual primary frequencies

and a real-time dual-band vibration generation algorithm.

1.1 Related Work

In the past 10 years, the potential of haptics for usability

enhancement has been persistently explored for mobile

devices. For example, Poupyrev et al. [2] developed a high-

performance vibrotactile actuator, named TouchEngine,

using a multiple-layer piezoelectric bending motor. They

proposed the use of vibrotactile feedback as ambient

sensory cues to assist user interaction. Rekimoto andSchwesig [3] added another input dimension by sensingthe finger pressure applied to a touchscreen and used thisinformation for modulating GUI responses. A usabilitystudy by Hoggan et al. [4] demonstrated that vibrotactilefeedback from virtual keypads improves the accuracy andcompletion time of text entry, while reducing mentalworkload. In addition, vibrotactile patterns can deliverabstract information, for example, tactile icons (tactons) thatindicate the phone alert type and priority [5]. A largenumber of haptic icons with high discriminability, learn-ability, and memorability can be designed using reliabledesign heuristics in the mobile context [6]. See [7] for acomprehensive review of the use of vibrotactile feedbackfor mobile devices.

The present study addresses multimodal UIs, wheremultiple sensory channels are stimulated for informationpresentation. Crossmodal icons that combine intuitivelysimilar earcons and tactons are good examples [8]. The topicof this work, enhancement of music listening experience bythe simultaneous playback of cutaneous vibration, also hasreceived increasing attention. Even though the aestheticvalues of vibrotactile music have not been elucidated [9],the industry took a rapid step forward and released severalproducts with this functionality onto the consumer market(e.g., the mobile phone Galaxy S3 and the MP3 player YP-P3from Samsung Electronics). The MOTIV studio for mobiledevices from Immersion Corp. is another notable commer-cial solution. The MOTIV studio provides a convenientauthoring feature of automatic vibration generation fromparsed sound information in a MIDI file or directly from awave file. The basic operations of these commercialproducts are the same as those proposed by Chang andO’Sullivan [10], where vibrotactile patterns are created fromsound signals in a low-frequency bass band. This approachwas reported to amplify the sense of beat and improve theperception of sound quality.

340 IEEE TRANSACTIONS ON HAPTICS, VOL. 6, NO. 3, JULY-SEPTEMBER 2013

. I. Hwang and S. Choi are with the Haptics and Virtual Reality Laboratory,Department of Computer Science and Engineering, POSTECH, Hyoja-dong, Nam-gu, Science Building #4-115, Pohang, Gyungsangbuk-do 790-784, Republic of Korea. E-mail: {inux, choism}@postech.ac.kr.

. H. Lee is with the Department of Industrial and Management Engineering,POSTECH, Hyoja-dong, Nam-gu, Science Building #4-314, Pohang,Gyungsangbuk-do 790-784, Republic of Korea.E-mail: hyelee@postech.ac.kr.

Manuscript received 29 Mar. 2012; revised 5 Feb. 2013; accepted 13 Feb. 2013;published online 27 Feb. 2013.Recommended for acceptance by D. Grant.For information on obtaining reprints of this article, please send e-mail to:toh@computer.org, and reference IEEECS Log Number TH-2012-03-0023.Digital Object Identifier no. 10.1109/TOH.2013.7.

1939-1412/13/$31.00 � 2013 IEEE Published by the IEEE CS, RAS, & CES

Overall, the current research status of simultaneousplayback of audio and vibrotactile music via a mobiledevice can be regarded as immature. High-performancecommercial vibration actuators that are suitable for thispurpose are not available. We also need signal conversionalgorithms that adequately consider the human perceptionof sound and vibration. The feasibility of their real-timeoperation with a mobile processor also needs to bevalidated. Furthermore, the subjective responses of usersto this new functionality remain largely unexplored.

In the more general haptics literature, transfer of speechinto vibrotactile stimuli has been actively studied since thepioneering work of Gault [11] in the early 20th century.Bernstein et al. [12] investigated the effectiveness of sixspeech-to-tactile transformation methods for conveying theintonation and stress information contained in speech.Brooks and Frost [13] found that lipreading correctnessimproved from 39 percent to 88 percent with the use of atactile vocoder. Their system was equipped with 16 voicecoilactuators, and each actuator conveys the speech informa-tion contained in one of the 16 logarithmically equidistantfrequency intervals between 200 and 8,000 Hz. Thisapproach was also used in [14] to directly transfer musicto multiple tactile stimulation sites on the torso of thehearing impaired.

Apart from this, research on computer music hascontinued to develop algorithms for automatic featureextraction from music. For instance, Schierer [15] proposeda method for extracting the tempo and beat of mbyiteratively matching original and reconfigured signals,providing 68 percent correctness for 60 songs of variousgenres. To identify the beat information in music, Mayor[16] used a filter bank and extracted the energy of each bandfollowed by a maximum correlation search with multiplehypotheses of the beat speed. Zils et al. [17] extractedpercussive sounds from polyphonic music by matchingsound patterns to simulated drum sounds. Ma et al. [18]developed a user attention model for video summarizationthat included an aural saliency model and attention modelsfor speech and music. However, at present, the applicabilityof computer music algorithms to simultaneous tactilefeedback remains unclear, requiring further investigation.

Finally, there has been an increasing interest in addingtactile feedback to multimedia content. For example,Kim et al. [19] developed an authoring system thatfacilitates design of the tactile effects that are playedsynchronously with the audiovisual content of a movie toprovide immersive multimedia experience. Our groupproposed a real-time video-to-vibrotactile translation algo-rithm that automatically estimates visually salient regionson a screen and emphasizes these regions with a 2D arrayof vibrators [20]. Such methods may contribute to the birthof the new field of haptic broadcasting [21], [22]. In thiswider context, our study belongs to the class of autono-mous tactile effect synthesis algorithms from audiocontent with real-time performance.

1.2 Article Overview

In the remainder of this paper, we present a haptic musicplayer for mobile devices developed to enrich music

listening experience. Our haptic music player has the

following four major features.First, we use a new miniature actuator called the dual-

mode actuator (DMA). The DMA can produce vibrations

composed of two principal frequencies, which can lead to

greater diversity in vibrotactile perception [23], [24], [25].

This is in contrast to the vast majority of commercial mobile

devices that use a simple actuator, for example, an eccentric

rotating mass (ERM) or a linear resonance actuator (LRA).

The dynamic performance of these actuators is insufficient

for creating expressive vibrotactile effects for haptic music.Second, our haptic music player enables real-time, on-the-

fly playback of vibrotactile effects. Since thousands of new

musical pieces are published every year, producing

vibrotactile music directly from musical sources without

any preprocessing is a highly desirable benefit. Our

vibration generation algorithms satisfy this requirement

using digital signal processing techniques with very low

computational complexity.Third, our haptic music player supports dual-band

vibration playback. As reviewed earlier, previous attempts

for haptic music playback relied on the rhythms or beats

extracted from the bass band of sound signals. This was

attributable partly to the performance limits of the vibration

actuators used. Other salient aspects of music, such as a

singer’s voice or guitar solo, were ignored. In our haptic

music player, the rhythmic variations in music are encoded

in a low-pitch vibration signal (bass band), whereas high-

frequency salient sounds are transmitted in a high-

frequency signal (treble band), both produced by one

DMA. To deal with high-frequency music saliency that

varies greatly among music genres, we introduce the

concept of a haptic equalizer. The haptic equalizer mixes

the signal energies from different frequency bands using

genre-dependent weights analogously to an audio equalizer.Finally, in our haptic music player, all the conversion

and scaling processes between sound and vibrotactile

signals are based on perceptual data taken from the relevant

literature. Human perception of vibrotactile stimuli is

complex and depends on various factors such as signal

frequency, contact site, and contact area. However, the

previous methods of vibrotactile music tended to control

the physical amplitude of vibrotactile stimulus, without

explicit consideration of their perceptual consequences. We

also compensate for the actuator input/output relationships

to minimize the perceptual distortions that might occur

otherwise because of the actuator dynamics and the human

perception process.In addition, we conducted a user study to evaluate the

perceptual merits of our dual-band vibrotactile music

playback in comparison with the conventional single-band

playback. Two types of actuators (LRA and DMA) were

used, and 16 musical pieces were selected to represent four

music genres (rock, dance, classical, and vocal; four pieces

each). The experimental results elucidated the benefits of

dual-band playback and their dependence on music genre,

also providing insights that can facilitate further improve-

ments of simultaneous audio-haptic music playback.

HWANG ET AL.: REAL-TIME DUAL-BAND HAPTIC MUSIC PLAYER FOR MOBILE DEVICES 341

2 VIBRATION ACTUATORS

This section describes the characteristics of the vibrationactuator used in our haptic music player.

2.1 Vibration Actuators for Mobile Devices

During the past decade, many types of vibration actuatorshave been developed for mobile devices. However, onlytwo types, ERM and LRA, are widely used in consumermobile devices. ERM is a DC motor that has a rotor with aneccentric mass distribution to induce large centrifugalacceleration. This structure produces 2D vibration in alloutgoing directions on the rotational plane of the rotor.ERMs are small and inexpensive. A drawback is that theirvibration frequency and amplitude are both determined byan input voltage level, so they cannot be controlledindependently. This is a serious obstacle against expressivevibrotactile rendering. They also have a slow and nonlinearresponse with large actuation delays. These problemsconfine the use of ERMs mostly for alerts. See [7] forfurther details.

An LRA is a voice-coil actuator with mass and springcomponents that are connected linearly along the same axis.Vibration is produced by the mechanical resonance of thetwo components. The vibration direction is parallel to theaxis. Since LRA has a fast and linear response, it has beenregarded as an adequate choice for touchscreen interaction.However, it has a very narrow frequency bandwidth (only afew hertz wide) centered at the resonance frequency. Thus,the perceptual impression of LRA vibrations is monotone,which precludes rendering of diverse vibrotactile pitchesfor music playback.

Efforts to embed more sophisticated actuators, forexample, piezoelectric and electro-active polymer actuators,into mobile devices are ongoing. However, their industrialadoption has been rare because of stringent industryrequirements, such as size, reliability, and durability toexternal shock.

2.2 Dual-Mode Actuator

DMA is a new vibration actuator developed by LGElectronics [26], [27]. The DMA is based on the sameworking principle as an LRA, but it has a more advanceddesign as shown in Fig. 1. The DMA includes two built-inmass-spring elements with different resonance frequencies.The two elements share the magnetic field of a common coillocated in the center. Each element responds only when thecommon input to the coil includes spectral energy around

its resonance frequency. Thus, a single voltage input withsuperimposed frequencies can control both frequencycomponents independently. This use of the common coil

enables compact size and small power consumption(10� 10� 3 mm; 0.1 W max) suitable for mobile devices.

Given the two resonance frequencies, f1 and f2, all themechanical parameters can be determined. As such, theDMA has a frequency response with two distinct peaks at

the two resonant frequencies. An example of response withf1 ¼ 150 Hz and f2 ¼ 223 Hz is shown in Fig. 2a. f2 wasselected to be within the frequency range to which humansare most sensitive [28]. To elicit distinctive sensations fromf2, f1 was chosen as the lowest frequency from f2 while

satisfying the requirements for actuator size and vibrationstrength. The DMA responses are fairly linear at both f1 andf2. Fig. 2b shows the time-domain response of the DMA fora superimposed sinusoidal input. The rising time and

falling time of the DMA are around 50 ms and 100 ms,respectively, similar to that of the LRA. This prototypemodel was used throughout this study.

We used the following voltage input V ðtÞ to drive theDMA:

V ðtÞ ¼ V1 sinð2�f1tÞ þ V2 sinð2�f2tÞ; ð1Þ

V1 þ V2 � Vrated; ð2Þ

where Vrated is the rated voltage of DMA. This superposition

rule creates a vibration output that contains the twofrequency components, f1 and f2. This dual-frequencystimulation is a unique advantage of DMA. Further detailson the mechanical design of DMA are available in its patent

specifications [26], [27].

342 IEEE TRANSACTIONS ON HAPTICS, VOL. 6, NO. 3, JULY-SEPTEMBER 2013

Fig. 1. Internal structure of the DMA.

Fig. 2. Example responses of the DMA.

2.3 Perception of Superimposed Vibration

The DMA can produce superimposed vibrations thatconsist of two frequency components, and this capabilitycan diversify the perceptual impression of the vibrations toa large extent. Bensmaıa et al. [23] studied the discrimin-ability between simple sinusoidal and superimposedvibrations and demonstrated that it can be very high(sensitivity index d0 as high as 3.0). This finding was furtherunderstood by the perceptual space of amplitude-modu-lated sinusoidal vibrations1 presented by our group [24].This distinctiveness results from the presence of a low-frequency beat phenomenon, which is easily detectable insimultaneously rendered sinusoidal stimuli that differ byonly a few hertz [25]. Our group also evaluated theconsonance of “vibrotactile chords” (vibration signalscomposed of multiple frequency components) by adaptingthe consonance concept of musical chords [29]. The resultssuggested that vibrotactile beat was responsible for theperception of dissonant vibrotactile signals. All of theseprevious studies indicate that simple sinusoidal vibrationsand superimposed vibrations can possess highly distinctiveperceptual characteristics. Thus, the DMA has a highpotential for expressive vibrotactile rendering in mobiledevices.

3 HAPTIC MUSIC PLAYER

The software structure and algorithms of the haptic musicplayer are described in this section.

3.1 Software Structure

Fig. 3 shows the overall structure and computationalprocesses of our haptic music player. The first step is toopen a music source file and store it as a timed sequence.Each element in the sequence represents a sound ampli-tude. The sequence is then partitioned into a number ofsegments with the same length. Fast Fourier transform(FFT) is applied to each signal segment to extract thespectral densities between 0 Hz and the Nyquist frequency(half of the music sampling frequency). The segment lengthcan be determined based on the processing power of thecomputing platform.

The spectral densities are divided into two bands, whichare denoted as “bass” and “treble” in Fig. 3. The bass bandis used to extract the beat information. This information isplayed through the superimposed, perceptually low-pitchvibrations of the DMA. The treble band, which is unique toour haptic music player, tracks high-frequency salient

features of music. These features are delivered by thehigh-frequency signal of DMA. To handle the saliencydependence on music genres, the treble band is furtherpartitioned into many subbands. The signal energies of thesubbands are then merged using a haptic equalizer withmixing weights dependent on music genre.

The final step is a nonlinear scaling procedure thatconverts the extracted audio energies into the voltagecommands of DMA in (1). The overall conversion processis: sound signal energy! auditory perceived magnitude!vibratory perceived magnitude! physical vibration ampli-tude ! voltage command amplitude to the DMA. Thistransformation uses appropriate perceptual data acquiredfrom the literature.

The entire procedure is repeated as a loop for each musicsignal segment until the end of music playback. For real-time performance, our computational algorithms are de-signed to be as efficient as possible while maintainingperceptual plausibility, as described further in the remain-der of this section. The current implementation is tailored tothe DMA, but it can be extended to other widebandvibration actuators.

3.2 Haptic Equalizer

Musical instruments have their own frequency band. Forexample, the general frequency band of human voice is 80-1,300 Hz, while that of a drum is 50-1,000 Hz. We alsoexamined the frequency ranges of various pop songs andfound that the bass accompaniment was in 50-200 Hz, vocalsound was in 200-1,500 Hz, while the treble percussivesound was around 4,000 Hz. Thus, to fully exploit the dual-vibration playback mode, feature extraction algorithmsneed to handle this wide frequency range. Our approachfor this requirement involves the use of a haptic equalizersimilar to the audio equalizer found in audio systems andtraditional tactile vocoders [13].

We partition a music signal of frequencies ranging from0 to 6,400 Hz into the bass and treble bands at a 200-Hzboundary. This is the most common setting for bass-trebleseparation in audio systems. The treble band is furtherdivided into five subbands, linearly in a log scale: 200-400,400-800, 800-1,600, 1,600-3,200, and 3,200-6,400 Hz. Separatemixing weights are assigned to each of the five frequencybands. Each weight determines the amplification gain of thecorresponding frequency band.

Table 1 shows the preset weights used in our hapticmusic player for four representative music genres: rock,dance, classical, and vocal. Their initial values were takenfrom the equalizer gains of a popular music player program(jetAudio; Cowon Systems Inc.). These weights were thentuned manually to better express the genre characteristics

HWANG ET AL.: REAL-TIME DUAL-BAND HAPTIC MUSIC PLAYER FOR MOBILE DEVICES 343

Fig. 3. Process loop of the haptic music player.

1. Amplitude-modulated and superimposed vibrations have similarsignal envelopes and perceptual characteristics. They become equivalent iftwo vibrations with the same amplitude are superimposed.

via touch. The preset weights for rock music emphasize thelow-frequency bass guitar sounds and the high-frequencypercussive drum sounds. The weights used for dance musicwere adjusted slightly from the rock preset for milderexpression. The classical preset has evenly distributedweights with some emphasis on 400-800 Hz, which is themain frequency band of many classical music instruments.The focus of the vocal preset is on the high tone voice of asinger. A user can freely adjust the preset weights accordingto their preference, thereby providing maximum controlto the user.

3.3 Modality Conversion and Intensity Scaling

The signal energies that are divided and weighted by thehaptic equalizer are converted into voltage commands toDMA, V1ðnÞ and V2ðnÞ in (1), following several computa-tional steps that rely on relevant perceptual data. Comparedwith previous approaches, this step is unique to our hapticmusic player.

3.3.1 Computation of Sound Signal Energies

Let EbassðnÞ be the sound signal energy of the bass band andEiðnÞ be that of the ith subband in the treble band. The firststep is to compute the signal energies by summating the

absolute spectral magnitudes over all frequencies in thecorresponding band.

3.3.2 Conversion to Auditory Perceived Magnitudes

We use a two-step procedure to convert the sound signalenergies to two auditory perceived magnitudes, LbassðnÞand LtrebleðnÞ, for the bass and treble bands, respectively.First, we compute intermediate auditory perceived magni-tudes, AbassðnÞ and AtrebleðnÞ, as

AbassðnÞ ¼EbassðnÞ

l

� �e; ð3Þ

AtrebleðnÞ ¼X5

i¼1

wiEiðnÞl

( )e

; ð4Þ

where l is the length of a signal segment and wi is theweight of the ith subband in the haptic equalizer. SinceEðnÞ represents the integrated energy in a wide frequencyband, its exact transformation to auditory loudness can bevery complex, and it is more so with perceptual weights.Equations (3) and (4) are plausible approximations based onStevens’ power law for fast computation. Stevens’ powerlaw provides a well-defined relation between physical andperceptual magnitudes for simple auditory stimuli [30].

In our implementation, the exponent e ¼ 0:67 was derivedfrom the experimental data for 3-kHz tone loudness [30].

Next, we calculate an additional gain gðnÞ for the trebleband and then determine LbassðnÞ and LtrebleðnÞ, such that

LbassðnÞ ¼ AbassðnÞ; ð5Þ

LtrebleðnÞ ¼ gðnÞAtrebleðnÞ: ð6Þ

During our initial implementation, we realized that sub-band sound is very salient in terms of perception when theenergy of the sub-band is significantly greater than those ofthe other sub-bands, for example, during the solo perfor-mance of an instrument. However, the total signal energy ofthe treble band reflected in AtrebleðnÞ may not be strongenough to deliver this dominant band effect. To compensatefor this, we introduce gðnÞð1 � gðnÞ � 2Þ. If AtrebleðnÞ � At,

gðnÞ ¼ 1þAt �AtrebleðnÞAt

wmaxEmaxðnÞP5i¼1 wiEiðnÞ

: ð7Þ

If AtrebleðnÞ > At; gðnÞ ¼ 1. Here, EmaxðnÞ represents theenergy of the sub-band with the maximum energy, andwmax is the weight of that sub-band. The rightmost termconverges to 1 if EmaxðnÞ approaches the total energy of thetreble band. The middle term that includes AtrebleðnÞreduces the effect of dominant band amplification if thetotal treble band energy increases, preventing overcom-pensation. Dominant band amplification is activated onlyif AtrebleðnÞ � At, also to avoid overcompensation. We setAt to the 90 percent level of the range of auditoryperceived magnitudes.

3.3.3 Conversion to Vibratory Perceived Magnitudes

Matching the scales of perceived magnitudes betweensound and vibration requires great care. The absolutesound levels in music files vary significantly from file tofile, and vibration actuators also have different ranges ofproducible vibration strength. Our approach relies on ourprevious study [31], which presented a psychophysicalmagnitude function of vibration frequency and amplitudeto the resulting perceived magnitude in a wide parameterrange for a mobile device held in the hand. One canmeasure the range of vibration amplitudes generated by anactuator at a given frequency, input this range into theperceived magnitude function, and obtain a range ofvibratory perceived magnitudes. The range of auditoryperceived magnitudes can then be scaled to match the rangeof vibratory perceived magnitudes. Nonetheless, the phy-sical signal level of music files is still beyond our control,which demands a “haptic volume” control.

After the scales are determined for the auditory andvibratory perceived magnitudes, we can compute thedesired perceived magnitudes of vibration, IbassðnÞ andItrebleðnÞ, from LbassðnÞ and LtrebleðnÞ as follows:

IbassðnÞ ¼ wbasscLbassðnÞ; ð8Þ

ItrebleðnÞ ¼ wtreblecLtrebleðnÞ; ð9Þ

where c is the cross-modal scaling constant, and wbass andwtreble are the amplification gains of the bass and treblebands in charge of the haptic volume control.

344 IEEE TRANSACTIONS ON HAPTICS, VOL. 6, NO. 3, JULY-SEPTEMBER 2013

TABLE 1Preset Weights of Haptic Equalizer for Music Genres

3.3.4 Rendering Mode Selection

To drive the DMA, a subtle adjustment is required on theperceived intensity because of its superposition mode.Activating the DMA with positive V1ðnÞ and V2ðnÞ using(1) generates superimposed vibration of the frequencies f1

and f2. The sensation of such vibration is much rougher andfeels like a lower frequency than f1 or f2 [24], [25].Therefore, we use this superposition mode to render basssignals. When the treble intensity is dominant, we onlyuse vibration frequency f2, i.e., V1ðnÞ ¼ 0. Furthermore, weobserved via a magnitude matching experiment that thesuperimposed vibration of two equally intensive vibrationswith frequencies f1 and f2 has a perceived magnitude thatis about 1.25 times higher than the perceived magnitude ofeach individual vibration. To compensate for this, we scaledown the individual perceived magnitudes of the twovibrations by 0.8 times in the superposition mode so thatthe resulting superimposed vibration would have the sameperceived magnitude as the bass band auditory perceivedmagnitude IbassðnÞ. In summary, the desired perceivedmagnitudes for two vibration frequencies f1 and f2, P1ðnÞand P2ðnÞ, are as follows: If IbassðnÞ > ItrebleðnÞ, then

P1ðnÞ ¼ 0:8IbassðnÞ and P2ðnÞ ¼ 0:8IbassðnÞ: ð10Þ

If IbassðnÞ � ItrebleðnÞ, then

P1ðnÞ ¼ 0 and P2ðnÞ ¼ ItrebleðnÞ: ð11Þ

An example demonstrating this rule is shown in Fig. 4.The high-frequency unit (for f2) is always activated,whereas the low-frequency unit (for f1) is activated onlywhen the bass component is dominant.

3.3.5 Conversion to Physical Vibration Amplitudes

The desired perceived magnitudes of the two vibrations,P1ðnÞ and P2ðnÞ, can be readily converted to the desiredvibration amplitudes at frequencies f1 and f2 using theinverse of the perceived magnitude function of vibratorystimuli [31].

3.3.6 Conversion to Voltage Command Amplitudes

The final step is to convert the desired vibration amplitudesto voltage command amplitudes for f1 and f2. To do this,we need the I/O mappings of the DMA for each frequency,

which are derived from input voltage amplitude to outputvibration amplitude measured when the DMA is attachedto a mobile device. This I/O calibration can be done easily(e.g., see [32]), especially for the DMA which has fairlylinear responses at both resonance frequencies. The inputvoltage amplitudes V1ðnÞ and V2ðnÞ are determined usingthese I/O relations.

3.4 Implementation and Processing Speed

We implemented the algorithms described above on an MSWindows platform, using MS Visual C++ 2008 withexternal libraries for music file I/O (Audiere 1.9.4) andFFT calculation (FFTw 3.2.2). The haptic music player ranon a desktop PC (3-GHz Intel Core 2 Duo) because of thedifficulty of custom signal I/O for the DMA on commercialmobile platforms.

A parameter critical for the performance of our hapticmusic player is the length of a music segment processedin each loop. Increasing the segment length improvesthe frequency resolution in spectral density estimation.However, it degrades the smoothness of vibration updatesand also leads to a longer processing time because of theincreased computational load for FFT and subsequentoperations.

After extensive tests, we set the length of a music segmentto 50 ms as the best trade-off. This value allows a 20-Hzupdate rate for vibration playback using each 2,205 samplesfrom a music source sampled at 44.1 kHz. This update rate issufficient for smooth transitions between music segmentswithout causing any perceptible discontinuity and also forfast responses synchronized with audio playback.

In our desktop system, a single loop took 0.3 ms onaverage for vibration command extraction. We also portedthese Windows codes to Android to assess viability onmobile platforms. When tested with a smartphone (Sam-sung Electronics; Galaxy S2; without generating vibrations),the Android version took 1.0 ms, on average, for processinga single loop, including the most time-consuming audio filedecoding. Compared to the 50-ms processing interval for20-Hz updates, this performance clearly allows for real-timerendering with a very low computational burden. Note thatthe current mobile devices are equipped with a fastermulticore CPU than one included in the smartphone usedfor our test.

4 USER STUDY

We evaluated the subjective performance of our dual-bandvibration extraction algorithm compared with the bass-band-only algorithm for four music genres via a user study.Details are presented in this section.

4.1 Methods

4.1.1 Participants

Twenty-four university students (12 males and 12 females)participated in this experiment. They were 18-30 years oldwith a mean 22.3 (SD 3.5). Young participants werepreferred as they are generally more enthusiastic inlistening to music and accepting new technology andinterfaces. All participants were daily users of a mobilephone with no known sensorimotor impairment. They werepaid KRW 20,000 (’ USD 17) for the experiment.

HWANG ET AL.: REAL-TIME DUAL-BAND HAPTIC MUSIC PLAYER FOR MOBILE DEVICES 345

Fig. 4. Example of input signal to a DMA for dual-band playback.

4.1.2 Apparatus

We used LRA (LG Innotek; model MVMU-A360G) andDMA (LG Electronics; a prototype model) as a vibrationactuator. The resonance frequency of LRA was 178 Hz,while those of DMA were 150 and 223 Hz. Each actuatorwas attached to one wide face of a handheld mockup madefrom acrylic resin (105� 45� 15 mm), as shown in Fig. 5.The actuators were not in direct contact with the partici-pants’ hand. Their input-output relations were calibratedusing a miniature accelerometer (Kistler; model 7894A500;7.5 g) attached to the center of the mockup. The actuatorswere controlled by a PC via a data acquisition board(National Instruments; model USB-6251) with a custom-made power amplifier. The sampling rate for signal I/Owas 10 kHz for faithful signal sampling and reconstruction.

4.1.3 Experimental Conditions

This study consisted of 16 experimental conditions(2 rendering modes � 2 actuators � 4 music genres) in awithin-subject design. For vibration rendering, we usedtwo methods: single- and dual-band modes (SINGLEversus DUAL). The single-band mode represents thecurrent standard and presents only a bass-band signal.The dual-band mode, the main function of our work,provides both bass- and treble-band vibrations, which isunique to our haptic music player.

As an actuator, LRA or DMA was used. When an LRAwas used, the single-band mode produced 178-Hz reso-nance vibrations using only bass signals, i.e., P1ðnÞ ¼IbassðnÞ and P2ðnÞ ¼ 0. In the dual-band mode, both bassand treble signals were encoded in the 178-Hz vibrationsusing the same algorithm as the DMA. The specialmagnitude adjustments required for the superpositionmode of DMA are not necessary for LRA, thus (10) becameP1ðnÞ ¼ IbassðnÞ and P2ðnÞ ¼ 0 and (11) remained intact.When the DMA was used, the single-band mode usedsuperimposed vibrations (mix of 150- and 223-Hz vibra-tions) to render bass signals. In the dual-band mode, basssignals were expressed by the superposition method, whiletreble signals were in 223-Hz vibrations. The maximumvibration amplitude was set to level 6 of the perceivedintensity model in [31] (about 0.5, 0.6, and 0.8 G at 150, 178,and 223 Hz, respectively).

In pilot experiments, we realized that participants’preference of vibration playback depends on music genreto a great extent. Thus, we included music genre as anindependent factor in the experiment. We selected four

genres of rock, dance, classical, and vocal, and chose fourmusic pieces per genre, as listed in Table 2. They arefamiliar music pieces to our Korean participants containingthe style representative of the corresponding genres. Forplayback, we trimmed 1 minute of each musical piece andconcatenated them for each music genre. Each music clipwas played with the preset weights of the correspondinggenre shown in Table 1. This set of equalizer gains wasfound by the experimenter to best express the genrecharacteristics via vibrotactile stimulation. For all music,haptic volume was set to be identical by the experimenter:c ¼ 0:001 and ðwbass : wtrebleÞ ¼ ð6 : 7Þ.

Both equalizer gains and haptic volume can affect thesubjective preference of vibration playback, but we wereunable to include them as independent factors in theexperiment because of the large number of continuousvariables involved (20 for equalizer gains and 3 for hapticvolume). Instead, we used the fixed values found by theexperimenter to be the best, concentrating more on our majorinterests (the effects of rendering method and actuator).

4.1.4 Subjective Performance Measures

We collected four subjective measures using a questionnairein a 0-100 continuous scale. They were as follows:Precision—“Did the vibration express the music precisely?”(0: very imprecise, 100: very precise); Harmony—“Was thevibration harmonious with the music?” (0: very inharmo-nious, 100: very harmonious); Fun—“Was the vibrationfun?” (0: very boring, 100: very fun); and Preference—“Didyou like the vibration?” (0: dislike very much, 100: like verymuch). The participants also described the subjectiveimpressions of vibrotactile feedback in a free form.

4.1.5 Procedure

Prior to the experiment, each participant was giveninstructions about the experimental procedures andexplanations of the meaning of the questions in thequestionnaire. A training session was then followed,where two songs that were not used in the main sessionswere played using each of the four vibration renderingconditions (2 modes � 2 actuators) for 2 min.

346 IEEE TRANSACTIONS ON HAPTICS, VOL. 6, NO. 3, JULY-SEPTEMBER 2013

Fig. 5. Handheld mockups with two vibration actuators (LRA and DMA)used in the user study.

TABLE 2The 16 Genre-Representative

Musical Pieces Used for Evaluation

The main experiment consisted of four sessions. Eachsession used one of the 4-min genre-representative musicclips. At the beginning, the music clip was played withoutvibrotactile playback using over-ear headphones to provideperceptual reference. The participant could adjust audiovolume to a comfortable level. Then, the music clip wasplayed with vibration using one of the four renderingconditions. After the playback, the participant answered thequestions on the rendering conditions using the question-naire sheets. For each performance metric, the participantgave a score by marking a position on a line labeled on bothends with their meanings. The participant had rest for a fewminutes to prevent tactile adaptation before proceeding tothe next rendering condition. This procedure was repeatedfour times with different rendering conditions.

To remove any possible order effects, we randomized theorders of the rendering conditions in each session and thoseof the music genres. The entire experiment took about2.5 hours, and each participant finished it in two days (twomain sessions per day).

4.2 Results

A within-subject three-way analysis of variance (ANOVA)was conducted, where rendering mode, actuator, and musicgenre were fixed-effect factors and subject was treated as arandom effect factor. To see the influence of the renderingmode (DUAL versus SINGLE) and actuator (DMA versusLRA) on the results more clearly, we also provide theanalysis results classified by music genre. In the analysis,we excluded data of one male participant, treating him asan outlier. Most of his data lied outside of the confidenceintervals, and he reported that he was a hip-hop dancertrained to respond to bass-beat sounds.

4.2.1 ANOVA Results

We presented the results of the three-way ANOVA with theeffect sizes (�2) of the main effects in Table 3. In allmeasures, the effects of rendering mode (R) and musicgenre (M) were strongly significant (p < 0:01), while theeffect of actuator (A) was not. Interactions of A�M andR�A�M were significant (p < 0:05) in fun and marginallysignificant (p < 0:1) in preference. R�A�M also hadmarginal significance in harmony.

Since our main interest was finding the effects of A andR, we then conducted a two-way ANOVA for each of thefour music genres. The results are summarized in Table 4.

In most cases, R had very significant effects, consistent withthe results of the three-way ANOVA (DUAL>SINGLE). Theeffect of A was significant in fun and preference of dancemusic and in preference of classical music. Marginalsignificance of A was also seen in harmony of classicalmusic and precision of vocal music. The interaction effectR�A was significant in fun and marginally significant inthe other three measures of dance music.

Overall, the results indicated that the dual-band render-ing, the main function of our haptic music player, improvedthe users’ evaluations of music listening compared with thecurrent standard of single-band, bass-only rendering.

4.2.2 Effects of Rendering Conditions

Score differences were analyzed among the four renderingconditions. The average evaluation scores in Fig. 6 showmuch higher scores for DUAL than for SINGLE in allmeasures, while the scores of the DMA and LRA weremostly comparable. We separated the scores by music genreas shown in Fig. 7 and conducted the Student-Newman-Keuls (SNK) multiple comparison test. Table 5 shows thegrouping results. We summarized noteworthy results below,emphasizing comparisons between DUAL-DMA (our maincontribution) and SINGLE-LRA (current standard).

For rock music, in precision and harmony, significantdifferences (p < 0:05) were found between DUAL-DMAand the two SINGLE conditions, and marginally significantdifferences (p < 0:1) were found between DUAL-DMA andDUAL-LRA, with the much higher scores of DUAL-DMA.

HWANG ET AL.: REAL-TIME DUAL-BAND HAPTIC MUSIC PLAYER FOR MOBILE DEVICES 347

TABLE 3Three-way ANOVA Results (F -ratios)

with Effect Size (in parentheses)

* : p < 0:10, ** : p < 0:05, *** : p < 0:01

TABLE 4Two-Way ANOVA Results for the Four Music Genres

Fig. 6. Average evaluation results of the four rendering conditions. Errorbars represent standard errors.

In particular, DUAL-DMA and SINGLE-LRA showedsignificant differences in precision and harmony. Hence, itcan be said that DUAL-DMA, which acquired the highestscores in all the four measures, was the best renderingmode for rock music.

For dance music, a significant difference was seenbetween SINGLE-DMA and the two DUAL conditions inprecision. In the other three measures, significant differ-ences were between SINGLE-DMA and the other threeconditions. In all of these cases, SINGLE-DMA exhibited thelowest scores. Between DUAL-DMA and SINGLE-LRA, nosignificant difference was present in any of the fourmeasures, although the scores of DUAL-DMA were higher.In summary, SINGLE-DMA was the lowest-rated renderingcondition for dance music, and the two DUAL conditionswere comparable without evident effects of the actuator.

For classical music, a significant difference was found inprecision between the DUAL and SINGLE conditions. Inharmony, fun, and preference, DUAL-LRA and the otherthree conditions showed significance differences. In all ofthese cases, DUAL-LRA resulted in the highest scores.DUAL-DMA and SINGLE-LRA showed significance differ-ences in precision and fun, with the higher scores ofDUAL-DMA.

For the vocal-oriented songs, a significant differenceoccurred in precision between SINGLE-DMA and the other

conditions, with the lowest score of SINGLE-DMA. In fun,there was a marginally significant difference betweenDUAL and SINGLE. No significant difference was seenbetween DUAL-DMA and SINGLE-LRA in any of the fourmeasures, despite the higher scores of DUAL-DMA. Hence,the DUAL conditions were generally better than theSINGLE conditions for vocal music.

Overall, our new rendering method, DUAL-DMA,scored the highest for rock, dance, and vocal music, exceptclassical music where DUAL-LRA obtained the best scores.DUAL-DMA also received higher scores than the currentstandard method, SINGLE-LRA in all the measures for allthe genres, with statistical significance in four (out of 16)cases. The benefit of DUAL-DMA was the most evident forrock music.

4.3 Discussion

The user study elucidated the benefit of our dual-bandhaptic music player for improving the music listeningexperience. The dual-band rendering with DMA acquiredhigh subjective scores, especially for rock music among thefour music genres. The evaluation results also indicatedadequate use of single-mode versus dual-mode renderingfor each music genre.

The comments of the participants collected after theexperiment were quite diverse. Common ones are reportedbelow along with the major experimental results. First of all,48 percent participants (11 of 23) said that a vibrationrendering method should be tailored to a music genre. Thisincluded a choice of single- or dual-mode and individuallycustomizable weights of the haptic equalizer.

Most participants evaluated vibration playback of thedual-band mode higher than that of the single-band mode.The bass components of music mainly contain regular beatsounds played by drums and bass instruments. Theparticipants reported frequently that regularly repeatedvibration beats under the single-band conditions weresomewhat flat and boring. This was some exacerbated withthe classical music that had low bass-band energy, wherethe single-band rendering conditions were not able toprovide clear vibrotactile sensations for beats. In contrast,the dual-mode rendering attempts to add the playback ofthe main aspects of music, such as theme and melody. Thisbehavior seems to have resulted in the large scoredifferences between DUAL and SINGLE for classical music.

Between the two DUAL conditions, the effects of the twoactuators greatly depended on music genre. The DMA

348 IEEE TRANSACTIONS ON HAPTICS, VOL. 6, NO. 3, JULY-SEPTEMBER 2013

Fig. 7. Evaluation results of the four rendering conditions by music genre. Error bars represent standard errors.

TABLE 5Grouping of Rendering Methods by the

SNK Test (� ¼ 0:05; � ¼ 0:1 in Parentheses)

The rendering methods represented by the same alphabet belonged tothe same group.

outperformed the LRA for rock music, while the LRA wasbetter for classical music. For rock music, many participantsreported that they liked short and highly contrastedvibrations instead of continuous vibrations. DMA couldexpress the intensive drum beats of rock music adequatelywith superimposed vibrations giving rough sensations withhigh-contrast to high-frequency vibrations. On the otherhand, many participants recommended fine and delicatevibrations for classical music. The rough feeling of super-imposed vibration does not seem to match with classicalmusic, while the smooth sensations of single sinusoidalvibration by LRA appear to be more preferred.

In addition, most participants expressed strong prefer-ence on vibration strength. During the user study, theexperimenter sets the vibration volume, and the participantswere not allowed to change it. Some participants com-plained of fatigue because of long strong vibration, whereasothers complained of vibration strength being too weak.

Finally, some notable descriptions of the participants forthe four rendering conditions are provided: 1) DUAL-LRA—“The vibration is smooth and matches well with themusic, but it is somewhat flat and boring.” 2) SINGLE-LRA—“The vibration is good at beat expression, but it is toosparse, regular, and weak.” 3) DUAL-DMA—“The expres-sion of bass beats is good. It is also more fun and feels like awell-tailored vibration to music.” 4) SINGLE-DMA—“Theexpression of strong bass beats is good. However, it focuseson the bass sound too much and is a little boring.” Thesesubjective reports suggested that bass beat expression usingthe superimposed vibration of DMA was distinct from thesimple sinusoidal vibration of LRA, making good impres-sions on many participants.

5 LIMITATIONS

Our current vibration generation algorithm is designed forreal-time processing on a mobile platform with relativelylow processing power. As such, it has several importantlimitations. In this section, we discuss these limitations, aswell as potential research directions to resolve the issues.

First of all, we noted during the user study that theusers’ expectations of good vibration playback are quitediverse. For example, some participants preferred vibrationplayback that faithfully followed the main melody of asong, whereas some others wanted vibration playback totrack a particular instrument. Our current vibrationextraction algorithm, which relies on sound energy integra-tion, is not capable of such explicit feature tracking;selective feature tracking requires much more sophisticatedalgorithms. In computer music research, there has beenactive research on automatic transcription of polyphonicmusic. In particular, some algorithms can automaticallyextract musical scores from music sources using thesampled sounds of an instrument, with about 70 percentaccuracy [33], [34]. Such musical scores can be transformedinto scores for vibration playback based on signal-levelconversion between sound and vibration, as demonstratedin a score-based vibration authoring tool we developedearlier [35], [36]. However, such approaches are likely torequire a significant amount of preprocessing and/oroffline authoring. Devising a real-time algorithm with a

reasonable trade-off between tracking accuracy and proces-sing speed will be an intriguing research topic.

Matching perceptual variables between sound andvibration while considering actuator limitations and aes-thetic quality also remains largely unexplored. Our ap-proach uses perceived magnitude as a medium, but othertime-related factors, for example, signal duration, may alsobe crucial. Our evaluation results also suggested that thebest relationships may depend on music genre. These issuesalso need significantly more attention, especially withwideband actuators becoming popular in mobile devices.

Another important practical problem is power consump-tion. Music listening with vibration playback is inevitably aprolonged task; thus, vibration patterns that can savepower, for example, short-duration patterns, are moredesirable if their perceptual value is comparable to thoseof other patterns. The incorporation of these requirementsinto automatic vibration generation demands furtherresearch before this new function can be actively employedin mobile devices.

The current haptic music player is our initial approach tovibrotactile music rendering. The results of this study arenot specific to DMA; they can be adapted to other mobileactuators with wide frequency bands. Alternatively, we canuse multiple LRAs that have different resonance frequen-cies to implement superimposed vibration. Apart fromthese, the vibration superposition approach is expected tohave a long-lasting merit because its perception is betterunderstood [24], [29] and considerably simpler than thelargely unexplored perception of wideband vibrotactilestimuli. Our haptic music player can also be extended toother applications that contain audio signals, such asmovies, games, and music videos.

6 CONCLUSIONS

In this paper, we presented a real-time dual-band hapticmusic player for mobile devices, as well as the evaluationresults of its computational and subjective performances.The rendering algorithm is designed for a new dual-modevibration actuator, while utilizing the perceptual benefit ofsuperimposed vibrations that deliver low-pitch, roughsensations. Major contributions include: 1) a real-timeaudio-to-tactile conversion algorithm that supports dual-band vibration playback, enabled by careful modalitymatching and scaling based on human perception; 2) auser study that assessed the benefits of dual-band renderingcompared with the current single-band rendering fordifferent music genres; and 3) addressing new researchissues in the domain of entertainment improvements formobile devices.

In the future, we plan to improve our haptic musicplayer based on auditory perception, for example, usingauditory saliency tracking [18], [37], with a widebandvibration actuator.

ACKNOWLEDGMENTS

This work was supported in part by LG Electronicsand the Mid-Career Researcher Program (Core)2013R1A2A2A01016907, the BRL Program 2012-0008835,the Pioneer Program 2012-0000455 from the NRF, and the

HWANG ET AL.: REAL-TIME DUAL-BAND HAPTIC MUSIC PLAYER FOR MOBILE DEVICES 349

ITRC Program 2013-H0301-13-3005 from the NIPA. Thesecond author was also supported by the Basic ScienceResearch Program 2012-0008835 from the NRF. Theauthors thank Moonchae Joung, Sunwook Kim, Kyun-ghun Hwang, and Jaecheon Sa of LG Electronics for theirgenerous support of DMA and technical assistance.

REFERENCES

[1] A. Gallace, H.Z. Tan, and C. Spence, “The Body Surface as aCommunication System: The State of the Art after 50 Years,”Presence: Teleoperators and Virtual Environments, vol. 16, no. 6,pp. 655-676, 2007.

[2] I. Poupyrev, S. Maruyama, and J. Rekimoto, “Ambient Touch:Designing Tactile Interfaces for Handheld Devices,” Proc. ACMSymp. User Interface Software and Technology (UIST ’02), pp. 51-60, 2002.

[3] J. Rekimoto and C. Schwesig, “PreSenseII: Bi-Directional Touchand Pressure Sensing Interactions with Tactile Feedback,” Proc.ACM Conf. Human Factors in Computing Systems (CHI ’06),pp. 1253-1258, 2006.

[4] E. Hoggan, S.A. Brewster, and J. Johnston, “Investigating theEffectiveness of Tactile Feedback for Mobile Touchscreens,” Proc.ACM Conf. Human Factors in Computing Systems (CHI ’08),pp. 1573-1582, 2008.

[5] L.M. Brown and T. Kaaresoja, “Feel Who’s Talking: Using Tactonsfor Mobile Phone Alerts,” Proc. ACM Conf. Human Factors inComputing Systems (CHI ’06), pp. 604-609, 2006.

[6] M. Enriquez and K.E. MacLean, “The Role of Choice in Long-itudinal Recall of Meaningful Tactile Signals,” Proc. IEEE EighthSymp. Haptic Interfaces for Virtual Environment and TeleoperatorSystems, pp. 49-56, 2008.

[7] J. Ryu, J. Jung, G. Park, and S. Choi, “Psychophysical Model forVibrotactile Rendering in Mobile Devices,” Presence: Teleoperatorsand Virtual Environments, vol. 19, no. 4, pp. 1-24, 2010.

[8] E. Hoggan and S.A. Brewster, “Crossmodal Icons for InformationDisplay,” Proc. ACM Conf. Human Factors in Computing Systems(CHI ’06), pp. 857-862, 2006.

[9] E. Gunther, “Skinscape: A Tool for Composition in the TactileModality,” Master’s thesis, Massachusetts Inst. of Technology,2001.

[10] A. Chang and C. O’Sullivan, “Audio-Haptic Feedback in MobilePhones,” Proc. ACM Conf. Human Factors in Computing Systems(CHI ’05), pp. 1264-1267, 2005.

[11] R.H. Gault, “Progress in Experiments on Tactual Interpretation ofOral Speech,” J. Abnormal Psychology and Social Psychology, vol. 19,no. 2, pp. 155-159, 1924.

[12] L.E. Bernstein, S.P. Eberhardt, and M.E. Demorest, “Single-Channel Vibrotactile Supplements to Visual Perception of Intona-tion and Stress,” J. Acoustical Soc. of Am., vol. 85, no. 1, pp. 397-405,1989.

[13] P.L. Brooks and B.J. Frost, “The Development and Evaluation of aTactile Vocoder for the Profoundly Deaf,” Canadian J. PublicHealth, vol. 77, pp. 108-113, 1986.

[14] M. Karam, F.A. Russo, and D.I. Fels, “Designing the ModelHuman Cochlea: An Ambient Crossmodal Audio-Tactile Dis-play,” IEEE Trans. Haptics, vol. 2, no. 3, pp. 160-169, July 2009.

[15] E.D. Scheirer, “Tempo and Beat Analysis of Acoustic MusicalSignals,” J. Acoustical Soc. of Am., vol. 103, no. 1, pp. 588-601, 1998.

[16] O. Mayor, “An Adaptive Real-Time Beat Tracking System forPolyphonic Pieces of Audio Using Multiple Hypotheses,” Proc.MOSART Workshop Current Research Directions in Computer Music,2001.

[17] A. Zils, F. Pachet, O. Delerue, and F. Gouyon, “AutomaticExtraction of Drum Tracks from Polyphonic Music Signals,” Proc.IEEE Int’l Conf. Web Delivering of Music, pp. 179-183, 2002.

[18] Y.F. Ma, X.S. Hua, L. Lu, and H.J. Zhang, “A Generic Framework ofUser Attention Model and Its Application in Video Summariza-tion,” IEEE Trans. Multimedia, vol. 7, no. 5, pp. 907-919, Oct. 2005.

[19] Y. Kim, J. Cha, I. Oakley, and J. Ryu, “Exploring Tactile Movies:An Initial Tactile Glove Design and Concept Evaluation,” IEEEMultimedia, vol. 16, no. 3, pp. 34-44, July-Sept. 2009.

[20] M. Kim, S. Lee, and S. Choi, “Saliency-Driven Tactile EffectAuthoring for Real-Time Visuotactile Feedback,” Proc. Int’l Conf.Haptics: Perception, Devices, Mobility, and Comm. (EuroHaptics ’12),pp. 258-269, 2012.

[21] J. Cha, Y.-S. Ho, Y. Kim, J. Ryu, and I. Oakley, “A Framework forHaptic Broadcasting,” IEEE Multimedia, vol. 16, no. 3, pp. 16-27,July-Sept. 2009.

[22] J. Cha, M. Eid, and A.E. Saddik, “Touchable 3D Video System,”ACM Trans. Multimedia Computing, Comm. and Applications, vol. 5,no. 4, pp. 29:1-29:25, 2009.

[23] S.J. Bensmaıa, M. Hollins, and J. Yau, “Vibrotactile Intensity andFrequency Information in the Pacinian System: A PsychophysicalModel,” Perception and Psychophysics, vol. 67, no. 5, pp. 828-841,2005.

[24] G. Park and S. Choi, “Perceptual Space of Amplitude-ModulatedVibrotactile Stimuli,” Proc. IEEE World Haptics Conf., pp. 59-64,2011.

[25] S.C. Lim, K.U. Kyung, and D.S. Kwon, “Effect of FrequencyDifference on Sensitivity of Beats Perception,” Experimental BrainResearch, vol. 211, pp. 11-19, 2012.

[26] LG Electronics, “Apparatus for Generating Vibration and MobileDevice Employing the Same,” Korean Patent Application, No. 10-2009-0003264, Jan. 2009.

[27] LG Electronics, “Apparatus and Method for Generating VibrationPattern,” US Patent Application, No. 12/840,988, July 2010.

[28] L.A. Jones and S.J. Lederman, Human Hand Function. Oxford Univ.Press, 2006.

[29] Y. Yoo, I. Hwang, and S. Choi, “Consonance Perception ofVibrotactile Chords: A Feasibility Study,” Proc. Sixth Int’l Conf.Haptic and Audio Interaction Design (HAID ’11), pp. 42-51, 2011.

[30] S.S. Stevens, G. Stevens, and L.E. Marks, Psychophysics: Introduc-tion to Its Perceptual, Neural, and Social Prospects. John Wiley &Sons, 1975.

[31] I. Hwang, J. Seo, M. Kim, and S. Choi, “Perceived Intensity ofTool-Transmitted Vibration: Effects of Amplitude and Fre-quency,” Proc. IEEE Int’l Symp. Haptic Visual-Audio Environmentsand Games (HAVE ’12), pp. 1-6, 2012.

[32] A. Israr, S. Choi, and H.Z. Tan, “Detection Threshold andMechanical Impedance of the Hand in a Pen-Hold Posture,” Proc.IEEE/RSJ Int’l Conf. Intelligent Robots and Systems (IROS ’06),pp. 472-477, 2006.

[33] A. Klapuri, “Automatic Music Transcription as We Know ItToday,” J. New Music Research, vol. 33, no. 3, pp. 269-282, 2004.

[34] G.E. Poliner, D.P.W. Ellis, A.F. Ehmann, E. Gomez, S. Streich, andB. Ong, “Melody Transcription from Music-Audio: Approachesand Evaluation,” IEEE Trans. Audio Speech and Language Processing,vol. 15, no. 4, pp. 1247-1256, May 2007.

[35] J. Lee, J. Ryu, and S. Choi, “Vibrotactile Score: A Score Metaphorfor Designing Vibrotactile Patterns,” Proc. IEEE World HapticsConf., pp. 302-307, 2009.

[36] J. Lee and S. Choi, “Evaluation of Vibrotactile Pattern DesignUsing Vibrotactile Score,” Proc. IEEE Haptics Symp., pp. 231-238,2012.

[37] G. Evangelopoulos, K. Rapantzikos, P. Maragos, Y. Avrithis, andA. Potamianos, “Audiovisual Attention Modeling and SalientEvent Detection,” Multimodal Processing and Interaction, vol. 33,no. 2, pp. 1-21, 2008.

Inwook Hwang received the BS and PhDdegrees in computer science and engineeringfrom the Pohang University of Science andTechnology (POSTECH) in 2006 and in 2013,respectively. His main research interests includehaptic perception and haptic feedback in mobiledevices. He is a student member of the IEEE.

350 IEEE TRANSACTIONS ON HAPTICS, VOL. 6, NO. 3, JULY-SEPTEMBER 2013

Hyeseon Lee received the BS degree fromSeoul National University in 1985, the MSdegree from the Statistics Department, CornellUniversity, in 1993, and the PhD degree fromKyungpook National University in 2008. She iscurrently a research professor in the Departmentof Industrial and Management Engineering,Pohang University of Science and Technology.She was a research programmer with theNational Opinion Research Center, University

of Chicago, in 1993-1996 and was a statistician in the Medical School,University of California, San Diego, in 2002 and 2004. She also was aproject investigator in the marketing division of Samsung Electronicsand developed customer scoring model in 2003. She receivedthe Academic Research Award from the Korean Statistical Societyin 2011. She currently serves as a member of Korean NationalStatistical Advisory Committee. Her major interest in statistics is thedata mining area, especially in nonlinear partial least square method forunbalanced data.

Seungmoon Choi received the BS and MSdegrees in control and instrumentation engineer-ing from Seoul National University in 1995and 1997, respectively, and the PhD degree inelectrical and computer engineering from Pur-due University in 2003. He is an associateprofessor of computer science and engineeringat the Pohang University of Science andTechnology. His research interests includehaptic rendering and perception, with emphasis

on kinesthetic rendering of hardness and texture, tactile rendering, skillmodeling and transfer, haptic augmented reality, mobile haptic interface,data haptization, and applied perception. His basic research has beenapplied to mobile devices, automobiles, virtual prototyping, and motion-based remote controllers. He is an associate editor of IEEE Transac-tions on Haptics and an editorial board member of Virtual Reality. Hewas a cochair of the IEEE Technical Committee on Haptics from 2009 to2011. He received the 2011 Early Career Award from the IEEETechnical Committee on Haptics and several best paper awards frompremium international conferences. He is a member of the IEEE.

. For more information on this or any other computing topic,please visit our Digital Library at www.computer.org/publications/dlib.

HWANG ET AL.: REAL-TIME DUAL-BAND HAPTIC MUSIC PLAYER FOR MOBILE DEVICES 351