340 IEEE TRANSACTIONS ON HAPTICS, VOL. 6, NO. 3, ??2015-08-17Real-Time Dual-Band Haptic Music Player for Mobile Devices Inwook Hwang, ... provide immersive multimedia experience. ... we present a haptic music player for mobile devices developed to enrich music

  • Published on

  • View

  • Download

Embed Size (px)


  • Real-Time Dual-Band HapticMusic Player for Mobile Devices

    Inwook Hwang, Student Member, IEEE, Hyeseon Lee, and Seungmoon Choi, Member, IEEE

    AbstractWe introduce a novel dual-band haptic music player for real-time simultaneous vibrotactile playback with music in mobile

    devices. Our haptic music player features a new miniature dual-mode actuator that can produce vibrations consisting of two principal

    frequencies and a real-time vibration generation algorithm that can extract vibration commands from a music file for dual-band

    playback (bass and treble). The algorithm uses a haptic equalizer and provides plausible sound-to-touch modality conversion based

    on human perceptual data. In addition, we present a user study carried out to evaluate the subjective performance (precision,

    harmony, fun, and preference) of the haptic music player, in comparison with the current practice of bass-band-only vibrotactile

    playback via a single-frequency voice-coil actuator. The evaluation results indicated that the new dual-band playback outperforms the

    bass-only rendering, also providing several insights for further improvements. The developed system and experimental findings have

    implications for improving the multimedia experience with mobile devices.

    Index TermsHaptic I/O, vibration, music, real time, dual band, dual-mode actuator


    MULTIMODAL sensory displays have great potential forimproving user interfaces and task performance.Visual and auditory displays are standard in the majorityof consumer electronic devices. Haptic displays are alsoapplied in an increasing number of applications, such asuser-interface (UI) components of mobile devices, specialeffects for entertainment, and information delivery invehicles. Haptic feedback is generally regarded as particu-larly effective in environments under sensory overload [1]and as ambient interfaces [2].

    This paper is in line with the recent research thrusts

    aiming to improve entertainment applications of mobile

    devices with haptic feedback. We present a haptic music

    player for vibrotactile music, which features the use of a

    miniature vibration actuator with dual primary frequencies

    and a real-time dual-band vibration generation algorithm.

    1.1 Related Work

    In the past 10 years, the potential of haptics for usability

    enhancement has been persistently explored for mobile

    devices. For example, Poupyrev et al. [2] developed a high-

    performance vibrotactile actuator, named TouchEngine,

    using a multiple-layer piezoelectric bending motor. They

    proposed the use of vibrotactile feedback as ambient

    sensory cues to assist user interaction. Rekimoto andSchwesig [3] added another input dimension by sensingthe finger pressure applied to a touchscreen and used thisinformation for modulating GUI responses. A usabilitystudy by Hoggan et al. [4] demonstrated that vibrotactilefeedback from virtual keypads improves the accuracy andcompletion time of text entry, while reducing mentalworkload. In addition, vibrotactile patterns can deliverabstract information, for example, tactile icons (tactons) thatindicate the phone alert type and priority [5]. A largenumber of haptic icons with high discriminability, learn-ability, and memorability can be designed using reliabledesign heuristics in the mobile context [6]. See [7] for acomprehensive review of the use of vibrotactile feedbackfor mobile devices.

    The present study addresses multimodal UIs, wheremultiple sensory channels are stimulated for informationpresentation. Crossmodal icons that combine intuitivelysimilar earcons and tactons are good examples [8]. The topicof this work, enhancement of music listening experience bythe simultaneous playback of cutaneous vibration, also hasreceived increasing attention. Even though the aestheticvalues of vibrotactile music have not been elucidated [9],the industry took a rapid step forward and released severalproducts with this functionality onto the consumer market(e.g., the mobile phone Galaxy S3 and the MP3 player YP-P3from Samsung Electronics). The MOTIV studio for mobiledevices from Immersion Corp. is another notable commer-cial solution. The MOTIV studio provides a convenientauthoring feature of automatic vibration generation fromparsed sound information in a MIDI file or directly from awave file. The basic operations of these commercialproducts are the same as those proposed by Chang andOSullivan [10], where vibrotactile patterns are created fromsound signals in a low-frequency bass band. This approachwas reported to amplify the sense of beat and improve theperception of sound quality.


    . I. Hwang and S. Choi are with the Haptics and Virtual Reality Laboratory,Department of Computer Science and Engineering, POSTECH, Hyoja-dong, Nam-gu, Science Building #4-115, Pohang, Gyungsangbuk-do 790-784, Republic of Korea. E-mail: {inux, choism}@postech.ac.kr.

    . H. Lee is with the Department of Industrial and Management Engineering,POSTECH, Hyoja-dong, Nam-gu, Science Building #4-314, Pohang,Gyungsangbuk-do 790-784, Republic of Korea.E-mail: hyelee@postech.ac.kr.

    Manuscript received 29 Mar. 2012; revised 5 Feb. 2013; accepted 13 Feb. 2013;published online 27 Feb. 2013.Recommended for acceptance by D. Grant.For information on obtaining reprints of this article, please send e-mail to:toh@computer.org, and reference IEEECS Log Number TH-2012-03-0023.Digital Object Identifier no. 10.1109/TOH.2013.7.

    1939-1412/13/$31.00 2013 IEEE Published by the IEEE CS, RAS, & CES

  • Overall, the current research status of simultaneousplayback of audio and vibrotactile music via a mobiledevice can be regarded as immature. High-performancecommercial vibration actuators that are suitable for thispurpose are not available. We also need signal conversionalgorithms that adequately consider the human perceptionof sound and vibration. The feasibility of their real-timeoperation with a mobile processor also needs to bevalidated. Furthermore, the subjective responses of usersto this new functionality remain largely unexplored.

    In the more general haptics literature, transfer of speechinto vibrotactile stimuli has been actively studied since thepioneering work of Gault [11] in the early 20th century.Bernstein et al. [12] investigated the effectiveness of sixspeech-to-tactile transformation methods for conveying theintonation and stress information contained in speech.Brooks and Frost [13] found that lipreading correctnessimproved from 39 percent to 88 percent with the use of atactile vocoder. Their system was equipped with 16 voicecoilactuators, and each actuator conveys the speech informa-tion contained in one of the 16 logarithmically equidistantfrequency intervals between 200 and 8,000 Hz. Thisapproach was also used in [14] to directly transfer musicto multiple tactile stimulation sites on the torso of thehearing impaired.

    Apart from this, research on computer music hascontinued to develop algorithms for automatic featureextraction from music. For instance, Schierer [15] proposeda method for extracting the tempo and beat of mbyiteratively matching original and reconfigured signals,providing 68 percent correctness for 60 songs of variousgenres. To identify the beat information in music, Mayor[16] used a filter bank and extracted the energy of each bandfollowed by a maximum correlation search with multiplehypotheses of the beat speed. Zils et al. [17] extractedpercussive sounds from polyphonic music by matchingsound patterns to simulated drum sounds. Ma et al. [18]developed a user attention model for video summarizationthat included an aural saliency model and attention modelsfor speech and music. However, at present, the applicabilityof computer music algorithms to simultaneous tactilefeedback remains unclear, requiring further investigation.

    Finally, there has been an increasing interest in addingtactile feedback to multimedia content. For example,Kim et al. [19] developed an authoring system thatfacilitates design of the tactile effects that are playedsynchronously with the audiovisual content of a movie toprovide immersive multimedia experience. Our groupproposed a real-time video-to-vibrotactile translation algo-rithm that automatically estimates visually salient regionson a screen and emphasizes these regions with a 2D arrayof vibrators [20]. Such methods may contribute to the birthof the new field of haptic broadcasting [21], [22]. In thiswider context, our study belongs to the class of autono-mous tactile effect synthesis algorithms from audiocontent with real-time performance.

    1.2 Article Overview

    In the remainder of this paper, we present a haptic musicplayer for mobile devices developed to enrich music

    listening experience. Our haptic music player has the

    following four major features.First, we use a new miniature actuator called the dual-

    mode actuator (DMA). The DMA can produce vibrations

    composed of two principal frequencies, which can lead to

    greater diversity in vibrotactile perception [23], [24], [25].

    This is in contrast to the vast majority of commercial mobile

    devices that use a simple actuator, for example, an eccentric

    rotating mass (ERM) or a linear resonance actuator (LRA).

    The dynamic performance of these actuators is insufficient

    for creating expressive vibrotactile effects for haptic music.Second, our haptic music player enables real-time, on-the-

    fly playback of vibrotactile effects. Since thousands of new

    musical pieces are published every year, producing

    vibrotactile music directly from musical sources without

    any preprocessing is a highly desirable benefit. Our

    vibration generation algorithms satisfy this requirement

    using digital signal processing techniques with very low

    computational complexity.Third, our haptic music player supports dual-band

    vibration playback. As reviewed earlier, previous attempts

    for haptic music playback relied on the rhythms or beats

    extracted from the bass band of sound signals. This was

    attributable partly to the performance limits of the vibration

    actuators used. Other salient aspects of music, such as a

    singers voice or guitar solo, were ignored. In our haptic

    music player, the rhythmic variations in music are encoded

    in a low-pitch vibration signal (bass band), whereas high-

    frequency salient sounds are transmitted in a high-

    frequency signal (treble band), both produced by one

    DMA. To deal with high-frequency music saliency that

    varies greatly among music genres, we introduce the

    concept of a haptic equalizer. The haptic equalizer mixes

    the signal energies from different frequency bands using

    genre-dependent weights analogously to an audio equalizer.Finally, in our haptic music player, all the conversion

    and scaling processes between sound and vibrotactile

    signals are based on perceptual data taken from the relevant

    literature. Human perception of vibrotactile stimuli is

    complex and depends on various factors such as signal

    frequency, contact site, and contact area. However, the

    previous methods of vibrotactile music tended to control

    the physical amplitude of vibrotactile stimulus, without

    explicit consideration of their perceptual consequences. We

    also compensate for the actuator input/output relationships

    to minimize the perceptual distortions that might occur

    otherwise because of the actuator dynamics and the human

    perception process.In addition, we conducted a user study to evaluate the

    perceptual merits of our dual-band vibrotactile music

    playback in comparison with the conventional single-band

    playback. Two types of actuators (LRA and DMA) were

    used, and 16 musical pieces were selected to represent four

    music genres (rock, dance, classical, and vocal; four pieces

    each). The experimental results elucidated the benefits of

    dual-band playback and their dependence on music genre,

    also providing insights that can facilitate further improve-

    ments of simultaneous audio-haptic music playback.



    This section describes the characteristics of the vibrationactuator used in our haptic music player.

    2.1 Vibration Actuators for Mobile Devices

    During the past decade, many types of vibration actuatorshave been developed for mobile devices. However, onlytwo types, ERM and LRA, are widely used in consumermobile devices. ERM is a DC motor that has a rotor with aneccentric mass distribution to induce large centrifugalacceleration. This structure produces 2D vibration in alloutgoing directions on the rotational plane of the rotor.ERMs are small and inexpensive. A drawback is that theirvibration frequency and amplitude are both determined byan input voltage level, so they cannot be controlledindependently. This is a serious obstacle against expressivevibrotactile rendering. They also have a slow and nonlinearresponse with large actuation delays. These problemsconfine the use of ERMs mostly for alerts. See [7] forfurther details.

    An LRA is a voice-coil actuator with mass and springcomponents that are connected linearly along the same axis.Vibration is produced by the mechanical resonance of thetwo components. The vibration direction is parallel to theaxis. Since LRA has a fast and linear response, it has beenregarded as an adequate choice for touchscreen interaction.However, it has a very narrow frequency bandwidth (only afew hertz wide) centered at the resonance frequency. Thus,the perceptual impression of LRA vibrations is monotone,which precludes rendering of diverse vibrotactile pitchesfor music playback.

    Efforts to embed more sophisticated actuators, forexample, piezoelectric and electro-active polymer actuators,into mobile devices are ongoing. However, their industrialadoption has been rare because of stringent industryrequirements, such as size, reliability, and durability toexternal shock.

    2.2 Dual-Mode Actuator

    DMA is a new vibration actuator developed by LGElectronics [26], [27]. The DMA is based on the sameworking principle as an LRA, but it has a more advanceddesign as shown in Fig. 1. The DMA includes two built-inmass-spring elements with different resonance frequencies.The two elements share the magnetic field of a common coillocated in the center. Each element responds only when thecommon input to the coil includes spectral energy around

    its resonance frequency. Thus, a single voltage input withsuperimposed frequencies can control both frequencycomponents independently. This use of the common coil

    enables compact size and small power consumption(10 10 3 mm; 0.1 W max) suitable for mobile devices.

    Given the two resonance frequencies, f1 and f2, all themechanical parameters can be determined. As such, theDMA has a frequency response with two distinct peaks at

    the two resonant frequencies. An example of response withf1 150 Hz and f2 223 Hz is shown in Fig. 2a. f2 wasselected to be within the frequency range to which humansare most sensitive [28]. To elicit distinctive sensations fromf2, f1 was chosen as the lowest freq...