Interactive Physically-Based Sound Design of 3D Model using Material Optimization (SCA'16)

Interactive Physically-Based Sound Design of 3D Modelusing Material Optimization

Kazuhiro Yamamoto Takeo IgarashiThe University of Tokyo

“How can we design the physically-based soundwith an intuitive way?”

Motivation

Physically-Based Sound Rendering

We focus on modal sound synthesis.

It produces perfectly synchronized sounds to various visual events via physical simulation.

Toward High-Quality Modal Contact Sound, Chanxi et al., SIGGRAPH 2011

Harmonic Shells: A Practical Nonlinear Sound Model for Near-Rigid Thin Shells,

Jeffrey et al., SIGGRAPH 2009

Toward High-Quality Modal Contact Sound

Changxi Zheng Doug L. James

Cornell University

Figure 1: A Rube-Goldberg contraption that demonstrates many challenging multibody contact sounds. A noisy block feeder (Left) withflexible tubes ejects marbles into a double helix of plastic chutes (Middle), which causes a cup to fill up, lifting a lever that drops a bunny intoa runaway shopping cart (Right) producing familiar clattering and clanging sounds due to deformable micro-collisions. Our approach canaccurately resolve modal vibrations and contact sounds using an asynchronous, adaptive, frictional contact solver.

AbstractContact sound models based on linear modal analysis are com-monly used with rigid body dynamics. Unfortunately, treating vi-brating objects as “rigid” during collision and contact processingfundamentally limits the range of sounds that can be computed, andcontact solvers for rigid body animation can be ill-suited for modalcontact sound synthesis, producing various sound artifacts. In thispaper, we resolve modal vibrations in both collision and frictionalcontact processing stages, thereby enabling non-rigid sound phe-nomena such as micro-collisions, vibrational energy exchange, andchattering. We propose a frictional multibody contact formulationand modified Staggered Projections solver which is well-suited tosound rendering and avoids noise artifacts associated with spatialand temporal contact-force fluctuations which plague prior meth-ods. To enable practical animation and sound synthesis of numer-ous bodies with many coupled modes, we propose a novel asyn-chronous integrator with model-level adaptivity built into the fric-tional contact solver. Vibrational contact damping is modeled toapproximate contact-dependent sound dissipation. Results are pro-vided that demonstrate high-quality contact resolution with sound.CR Categories: I.3.5 [Computer Graphics]: Computational Geometry andObject Modeling—Physically based modeling; I.6.8 [Simulation and Mod-eling]: Types of Simulation—Animation; H.5.5 [Information Systems]: In-formation Interfaces and Presentation—Sound and Music Computing

Keywords: Sound synthesis; contact sounds; modal analysis; asyn-chronous integration; frictional contact

1 IntroductionSound models based on linear modal vibrations are widely used toefficiently synthesize plausible contact sounds for so-called rigid

bodies in computer animation and interactive virtual environments.Unfortunately, there still remain a number of significant contact-related deficiencies that limit the realism of modal contact soundsin practice. To begin with, for speed and simplicity, modal soundmodels are usually just excited by using contact force impulsesfrom rigid body contact solvers. In reality, there is no such thing asa “rigid” object, and the same small vibrations that produce soundalso play an important role in producing rich contact events: micro-collisions, chattering, squeaking, coupled vibrations, contact damp-ing, etc. Ignoring contact-level vibrations is the source of manysound-related deficiencies, as these small vibrations can be visuallyinconsequential but aurally significant. For example, pounding ona seemingly “rigid” dinner table can shake dishes—and may alsoupset your friends (see Figure 2). Frictional contact and deforma-tion coupling is also important for sound; for example, slip-stickphenomena is responsible for many familiar squeaking and scrap-ing sounds, e.g., fingernails scraping on a chalkboard. Resolvingthese vibrational contact effects is challenging due to the need toresolve deformable collisions and contact at high temporal rates.

Even in seemingly rigid scenarios, such as an object resting on aplane, current contact solver implementations can generate tem-porally incoherent contact impulses which lead to sound artifacts,such as resting objects that strangely humm or buzz when integratedat near-audio rates. These artifacts are a consequence of the fun-damental non-uniqueness of rigid body contact forces (e.g., staticindeterminacy) which can lead to point-like and nonphysical con-tact force (traction) distributions. Additionally, rigid-body contactimpulses can exhibit nonphysical temporal fluctuations, which leadto noise-related sound artifacts (especially with iterative contact so-lution techniques) that must be dissipated artificially.

Moreover, the sound of a resting object should also depend on itscontact state, and how contacts oppose surface vibrations. For ex-ample, a coffee mug exhibits distinctive vibrational damping whenplaced in different orientations on surfaces (see Figure 4). This con-tact damping phenomena involves complex vibrational and contactcoupling effects, and is ignored in current sound models or handledin ad hoc ways, e.g., “increase damping when in contact.”

In this paper, we propose the first approach to address all of theseconcerns and enable richer contact sounds (see Figure 3). We adopta flexible multibody dynamics formulation, wherein each seem-ingly rigid object is allowed to deform with linear modal vibrations.

Harmonic Shells: A Practical Nonlinear Sound Model for Near-Rigid Thin Shells

Jeffrey N. Chadwick Steven S. AnCornell University

Doug L. James

Figure 1: Crash! Our physically based sound renderings of thin shells produce characteristic “crashing” and “rumbling” sounds when animated using rigidbody dynamics. We synthesize nonlinear modal vibrations using an efficient reduced-order dynamics model that captures important nonlinear mode coupling.High-resolution sound field approximations are generated using far-field acoustic transfer (FFAT) maps, which are precomputed using efficient fast Helmholtzmultipole methods, and provide cheap evaluation of detailed low- to high-frequency acoustic transfer functions for realistic sound rendering.

AbstractWe propose a procedural method for synthesizing realistic soundsdue to nonlinear thin-shell vibrations. We use linear modal analysisto generate a small-deformation displacement basis, then couple themodes together using nonlinear thin-shell forces. To enable audio-rate time-stepping of mode amplitudes with mesh-independent cost,we propose a reduced-order dynamics model based on a thin-shellcubature scheme. Limitations such as mode locking and pitch glideare addressed. To support fast evaluation of mid-frequency mode-based sound radiation for detailed meshes, we propose far-fieldacoustic transfer maps (FFAT maps) which can be precomputedusing state-of-the-art fast Helmholtz multipole methods. Familiarexamples are presented including rumbling trash cans and plasticbottles, crashing cymbals, and noisy sheet metal objects, each withincreased richness over linear modal sound models.CR Categories: I.3.5 [Computer Graphics]: Computa-tional Geometry and Object Modeling—Physically based mod-eling; I.6.8 [Simulation and Modeling]: Types of Simulation—Animation; H.5.5 [Information Systems]: Information Interfacesand Presentation—Sound and Music ComputingKeywords: Sound synthesis; thin shells; contact sounds;modal analysis; dimensional model reduction; subspace integra-tion; acoustic transfer; Helmholtz equation

1 IntroductionLinear modal sound models are widely used for rigid bodies in com-puter animation and virtual environments [van den Doel et al. 2001;O’Brien et al. 2002; Bonneel et al. 2008], and when combined withacoustic transfer models for sound radiation [James et al. 2006] they

can provide convincing physically based sound sources, especiallyfor pure ringing tones such as chimes, bells, or “knocks.” Unfor-tunately, we lack effective sound models for a broad class of noisyvirtual objects: thin shells (objects with thicknesses orders of mag-nitude smaller than their other dimensions). Thin shells are verycommon in real and virtual environments, and produce rich andeasily recognizable impact sounds: sheet metal objects (trash cans,oil drums, tin roofs, machinery), plastic containers (water bottles),musical instruments (cymbals), etc. Their rich nonlinear vibrationsproduce proverbial “crashes” and “rumbles” that are poorly approx-imated by linear modal sound models which lack nonlinear modecoupling. To make matters worse, thin shells are often very loudand important sound sources due to their ability to vibrate and radi-ate sound so effectively, e.g., consider a metal roof pelted by hail.Alas, their expensive nonlinear dynamics have made thin shellscomputationally impractical for physically based sound synthesis.

In this paper, we propose an efficient method for synthesizing re-alistic sounds from thin-shell structures undergoing small but non-linear vibrations. Given a description of an object’s geometry andmaterial properties, we compute linear vibration modes, then cou-ple these modes together using the nonlinear thin-shell force model.To accelerate nonlinear modal dynamics, we optimize a thin-shellcubature scheme to evaluate reduced-order shell forces at costs in-dependent of the geometric complexity of the model. We show thatthe complex internal dynamics of thin-shell models can be approxi-mated with sufficient accuracy and efficiency to allow practical syn-thesis of plausible thin-shell sounds. We also address sound-relatedlocking effects that arise when simulating nonlinear modal dynam-ics that might produce pitch-glide artifacts in general animations.

Our nonlinear reduced-order dynamics model can synthesize modalvibrations using hundreds of modes, which then drive sound radi-ation. Unfortunately the estimation of sound wave radiation viaprior acoustic transfer models is complicated for two reasons: (1)the nonlinear mode vibrations are no longer linear harmonics, and(2) higher frequency acoustic transfer with high-resolution meshesis expensive to precompute, represent, and evaluate at runtime.First, we observe that nonlinear thin-shell vibrations produced byour animations exhibit frequency-localized modes for which lin-ear frequency-domain radiation models still provide a plausible ap-proximation. Second, we propose far-field acoustic transfer maps

Rigid-Body Fracture Sound with Precomputed Soundbanks

Changxi Zheng Doug L. James

Cornell University

Figure 1: SMASH! We synthesize the violent fracture and impact sounds of a glass table setting smashed into over 300 pieces (see soundspectrogram). We use time-varying rigid-body sound models to approximate this brittle fracture sound by a superposition of 4046 modalvibrations (up to 14kHz). To avoid thousand-mode modal analysis and acoustic transfer costs for complex fracture geometry, we use soundproxies sampled from Precomputed Rigid-Body Soundbanks, here producing plausible fracture sound models at almost 500⇥ speedup.

Abstract

We propose a physically based algorithm for synthesizing soundssynchronized with brittle fracture animations. Motivated by lab-oratory experiments, we approximate brittle fracture sounds us-ing time-varying rigid-body sound models. We extend methodsfor fracturing rigid materials by proposing a fast quasistatic stresssolver to resolve near-audio-rate fracture events, energy-based frac-ture pattern modeling and estimation of “crack”-related fracture im-pulses. Multipole radiation models provide scalable sound radia-tion for complex debris and level of detail control. To reduce sound-model generation costs for complex fracture debris, we proposePrecomputed Rigid-Body Soundbanks comprised of precomputedellipsoidal sound proxies. Examples and experiments are presentedthat demonstrate plausible and affordable brittle fracture sounds.

CR Categories: I.3.5 [Computer Graphics]: Computa-tional Geometry and Object Modeling—Physically based mod-eling; I.6.8 [Simulation and Modeling]: Types of Simulation—Animation; H.5.5 [Information Systems]: Information Interfacesand Presentation—Sound and Music Computing

1 Introduction

Brittle fracture is an important and loud part of physically basedanimation and interactive virtual environments. Unfortunately wedo not know how to procedurally synthesize fracture sounds auto-matically and efficiently. Current sound production methods relyinstead on audio recordings of fracture events, which can inheritshortcomings of implausibility, lack of physical synchronization,and large memory footprints to avoid repetition.

In this paper, we propose the first physically based approach for au-tomatic synthesis of synchronized brittle fracture sounds for com-puter animation (see Figure 1). Despite the familiar complexity of3D fracture animation, our fracture sound synthesis method inher-its the simplicity of rigid-body sound synthesis. Based on observa-tions from laboratory fracture experiments with high-speed videoand sound recordings (see Figure 2), we hypothesize that brittlefracture sounds can be efficiently and effectively approximated bytime-varying rigid-body sound models (see Figure 3). Our rigid-body fracture sound synthesis has three parts: (1) a fracture prepro-cess which generates rigid-bodies with contact and “crack”-relatedfracture impulses; (2) a parallel sound model generation phase con-sisting of modal and acoustic transfer analysis; and (3) a sound syn-thesis phase where sounds are rendered at the listener’s position.

We leverage prior work on fracturing rigid materials [Bao et al.2007], and propose a sparse, direct, least-squares solver for therank-deficient elastostatic problem (Ku = f ) to resolve fracturesound events at near audio rates. An energy-based fracture modelis used to model plausible fracture energy and sound generation,and also to estimate stress-based fracture impulses which excite theinitially silent sound models of rigid-body debris to produce charac-teristically loud “crack” sounds (see Figure 2). The fracture simu-

Rigid-Body Fracture Sound with Precomputed Soundbanks, Changxi et al., SIGGRAPH Asia 2010

Modal Sound Synthesis

for quasi-rigid body sound (e.g., collision, bounce, and scratch)

An Object

Vibration Analysis(Modal Analysis)

Sound

Modal Parameters

graphics applications. With these conditions, diagonaliz-ing equation (1) becomes equivalent to solving a general-ized symmetric eigenproblem with symmetric, positive-definite matrices. Cook, Malkus, and Plesha describe theprocess in detail and we only repeat the end result here.

With the restriction of Rayleigh damping equation (1)may be rewritten as:

K(d + ↵1˙d) + M(↵2

˙d +

¨d) = f , (2)

where ↵1 and ↵2 are the Rayleigh coefficients. Let thecolumns of W be the solution to the generalized sym-metric eigenproblem Kx + �Mx = 0 and ⇤ be thediagonal matrix of eigenvalues1, then equation (2) maybe transformed to:

⇤(z + ↵1 ˙z) + (↵2 ˙z +

¨z) = g , (3)

where z = W�1d is the vector of modal coordinatesand g = W Tf is the external force vector in the modalcoordinate system.

Each row of equation (3) corresponds to a single scalarsecond-order differential equation:

�izi + (↵1�i + ↵2)zi + zi = gi . (4)

The analytical solutions to each equation are

zi = c1et!+

i+ c2e

t!�i (5)

where c1 and c2 are arbitrary (complex) constants, and !i

is the complex frequency given by

!±i =

�(↵1�i + ↵2) ±p

(↵1�i + ↵2)2 � 4�i

2

. (6)

The absolute value of the imaginary part of !i is the fre-quency (in radians/second, not Hertz) of the mode, andthe real part is the mode’s decay rate. In the special casewhere the term under the radical in equation (6) is zero,we have !+

i = !�i , which gives the critically dampedsolution:

zi = c1tet!i

+ c2et!i . (7)

The columns of W are the vibrational modes of theobject being modeled. (See figure 4.) Each mode has theproperty that a displacement or velocity over the objectthat is a scalar multiple of the mode will produce an ac-celeration that is also a scalar multiple of the mode. Thisproperty means that the modes do not interact with eachother, which is why decoupling the system into a set ofindependent oscillators was possible. The eigenvalue foreach mode is the ratio of the mode’s elastic stiffness to themode’s mass, and it is the square of the mode’s naturalfrequency (in radians per second). In general the eigen-values will be positive, but for each free body in the sys-tem there will be six zero eigenvalues that correspond to

1 Equivalently let W = L�TV where M = LLT (Choleskydecomposition) and V ⇤V T = L�1KL�T (symmetric eigendecom-position).

Figure 4: The two rows show a side and top view of

a bowl along with three of the bowl’s first vibrational

modes. The modes selected for the illustration are the

first three non-rigid modes with distinct eigenvalues that

are excited by a transverse impulse to the bowl’s rim.

the body’s six rigid-body modes. The rigid-body eigen-values are zero because a rigid-body displacement willnot generate any elastic forces.

The decoupled system of equations is not an approxi-mation of the original linear system, it will generate ex-actly the same results as the original linear system. Ofcourse the linear system may have been an approxima-tion to some initial nonlinear one, but any problem thatcould be solved using equation (2) could also be solvedwith equation (3). Furthermore, simulation that wouldhave required numerical time integration of equation (1)can now be solved without integration using the analyti-cal solutions in equations (5) or (7).

3.2 Discarding ModesAlthough decoupling equation (1) and then solving eachof the resulting components analytically provides signifi-cant benefits, we can derive additional benefit by consid-ering whether or not each of these components is needed.In particular we can discard modes that will have no sig-nificant effect on the phenomena we wish to model.

If the eigenvalue, �i, associated with a particular modeis large, then the force required to cause a discernibledisplacement of that mode will also be large. We canexpect that in a given environment there will be bothan upper bound on the magnitude of the forces encoun-tered and a lower limit on the amplitude of observablemovement. For example, if modeling an indoor envi-ronment we would not expect to encounter forces in ex-cess of 60, 000 N (the braking force of a large truck), andwe would not be able to observe displacements less thanabout 0.1 mm. Thus if ||wi||2/�i < min res/ max frc

for some mode then that mode’s behavior will be unob-servable.

The imaginary part of !i determines the frequency thata mode will vibrate at. Modes that vibrate at more thanhalf the display’s frame rate will cause temporal aliasing.

Removing modes that are too stiff and/or too high fre-quency to be observed will not change the appearance of

Graphics Interface 2003 250



K(d + ↵1˙d) + M(↵2

˙d +

¨d) = f , (2)


⇤(z + ↵1 ˙z) + (↵2 ˙z +

¨z) = g , (3)



�izi + (↵1�i + ↵2)zi + zi = gi . (4)


zi = c1et!+

i+ c2e

t!�i (5)



!±i =

�(↵1�i + ↵2) ±p

(↵1�i + ↵2)2 � 4�i

2

. (6)



zi = c1tet!i

+ c2et!i . (7)


















K(d + ↵1˙d) + M(↵2

˙d +

¨d) = f , (2)


⇤(z + ↵1 ˙z) + (↵2 ˙z +

¨z) = g , (3)



�izi + (↵1�i + ↵2)zi + zi = gi . (4)


zi = c1et!+

i+ c2e

t!�i (5)



!±i =

�(↵1�i + ↵2) ±p

(↵1�i + ↵2)2 � 4�i

2

. (6)



zi = c1tet!i

+ c2et!i . (7)


















K(d + ↵1˙d) + M(↵2

˙d +

¨d) = f , (2)


⇤(z + ↵1 ˙z) + (↵2 ˙z +

¨z) = g , (3)



�izi + (↵1�i + ↵2)zi + zi = gi . (4)


zi = c1et!+

i+ c2e

t!�i (5)



!±i =

�(↵1�i + ↵2) ±p

(↵1�i + ↵2)2 � 4�i

2

. (6)



zi = c1tet!i

+ c2et!i . (7)
















+

modal frequency: f0modal amplitude: a0modal damping : d0



Modal Sound Synthesis

for quasi-rigid body sound (e.g., collision, bounce, and scratch)

An Object

Vibration Analysis(Modal Analysis)

Sound

Modal Parameters



K(d + ↵1˙d) + M(↵2

˙d +

¨d) = f , (2)


⇤(z + ↵1 ˙z) + (↵2 ˙z +

¨z) = g , (3)



�izi + (↵1�i + ↵2)zi + zi = gi . (4)


zi = c1et!+

i+ c2e

t!�i (5)



!±i =

�(↵1�i + ↵2) ±p

(↵1�i + ↵2)2 � 4�i

2

. (6)



zi = c1tet!i

+ c2et!i . (7)


















K(d + ↵1˙d) + M(↵2

˙d +

¨d) = f , (2)


⇤(z + ↵1 ˙z) + (↵2 ˙z +

¨z) = g , (3)



�izi + (↵1�i + ↵2)zi + zi = gi . (4)


zi = c1et!+

i+ c2e

t!�i (5)



!±i =

�(↵1�i + ↵2) ±p

(↵1�i + ↵2)2 � 4�i

2

. (6)



zi = c1tet!i

+ c2et!i . (7)


















K(d + ↵1˙d) + M(↵2

˙d +

¨d) = f , (2)


⇤(z + ↵1 ˙z) + (↵2 ˙z +

¨z) = g , (3)



�izi + (↵1�i + ↵2)zi + zi = gi . (4)


zi = c1et!+

i+ c2e

t!�i (5)



!±i =

�(↵1�i + ↵2) ±p

(↵1�i + ↵2)2 � 4�i

2

. (6)



zi = c1tet!i

+ c2et!i . (7)


















K(d + ↵1˙d) + M(↵2

˙d +

¨d) = f , (2)


⇤(z + ↵1 ˙z) + (↵2 ˙z +

¨z) = g , (3)



�izi + (↵1�i + ↵2)zi + zi = gi . (4)


zi = c1et!+

i+ c2e

t!�i (5)



!±i =

�(↵1�i + ↵2) ±p

(↵1�i + ↵2)2 � 4�i

2

. (6)



zi = c1tet!i

+ c2et!i . (7)
















+




Difficulty of Physically-Based Sound Design


？Material Parameters


？Material Parameters Physical Simulation

？

Our Approach

Example-based interactive frameworks using material optimization.

Input:3D Model and Target Sound

Our Approach



Sound 4

Position 4

Sound 2

Position 2

Sound 3

Position 3

Sound 1

Position 1

Our Approach



Sound 4

Position 4

Sound 2

Position 2

Sound 3

Position 3

Sound 1

Position 1

Output:Material Distribution

Our Approach



Sound 4

Position 4

Sound 2

Position 2

Sound 3

Position 3

Sound 1

Position 1

What is the “Material” ?

We only optimize Young’s modulus.

• Density

• Young’s modulus

• Poisson’s rate.

What is the “Material” ?

We only optimize Young’s modulus.

• Density

• Young’s modulus

• Poisson’s rate.

Related Work

Foleyautomatic: physically-based sound effects for interactive simulation and animation,

Pai et al., 2001

Example-guided physically based modal sound synthesis, Ren et al., 2013

TimbreFields: 3D Interactive Sound Models for Real-Time Audio., Corbett et al., 2007

Impulse responses were recorded digitally as 16 bitaudio at a sampling rate of 44.1 KHz. The microphonewas mounted on a programmable robotic arm, whichautomatically moved the microphone to a variety of lis-tening locations for a given contact location. The sole-noid was synchronized with the robotic arm to expeditethe recording process. All the microphone and excita-tion control programming was done in advance. Thestrategy was to gather all the impulse responses within arobotic environment so as to minimize errors intro-duced through human control.

Before each impulse response recording, 0.5 s of am-bient sound was recorded to help identify spuriousmodes in the background noise. This precaution wasnecessary because the measuring environment containeda lot of acoustic noise from machine fans and powermotor control equipment.

This process produced a set of acoustic responsesamples for different contact and listener locationsdistributed across the 5D timbre space. The modalmodel for the entire object was then constructed inseveral phases.

3.1 Estimation of the Modes

The estimation of the modal model parametersfrom the measurements was achieved in the followingthree phases.

First, we estimated the modal model parameters foreach sound sample separately. This is done by comput-ing the windowed discrete Fourier transform (Gabortransform) of the signal—extracting the frequencies,dampings, and amplitudes by fitting each frequency binwith a sum of a small number (e.g., four) of dampedcomplex exponentials. The parameter fitting method iscapable of very accurate frequency reconstructions andis able to resolve very close modes. Close modes are verycommon in artificial objects that have approximate sym-metries resulting in mode degeneracy. Manufacturingimpurities break this symmetry, splitting the frequenciesby a small amount. These nearby frequencies are dis-tinctly audible as beating, or “shimmering” sounds andsignificantly enhance the realism of the synthesizedsound.

Second, we determined which modes are actually au-dible, using a rough model of auditory masking (Doel,Pai, Adam, Kortchmar, & Pichora-Fuller, 2002; Doel,Knott, & Pai, 2004), and retained only the audiblemodes. This step is necessary because the parameter fit-ting algorithm produces many spurious modes.

Third, we merged all the modal models, using a sim-ple model of human frequency discrimination, whichresults in a single frequency and damping model for theentire object, and a discrete sampling of the timbre fielda on the 5D interaction space. In theory, all the modelsshould share the same set of frequencies and dampings,but due to noise they will not be precisely the same,motivating this third step.

We now describe the details of the parameter fitting.To estimate the modal content of a single recorded

impulse response s(t), we first compute the windowedFourier transform. We use a Blackman-Harris windowof length Tw, and Noverlap ! 4 windows are taken to beoverlapping, giving a “hopsize” TH of TH ! Tw/Noverlap.(The hopsize is the amount by which consecutive win-dows are apart.) The window size Tw is chosen to be 46ms, which is appropriate for audio analysis. Let us de-

Figure 3. An automated measuring system was used to acquire thedata. A robot arm moves the microphone to a preprogrammed set oflocations. An impulse force was then automatically applied to theobject with the solenoid and sounds were recorded for subsequentanalysis.

648 PRESENCE: VOLUME 16, NUMBER 6

1:2 • Z. Ren et al.

(e)(d)(c)(b)(a)

Fig. 1. From the recording of a real-world object (a), our framework is able to find the material parameters and generates similar sound for a replicate object(b). The same set of parameters can be transfered to various virtual objects to produce sounds with the same material quality ((c), (d), (e)).

(such as stiffness, damping coefficients, and mass density) that canbe directly used in modal analysis.

As a result, for objects with different geometries and runtimeinteractions, different sets of modes are generated or excited differ-ently, and different sounds are produced. However, if the materialproperties are the same, they should all sound like coming fromthe same material. For example, a plastic plate being hit, a plasticball being dropped, and a plastic box sliding on the floor generatedifferent sounds, but they all sound like plastic, as they have thesame material properties. Therefore, if we can deduce the materialproperties from a recorded sound and transfer them to differentobjects with rich interactions, the intrinsic quality of the originalsounding material is preserved. Our method can also compensate thedifferences between the example audio and the modal-synthesizedsound. Both the material parameters and the residual compensationare capable of being transfered to virtual objects of varying sizesand shapes and capture all forms of interactions. Figure 1 shows anexample of our framework. From one recorded impact sound (Fig-ure 1(a)), we estimated material parameters, which can be directlyapplied to various geometries (Figures 1(c), 1(d), 1(e)) to generateaudio effects that automatically reflect the shape variation while stillpreserving the same sense of material. Figure 2 depicts the pipelineof our approach, and its various stages are explained next.

Feature extraction. Given a recorded impact audio clip, fromwhich we first extract some high-level features, namely, a set ofdamped sinusoids with constant frequencies, dampings, and initialamplitudes (Section 4). These features are then used to facilitateestimation of the material parameters (Section 5), and guide theresidual compensation process (Section 6).

Parameter estimation. Due to the constraints of the sound syn-thesis model, we assume a limited input from just one recordingand it is challenging to estimate the material parameters from oneaudio sample. To do so, a virtual object of the same size and shapeas the real-world object used in recording the example audio iscreated. Each time an estimated set of parameters are applied tothe virtual object for a given impact, the generated sound, as wellas the feature information of the resonance modes, are comparedwith the real-world example sound and extracted features, respec-tively, using a difference metric. This metric is designed based onpsychoacoustic principles, and aimed at measuring both the audiomaterial resemblance of two objects and the perceptual similaritybetween two sound clips. The optimal set of material parameters isthereby determined by minimizing this perceptually inspired metric

Fig. 2. Overview of the example-guided sound synthesis framework(shown in the blue block): Given an example audio clip as input, features areextracted. They are then used to search for the optimal material parametersbased on a perceptually inspired metric. A residual between the recorded au-dio and the modal synthesis sound is calculated. At runtime, the excitation isobserved for the modes. Corresponding rigid-body sounds that have a similaraudio quality as the original sounding materials can be automatically synthe-sized. A modified residual is added to generate a more realistic final sound.

function (see Section 5). These parameters are readily transferableto other virtual objects of various geometries undergoing rich inter-actions, and the synthesized sounds preserve the intrinsic quality ofthe original sounding material.

Residual compensation. Finally, our approach also accounts forthe residual, that is, the approximated differences between the real-world audio recording and the modal synthesis sound with the esti-mated parameters. First, the residual is computed using the extractedfeatures, the example recording, and the synthesized audio. Thenat runtime, the residual is transfered to various virtual objects. Thetransfer of residual is guided by the transfer of modes, and natu-rally reflects the geometry and runtime interaction variation (seeSection 6).

Our key contributions are summarized as follows.

—A feature-guided parameter estimation framework to determinethe optimal material parameters can be used in existing modalsound synthesis applications.

ACM Transactions on Graphics, Vol. 32, No. 1, Article 1, Publication date: January 2013.

from many sound clips (measured)

from a sound clip

Example-based sound design for modal sound synthesis

Related Work


Pai et al., 2001
















(e)(d)(c)(b)(a)













from a sound clip


It requires expensive measurement procedure.

Related Work


Pai et al., 2001
















(e)(d)(c)(b)(a)













from a sound clip


It requires expensive measurement procedure.

It requires same object in real world

Demo

Demo

Demo

Demo

Demo

Demo

Demo

Demo

Demo

Demo

The system allows multiple assignments of example sounds.

Demo


Demo


Demo


Contributions

• An example-based framework for designing physically-based sound

• Fast approximate modal analysis for an interactive simulation

• Extended data-driven FEM using regression forests

• Hierarchical component mode synthesis with error correction

• Handling a large range of continuous material settings. • Constant evaluation cost.• High generalization ability.

• Parallel• Accurate

Modal Analysis

Algorithm OverviewKazuhiko Yamamoto & Takeo Igarashi / Interactive Physically-Based Sound Design of 3D Modelusing Material Optimization

Data-Driven Online Coarsening (§7.1)

Hierarchical Component Mode Synthesis (§7.2)

Runtime

Fast Approximate Modal Analysis (§7)

The evaluation of the objective function and Gradient Computation (§6)

+

…

For Model(Independent of Materials)

Input: Surface Mesh

Eigenvectors of Volumetric Mesh Laplacian

Training of Mapping (§7.1.3)

For Material Set(Independent of Models)

Voxelization and Eigen Decomposition (§7.2)

Sample Position 1

User Input(Target Sound)

Simulated Sound

Material Optimization (§6)

New Material Distribution

Similarity Score Computation (§5)

Used for Material Reduction (§6)

Used for Domain Decomposition (§7.2)

EigenpairsSample Position 2

Sample Position 3

Timbre

Precomputation

Regression Forests

Figure 2: Algorithm Overview. Our optimization algorithm consists of the precomputation and runtime. An iteration of our optimizationprocedure at runtime consists of 3 steps. First, the system computes modal analysis to obtain the vibrational property of the object. Second, itcomputes the similarity score between the simulated sounds of the object and user specified target sounds. According to this similarity score,the system updates the material distribution inside the object to minimise the cost.

2. Related WorkParameter Acquisition for Modal Sound Synthesis: To deter-mine the material parameters used in modal sound synthesis, Paiet al. [PvdDJ∗01] and Corbett et al. [CvdDLH07] acquired the pa-rameters from actually measured impact sound data, and interpo-lated them in auditory space. A robotic actuated device is used toapply impulses on a real object at a large number of sample points,and map the recorded impact sounds to virtual objects. However,the measurement procedure of such a huge number of samples foran object and manipulating them are prohibitively expensive. Fo-leyAutomatic [vdDKP01] also employed similar approach, but in-terpolated them in modal space for achieving rich sound interac-tions. However, they also require sufficient amount of samples toestimate the modal function on the surface. Same example soundcan be reused at different locations, but it causes a lack of the soundvariations when the object interacts with other objects at variouslocations. This problem becomes profound when the model to bedesigned has a larger scale.

To avoid measuring such a huge number of parameters for oneobject from many audio clips, Lloyd et al. [LRG11] proposed adata-driven approach to assign the sound of an object from only oneaudio clip. They estimated the modal parameters from the audioclip, and at runtime, they randomized the mixture gains of all thetracked modes to generate imaginary varied sounds when hittingdifferent locations on the object. However, this method producesunnatural artifacts because the sounds are not consistent with hitpoints.

As another approach, Ren et al. [RYL13] proposed a method toestimate the material specific parameter (Rayleigh damping param-eters) directly instead of modal parameters from a audio clip underthe assumption of uniform material distribution inside the object.The advantage of their approach is that it enables the estimatedmaterial parameters to be transferred to different shapes. However,their approach requires that the real object have exactly the sameshape as the virtual model to be estimated and should be easy toprepare. These requirements are impractical to implement in actualscenes, which is considered in this paper.

Vibrational Property Optimization: To obtain the desired vi-brational property of an object, Yamasaki et al. [YNY∗10] opti-mized the shape and topology of an industrial structure using lev-

elset optimization, and controlled the several lowest eigenfrequen-cies. Yua et al. [YJKK10, YJK13] optimized the topology of a vi-olin’s body as specific thin shell structure to control the mode fre-quencies and amplitudes (mode vectors) that are expected to belargely contributed to the timbre. Bharaj et al. [BLT∗15] optimizedthe shape of a common elastic structure to control both a few modefrequencies as well as their amplitudes for fabricating metal percus-sion instruments. Our formulation is similar to theirs, but there arefour differences. 1: We control a much larger number of modes fordramatically changing the sound’s timbre and sacrificing the fab-rication possibility. 2: We optimize the material distribution whilemaintaining the shape whereas they optimize the shape. 3: Our op-timization runs at an interactive rate that is enabled by an expansionof data-driven finite elements method (FEM) [CLSM15] and highlyparallelized hierarchical component mode synthesis. 4: Our objec-tive function considers the perceptual differences of two soundswhereas they use square distances of frequencies and amplitudes.

Modal Analysis: is a well-studied technique in both computergraphics and engineering. It solves the generalized eigenproblem ofthe finite element stiffness and mass matrices to obtain the vibra-tional frequencies and the corresponding deformations [HSO03].Because modal analysis is a time-consuming operation, it is usu-ally used for only the precomputation phase. As some exceptions,Umetani et al. [UMIT10] introduced 2D modal analysis into an in-teractive design tool for percussion instrument by limiting the fun-damental mode computation. Maxwell and Bindel [MB07] com-puted quasi-3D modal analysis of thin shell structure percussioninstruments including the several overtones at a quasi-interactiverate. We introduced 3D modal analysis of a more complex struc-ture into an interactive application.

Many studies focused on the improvement of the computa-tional efficiency of modal analysis. A powerful solution is the do-main decomposition approach called the component mode syn-thesis method (CMS) [Hur65]. CMS decomposes a large probleminto many small problems of subdomains and merges them. Thereare several variations of CMS according to how the boundariesbetween subdomains are treated [CP88, YVC13]. The major ap-proach is the Craig-Bamptom method [BC68] that treats the inter-faces of subdomains as fixed. However, finding an optimal divi-sion of a mesh in subdomains is non-trivial, and it should be often

c⃝ 2016 The Author(s)Eurographics Proceedings c⃝ 2016 The Eurographics Association.

Kazuhiko Yamamoto & Takeo Igarashi / Interactive Physically-Based Sound Design of 3D Modelusing Material Optimization



Runtime



+

…


Input: Surface Mesh





Sample Position 1


Simulated Sound







Sample Position 3

Timbre

Precomputation

Regression Forests













Runtime



+

…


Input: Surface Mesh





Sample Position 1


Simulated Sound







Sample Position 3

Timbre

Precomputation

Regression Forests










A FEM Mesh

Target Sounds → Target Parameters




Runtime



+

…


Input: Surface Mesh





Sample Position 1


Simulated Sound







Sample Position 3

Timbre

Precomputation

Regression Forests










Simulated Sound

Material Optimization


Perceptual Differencesand Gradients

→ (f1, a1, d1)

(modal parameters)

→ (f2, a2, d2)

→ (f3, a3, d3)

Modal Analysis




Runtime



+

…


Input: Surface Mesh





Sample Position 1


Simulated Sound







Sample Position 3

Timbre

Precomputation

Regression Forests













Runtime



+

…


Input: Surface Mesh





Sample Position 1


Simulated Sound







Sample Position 3

Timbre

Precomputation

Regression Forests













Runtime



+

…


Input: Surface Mesh





Sample Position 1


Simulated Sound







Sample Position 3

Timbre

Precomputation

Regression Forests










A FEM Mesh





Runtime



+

…


Input: Surface Mesh





Sample Position 1


Simulated Sound







Sample Position 3

Timbre

Precomputation

Regression Forests










Simulated Sound




→ (f1, a1, d1)

(modal parameters)

→ (f2, a2, d2)

→ (f3, a3, d3)

Modal Analysis




Runtime



+

…


Input: Surface Mesh





Sample Position 1


Simulated Sound







Sample Position 3

Timbre

Precomputation

Regression Forests













Runtime



+

…


Input: Surface Mesh





Sample Position 1


Simulated Sound







Sample Position 3

Timbre

Precomputation

Regression Forests













Runtime



+

…


Input: Surface Mesh





Sample Position 1


Simulated Sound







Sample Position 3

Timbre

Precomputation

Regression Forests










A FEM Mesh





Runtime



+

…


Input: Surface Mesh





Sample Position 1


Simulated Sound







Sample Position 3

Timbre

Precomputation

Regression Forests










Simulated Sound




→ (f1, a1, d1)

(modal parameters)

→ (f2, a2, d2)

→ (f3, a3, d3)

Modal Analysis




Runtime



+

…


Input: Surface Mesh





Sample Position 1


Simulated Sound







Sample Position 3

Timbre

Precomputation

Regression Forests













Runtime



+

…


Input: Surface Mesh





Sample Position 1


Simulated Sound







Sample Position 3

Timbre

Precomputation

Regression Forests













Runtime



+

…


Input: Surface Mesh





Sample Position 1


Simulated Sound







Sample Position 3

Timbre

Precomputation

Regression Forests










A FEM Mesh





Runtime



+

…


Input: Surface Mesh





Sample Position 1


Simulated Sound







Sample Position 3

Timbre

Precomputation

Regression Forests










Simulated Sound




→ (f1, a1, d1)

(modal parameters)

→ (f2, a2, d2)

→ (f3, a3, d3)

Modal Analysis




Runtime



+

…


Input: Surface Mesh





Sample Position 1


Simulated Sound







Sample Position 3

Timbre

Precomputation

Regression Forests













Runtime



+

…


Input: Surface Mesh





Sample Position 1


Simulated Sound







Sample Position 3

Timbre

Precomputation

Regression Forests













Runtime



+

…


Input: Surface Mesh





Sample Position 1


Simulated Sound







Sample Position 3

Timbre

Precomputation

Regression Forests










A FEM Mesh





Runtime



+

…


Input: Surface Mesh





Sample Position 1


Simulated Sound







Sample Position 3

Timbre

Precomputation

Regression Forests










Simulated Sound




→ (f1, a1, d1)

(modal parameters)

→ (f2, a2, d2)

→ (f3, a3, d3)

Modal Analysis




Runtime



+

…


Input: Surface Mesh





Sample Position 1


Simulated Sound







Sample Position 3

Timbre

Precomputation

Regression Forests













Runtime



+

…


Input: Surface Mesh





Sample Position 1


Simulated Sound







Sample Position 3

Timbre

Precomputation

Regression Forests













Runtime



+

…


Input: Surface Mesh





Sample Position 1


Simulated Sound







Sample Position 3

Timbre

Precomputation

Regression Forests










A FEM Mesh





Runtime



+

…


Input: Surface Mesh





Sample Position 1


Simulated Sound







Sample Position 3

Timbre

Precomputation

Regression Forests










Simulated Sound




→ (f1, a1, d1)

(modal parameters)

→ (f2, a2, d2)

→ (f3, a3, d3)

Similar to [Bharaj et al. ’15].


Frequency Differences(Critical band rate)


Position 1

Position 2

Position 3

Position 4

Frequency [Hz]

Amplitude [dB]

Frequency [Hz]

Amplitude [dB]

Frequency [Hz]

Amplitude [dB]

Frequency [Hz]

Amplitude [dB]

Sound Assigned to Position 1




Figure 3: The user interface view. The left pane allows the userto assign the target sounds for the model and preview the contactsound while the right views represent the power spectrums of as-signed sounds (green) and the sounds when the positions the userselected are struck (red). The black arrows on the left pane repre-sent the positions the user assigned targe sounds.

undertaken manually for improving the accuracy. It requires addi-tional expertise and manual efforts by the user. Our approach doesnot distinguish between subdomains and boundaries, and automat-ically decomposes it as a hierarchical structure and merges themin parallel. In addition, we improve the accuracy using the fast er-ror correction algorithm, which consists of a combination of thesubspace iteration method [Bat13] and sparse mass-Gram-Schmidtprocess [YLX∗15].

3. User Workflow

This section describes the user workflow of our interactivephysically-based sound design tool. Please see the supplementalvideo for an interactive demonstration. As seen in the screen cap-ture shown in Figure 3, the user first provides a 3D surface model asan input. The system automatically voxelizes it and converts it intoa uniform hexahedral finite element mesh, and executes precom-putations as described in the next section. Next, the user selects avertex position on the surface of the mesh using the mouse, andassigns a sound clip to the position by a drag-and-drop operation.The sound clip defines the sound to be rendered when the modelis stuck at the position. The assigned sound clip can be either apre-recorded real sound (exists in the real world) or an artificialsound (e.g., sound generated by sound synthesizer), but it needs tobe an attenuated contact-like sound (free vibrational sound causedby single impulse. An impulse response is ideal). The system al-lows the user to select multiple positions for each correspondingsound clip. After assigning sounds, the user presses the "optimize"button, and the system optimizes the material distribution insidethe model to obtain the desired sound properties. Finally, the sys-tem exports the optimized embedded finite element mesh for thesurface model with the eigenpairs, and the user can use it for modalsound synthesis.

The optimization gradually progresses at an interactive rate. Thesystem visualizes the current material distribution inside the modelby colors and the resulting sounds when the sample positions arestruck by power spectrums. The user also can check the sound byclicking the mouse on the mesh surface during optimization at anytime. The user can stop the optimization procedure at an arbitrarytiming, reassign another sound to a new sample point, and restartthe optimization iteratively. In this way, the user can interactivelydesign the physically-based sound for a 3D object as if it were asound synthesizer.

4. Algorithm Overview

Figure 2 shows an overview of our optimization algorithm. Our al-gorithm consists of two stages: the precomputation stage and theruntime. The precomputation stage consists of two parts. One isprecomputation for each material set (independent of models), andit constructs regression forests for data-driven FEM. The regres-sion forests are used for online mesh coarsening using data-drivenFEM (§7.1). The other is precomputation for each input model (in-dependent of materials), and it involves voxelizing the model into ahexahedral FEM mesh and computation of the eigenvectors of thevolumetric Laplacian of the mesh following [XLCB15]. The eigen-vectors of the volumetric Laplacian are used for material reduction(§6), and mesh segmentation (§7.2).

At runtime, the system minimizes the perceptual difference be-tween the user-specified input sound and simulated sound by it-erative optimization of material distribution (§6). We consider vi-brational property (mode frequencies and amplitudes) to measureperceptual difference (§5). We optimized Young’s modulus at eachelement of FEM, and we kept the densities and Poisson’s ratiosconstant for simplicity. At each iteration, it is necessary to exe-cute a modal analysis of the model to compute the resulting sound.Conventional modal analysis solves the generalized eigenproblemof large stiffness and mass matrices, but it is prohibitively expen-sive and impractical to use during iterative optimization. To ad-dress this, we propose a fast approximate modal analysis based ona combination of data-driven FEM using regression forests (§7.1)and hierarchical component mode synthesis method including errorcorrection (§7.2).

5. Problem Formulation

When the user assigns a sound clip onto a sample position, thesystem extracts the parameters of the sound’s timbre from it. Anattenuated contact sound can be parameterized by modal parame-ters (frequencies, amplitudes, and dampings). For the details of themodal parameters, please see Appendix 1. We employ Ren et al.’stechnique [RYL13] to extract these parameters from a sound clip.We also extract the residual parameters following them. After T as-signments, the system has N sorted mode frequencies of assignedsounds (F1, ...,FN), corresponding dampings (D1, ...,DN), corre-sponding residuals (R1, ...,RN), and corresponding amplitudes atT sample positions (A1

1, ...,A1N), ...,(A

T1 , ...,A

TN). We call these ex-

tracted parameters as target parameters.

For a given finite element mesh, we compute the first Nmode frequencies ( f1, ..., fN) and corresponding amplitudes at Tsample positions (a1

1, ...,a1N), ...,(a

T1 , ...,a

TN) using modal analysis.

The modal analysis computes a generalized eigenproblem: KU =ΛMU , where K and M denote the stiffness and mass matrix respec-tively and Λ and U denote the eigenvalues and the correspondingeigenvectors. To compute the mode amplitudes, we assume eachsample position pi (i = 1, ...,Np) is struck by a unit force impulsef pi

n which has the inverse direction of the surface normal n at theposition. Then, the k-th mode amplitude at the position pi is repre-sented as api

k = uTk f pi

n , where uk is the k-th eigenvector.

Using the target frequencies F , amplitudes A and simulated pa-rameters, our objective function for minimizing the perceptual dif-ference of the mode frequencies is represented as

E f =12

N

∑i=2

!Bark(s f fi)−Bark(Fi)

"2 (1)

where Bark( f ) is a function to transform the frequency to criticalband rate [bark] [ZF99], and s f = F ′

1/ f1 is the scaling factor. The


Amplitude Differences


objective function for amplitudes is also obtained using the bal-ances with other mode amplitudes at the position

Ea =12

T

∑j=1

N

∑i=2

!a j

i

a jmax−

A ji

A jmax

"2

. (2)

where a jmax and A j

max denote the largest amplitude at the positionj of the simulated and target’s modes respectively. These formu-lations are similar to [BLT∗15]; however, we use the perceptualmetrics whereas they use square distances of frequencies and am-plitudes. We minimize these functions by optimizing the Young’smodulus Ye ∈ RM at each finite element e, where M denotes thenumber of the elements. Finally, our design problem is formulatedas

argminYe

: w f E f +waEa, sub ject to : Ye > 0 (3)

where w f and wa denote the positive weights.

Note that we do not optimize damping parameters. We insteadreuse the estimated damping from the assigned sound clips asmode-dependent damping. This means that our damping is not spa-tially constant. This setting is physically incorrect, but it makes theproblem simpler.

6. Material Optimization

The optimization of element-wise material parameters is imprac-tical. To reduce the design space of material parameters, we in-troduce the reduction technique of [XLCB15]. The technique ex-presses the Young’s modulus as Y = Φz using the eigenvectors ofthe volumetric mesh Laplacian Φ ∈ RM×m, and uses the general-ized material parameters z ∈ Rm, (m << M) for the optimization.Then, our design problem can be rewritten in the reduced space as

argminz

: w f E f +waEa +wrR, R =12

zT Qz (4)

where wr is a weight, R is the regularization term, and Q is thereduced Laplacian matrix which is diagonal and its entries consistof the eigenvalues of the volumetric mesh Laplacian (please see[XLCB15] for the details). This material reduction also has a meritto reduce the over-fitting problem.

We solve our design problem Eq. (4) by decomposing it into twoproblems min : E f and min : Ea, and minimizing them alternately.We employ a hybrid optimization scheme [CLJ09] of evolutionalstrategies (we used CMA-ES [HMK03]) and gradient descent ap-proach (we employed the Quasi Newton method). For the detailsof the gradient computation and this hybrid scheme, please see Ap-pendix 2, 3.

7. Fast Approximate Modal Analysis

At each iteration during our optimization, a modal analysis is re-quired for the evaluation of the objective function and its gradient.However, standard modal analysis (solving a generalized eigen-problem of large stiffness and mass matrices) is prohibitively ex-pensive and impossible to execute at an interactive rate. To addressthis, we present a method that combines extended data-driven on-line coarsening of finite elements (§7.1) and highly parallelized hi-erarchical component mode synthesis (§7.2).

7.1. Data-Driven FEM using Regression Forests

In this section, we explain the data-driven online coarsening ofthe FEM mesh. It takes the detailed voxel mesh (2× 2× 2× cube

4 Cubature Points

× ×

× ×× ×

× ×

DDFEM(e1,…,e4)e1 e2

e3 e4

E1 E2

E3 E4

16 Cubature Points

4 fine materials (in 2D) 4 coarse materials

× ×

× ×

× ×

× ×

× ×

× ×

Figure 4: Data-Driven Coarsening [Chen et al. 2015] (in 2D illus-tration). The function DDFEM() takes four material parameters(e1,e2,e3,e4) of fine four elements (left) and returns correspondingfour coarse material parameters (E1,E2,E3,E4) at the quadraturepoints (right) to minimize the error.

= = =

= = =

90° 90° 90°

Re

fle

cte

d V

ari

ati

on

s

Rotated Variations

e1 e2

e4 e3

e1 e2

e4 e3

e1 e2

e4 e3

e1 e2

e4 e3

e1 e2

e4 e3

e1 e2

e4 e3

e1 e2

e4 e3

e1 e2

e4 e3

= = = =

Figure 5: Eight equivalent cell variations (in 2D illustration). Thetop row represents four rotated variations and the bottom row rep-resents four reflected variations.

elements) as input and generates a coarse approximated mesh (acube element) as output using the the material parameter map-ping learned from training data in the precomputation step (Fig-ure 4). The concept of our data driven FEM coarsening is basedon [CLSM15]. The goal of their data-driven FEM is obtaining

(E1, ...,E8) = DDFEM(e1, ..,e8) (5)

where DDFEM() is a function that takes eight material parame-ters (e1, ...,e8) of a detailed mesh and returns the correspondingeight coarse material parameters (E1, ...,E8) at the cubature pointsto minimize the error. Their system computes this function for allpossible input values in precomputation and stores the result in themain memory. The system then evaluates this function referring thememory at runtime. It aggressively accelerates FEM while main-taining the accuracy by reducing the Dofs (24/81) and the numberof the cubature points (8/64) although the total number of the mate-rial parameters remains unchanged between the detailed and coarsemesh. However, in their approach, given N discrete materials, thenumber of material combinations becomes N8. Although they alsoproposed a compression algorithm by retaining only the small num-ber of representative material combinations, it still cannot be usedfor our material optimization that requires a large range of contin-uous material settings. Additionally, it is non-trivial to obtain anactual value from such representative materials. To address this,we present three techniques: 1: Overlapping Free Cell Ordering, 2:Scaling Factor Separation, 3: Regression Forests. The former twotechniques reduce the parameter space of the feature vector e (thedetailed eight material parameters) for efficient machine learning,and the last technique enables handling of a large amount of datasetwith a constant evaluation cost.

7.1.1. Overlapping Free Cell Ordering

As shown in Figure 5, the rotated and reflected variations of a ma-terial setting are basically equivalent. To enumerate such patternsincreases the parameter space of the feature vector unnecessarilyand it should be reduced for efficiency. To address this, we defineOverlapping Free Cell Ordering algorithm which makes explicitconsideration of rotated and reflected patterns unnecessary.

First, we redefine the data-driven function DDFEM() Eq.5 as.

Ei = DDFEMi(e1, ..,e8), i = 1,2, ...,8. (6)




Ea =12

T

∑j=1

N

∑i=2

!a j

i

a jmax−

A ji

A jmax

"2

. (2)



argminYe






argminz


zT Qz (4)







4 Cubature Points

× ×

× ×× ×

× ×


e3 e4

E1 E2

E3 E4

16 Cubature Points


× ×

× ×

× ×

× ×

× ×

× ×


= = =

= = =

90° 90° 90°

Re

fle

cte

d V

ari

ati

on

s

Rotated Variations

e1 e2

e4 e3

e1 e2

e4 e3

e1 e2

e4 e3

e1 e2

e4 e3

e1 e2

e4 e3

e1 e2

e4 e3

e1 e2

e4 e3

e1 e2

e4 e3

= = = =



(E1, ...,E8) = DDFEM(e1, ..,e8) (5)





Ei = DDFEMi(e1, ..,e8), i = 1,2, ...,8. (6)


We minimize:

Y: Young’s modulus

N: optimized modes, T: sample positions

We minimize perceptual differencesbetween simulated and target modal parameters.





Position 1

Position 2

Position 3

Position 4

Frequency [Hz]

Amplitude [dB]

Frequency [Hz]

Amplitude [dB]

Frequency [Hz]

Amplitude [dB]

Frequency [Hz]

Amplitude [dB]







3. User Workflow








1, ...,A1N), ...,(A

T1 , ...,A




1, ...,a1N), ...,(a

T1 , ...,a




k = uTk f pi



E f =12

N

∑i=2


"2 (1)







Ea =12

T

∑j=1

N

∑i=2

!a j

i

a jmax−

A ji

A jmax

"2

. (2)



argminYe






argminz


zT Qz (4)







4 Cubature Points

× ×

× ×× ×

× ×


e3 e4

E1 E2

E3 E4

16 Cubature Points


× ×

× ×

× ×

× ×

× ×

× ×


= = =

= = =

90° 90° 90°

Re

fle

cte

d V

ari

ati

on

s

Rotated Variations

e1 e2

e4 e3

e1 e2

e4 e3

e1 e2

e4 e3

e1 e2

e4 e3

e1 e2

e4 e3

e1 e2

e4 e3

e1 e2

e4 e3

e1 e2

e4 e3

= = = =



(E1, ...,E8) = DDFEM(e1, ..,e8) (5)





Ei = DDFEMi(e1, ..,e8), i = 1,2, ...,8. (6)




Ea =12

T

∑j=1

N

∑i=2

!a j

i

a jmax−

A ji

A jmax

"2

. (2)



argminYe






argminz


zT Qz (4)







4 Cubature Points

× ×

× ×× ×

× ×


e3 e4

E1 E2

E3 E4

16 Cubature Points


× ×

× ×

× ×

× ×

× ×

× ×


= = =

= = =

90° 90° 90°

Re

fle

cte

d V

ari

ati

on

s

Rotated Variations

e1 e2

e4 e3

e1 e2

e4 e3

e1 e2

e4 e3

e1 e2

e4 e3

e1 e2

e4 e3

e1 e2

e4 e3

e1 e2

e4 e3

e1 e2

e4 e3

= = = =



(E1, ...,E8) = DDFEM(e1, ..,e8) (5)





Ei = DDFEMi(e1, ..,e8), i = 1,2, ...,8. (6)


We minimize:








Position 1

Position 2

Position 3

Position 4

Frequency [Hz]

Amplitude [dB]

Frequency [Hz]

Amplitude [dB]

Frequency [Hz]

Amplitude [dB]

Frequency [Hz]

Amplitude [dB]







3. User Workflow








1, ...,A1N), ...,(A

T1 , ...,A




1, ...,a1N), ...,(a

T1 , ...,a




k = uTk f pi



E f =12

N

∑i=2


"2 (1)







Ea =12

T

∑j=1

N

∑i=2

!a j

i

a jmax−

A ji

A jmax

"2

. (2)



argminYe






argminz


zT Qz (4)







4 Cubature Points

× ×

× ×× ×

× ×


e3 e4

E1 E2

E3 E4

16 Cubature Points


× ×

× ×

× ×

× ×

× ×

× ×


= = =

= = =

90° 90° 90°

Re

fle

cte

d V

ari

ati

on

s

Rotated Variations

e1 e2

e4 e3

e1 e2

e4 e3

e1 e2

e4 e3

e1 e2

e4 e3

e1 e2

e4 e3

e1 e2

e4 e3

e1 e2

e4 e3

e1 e2

e4 e3

= = = =



(E1, ...,E8) = DDFEM(e1, ..,e8) (5)





Ei = DDFEMi(e1, ..,e8), i = 1,2, ...,8. (6)




Ea =12

T

∑j=1

N

∑i=2

!a j

i

a jmax−

A ji

A jmax

"2

. (2)



argminYe






argminz


zT Qz (4)







4 Cubature Points

× ×

× ×× ×

× ×


e3 e4

E1 E2

E3 E4

16 Cubature Points


× ×

× ×

× ×

× ×

× ×

× ×


= = =

= = =

90° 90° 90°

Re

fle

cte

d V

ari

ati

on

s

Rotated Variations

e1 e2

e4 e3

e1 e2

e4 e3

e1 e2

e4 e3

e1 e2

e4 e3

e1 e2

e4 e3

e1 e2

e4 e3

e1 e2

e4 e3

e1 e2

e4 e3

= = = =



(E1, ...,E8) = DDFEM(e1, ..,e8) (5)





Ei = DDFEMi(e1, ..,e8), i = 1,2, ...,8. (6)


We minimize:








Position 1

Position 2

Position 3

Position 4

Frequency [Hz]

Amplitude [dB]

Frequency [Hz]

Amplitude [dB]

Frequency [Hz]

Amplitude [dB]

Frequency [Hz]

Amplitude [dB]







3. User Workflow








1, ...,A1N), ...,(A

T1 , ...,A




1, ...,a1N), ...,(a

T1 , ...,a




k = uTk f pi



E f =12

N

∑i=2


"2 (1)







Ea =12

T

∑j=1

N

∑i=2

!a j

i

a jmax−

A ji

A jmax

"2

. (2)



argminYe






argminz


zT Qz (4)







4 Cubature Points

× ×

× ×× ×

× ×


e3 e4

E1 E2

E3 E4

16 Cubature Points


× ×

× ×

× ×

× ×

× ×

× ×


= = =

= = =

90° 90° 90°

Re

fle

cte

d V

ari

ati

on

s

Rotated Variations

e1 e2

e4 e3

e1 e2

e4 e3

e1 e2

e4 e3

e1 e2

e4 e3

e1 e2

e4 e3

e1 e2

e4 e3

e1 e2

e4 e3

e1 e2

e4 e3

= = = =



(E1, ...,E8) = DDFEM(e1, ..,e8) (5)





Ei = DDFEMi(e1, ..,e8), i = 1,2, ...,8. (6)




Ea =12

T

∑j=1

N

∑i=2

!a j

i

a jmax−

A ji

A jmax

"2

. (2)



argminYe






argminz


zT Qz (4)







4 Cubature Points

× ×

× ×× ×

× ×


e3 e4

E1 E2

E3 E4

16 Cubature Points


× ×

× ×

× ×

× ×

× ×

× ×


= = =

= = =

90° 90° 90°

Re

fle

cte

d V

ari

ati

on

s

Rotated Variations

e1 e2

e4 e3

e1 e2

e4 e3

e1 e2

e4 e3

e1 e2

e4 e3

e1 e2

e4 e3

e1 e2

e4 e3

e1 e2

e4 e3

e1 e2

e4 e3

= = = =



(E1, ...,E8) = DDFEM(e1, ..,e8) (5)





Ei = DDFEMi(e1, ..,e8), i = 1,2, ...,8. (6)


We minimize:





We solve this problem by material reduction [Xu et al. ’15], and a hybrid optimization scheme of evolutional strategies (CMA-ES)

and gradient descent (quasi-Newton).



Ea =12

T

∑j=1

N

∑i=2

!a j

i

a jmax−

A ji

A jmax

"2

. (2)



argminYe






argminz


zT Qz (4)







4 Cubature Points

× ×

× ×× ×

× ×


e3 e4

E1 E2

E3 E4

16 Cubature Points


× ×

× ×

× ×

× ×

× ×

× ×


= = =

= = =

90° 90° 90°

Re

fle

cte

d V

ari

ati

on

s

Rotated Variations

e1 e2

e4 e3

e1 e2

e4 e3

e1 e2

e4 e3

e1 e2

e4 e3

e1 e2

e4 e3

e1 e2

e4 e3

e1 e2

e4 e3

e1 e2

e4 e3

= = = =



(E1, ...,E8) = DDFEM(e1, ..,e8) (5)





Ei = DDFEMi(e1, ..,e8), i = 1,2, ...,8. (6)


We minimize:


Modal Analysis




Runtime



+

…


Input: Surface Mesh





Sample Position 1


Simulated Sound







Sample Position 3

Timbre

Precomputation

Regression Forests













Runtime



+

…


Input: Surface Mesh





Sample Position 1


Simulated Sound







Sample Position 3

Timbre

Precomputation

Regression Forests













Runtime



+

…


Input: Surface Mesh





Sample Position 1


Simulated Sound







Sample Position 3

Timbre

Precomputation

Regression Forests










A FEM Mesh





Runtime



+

…


Input: Surface Mesh





Sample Position 1


Simulated Sound







Sample Position 3

Timbre

Precomputation

Regression Forests










Simulated Sound




→ (f1, a1, d1)

(modal parameters)

→ (f2, a2, d2)

→ (f3, a3, d3)

Modal Analysis




Runtime



+

…


Input: Surface Mesh





Sample Position 1


Simulated Sound







Sample Position 3

Timbre

Precomputation

Regression Forests













Runtime



+

…


Input: Surface Mesh





Sample Position 1


Simulated Sound







Sample Position 3

Timbre

Precomputation

Regression Forests













Runtime



+

…


Input: Surface Mesh





Sample Position 1


Simulated Sound







Sample Position 3

Timbre

Precomputation

Regression Forests










A FEM Mesh





Runtime



+

…


Input: Surface Mesh





Sample Position 1


Simulated Sound







Sample Position 3

Timbre

Precomputation

Regression Forests










Simulated Sound




→ (f1, a1, d1)

(modal parameters)

→ (f2, a2, d2)

→ (f3, a3, d3)

KU = ΛMUK: Stiffness MatrixM: Mass Matrix

Expensive




Runtime



+

…


Input: Surface Mesh





Sample Position 1


Simulated Sound







Sample Position 3

Timbre

Precomputation

Regression Forests













Runtime



+

…


Input: Surface Mesh





Sample Position 1


Simulated Sound







Sample Position 3

Timbre

Precomputation

Regression Forests










Mesh Coarsening with Data-Driven FEM

Fast Approximate Modal Analysis

Hierarchical Component Mode Synthesis




Runtime



+

…


Input: Surface Mesh





Sample Position 1


Simulated Sound







Sample Position 3

Timbre

Precomputation

Regression Forests










+

Data-Driven FEM using Regression Forests




Runtime



+

…


Input: Surface Mesh





Sample Position 1


Simulated Sound







Sample Position 3

Timbre

Precomputation

Regression Forests













Runtime



+

…


Input: Surface Mesh





Sample Position 1


Simulated Sound







Sample Position 3

Timbre

Precomputation

Regression Forests










Machine Learning

Detailed mesh Coarse mesh

Based on “Data-Driven Finite Elements for Geometry and Material Design” , Chen et al. SIGGRAPH 2015

Data-Driven FEM (DDFEM) [Chen et al. 15]Kazuhiko Yamamoto & Takeo Igarashi / Interactive Physically-Based Sound Design of 3D Modelusing Material Optimization


Ea =12

T

∑j=1

N

∑i=2

!a j

i

a jmax−

A ji

A jmax

"2

. (2)



argminYe






argminz


zT Qz (4)







4 Cubature Points

× ×

× ×× ×

× ×


e3 e4

E1 E2

E3 E4

16 Cubature Points


× ×

× ×

× ×

× ×

× ×

× ×


= = =

= = =

90° 90° 90°

Re

fle

cte

d V

ari

ati

on

s

Rotated Variations

e1 e2

e4 e3

e1 e2

e4 e3

e1 e2

e4 e3

e1 e2

e4 e3

e1 e2

e4 e3

e1 e2

e4 e3

e1 e2

e4 e3

e1 e2

e4 e3

= = = =



(E1, ...,E8) = DDFEM(e1, ..,e8) (5)





Ei = DDFEMi(e1, ..,e8), i = 1,2, ...,8. (6)


Pushing Twisting12x Faster 15x Faster

(a) (b)

(c) (d)

Figure 10: Examples of pushing a cube (a - Initial State, c - Com-pressed) and twisting a bar (b - Initial State, d - Compressed), bothwith heterogeneous material distribution. We compare DDFEM toNaıve Coarsening and the ground-truth, High-Res Simulation. Werender wire frames to show the simulation meshes.

1 Level of Coarsening 2 Levels of Coarsening13x Faster 23x Faster

(a) (b)

(c) (d)

Figure 11: Bending a heterogeneous bar: We compare DDFEMto Naıve Coarsening and a High-Res Simulation. Subfigures (a, c)show comparison for 1 level of coarsening, and (b, d) show 2 levelsof coarsening. The naıve coarsening approach results in a muchstiffer behavior, whereas our fitted model more closely approximatesthe fine model.

finer element, 0p contains 2 material moduli plus C,v.) For furtherrecursive levels, we can limit ourselves to 320 values per material.Our current 3 material database is 4 megabytes in size.

6.2 Simulation Results

We show results from elastostatic simulations performed usingDDFEM. We also demonstrate its performance advantages overhigh-resolution simulations. We render wire frames to show thediscretizations of the high-resolution and coarse meshes. We firstshow examples of two simple simulations, the pushing and twist-ing of a rectangular object with heterogeneous, layered materialdistribution (Fig. 10). Note that in all cases DDFEM qualitativelymatches the behavior of the high-resolution simulation. We alsocompare the performance of DDFEM to a naıve coarsening methodthat uses the material properties from 2⇥2⇥2 element blocks of thehigh-resolution simulation mesh at each corresponding quadraturepoint. In our supplemental video we compare to a second baselinemodel which averages material parameters inside each coarse ele-ment. This average model is less accurate than the Naıve model in


(a) (b)

(c) (d)

Figure 12: Compressing a heterogeneous slab using Naıve Coars-ening (1 level and 2 levels of coarsening), DDFEM (1 level and 2levels of coarsening) and a High-Res Simulation. The top, darkerlayer is stiffer, causing the object to buckle. The bottom verticesare constrained to stay on the floor. Figure (a,b) shows the slabsbefore compression, figure (c,d) shows the slabs after compressionand figure. Notice that, after 1 level of coarsening, Naıve Coarsen-ing neither compresses nor buckles as much as either DDFEM orHigh-Res Simulation. After 2 levels of coarsening, the buckling be-havior is lost. The Naıve Coarsening fails to capture the compressivebehavior of High-Res Simulation, whereas DDFEM does.


(a) (b)

(c) (d)

Figure 13: Simulating a bar with an embedded set of fibers usingNaıve Coarsening, DDFEM and a High-Res Simulation. Note thatDDFEM captures the characteristic twisting motion of the bar betterthan Naıve Coarsening. (a,b) shows the initial state of both barswhile (c,d) shows the deformed state after pulling on the top of thebars.

all cases.Naıve approaches often exhibit pathological stiffness for heteroge-neous materials (illustrated by the lack of compression of the boxand lack of twisting of the bar) [Nesme et al. 2009]. In these cases,DDFEM yields good speed ups while maintaining accuracy. For asingle level of coarsening we achieve 8 times or greater speed upsfor all examples. Performance numbers and mean errors are listedin Table 1. Since the fine simulation and the coarse simulation havedifferent numbers of vertices, we create a fine mesh from the coarsesimulation by trilinearly interpolating the fine vertices using thecoarse displacements. The errors are measured by computing the av-erage vertex distance relative to the longest dimension of the bound-ing box in rest shape. We also examine the behavior of DDFEM

“Data-Driven Finite Elements for Geometry and Material Design” , Chen et al. SIGGRAPH 2015

Data-Driven FEM (DDFEM) [Chen et al. 15]

Given N materials, the number of combinations become N8

It is impractical to use for our problem, because it requires a large range of continuous material settings.

Extended Data-Driven FEM (DDFEM*)

To Reduce the material space

1. Overlapping-Free Cell Ordering

2. Scaling Factor Separation

3. Regression Forests To handle large amount of datasetwith a constant evaluation cost

1. Overlapping-Free Cell Ordering



Ea =12

T

∑j=1

N

∑i=2

!a j

i

a jmax−

A ji

A jmax

"2

. (2)



argminYe






argminz


zT Qz (4)







4 Cubature Points

× ×

× ×× ×

× ×


e3 e4

E1 E2

E3 E4

16 Cubature Points


× ×

× ×

× ×

× ×

× ×

× ×


= = =

= = =

90° 90° 90°R

efl

ec

ted

Va

ria

tio

ns

Rotated Variations

e1 e2

e4 e3

e1 e2

e4 e3

e1 e2

e4 e3

e1 e2

e4 e3

e1 e2

e4 e3

e1 e2

e4 e3

e1 e2

e4 e3

e1 e2

e4 e3

= = = =



(E1, ...,E8) = DDFEM(e1, ..,e8) (5)





Ei = DDFEMi(e1, ..,e8), i = 1,2, ...,8. (6)


Basically equivalent cell variations (in 2D)

should be emitted from the dataset.

1. Overlapping-Free Cell OrderingKazuhiko Yamamoto & Takeo Igarashi / Interactive Physically-Based Sound Design of 3D Modelusing Material Optimization

The Origin Cell (i 1)e1 e’1

1: Initial ordering

Reordering (i = 1)

e1 e2

e4 e3

e’1 e2

e4 e3

2: Compare the neighbors

e’1 e’2

e’3 e’4

e’1 e’3

e’2 e’4

e’1 e’2

e’3 e3

e’1 e’3

e’2 e3

3: If e2 e4

3: otherwise

4: Fill rests

Figure 6: Overlapping Free Cell Ordering (in 2D illustration). 1:At the i-th cell evaluation (in this example, we assume i = 2), eibecomes the origin cell e′1. 2: we compare the material values ofthe adjacent cells. 3: the smaller cell becomes e′2 and the otherbecomes e′3. 4: The left cell becomes e′4.

Our data-driven FEM function returns a scalar while Eq.5 outputsa R8 vector. It means we repeat this DDFEM() evaluation eighttimes to convert a detailed 2× 2× 2 element into a coarse ele-ment. Next, we reorder the numbering of the eight cells by eachDDFEM() evaluation. We show this operation as a 2D example inFigure 6. The indices of the cells are defined in a local R3 spacecoordinate. At the i-th evaluation within the eight evaluations, wedefine the i-th cell as the origin e′1. Then, we compare the value ofthe Young’s modulus of the three adjacent cells of the origin cell (in2D, two cells), and define the index the cell who has the smallestvalue as e′2, the cell who has the secondary smallest value as e′3, andthe other cell as e′4. Finally, we decide the ordering of the rest fourcells by the following rule: The cell that is adjacent to e′2 and e′3becomes e′5. The cell that is adjacent to e′3 and e′4 becomes e′6. Thecell that is adjacent to e′2 and e′4 becomes e′7. The last one becomese′8.

Then, using these reordered parameters, our DDFEM() functionis redefined again as

Ei = DDFEMi(e′1,e

′2, ...,e

′8), e′1 = ei (7)

By using this representation, we can avoid explicit enumeration ofthe eight rotated and eight reflected patterns of a material pattern,and reduce the input parameter space in 3D at both training andruntime. For dataset generation at the training, we first determinethe value at the origin cell, and seed the values at the three cells e′2,e′3, e′4 to be e′2 ≤ e′3 ≤ e′4, and the rest of the values are randomlyseeded.

7.1.2. Scaling Factor Separation

Young’s modulus has a large range of the value 10−2 (Rub-ber)∼ 103 (Diamond) GPa while Poisson’s ratio has a small range(−1/2, 1/2). It is difficult to treat a practical amount of data forsuch a large range during training. To avoid this, we dramaticallyreduce the training size by separating the scale factor.

Based on [CLSM15], our DDFEM() is constructed to minimizethe square difference of the integral of the strain energy densityfunctions between the detailed and coarse meshes.

argminEi

∑f∈F

!8

∑i=1

wivci ( f ,Ei)−

8

∑j=1

8

∑i=1

wivdji( f ,e′j)

"2

(8)

where w denotes the cubature weights, F denotes a set of randomlysampled external forces, and vc and vd represent the strain energydensity function of the coarse and detailed mesh respectively. Here,the strain energy density function in linear elastic is representedas v( f ,e) = K(e)u(e)2 = K(e)(K−1(e) f )2 where K(e) and f arethe stiffness matrix and the external forces respectively. In addi-tion, multiplying e by a scalar s, v( f ,s · e) = K(s · e)u(s · e)2 =

sK(e)((sK(e))−1 f )2 = v( f ,e)/s because K() is the linear functionof e. Then, the minimization problem

argminEi

∑f∈F

!8

∑i=1

wivci ( f ,Ei)−

8

∑j=1

8

∑i=1

wivdji( f ,s · e′j)

"2

(9)

is equivalent to

argminE′

i =Ei/s∑f∈F

!8

∑i=1

wivci ( f ,E′

i )−8

∑j=1

8

∑i=1

wivdji( f ,e′j)

"2

(10)

This means that we can separate the input parameter space of ourDDFEM() problem by the multiplication of the value of the origincell as a scale factor and their quotients. Finally, we can obtain ourDDFEM() function as

Ei = ei ·DDFEMi

#e′2ei, ...,

e′8ei

$(11)

An advantage of this representation is that it reduces not only therange of dataset but also the dimensions of the feature vector fromR8 to R7. Note that we assume our model as linear elastics althoughthe original DDFEM() treats nonlinearity because the vibrationalanalysis discussed in this paper is a linear analysis. Introducing thenonlinearity for large deformation is a future work.

7.1.3. Regression Forests

In contract with Chen et al.’s method [CLSM15], we do not con-struct the database of data-driven materials because of two rea-sons. First, their database approach cannot handle the inputs thatare not included in the training dataset because it has no general-ization ability. Second, the evaluation cost at runtime is increased ata rate proportional to the amount of the dataset although the amountof the dataset should be increased for handling more material pat-terns. To address these problems, we train our DDFEM() func-tion using two regression forests. Our regression forests are similarto [LJS∗15] which construct each tree through two steps training:tree structure construction with a subset of learning data and least-square solve for the regression coefficients at each leaf node withall the dataset. The regression forest has an advantage of constantcost evaluation even if the amount of the dataset is increased. Fi-nally, our DDFEM() becomes

%Ei = e ·Reg1(

e′2e , ...,

e′8e ) (ei = 0)

Ei = ei ·Reg2(e′2ei, ...,

e′8ei) (ei > 0)

(12)

where Reg() represents the regression function, and e is the averageof the Young’s modulus in the target eight cells.

7.2. Hierarchical Component Mode Synthesis

After coarsening the mesh, we compute modal analysis using anovel hierarchical component mode synthesis method (HCMS) in-cluding an efficient error correction algorithm (Figure 7). It takesthe coarse voxel mesh as input and solves a generalized eigenprob-lem via hierarchical merging. It first decomposes the mesh intosmall components and solves a generalized eigenproblem for eachcomponent. It then hierarchically merges adjacent components andsolves generalized eigenproblems for the merged component. Con-ventional CMS [BC68] computes the eigenmodes of a structureby combining several small local subdomains after decomposing itinto several small subdomains. Our HCMS decomposes a structureinto finer subdomains compared to conventional CMS to increasethe computational efficiency while sacrificing accuracy. To com-pensate for the loss of accuracy, we apply a subspace iterative errorcorrection using the result of HCMS as an initial solution.




1: Initial ordering

Reordering (i = 1)

e1 e2

e4 e3

e’1 e2

e4 e3


e’1 e’2

e’3 e’4

e’1 e’3

e’2 e’4

e’1 e’2

e’3 e3

e’1 e’3

e’2 e3

3: If e2 e4

3: otherwise

4: Fill rests




Ei = DDFEMi(e′1,e

′2, ...,e

′8), e′1 = ei (7)





argminEi

∑f∈F

!8

∑i=1

wivci ( f ,Ei)−

8

∑j=1

8

∑i=1

wivdji( f ,e′j)

"2

(8)



argminEi

∑f∈F

!8

∑i=1

wivci ( f ,Ei)−

8

∑j=1

8

∑i=1


"2

(9)

is equivalent to

argminE′

i =Ei/s∑f∈F

!8

∑i=1

wivci ( f ,E′

i )−8

∑j=1

8

∑i=1

wivdji( f ,e′j)

"2

(10)


Ei = ei ·DDFEMi

#e′2ei, ...,

e′8ei

$(11)




%Ei = e ·Reg1(

e′2e , ...,

e′8e ) (ei = 0)


e′8ei) (ei > 0)

(12)





sorted by the Young’s modulus.

Please see the paper for the details of the sorting procedure.



Ea =12

T

∑j=1

N

∑i=2

!a j

i

a jmax−

A ji

A jmax

"2

. (2)



argminYe






argminz


zT Qz (4)







4 Cubature Points

× ×

× ×× ×

× ×


e3 e4

E1 E2

E3 E4

16 Cubature Points


× ×

× ×

× ×

× ×

× ×

× ×


= = =

= = =

90° 90° 90°

Re

fle

cte

d V

ari

ati

on

s

Rotated Variations

e1 e2

e4 e3

e1 e2

e4 e3

e1 e2

e4 e3

e1 e2

e4 e3

e1 e2

e4 e3

e1 e2

e4 e3

e1 e2

e4 e3

e1 e2

e4 e3

= = = =



(E1, ...,E8) = DDFEM(e1, ..,e8) (5)





Ei = DDFEMi(e1, ..,e8), i = 1,2, ...,8. (6)


Young’s modulus has a large range of the value (10-2 ~ 103 GPa)


It is difficult to treat such a large range by DDFEM directly !!




1: Initial ordering

Reordering (i = 1)

e1 e2

e4 e3

e’1 e2

e4 e3


e’1 e’2

e’3 e’4

e’1 e’3

e’2 e’4

e’1 e’2

e’3 e3

e’1 e’3

e’2 e3

3: If e2 e4

3: otherwise

4: Fill rests




Ei = DDFEMi(e′1,e

′2, ...,e

′8), e′1 = ei (7)





argminEi

∑f∈F

!8

∑i=1

wivci ( f ,Ei)−

8

∑j=1

8

∑i=1

wivdji( f ,e′j)

"2

(8)



argminEi

∑f∈F

!8

∑i=1

wivci ( f ,Ei)−

8

∑j=1

8

∑i=1


"2

(9)

is equivalent to

argminE′

i =Ei/s∑f∈F

!8

∑i=1

wivci ( f ,E′

i )−8

∑j=1

8

∑i=1

wivdji( f ,e′j)

"2

(10)


Ei = ei ·DDFEMi

#e′2ei, ...,

e′8ei

$(11)




%Ei = e ·Reg1(

e′2e , ...,

e′8e ) (ei = 0)


e′8ei) (ei > 0)

(12)





F: force set SED of coarse mesh

DDFEM() is designed to minimize the square difference of the integral of the strain energy density (SED).

SED of fine mesh



In linear elastics:


1: Initial ordering

e1 e2

e4 e3

e1 e’1

e4 e3


e’2 e’1

e’4 e’3

e’3 e’1

e’4 e’2

e’2 e’1

e4 e3

e’3 e’1

e4 e’2

3: If e1 ≦ e3

3: otherwise

4: Fill rests

＊

The Origin Cell (i = 2)e2 → e’1




Ei = DDFEMi(e′1,e

′2, ...,e

′8), e′1 = ei (7)





argminEi

∑f∈F

!8

∑i=1

wivci ( f ,Ei)−

8

∑j=1

8

∑i=1

wivdji( f ,e′j)

"2

(8)



argminEi

∑f∈F

!8

∑i=1

wivci ( f ,Ei)−

8

∑j=1

8

∑i=1


"2

(9)

is equivalent to

argminE′

i =Ei/s∑f∈F

!8

∑i=1

wivci ( f ,E′

i )−8

∑j=1

8

∑i=1

wivdji( f ,e′j)

"2

(10)


Ei = ei ·DDFEMi

#e′2ei, ...,

e′8ei

$(11)




%Ei = e ·Reg1(

e′2e , ...,

e′8e ) (ei = 0)


e′8ei) (ei > 0)

(12)






1: Initial ordering

e1 e2

e4 e3

e1 e’1

e4 e3


e’2 e’1

e’4 e’3

e’3 e’1

e’4 e’2

e’2 e’1

e4 e3

e’3 e’1

e4 e’2

3: If e1 ≦ e3

3: otherwise

4: Fill rests

＊





Ei = DDFEMi(e′1,e

′2, ...,e

′8), e′1 = ei (7)





argminEi

∑f∈F

!8

∑i=1

wivci ( f ,Ei)−

8

∑j=1

8

∑i=1

wivdji( f ,e′j)

"2

(8)



argminEi

∑f∈F

!8

∑i=1

wivci ( f ,Ei)−

8

∑j=1

8

∑i=1


"2

(9)

is equivalent to

argminE′

i =Ei/s∑f∈F

!8

∑i=1

wivci ( f ,E′

i )−8

∑j=1

8

∑i=1

wivdji( f ,e′j)

"2

(10)


Ei = ei ·DDFEMi

#e′2ei, ...,

e′8ei

$(11)




%Ei = e ·Reg1(

e′2e , ...,

e′8e ) (ei = 0)


e′8ei) (ei > 0)

(12)





s: scholar


In linear elastics:


1: Initial ordering

e1 e2

e4 e3

e1 e’1

e4 e3


e’2 e’1

e’4 e’3

e’3 e’1

e’4 e’2

e’2 e’1

e4 e3

e’3 e’1

e4 e’2

3: If e1 ≦ e3

3: otherwise

4: Fill rests

＊





Ei = DDFEMi(e′1,e

′2, ...,e

′8), e′1 = ei (7)





argminEi

∑f∈F

!8

∑i=1

wivci ( f ,Ei)−

8

∑j=1

8

∑i=1

wivdji( f ,e′j)

"2

(8)



argminEi

∑f∈F

!8

∑i=1

wivci ( f ,Ei)−

8

∑j=1

8

∑i=1


"2

(9)

is equivalent to

argminE′

i =Ei/s∑f∈F

!8

∑i=1

wivci ( f ,E′

i )−8

∑j=1

8

∑i=1

wivdji( f ,e′j)

"2

(10)


Ei = ei ·DDFEMi

#e′2ei, ...,

e′8ei

$(11)




%Ei = e ·Reg1(

e′2e , ...,

e′8e ) (ei = 0)


e′8ei) (ei > 0)

(12)






1: Initial ordering

e1 e2

e4 e3

e1 e’1

e4 e3


e’2 e’1

e’4 e’3

e’3 e’1

e’4 e’2

e’2 e’1

e4 e3

e’3 e’1

e4 e’2

3: If e1 ≦ e3

3: otherwise

4: Fill rests

＊





Ei = DDFEMi(e′1,e

′2, ...,e

′8), e′1 = ei (7)





argminEi

∑f∈F

!8

∑i=1

wivci ( f ,Ei)−

8

∑j=1

8

∑i=1

wivdji( f ,e′j)

"2

(8)



argminEi

∑f∈F

!8

∑i=1

wivci ( f ,Ei)−

8

∑j=1

8

∑i=1


"2

(9)

is equivalent to

argminE′

i =Ei/s∑f∈F

!8

∑i=1

wivci ( f ,E′

i )−8

∑j=1

8

∑i=1

wivdji( f ,e′j)

"2

(10)


Ei = ei ·DDFEMi

#e′2ei, ...,

e′8ei

$(11)




%Ei = e ·Reg1(

e′2e , ...,

e′8e ) (ei = 0)


e′8ei) (ei > 0)

(12)





s: scholar

Then,



1: Initial ordering

Reordering (i = 1)

e1 e2

e4 e3

e’1 e2

e4 e3


e’1 e’2

e’3 e’4

e’1 e’3

e’2 e’4

e’1 e’2

e’3 e3

e’1 e’3

e’2 e3

3: If e2 e4

3: otherwise

4: Fill rests




Ei = DDFEMi(e′1,e

′2, ...,e

′8), e′1 = ei (7)





argminEi

∑f∈F

!8

∑i=1

wivci ( f ,Ei)−

8

∑j=1

8

∑i=1

wivdji( f ,e′j)

"2

(8)



argminEi

∑f∈F

!8

∑i=1

wivci ( f ,Ei)−

8

∑j=1

8

∑i=1


"2

(9)

is equivalent to

argminE′

i =Ei/s∑f∈F

!8

∑i=1

wivci ( f ,E′

i )−8

∑j=1

8

∑i=1

wivdji( f ,e′j)

"2

(10)


Ei = ei ·DDFEMi

#e′2ei, ...,

e′8ei

$(11)




%Ei = e ·Reg1(

e′2e , ...,

e′8e ) (ei = 0)


e′8ei) (ei > 0)

(12)





is equivalent to



1: Initial ordering

Reordering (i = 1)

e1 e2

e4 e3

e’1 e2

e4 e3


e’1 e’2

e’3 e’4

e’1 e’3

e’2 e’4

e’1 e’2

e’3 e3

e’1 e’3

e’2 e3

3: If e2 e4

3: otherwise

4: Fill rests




Ei = DDFEMi(e′1,e

′2, ...,e

′8), e′1 = ei (7)





argminEi

∑f∈F

!8

∑i=1

wivci ( f ,Ei)−

8

∑j=1

8

∑i=1

wivdji( f ,e′j)

"2

(8)



argminEi

∑f∈F

!8

∑i=1

wivci ( f ,Ei)−

8

∑j=1

8

∑i=1


"2

(9)

is equivalent to

argminE′

i =Ei/s∑f∈F

!8

∑i=1

wivci ( f ,E′

i )−8

∑j=1

8

∑i=1

wivdji( f ,e′j)

"2

(10)


Ei = ei ·DDFEMi

#e′2ei, ...,

e′8ei

$(11)




%Ei = e ·Reg1(

e′2e , ...,

e′8e ) (ei = 0)


e′8ei) (ei > 0)

(12)






In linear elastics:


1: Initial ordering

e1 e2

e4 e3

e1 e’1

e4 e3


e’2 e’1

e’4 e’3

e’3 e’1

e’4 e’2

e’2 e’1

e4 e3

e’3 e’1

e4 e’2

3: If e1 ≦ e3

3: otherwise

4: Fill rests

＊





Ei = DDFEMi(e′1,e

′2, ...,e

′8), e′1 = ei (7)





argminEi

∑f∈F

!8

∑i=1

wivci ( f ,Ei)−

8

∑j=1

8

∑i=1

wivdji( f ,e′j)

"2

(8)



argminEi

∑f∈F

!8

∑i=1

wivci ( f ,Ei)−

8

∑j=1

8

∑i=1


"2

(9)

is equivalent to

argminE′

i =Ei/s∑f∈F

!8

∑i=1

wivci ( f ,E′

i )−8

∑j=1

8

∑i=1

wivdji( f ,e′j)

"2

(10)


Ei = ei ·DDFEMi

#e′2ei, ...,

e′8ei

$(11)




%Ei = e ·Reg1(

e′2e , ...,

e′8e ) (ei = 0)


e′8ei) (ei > 0)

(12)






1: Initial ordering

e1 e2

e4 e3

e1 e’1

e4 e3


e’2 e’1

e’4 e’3

e’3 e’1

e’4 e’2

e’2 e’1

e4 e3

e’3 e’1

e4 e’2

3: If e1 ≦ e3

3: otherwise

4: Fill rests

＊





Ei = DDFEMi(e′1,e

′2, ...,e

′8), e′1 = ei (7)





argminEi

∑f∈F

!8

∑i=1

wivci ( f ,Ei)−

8

∑j=1

8

∑i=1

wivdji( f ,e′j)

"2

(8)



argminEi

∑f∈F

!8

∑i=1

wivci ( f ,Ei)−

8

∑j=1

8

∑i=1


"2

(9)

is equivalent to

argminE′

i =Ei/s∑f∈F

!8

∑i=1

wivci ( f ,E′

i )−8

∑j=1

8

∑i=1

wivdji( f ,e′j)

"2

(10)


Ei = ei ·DDFEMi

#e′2ei, ...,

e′8ei

$(11)




%Ei = e ·Reg1(

e′2e , ...,

e′8e ) (ei = 0)


e′8ei) (ei > 0)

(12)





s: scholar

Then,



1: Initial ordering

Reordering (i = 1)

e1 e2

e4 e3

e’1 e2

e4 e3


e’1 e’2

e’3 e’4

e’1 e’3

e’2 e’4

e’1 e’2

e’3 e3

e’1 e’3

e’2 e3

3: If e2 e4

3: otherwise

4: Fill rests




Ei = DDFEMi(e′1,e

′2, ...,e

′8), e′1 = ei (7)





argminEi

∑f∈F

!8

∑i=1

wivci ( f ,Ei)−

8

∑j=1

8

∑i=1

wivdji( f ,e′j)

"2

(8)



argminEi

∑f∈F

!8

∑i=1

wivci ( f ,Ei)−

8

∑j=1

8

∑i=1


"2

(9)

is equivalent to

argminE′

i =Ei/s∑f∈F

!8

∑i=1

wivci ( f ,E′

i )−8

∑j=1

8

∑i=1

wivdji( f ,e′j)

"2

(10)


Ei = ei ·DDFEMi

#e′2ei, ...,

e′8ei

$(11)




%Ei = e ·Reg1(

e′2e , ...,

e′8e ) (ei = 0)


e′8ei) (ei > 0)

(12)





is equivalent to



1: Initial ordering

Reordering (i = 1)

e1 e2

e4 e3

e’1 e2

e4 e3


e’1 e’2

e’3 e’4

e’1 e’3

e’2 e’4

e’1 e’2

e’3 e3

e’1 e’3

e’2 e3

3: If e2 e4

3: otherwise

4: Fill rests




Ei = DDFEMi(e′1,e

′2, ...,e

′8), e′1 = ei (7)





argminEi

∑f∈F

!8

∑i=1

wivci ( f ,Ei)−

8

∑j=1

8

∑i=1

wivdji( f ,e′j)

"2

(8)



argminEi

∑f∈F

!8

∑i=1

wivci ( f ,Ei)−

8

∑j=1

8

∑i=1


"2

(9)

is equivalent to

argminE′

i =Ei/s∑f∈F

!8

∑i=1

wivci ( f ,E′

i )−8

∑j=1

8

∑i=1

wivdji( f ,e′j)

"2

(10)


Ei = ei ·DDFEMi

#e′2ei, ...,

e′8ei

$(11)




%Ei = e ·Reg1(

e′2e , ...,

e′8e ) (ei = 0)


e′8ei) (ei > 0)

(12)






In linear elastics:


1: Initial ordering

e1 e2

e4 e3

e1 e’1

e4 e3


e’2 e’1

e’4 e’3

e’3 e’1

e’4 e’2

e’2 e’1

e4 e3

e’3 e’1

e4 e’2

3: If e1 ≦ e3

3: otherwise

4: Fill rests

＊





Ei = DDFEMi(e′1,e

′2, ...,e

′8), e′1 = ei (7)





argminEi

∑f∈F

!8

∑i=1

wivci ( f ,Ei)−

8

∑j=1

8

∑i=1

wivdji( f ,e′j)

"2

(8)



argminEi

∑f∈F

!8

∑i=1

wivci ( f ,Ei)−

8

∑j=1

8

∑i=1


"2

(9)

is equivalent to

argminE′

i =Ei/s∑f∈F

!8

∑i=1

wivci ( f ,E′

i )−8

∑j=1

8

∑i=1

wivdji( f ,e′j)

"2

(10)


Ei = ei ·DDFEMi

#e′2ei, ...,

e′8ei

$(11)




%Ei = e ·Reg1(

e′2e , ...,

e′8e ) (ei = 0)


e′8ei) (ei > 0)

(12)






1: Initial ordering

e1 e2

e4 e3

e1 e’1

e4 e3


e’2 e’1

e’4 e’3

e’3 e’1

e’4 e’2

e’2 e’1

e4 e3

e’3 e’1

e4 e’2

3: If e1 ≦ e3

3: otherwise

4: Fill rests

＊





Ei = DDFEMi(e′1,e

′2, ...,e

′8), e′1 = ei (7)





argminEi

∑f∈F

!8

∑i=1

wivci ( f ,Ei)−

8

∑j=1

8

∑i=1

wivdji( f ,e′j)

"2

(8)



argminEi

∑f∈F

!8

∑i=1

wivci ( f ,Ei)−

8

∑j=1

8

∑i=1


"2

(9)

is equivalent to

argminE′

i =Ei/s∑f∈F

!8

∑i=1

wivci ( f ,E′

i )−8

∑j=1

8

∑i=1

wivdji( f ,e′j)

"2

(10)


Ei = ei ·DDFEMi

#e′2ei, ...,

e′8ei

$(11)




%Ei = e ·Reg1(

e′2e , ...,

e′8e ) (ei = 0)


e′8ei) (ei > 0)

(12)





s: scholar

Then,



1: Initial ordering

Reordering (i = 1)

e1 e2

e4 e3

e’1 e2

e4 e3


e’1 e’2

e’3 e’4

e’1 e’3

e’2 e’4

e’1 e’2

e’3 e3

e’1 e’3

e’2 e3

3: If e2 e4

3: otherwise

4: Fill rests




Ei = DDFEMi(e′1,e

′2, ...,e

′8), e′1 = ei (7)





argminEi

∑f∈F

!8

∑i=1

wivci ( f ,Ei)−

8

∑j=1

8

∑i=1

wivdji( f ,e′j)

"2

(8)



argminEi

∑f∈F

!8

∑i=1

wivci ( f ,Ei)−

8

∑j=1

8

∑i=1


"2

(9)

is equivalent to

argminE′

i =Ei/s∑f∈F

!8

∑i=1

wivci ( f ,E′

i )−8

∑j=1

8

∑i=1

wivdji( f ,e′j)

"2

(10)


Ei = ei ·DDFEMi

#e′2ei, ...,

e′8ei

$(11)




%Ei = e ·Reg1(

e′2e , ...,

e′8e ) (ei = 0)


e′8ei) (ei > 0)

(12)





is equivalent to



1: Initial ordering

Reordering (i = 1)

e1 e2

e4 e3

e’1 e2

e4 e3


e’1 e’2

e’3 e’4

e’1 e’3

e’2 e’4

e’1 e’2

e’3 e3

e’1 e’3

e’2 e3

3: If e2 e4

3: otherwise

4: Fill rests




Ei = DDFEMi(e′1,e

′2, ...,e

′8), e′1 = ei (7)





argminEi

∑f∈F

!8

∑i=1

wivci ( f ,Ei)−

8

∑j=1

8

∑i=1

wivdji( f ,e′j)

"2

(8)



argminEi

∑f∈F

!8

∑i=1

wivci ( f ,Ei)−

8

∑j=1

8

∑i=1


"2

(9)

is equivalent to

argminE′

i =Ei/s∑f∈F

!8

∑i=1

wivci ( f ,E′

i )−8

∑j=1

8

∑i=1

wivdji( f ,e′j)

"2

(10)


Ei = ei ·DDFEMi

#e′2ei, ...,

e′8ei

$(11)




%Ei = e ·Reg1(

e′2e , ...,

e′8e ) (ei = 0)


e′8ei) (ei > 0)

(12)







1: Initial ordering

Reordering (i = 1)

e1 e2

e4 e3

e’1 e2

e4 e3


e’1 e’2

e’3 e’4

e’1 e’3

e’2 e’4

e’1 e’2

e’3 e3

e’1 e’3

e’2 e3

3: If e2 e4

3: otherwise

4: Fill rests




Ei = DDFEMi(e′1,e

′2, ...,e

′8), e′1 = ei (7)





argminEi

∑f∈F

!8

∑i=1

wivci ( f ,Ei)−

8

∑j=1

8

∑i=1

wivdji( f ,e′j)

"2

(8)



argminEi

∑f∈F

!8

∑i=1

wivci ( f ,Ei)−

8

∑j=1

8

∑i=1


"2

(9)

is equivalent to

argminE′

i =Ei/s∑f∈F

!8

∑i=1

wivci ( f ,E′

i )−8

∑j=1

8

∑i=1

wivdji( f ,e′j)

"2

(10)


Ei = ei ·DDFEMi

#e′2ei, ...,

e′8ei

$(11)




%Ei = e ·Reg1(

e′2e , ...,

e′8e ) (ei = 0)


e′8ei) (ei > 0)

(12)







1: Initial ordering

Reordering (i = 1)

e1 e2

e4 e3

e’1 e2

e4 e3


e’1 e’2

e’3 e’4

e’1 e’3

e’2 e’4

e’1 e’2

e’3 e3

e’1 e’3

e’2 e3

3: If e2 e4

3: otherwise

4: Fill rests




Ei = DDFEMi(e′1,e

′2, ...,e

′8), e′1 = ei (7)





argminEi

∑f∈F

!8

∑i=1

wivci ( f ,Ei)−

8

∑j=1

8

∑i=1

wivdji( f ,e′j)

"2

(8)



argminEi

∑f∈F

!8

∑i=1

wivci ( f ,Ei)−

8

∑j=1

8

∑i=1


"2

(9)

is equivalent to

argminE′

i =Ei/s∑f∈F

!8

∑i=1

wivci ( f ,E′

i )−8

∑j=1

8

∑i=1

wivdji( f ,e′j)

"2

(10)


Ei = ei ·DDFEMi

#e′2ei, ...,

e′8ei

$(11)




%Ei = e ·Reg1(

e′2e , ...,

e′8e ) (ei = 0)


e′8ei) (ei > 0)

(12)








Runtime



+

…


Input: Surface Mesh





Sample Position 1


Simulated Sound







Sample Position 3

Timbre

Precomputation

Regression Forests










3. Regression Forests

✔ A large dataset handling

✔ Constant evaluation cost

✔ High generalization ability

Data-driven fluid simulations using regression forests, Ladický et al., SIGGRAPH Asia ‘15

3. Regression Forests



1: Initial ordering

Reordering (i = 1)

e1 e2

e4 e3

e’1 e2

e4 e3


e’1 e’2

e’3 e’4

e’1 e’3

e’2 e’4

e’1 e’2

e’3 e3

e’1 e’3

e’2 e3

3: If e2 e4

3: otherwise

4: Fill rests




Ei = DDFEMi(e′1,e

′2, ...,e

′8), e′1 = ei (7)





argminEi

∑f∈F

!8

∑i=1

wivci ( f ,Ei)−

8

∑j=1

8

∑i=1

wivdji( f ,e′j)

"2

(8)



argminEi

∑f∈F

!8

∑i=1

wivci ( f ,Ei)−

8

∑j=1

8

∑i=1


"2

(9)

is equivalent to

argminE′

i =Ei/s∑f∈F

!8

∑i=1

wivci ( f ,E′

i )−8

∑j=1

8

∑i=1

wivdji( f ,e′j)

"2

(10)


Ei = ei ·DDFEMi

#e′2ei, ...,

e′8ei

$(11)




%Ei = e ·Reg1(

e′2e , ...,

e′8e ) (ei = 0)


e′8ei) (ei > 0)

(12)





Finally, our DDFEM becomes

Reg(): regression function

Data-driven fluid simulations using regression forests, Ladický et al., SIGGRAPH Asia ‘15




Runtime



+

…


Input: Surface Mesh





Sample Position 1


Simulated Sound







Sample Position 3

Timbre

Precomputation

Regression Forests













Runtime



+

…


Input: Surface Mesh





Sample Position 1


Simulated Sound







Sample Position 3

Timbre

Precomputation

Regression Forests










Mesh Coarsening with Data-Driven FEM

Fast Approximate Mode Analysis

Hierarchical Component Mode Synthesis




Runtime



+

…


Input: Surface Mesh





Sample Position 1


Simulated Sound







Sample Position 3

Timbre

Precomputation

Regression Forests










+

Hierarchical Component Mode Synthesis (HCMS)


Error Correction (§7.2.1)

Domain Decomposition (§7.2)

Coarse Voxel Mesh(The Output of DDFEM*)

Subdomains

Local Eigen Problem Solves in Parallel(D: Local Eigenvalues, U: Local Eigenvectors)

{D1,U1}

{D2,U2}

{D3,U3}

{D4,U4}

{D5,U5}

{D6,U6}

{D7,U7}

{D8,U8}

{D9,U9}

Hierarchical Merge by Eq. (15)

{D11,U11}

{D12,U12}

{D13,U13}

{D14,U14}

{D22,U22}

{D21,U21}

{D31,U31}

{D41,U41}

Merge adjacent domains in parallel

Global Solution

Figure 7: Hierarchical Component Mode Synthesis. After coarsening the mesh, we decompose it into many subdomains and hierarchicallymerges them with reducing their DoFs in parallel. Finally, we improve the accuracy using an error correction algorithm.

To simplify the explanation for our HCMS, we first beginwith assuming that a model can be decomposed into two non-overlapping domains S1 and S2 as in conventional CMS, and theeigenpairs of each domain are already known. Under this assump-tion, the entire stiffness matrix Ktotal and the entire mass matrixMtotal can be represented as

Ktotal =

!K11 K12K12 K22

", Mtotal =

!M1 00 M2

"(13)

where K11, K22 and M1, M2 denote the local stiffness and mass ma-trices of each sub-domain respectively. K12 and K21 are the inter-face matrices that connect the domains S1 and S2. If the eigenvec-tors of each domain U1 and U2 are already known, we can rewritethe Eq. (13) using the reduced matrices of each domain with re-maining the lower frequency modes as

K′total =

!D1 U1T K12U2

U2T KT12U1 D2

"(14)

where D1 =U1T K11U1 and D2 =U2T K22U2 are diagonal matri-ces in which each diagonal entry is the eigenvalue of the respectivesubdomain. Note that the entire mass matrix also takes the sameform for, UT

1 M1U1 = I, and UT2 M2U2 = I, meaning that the en-

tire mass matrix becomes an identity matrix. Athough conventionalCMS distinguishes the interface of adjacent subdomains and sub-domains, and assumes the interface as fixed [BC68] or considersthe boundary modes [YXG∗13], our approach neither distinguishthem nor fix the interface, and does not treat the interface explic-itly. We can obtain a reduced eigenproblem of the entire structureas K′

totalU ′total = ΛtotalU ′

total , where Λtotal is a diagonal matrixin which each diagonal entry is the eigenvalues of the entire do-main. We solve this reduced eigenproblem, and finally recover theglobal eigenvectors by

Utotal =

!U1U2

"U ′

total . (15)

We apply the pair wise merger explained above in a hierarchicalmanner. We divide a large structure into many small subdomainsand merge them in a hierarchical manner (Figure 7). The systemfirst decomposes the volumetric mesh after coarsening into manysmall subdomains S1,S2, ...,SN by a domain decomposition. To de-compose a mesh, we use [WLAT14] by expanding it into volu-metric mesh, which decomposes the mesh by K-Means++ cluster-ing [AV07] of the eigenvectors of the volumetric mesh Laplacian.It requires no additional precomputation costs since the volumetricmesh Laplacian has been already obtained at the precomputationstage as described in §4.

We compute the local generalized eigenproblem of N sub-domains in parallel and reduce the DoFs using the eigenvectors ateach subdomain. Next, we iteratively merge two adjacent subdo-mains by Eq. (14), and solve the reduced eigenproblem, and Eq.

(15) to obtain the eigenvectors of the merged subdomain. This pro-cedure also can be executed in parallel until all the subdomains aremerged. Finally, we merge all the subdomains and obtain the ap-proximate eigenvector of the entire structure. The order of merg-ing subdomains is irrelevant in our algorithm because the errorcaused by suboptimal order will be fixed later in our error correc-tion (§7.2.1). We note that this hierarchical merging procedure isnew. We implemented local eigenprobrem solves of each subdo-main by a combination of incomplete Lanczos matrix triangulationand QR method.

7.2.1. Error Correction

HCMS is just an approximation method and sacrifices the accu-racy for computational efficiency. To correct this error, we intro-duce the subspace iteration method [Bat13] using reduced massGram-Shmidzt process [YLX∗15]. We set approximated eigenvec-tors of HCMS as the starting iteration vectors X0 and execute thefollowing iteration k = 1,2,3... until it converges.

Solve PCG : KU = MXk−1 (16)U ← ReducedMGS(U) (17)

K′ =UT KU , M′ =UT MU (18)Solve QR : K′Q = ΛM′Q (19)

Xk = U +Σ(UQ−U) (20)Xk ← ReducedMGS(Xk) (21)

where ReducedMGS() is the reduced mass Modified Gram-Schmidt process to orthogonalize the eigenvectors [YLX∗15], Σdenotes a diagonal matrix in which each diagonal correspondsto the overrelaxation weight of the i-th eigenvalue to acceleratethe convergence [BR80]. We solve the first line Eq. (16) by in-complete cholesky factorized pre-conditioned conjugate gradientmethod with respect to each column vector in parallel, and imple-ment the QR method Eq. (19) on GPU.

8. Results

8.1. Validation of Modal Analysis

In this subsection, we verify the accuracy and computational ef-ficiency of our fast approximate modal analysis. As the groundtruth, we used the result of the full-DoF standard modal analysisusing ARPack (with sufficiently fine-resolution uniform hexahe-dral mesh). We used CPU: Intel Core i7 2.6 GHz, RAM: 16GB,GPU: NVIDIA GeForce GT 750M as the equipments in §8 exclud-ing the DDFEM trainings. We set the Poisson’s ratio as 0.25 andthe density as 1.0 kg/m3 for all experiments.

Data-Driven FEM: We call our data-driven FEM as extendeddata-driven FEM (DDFEM*) for distinguishing from Chen et al.’smethod [CLSM15] (DDFEM). We used two regression forests, and


HCMS takes the coarse voxel mesh as input and solves a generalized eigenproblem via hierarchical merging.






Subdomains


{D1,U1}

{D2,U2}

{D3,U3}

{D4,U4}

{D5,U5}

{D6,U6}

{D7,U7}

{D8,U8}

{D9,U9}


{D11,U11}

{D12,U12}

{D13,U13}

{D14,U14}

{D22,U22}

{D21,U21}

{D31,U31}

{D41,U41}


Global Solution



Ktotal =

!K11 K12K12 K22

", Mtotal =

!M1 00 M2

"(13)


K′total =

!D1 U1T K12U2

U2T KT12U1 D2

"(14)






Utotal =

!U1U2

"U ′

total . (15)










8. Results





HCMS takes the coarse voxel mesh as input and solves a generalized eigenproblem via hierarchical merging.

Hierarchical Component Mode Synthesis (HCMS)subdomain 1 subdomain 2

Eigenvalues of A : Λ1Eigenvectors of A: U1

Eigenvalues of B : Λ2Eigenvectors of B: U2











Subdomains


{D1,U1}

{D2,U2}

{D3,U3}

{D4,U4}

{D5,U5}

{D6,U6}

{D7,U7}

{D8,U8}

{D9,U9}


{D11,U11}

{D12,U12}

{D13,U13}

{D14,U14}

{D22,U22}

{D21,U21}

{D31,U31}

{D41,U41}


Global Solution



Ktotal =

!K11 K12K12 K22

", Mtotal =

!M1 00 M2

"(13)


K′total =

!D1 U1T K12U2

U2T KT12U1 D2

"(14)






Utotal =

!U1U2

"U ′

total . (15)










8. Results









Subdomains


{D1,U1}

{D2,U2}

{D3,U3}

{D4,U4}

{D5,U5}

{D6,U6}

{D7,U7}

{D8,U8}

{D9,U9}


{D11,U11}

{D12,U12}

{D13,U13}

{D14,U14}

{D22,U22}

{D21,U21}

{D31,U31}

{D41,U41}


Global Solution



Ktotal =

!K11 K12K12 K22

", Mtotal =

!M1 00 M2

"(13)


K′total =

!D1 U1T K12U2

U2T KT12U1 D2

"(14)






Utotal =

!U1U2

"U ′

total . (15)










8. Results





≒

where





Subdomains


{D1,U1}

{D2,U2}

{D3,U3}

{D4,U4}

{D5,U5}

{D6,U6}

{D7,U7}

{D8,U8}

{D9,U9}


{D11,U11}

{D12,U12}

{D13,U13}

{D14,U14}

{D22,U22}

{D21,U21}

{D31,U31}

{D41,U41}


Global Solution



Ktotal =

!K11 K12K12 K22

", Mtotal =

!M1 00 M2

"(13)


K′total =

!D1 U1T K12U2

U2T KT12U1 D2

"(14)






Utotal =

!U1U2

"U ′

total . (15)










8. Results









Subdomains


{D1,U1}

{D2,U2}

{D3,U3}

{D4,U4}

{D5,U5}

{D6,U6}

{D7,U7}

{D8,U8}

{D9,U9}


{D11,U11}

{D12,U12}

{D13,U13}

{D14,U14}

{D22,U22}

{D21,U21}

{D31,U31}

{D41,U41}


Global Solution



Ktotal =

!K11 K12K12 K22

", Mtotal =

!M1 00 M2

"(13)


K′total =

!D1 U1T K12U2

U2T KT12U1 D2

"(14)






Utotal =

!U1U2

"U ′

total . (15)










8. Results





Both D matrices become diagonal matrices in which each diagonal entry is the eigenvalue of the respective subdomain.

Reduced SystemEntire System





Subdomains


{D1,U1}

{D2,U2}

{D3,U3}

{D4,U4}

{D5,U5}

{D6,U6}

{D7,U7}

{D8,U8}

{D9,U9}


{D11,U11}

{D12,U12}

{D13,U13}

{D14,U14}

{D22,U22}

{D21,U21}

{D31,U31}

{D41,U41}


Global Solution



Ktotal =

!K11 K12K12 K22

", Mtotal =

!M1 00 M2

"(13)


K′total =

!D1 U1T K12U2

U2T KT12U1 D2

"(14)






Utotal =

!U1U2

"U ′

total . (15)










8. Results





=








Subdomains


{D1,U1}

{D2,U2}

{D3,U3}

{D4,U4}

{D5,U5}

{D6,U6}

{D7,U7}

{D8,U8}

{D9,U9}


{D11,U11}

{D12,U12}

{D13,U13}

{D14,U14}

{D22,U22}

{D21,U21}

{D31,U31}

{D41,U41}


Global Solution



Ktotal =

!K11 K12K12 K22

", Mtotal =

!M1 00 M2

"(13)


K′total =

!D1 U1T K12U2

U2T KT12U1 D2

"(14)






Utotal =

!U1U2

"U ′

total . (15)










8. Results









Subdomains


{D1,U1}

{D2,U2}

{D3,U3}

{D4,U4}

{D5,U5}

{D6,U6}

{D7,U7}

{D8,U8}

{D9,U9}


{D11,U11}

{D12,U12}

{D13,U13}

{D14,U14}

{D22,U22}

{D21,U21}

{D31,U31}

{D41,U41}


Global Solution



Ktotal =

!K11 K12K12 K22

", Mtotal =

!M1 00 M2

"(13)


K′total =

!D1 U1T K12U2

U2T KT12U1 D2

"(14)






Utotal =

!U1U2

"U ′

total . (15)










8. Results





≒

where





Subdomains


{D1,U1}

{D2,U2}

{D3,U3}

{D4,U4}

{D5,U5}

{D6,U6}

{D7,U7}

{D8,U8}

{D9,U9}


{D11,U11}

{D12,U12}

{D13,U13}

{D14,U14}

{D22,U22}

{D21,U21}

{D31,U31}

{D41,U41}


Global Solution



Ktotal =

!K11 K12K12 K22

", Mtotal =

!M1 00 M2

"(13)


K′total =

!D1 U1T K12U2

U2T KT12U1 D2

"(14)






Utotal =

!U1U2

"U ′

total . (15)










8. Results









Subdomains


{D1,U1}

{D2,U2}

{D3,U3}

{D4,U4}

{D5,U5}

{D6,U6}

{D7,U7}

{D8,U8}

{D9,U9}


{D11,U11}

{D12,U12}

{D13,U13}

{D14,U14}

{D22,U22}

{D21,U21}

{D31,U31}

{D41,U41}


Global Solution



Ktotal =

!K11 K12K12 K22

", Mtotal =

!M1 00 M2

"(13)


K′total =

!D1 U1T K12U2

U2T KT12U1 D2

"(14)






Utotal =

!U1U2

"U ′

total . (15)










8. Results





Both D matrices become diagonal matrices in which each diagonal entry is the eigenvalue of the respective subdomain.

Reduced SystemEntire System





Subdomains


{D1,U1}

{D2,U2}

{D3,U3}

{D4,U4}

{D5,U5}

{D6,U6}

{D7,U7}

{D8,U8}

{D9,U9}


{D11,U11}

{D12,U12}

{D13,U13}

{D14,U14}

{D22,U22}

{D21,U21}

{D31,U31}

{D41,U41}


Global Solution



Ktotal =

!K11 K12K12 K22

", Mtotal =

!M1 00 M2

"(13)


K′total =

!D1 U1T K12U2

U2T KT12U1 D2

"(14)






Utotal =

!U1U2

"U ′

total . (15)










8. Results





=






Subdomains


{D1,U1}

{D2,U2}

{D3,U3}

{D4,U4}

{D5,U5}

{D6,U6}

{D7,U7}

{D8,U8}

{D9,U9}


{D11,U11}

{D12,U12}

{D13,U13}

{D14,U14}

{D22,U22}

{D21,U21}

{D31,U31}

{D41,U41}


Global Solution



Ktotal =

!K11 K12K12 K22

", Mtotal =

!M1 00 M2

"(13)


K′total =

!D1 U1T K12U2

U2T KT12U1 D2

"(14)






Utotal =

!U1U2

"U ′

total . (15)










8. Results









Subdomains


{D1,U1}

{D2,U2}

{D3,U3}

{D4,U4}

{D5,U5}

{D6,U6}

{D7,U7}

{D8,U8}

{D9,U9}


{D11,U11}

{D12,U12}

{D13,U13}

{D14,U14}

{D22,U22}

{D21,U21}

{D31,U31}

{D41,U41}


Global Solution



Ktotal =

!K11 K12K12 K22

", Mtotal =

!M1 00 M2

"(13)


K′total =

!D1 U1T K12U2

U2T KT12U1 D2

"(14)






Utotal =

!U1U2

"U ′

total . (15)










8. Results





Reduced Generalized EigenProblem:

Approximated EigenVectors of The Entire System:

Hierarchical Component Mode Synthesis (HCMS)A coarse mesh


SubDomain 1 SubDomain 2 SubDomain 3 SubDomain 4

Domain Decomposition




MergedDomain 1 MergedDomain 2





EntireMesh


In parallel:

In parallel:




EntireMesh






Subdomains


{D1,U1}

{D2,U2}

{D3,U3}

{D4,U4}

{D5,U5}

{D6,U6}

{D7,U7}

{D8,U8}

{D9,U9}


{D11,U11}

{D12,U12}

{D13,U13}

{D14,U14}

{D22,U22}

{D21,U21}

{D31,U31}

{D41,U41}


Global Solution



Ktotal =

!K11 K12K12 K22

", Mtotal =

!M1 00 M2

"(13)


K′total =

!D1 U1T K12U2

U2T KT12U1 D2

"(14)






Utotal =

!U1U2

"U ′

total . (15)










8. Results





After the merging of HCMS, we reduce the error by an error correction.






Subdomains


{D1,U1}

{D2,U2}

{D3,U3}

{D4,U4}

{D5,U5}

{D6,U6}

{D7,U7}

{D8,U8}

{D9,U9}


{D11,U11}

{D12,U12}

{D13,U13}

{D14,U14}

{D22,U22}

{D21,U21}

{D31,U31}

{D41,U41}


Global Solution



Ktotal =

!K11 K12K12 K22

", Mtotal =

!M1 00 M2

"(13)


K′total =

!D1 U1T K12U2

U2T KT12U1 D2

"(14)






Utotal =

!U1U2

"U ′

total . (15)










8. Results





After the merging of HCMS, we reduce the error by an error correction.

Σ: over relaxation weights

Error Correction





Subdomains


{D1,U1}

{D2,U2}

{D3,U3}

{D4,U4}

{D5,U5}

{D6,U6}

{D7,U7}

{D8,U8}

{D9,U9}


{D11,U11}

{D12,U12}

{D13,U13}

{D14,U14}

{D22,U22}

{D21,U21}

{D31,U31}

{D41,U41}


Global Solution



Ktotal =

!K11 K12K12 K22

", Mtotal =

!M1 00 M2

"(13)


K′total =

!D1 U1T K12U2

U2T KT12U1 D2

"(14)






Utotal =

!U1U2

"U ′

total . (15)










8. Results





X0 ← Approximate EigenVectors of HCMS

and iterate…

The subspace iteration method - revisited, Bathe et al., Computers and Structure ’13


Error Correction





Subdomains


{D1,U1}

{D2,U2}

{D3,U3}

{D4,U4}

{D5,U5}

{D6,U6}

{D7,U7}

{D8,U8}

{D9,U9}


{D11,U11}

{D12,U12}

{D13,U13}

{D14,U14}

{D22,U22}

{D21,U21}

{D31,U31}

{D41,U41}


Global Solution



Ktotal =

!K11 K12K12 K22

", Mtotal =

!M1 00 M2

"(13)


K′total =

!D1 U1T K12U2

U2T KT12U1 D2

"(14)






Utotal =

!U1U2

"U ′

total . (15)










8. Results






and iterate…Full-DoF stiffness and mass matrices



Error Correction





Subdomains


{D1,U1}

{D2,U2}

{D3,U3}

{D4,U4}

{D5,U5}

{D6,U6}

{D7,U7}

{D8,U8}

{D9,U9}


{D11,U11}

{D12,U12}

{D13,U13}

{D14,U14}

{D22,U22}

{D21,U21}

{D31,U31}

{D41,U41}


Global Solution



Ktotal =

!K11 K12K12 K22

", Mtotal =

!M1 00 M2

"(13)


K′total =

!D1 U1T K12U2

U2T KT12U1 D2

"(14)






Utotal =

!U1U2

"U ′

total . (15)










8. Results






and iterate…

Expediting Precomputation for Reduced Deformable Simulation, Yang et al. SIGGRAPH Asia ’15.


Ground truth

CMS without EC

CMS with EC (4 iterations)CMS with EC (8 iterations)

CMS with EC (16 iterations)

Eigenvalue

Mode Number0 50 100 150 200

0

0.5

1

1.5

2

2.5 x 104

Eigenvalue

Mode Number

Ground truth

CMS without EC




Random starting vectors (16 iterations)

The Result of Error Correction




Runtime



+

…


Input: Surface Mesh





Sample Position 1


Simulated Sound







Sample Position 3

Timbre

Precomputation

Regression Forests










It converges very quickly in only a few iterations,

and efficiently reduces the error. Ground truth

Ground truth

CMS without EC

CMS with EC (4 iterations)CMS with EC (8 iterations)


Eigenvalue

Mode Number0 50 100 150 200

0

0.5

1

1.5

2

2.5 x 104

Eigenvalue

Mode Number

Ground truth

CMS without EC




Random starting vectors (16 iterations)

The Result of Error Correction




Runtime



+

…


Input: Surface Mesh





Sample Position 1


Simulated Sound







Sample Position 3

Timbre

Precomputation

Regression Forests










It converges very quickly in only a few iterations,

and efficiently reduces the error. Ground truth

Results

Fast Approximate Modal Analysis


Model DoF Precomputation ARPack DDFEM HCMS HCMS + EC DDFEM + HCMS + EC

Stanford Bunny 31419 18.8m 4.1h 15.3m 9.8m 28.4m 2.6s

Asian Dragon 6009 23s 14.8m 2.1m 1.1m 4.3m 0.86s

Utah Teapot 12057 1.5m 52.8m 5.7m 4.9m 17.2m 1.1s

Chinese Dragon 11394 50s 21.9m 3.2m 3.3m 12.5m 1.4s

Pitcher 15927 2.3m 37,6m 6.4m 3.4m 17.8m 2.3s

Snare Drum 35484 21.5m 3.9h 12.3m 9.7m 25.6m 2.4s

* *

Figure 11: Computation time comparison of modal analysis. We computed the first 256 modes for all model.

0 50 100 150 200 250 3000

0.005

0.01

0.015

0.02

0.025

0 5 10 15 20 25 30 35 40 45 500

0.005

0.01

0.015

0.02

0.025

# Time Step

Ma

gn

itud

e o

f T

he

Dis

pla

cem

en

ts

Ours1.4s

(blue lines)

Ground Truth21.9m

(red lines)

Figure 12: Comparison of between two deformation trajectoriesof the dragon’s nose (red circle at the top thumbnails) producedwith standard modal derivatives (red) and our method (DDFEM*+ HCMS + EC) (blue). The white arrows at the top thumbnailsrepresent the applied force impulse to drive them.

object in both lower and higher frequency domains, and the errorcorrection algorithm successfully brings the approximate solutionclose to to the ground truth within an additional few minutes.

Combination of Extended Data-Driven FEM and HCMS:Figure 11 shows the computational times of each modal analy-sis method using DDFEM*, HCMS with/without EC, and theircombination with several models, respectively, while ARPack de-notes the standard modal analysis (conventional method). The pre-computation column in Figure 11 represents the precomputationtimes taken for each model (The voxelization and the eigenproblemsolves for the volumetric Laplacian matrix). The computation timesof DDFEM* include the time for the online coarsening procedure.We achieve two orders of magnitudes acceleration with each ofDDFEM* and HCMS + EC respectively, and three orders of mag-nitudes acceleration in total compared to the conventional modalanalysis by their combination. The exact computational time usingthe combination method becomes 0.5∼3.0 secs, which is accept-able for interactive evaluation.

Figure 12 shows comparison between two deformation trajecto-ries produced with standard modal derivatives (ground truth) andour fast approximate modal analysis method after applying a unitforce impulse at the location pointed by the white arrow in the topthumbnails. We also provide the time series of magnitudes of thedisplacement at the dragon’s nose in Figure 12:Bottom. The twotrajectories plot quite a similar form, which shows that our methodgives good approximation for the conventional approach with muchfaster operation.

Finally, Figure 13 shows the spectrograms of the sound producedby a rigid body physics animation using ground truth and our com-

Standard Modal Analysis 4.1 hours

Fast Approximate Modal Analysis (Ours)2.6 seconds

Time [s]Time [s]

Fre

qu

en

cy [

Hz]

Fre

qu

en

cy [

Hz]

Figure 13: Comparison of the modal sound synthesis from a sim-ple rigid body physics simulation between standard modal analysis(left) and our fast approximate modal analysis (right).

bination method. Naturally, the spectrogram of the ground truth in-cludes more high frequency components than that of ours becauseit uses a detailed FEM mesh. However, our result can capture a bet-ter portion of the major components enough for our optimizationproblem.

8.2. Physically-Based Sound Design

In this subsection, we demonstrate our physically-based sound de-sign framework. For all the examples, we used the results of our fastapproximate modal analysis to render the sound. We set the weightsin Eq. (4) to w f = 1.0, wa = 10.0, wr = 10−5 in our experiment.

Basic Sound Assignment: Figure 14 shows an example of as-signing two sounds to a frying pan model. The frying pan consistsof a handle made of wood and plate made of iron. We stuck thepan at the handle and the plate, and recorded the respective sounds.We used these recorded sounds as input to the system. In this ex-ample, 30 extracted target modes were extracted from these twosound clips, and we controlled the first 30 modes of the model ex-cluding the six rigid modes. The two spectrograms at the top rowin Figure 14 represent the rendered sounds when each position isstruck before the optimization. The spectrograms at the middle roware target sounds, and at the bottom row are the results after oneminute of optimization. Apparently, the two spectrograms after theoptimization closely resemble each target sound. In addition, evenif a different position from the one assigned is struck, the soundcharacteristics of the target sounds near the position is producedin a physically plausible manner (Figure 14:Bottom). This resultshows that over-fitting is not a serious problem in our optimization.Furthermore, our approach requires less amount of example soundsfor designing the sound of an object, which reduces the user’s ef-fort. For example, in [vdDKP01], there is a frying pan examplewhich is similar to our experiment. They used five example soundsto design the sound of the plate alone (except the handle) while weused only one example sound for each part.

Interactive Editing: Next, we demonstrate an example of theinteractive editing procedure of physically-based sound using a








Pitcher 15927 2.3m 37,6m 6.4m 3.4m 17.8m 2.3s


* *


0 50 100 150 200 250 3000

0.005

0.01

0.015

0.02

0.025

0 5 10 15 20 25 30 35 40 45 500

0.005

0.01

0.015

0.02

0.025

# Time Step

Ma

gn

itud

e o

f T

he

Dis

pla

cem

en

ts

Ours1.4s

(blue lines)

Ground Truth21.9m

(red lines)








Time [s]Time [s]

Fre

qu

en

cy [

Hz]

Fre

qu

en

cy [

Hz]














Pitcher 15927 2.3m 37,6m 6.4m 3.4m 17.8m 2.3s


* *


0 50 100 150 200 250 3000

0.005

0.01

0.015

0.02

0.025

0 5 10 15 20 25 30 35 40 45 500

0.005

0.01

0.015

0.02

0.025

# Time Step

Ma

gn

itud

e o

f T

he

Dis

pla

cem

en

ts

Ours1.4s

(blue lines)

Ground Truth21.9m

(red lines)








Time [s]Time [s]

Fre

qu

en

cy [

Hz]

Fre

qu

en

cy [

Hz]








Computation Time (ARPack vs Ours)

OursCPU: Intel Core i7 2.6 GHz, RAM: 16GB, GPU: NVIDIA GeForce GT 750M







Pitcher 15927 2.3m 37,6m 6.4m 3.4m 17.8m 2.3s


* *


0 50 100 150 200 250 3000

0.005

0.01

0.015

0.02

0.025

0 5 10 15 20 25 30 35 40 45 500

0.005

0.01

0.015

0.02

0.025

# Time Step

Ma

gn

itud

e o

f T

he

Dis

pla

cem

en

ts

Ours1.4s

(blue lines)

Ground Truth21.9m

(red lines)








Time [s]Time [s]

Fre

qu

en

cy [

Hz]

Fre

qu

en

cy [

Hz]














Pitcher 15927 2.3m 37,6m 6.4m 3.4m 17.8m 2.3s


* *


0 50 100 150 200 250 3000

0.005

0.01

0.015

0.02

0.025

0 5 10 15 20 25 30 35 40 45 500

0.005

0.01

0.015

0.02

0.025

# Time Step

Ma

gn

itud

e o

f T

he

Dis

pla

cem

en

ts

Ours1.4s

(blue lines)

Ground Truth21.9m

(red lines)








Time [s]Time [s]

Fre

qu

en

cy [

Hz]

Fre

qu

en

cy [

Hz]














Pitcher 15927 2.3m 37,6m 6.4m 3.4m 17.8m 2.3s


* *


0 50 100 150 200 250 3000

0.005

0.01

0.015

0.02

0.025

0 5 10 15 20 25 30 35 40 45 500

0.005

0.01

0.015

0.02

0.025

# Time Step

Ma

gn

itud

e o

f T

he

Dis

pla

cem

en

ts

Ours1.4s

(blue lines)

Ground Truth21.9m

(red lines)








Time [s]Time [s]

Fre

qu

en

cy [

Hz]

Fre

qu

en

cy [

Hz]








Computation Time (ARPack vs Ours)

Three orders of magnitude acceleration

OursCPU: Intel Core i7 2.6 GHz, RAM: 16GB, GPU: NVIDIA GeForce GT 750M

Comparison (Deformation)

Ground Truth (21.9 minutes) Ours (1.4 seconds)

Comparison (Deformation)

Ground Truth (21.9 minutes) Ours (1.4 seconds)

Comparison (Sound Simulation)

Ground Truth (4.1 hours) Ours (2.6 seconds)











Physically-Based Sound Design


Interactive Editing


Interactive Editing


Interactive Editing


Imaginary Sound Assignment


Imaginary Sound Assignment


Complicated Scenario Example


Complicated Scenario Example

• Our optimization is not suited for fabrication.

• We only optimized Young’s modulus.

• We did not consider the radiation and propagation.

Limitation

Summery• An example-based framework for designing physically-based sound

• Fast approximate modal analysis for an interactive simulation

• Extended data-driven FEM using regression forests

• Hierarchical component mode synthesis with error correction

• Handling a large range of continuous material settings. • Constant evaluation cost.• High generalization ability.

• Parallel• Accurate

Thanks

Science

Interactive Physically-Based Sound Design of 3D Model using Material Optimization (SCA'16)