Extended Source-Filter Model of Harmonic Instruments for Sound Synthesis, Transformation and

Extended Source-Filter Model of Harmonic Instruments for Sound Synthesis, Transformation and Interpolation Transformation and Interpolation
14 July 2012
Henrik Hahn, Axel Robel IRCAM - CNRS - UMR 9912 - STMS, Paris, France
Extended Source-Filter Model of Harmonic Instruments
2/24
Introduction
Conclusions
3/24
Introduction
Conclusions
4/24
Sample Based Synthesis (Overview)
I an electronic instrument I based on ’playback’ of prerecorded
instrument sounds I playback is triggered by some input
device (MIDI Keyboard)
I synthesis sounds static I no expressive control
4/24
4/24
4/24
4/24
4/24
4/24
4/24
5/24
Sample Based Synthesis (State of the art)
I recordings on a semitone scale I recordings at several intensities I transformations based on local
Source-Filter approach
I soundspace is does not contain knowledge about intermediate values.
I transformations do not account for real instrument characteristics
... P
I
pp
ff ...
XA,1
5/24
I : Intensity {a=1...A} P : Pitch {b=1...B}
Xa,b : Sound Sample
XA,1
5/24
Xa,b : Sound Sample
XA,1
5/24
Xa,b : Sound Sample F : Filter
... P
I
pp
ff ...
F(ω)1,B X1,B
F(ω)1,B X1,1
5/24
... P
I
pp
ff ...
F(ω)1,B X1,B
F(ω)1,B X1,1
5/24
... P
I
pp
ff ...
F(ω)1,B X1,B
F(ω)1,B X1,1
6/24
Sample Based Synthesis (Proposed Method)
I usage of State of the Art databases I parametric model to describe the
whole instrument sound characteristic along pitch / global intensity
I account for temporal evolution of a sound (ASR) denoted local Intensity
I separately treat harmonic and noise components
I model shall learn its parameters from the database
Xa,b : Sound Sample
XA,1
6/24
Xa,b : Sound Sample
XA,1
6/24
Xa,b : Sound Sample
XA,1
6/24
IG : Global Intensity
P : Pitch
Xa,b : Sound Sample S : Partial Function k : Partial Index R : Resonance Filter F : Filter
IL : Local Intensity
XA,1
6/24
P : Pitch
XA,1
6/24
P : Pitch
XA,1
7/24
I Transformations based on actual instrument characteristics
I Sound synthesis with continuous pitch and intensity values
I Interpolation between sounds I Cross synthesis between different
instruments I ...
XA,1
7/24
instruments I ...
XA,1
7/24
instruments I ...
XA,1
7/24
instruments I ...
XA,1
7/24
instruments I ...
XA,1
7/24
instruments I ...
XA,1
8/24
Introduction
Conclusions
9/24
System Overview
I Global Intensity / Pitch I Local Intensity (ASR scheme)
I Model adaption for harmonic/noise component
I Remove estimated instrument sound from signal components
I yields database of ’flat’ residual sounds
I Interpolate 2 ’flat’ residuals (harmonic / noise separately)
I Apply any parameter change to estimate new envelopes to use on ’flat’ residuals
Sound DB
9/24
System Overview
Sound DB
SinusoidsSinusoids ResidualsResiduals
9/24
System Overview
Sound DB
Control Parameter
9/24
System Overview
Sound DB
Control Parameter
9/24
System Overview
Sound DB
Control Parameter
9/24
System Overview
Sound DB
Control Parameter
Harmonic Model
Noise Model
9/24
System Overview
Sound DB
Control Parameter
Harmonic Model
Noise Model
Residuals
9/24
System Overview
Sound DB
Control Parameter
Harmonic Model
Noise Model
Residuals
9/24
System Overview
Sound DB
Control Parameter
Harmonic Model
Noise Model
Transformed Sound
9/24
System Overview
Sound DB
Control Parameter
Harmonic Model
Noise Model
Transformed Sound
10/24
Signal Analysis
Harmonics/Noise Segregation
I Partials are modeled as amplitude and frequency function per partial k over time n:
A(k , n) | f (k , n)
I Noise is modeled as envelope using its smoothed Short Time Cepstrum C(l, n)
C(l, n)
10/24
Signal Analysis
A(k , n) | f (k , n)
C(l, n)
10/24
Signal Analysis
A(k , n) | f (k , n)
C(l, n)
11/24
Parameter Analysis
Global Intensity / Pitch Analysis I Obtained from meta data provided by the Database
Local Intensity I Local intensity reflects amplitude
envelope over time: IL(n). I Threshold method to determine
attack/release time frames nA, nR
← n A
n R
d B
scheme to define ns = {na, nr}
11/24
Parameter Analysis
← n A
n R
d B
11/24
Parameter Analysis
← n A
n R
d B
11/24
Parameter Analysis
← n A
n R
d B
12/24
Harmonic Model
Features by partial index k I function for each k depending on pitch m
(MIDI) and both intensities IG and IL I separate functions s for attack-sustain and
sustain-release I may refer to a vibrating string / air pipe
Partial function
Sk,s(IG, IL,m)
Features by frequency f I invariant filter I refers mainly to the instrument corpus
Resonance filter
I using log-domain values
12/24
Harmonic Model
Partial function
Sk,s(IG, IL,m)
Resonance filter
12/24
Harmonic Model
Partial function
Sk,s(IG, IL,m)
Resonance filter
12/24
Harmonic Model
Partial function
Sk,s(IG, IL,m)
Resonance filter
12/24
Harmonic Model
Partial function
Sk,s(IG, IL,m)
Resonance filter
12/24
Harmonic Model
Partial function
Sk,s(IG, IL,m)
Resonance filter
12/24
Harmonic Model
Partial function
Sk,s(IG, IL,m)
Resonance filter
12/24
Harmonic Model
Partial function
Sk,s(IG, IL,m)
Resonance filter
13/24
Harmonic Model
Ak,s(IG, IL,m, f (k , n)) = Sk,s(IG, IL,m) + R(f (k , n))
I Model of partial function using tensor-product B-splines:
Sk,s(IG, IL,m) =
P,Q,T∑ p,q,t
0.5
1
B-Spline functions for Bp (IG), Bq (IL), Bt (m)
13/24
Harmonic Model
I Model of partial function using tensor-product B-splines:
Sk,s(IG, IL,m) =
P,Q,T∑ p,q,t
0.5
1
14/24
Harmonic Model
I model of resonance filter using one-dimensional B-splines
R(f (k , n)) = V∑ v
Bv (f (k , n)) · λv
2 4 6 8 10 12 14 16 18 20 22 0
0.5
1
B-Spline functions for Bv (f (k, n))
14/24
Harmonic Model
I model of resonance filter using one-dimensional B-splines
R(f (k , n)) = V∑ v
Bv (f (k , n)) · λv
2 4 6 8 10 12 14 16 18 20 22 0
0.5
1
B-Spline functions for Bv (f (k, n))
15/24
Noise Model
I Cepstral coefficients are described using a single tensor-product B-spline model:
Ck,s(IG, IL,m) =
P,Q,T∑ p,q,t
0.5
1
16/24
Parameter Estimation
Oh = 1 2
2∑ s=1
K ,Ns∑ k,ns
On = 1 2
2∑ s=1
|C(l, ns)− Ck,s(IG, IL(ns),m)|2
17/24
Introduction
Conclusions
18/24
Model Results: Trumpet
−20
−10
0
10
20
30
B
19/24
Introduction
Conclusions
20/24
Subjective Evaluation
I Interpolation between different pitches (12st and 24st) I Interpolation between different intensities (pp-mf , mf -ff , pp-ff )
I Sequence of 3 sounds has always been presented, framing the interpolated by their original sounds.
I Each sequence was presented twice. Once containing the transformed and once the original counterpart
I Participants were asked to judge for any audible artifacts and convincingness
Clarinet: mf -ff Trumpet pp-ff Clarinet A#3-A#5
20/24
20/24
20/24
20/24
20/24
20/24
20/24
Clarinet_Intensity_ShortDist.mov
Subjective Evaluation: Results
I Measured the Mean Opinion Score for both instruments at once I Org represents original samples, Mod1 and Mod2 represent synthesized ones.
Org Mod 1.5
2
2.5
3
3.5
Mod2: pp-ff
I MOS for original value way too low. Need for a new test with different setup
21/24
Org Mod 1.5
2
2.5
3
3.5
Mod2: pp-ff
21/24
Org Mod 1.5
2
2.5
3
3.5
Mod2: pp-ff
21/24
Org Mod 1.5
2
2.5
3
3.5
Mod2: pp-ff
22/24
Introduction
Conclusions
23/24
Conclusions
We presented I A parametric model for harmonic instruments I A model which separately represents harmonic and noise components utilizing
tensor-product B-splines I An harmonic model separately representing features by partial index and
frequency I An objective function to estimate model parameters iteratively I A subjective evaluation showing promising results
I More instruments need to be adressed (Strings, Piano, Guitar, ...) I A subjective evaluation needs to be repeated with a different setup
23/24
Conclusions
23/24
Conclusions
23/24
Conclusions
23/24
Conclusions
23/24
Conclusions
23/24
Conclusions
23/24
Conclusions
24/24
Fin
Thanks for listening
Introduction

Documents

Extended Source-Filter Model of Harmonic Instruments for Sound Synthesis, Transformation and