1
1LSA 352 Summer 2007
LSA 352Speech Recognition and Synthesis
Dan Jurafsky
Lecture 4: Waveform Synthesis(in Concatenative TTS)
IP Notice: many of these slides come directly from Richard Sproat’sslides, and others (and some of Richard’s) come from Alan Black’sexcellent TTS lecture notes. A couple also from Paul Taylor
2LSA 352 Summer 2007
Goal of Today’s Lecture
Given:String of phonesProsody
– Desired F0 for entire utterance– Duration for each phone– Stress value for each phone, possibly accent value
Generate:Waveforms
3LSA 352 Summer 2007
Outline: Waveform Synthesis inConcatenative TTS
Diphone SynthesisBreak: Final ProjectsUnit Selection Synthesis
Target costUnit cost
JoiningDumbPSOLA
4LSA 352 Summer 2007
The hourglass architecture
5LSA 352 Summer 2007
Internal Representation:Input to Waveform Wynthesis
6LSA 352 Summer 2007
Diphone TTS architecture
Training:Choose units (kinds of diphones)Record 1 speaker saying 1 example of each diphoneMark the boundaries of each diphones,
– cut each diphone out and create a diphone database
Synthesizing an utterance,grab relevant sequence of diphones from databaseConcatenate the diphones, doing slight signalprocessing at boundariesuse signal processing to change the prosody (F0,energy, duration) of selected sequence of diphones
2
7LSA 352 Summer 2007
Diphones
Mid-phone is more stable than edge:
8LSA 352 Summer 2007
Diphones
mid-phone is more stable than edgeNeed O(phone2) number of units
Some combinations don’t exist (hopefully)ATT (Olive et al. 1998) system had 43 phones
– 1849 possible diphones– Phonotactics ([h] only occurs before vowels), don’t need
to keep diphones across silence– Only 1172 actual diphones
May include stress, consonant clusters– So could have more
Lots of phonetic knowledge in design
Database relatively small (by today’s standards)Around 8 megabytes for English (16 KHz 16 bit)
Slide from Richard Sproat
9LSA 352 Summer 2007
Voice
SpeakerCalled a voice talent
Diphone databaseCalled a voice
10LSA 352 Summer 2007
Designing a diphone inventory:Nonsense words
Build set of carrier words:pau t aa b aa b aa paupau t aa m aa m aa paupau t aa m iy m aa paupau t aa m iy m aa paupau t aa m ih m aa pau
Advantages:Easy to get all diphonesLikely to be pronounced consistently
– No lexical interference
Disadvantages:(possibly) bigger databaseSpeaker becomes bored
Slide from Richard Sproat
11LSA 352 Summer 2007
Designing a diphone inventory:Natural words
Greedily select sentences/words:Quebecois argumentsBrouhaha abstractionsArkansas arranging
Advantages:Will be pronounced naturallyEasier for speaker to pronounceSmaller database? (505 pairs vs. 1345 words)
Disadvantages:May not be pronounced correctly
Slide from Richard Sproat 12LSA 352 Summer 2007
Making recordings consistent:
Diiphone should come from mid-wordHelp ensure full articulation
Performed consistentlyConstant pitch (monotone), power, duration
Use (synthesized) prompts:Helps avoid pronunciation problemsKeeps speaker consistentUsed for alignment in labeling
Slide from Richard Sproat
3
13LSA 352 Summer 2007
Building diphone schemata
Find list of phones in language:Plus interesting allophonesStress, tons, clusters, onset/coda, etcForeign (rare) phones.
Build carriers for:Consonant-vowel, vowel-consonantVowel-vowel, consonant-consonantSilence-phone, phone-silenceOther special cases
Check the output:List all diphones and justify missing onesEvery diphone list has mistakes
Slide from Richard Sproat 14LSA 352 Summer 2007
Recording conditions
Ideal:Anechoic chamberStudio quality recordingEGG signal
More likely:Quiet roomCheap microphone/sound blasterNo EGGHeadmounted microphone
What we can do:Repeatable conditionsCareful setting on audio levels
Slide from Richard Sproat
15LSA 352 Summer 2007
Labeling Diphones
Run a speech recognizer in forced alignment modeForced alignment:
– A trained ASR system– A wavefile– A word transcription of the wavefile– Returns an alignment of the phones in the words to the wavefile.
Much easier than phonetic labeling:The words are definedThe phone sequence is generally definedThey are clearly articulatedBut sometimes speaker still pronounces wrong, so need to check.
Phone boundaries less important+- 10 ms is okay
Midphone boundaries importantWhere is the stable partCan it be automatically found?
Slide from Richard Sproat 16LSA 352 Summer 2007
Diphone auto-alignment
Givensynthesized promptsHuman speech of same prompts
Do a dynamic time warping alignment of the twoUsing Euclidean distance
Works very well 95%+Errors are typically large (easy to fix)Maybe even automatically detected
Malfrere and Dutoit (1997)
Slide from Richard Sproat
17LSA 352 Summer 2007
Dynamic Time Warping
Slide from Richard Sproat 18LSA 352 Summer 2007
Finding diphone boundaries
Stable part in phonesFor stops: one third inFor phone-silence: one quarter inFor other diphones: 50% in
In time alignment case:Given explicit known diphone boundaries in prompt in the labelfileUse dynamic time warping to find same stable point in newspeech
Optimal couplingTaylor and Isard 1991, Conkie and Isard 1996Instead of precutting the diphones– Wait until we are about to concatenate the diphones together– Then take the 2 complete (uncut diphones)– Find optimal join points by measuring cepstral distance at potential
join points, pick best
Slide modified from Richard Sproat
4
19LSA 352 Summer 2007
Diphone boundaries in stops
Slide from Richard Sproat 20LSA 352 Summer 2007
Diphone boundaries in endphones
Slide from Richard Sproat
21LSA 352 Summer 2007
Concatenating diphones:junctures
If waveforms are very different, will perceive a click atthe junctures
So need to window them
Also if both diphones are voicedNeed to join them pitch-synchronously
That means we need to know where each pitchperiod begins, so we can paste at the same place ineach pitch period.
Pitch marking or epoch detection: mark whereeach pitch pulse or epoch occurs
– Finding the Instant of Glottal Closure (IGC)
(note difference from pitch tracking)
22LSA 352 Summer 2007
Epoch-labeling
An example of epoch-labeling useing “SHOW PULSES”in Praat:
23LSA 352 Summer 2007
Epoch-labeling:Electroglottograph (EGG)
Also called laryngographor Lx
Device that straps onspeaker’s neck near thelarynxSends small highfrequency currentthrough adam’s appleHuman tissue conductswell; air not as wellTransducer detects howopen the glottis is (I.e.amount of air betweenfolds) by measuringimpedence.
Picture from UCLA Phonetics Lab
24LSA 352 Summer 2007
Less invasive way to do epoch-labeling
Signal processingE.g.:BROOKES, D. M., AND LOKE, H. P. 1999. Modelling energyflow in the vocal tract with applications to glottal closure andopening detection. In ICASSP 1999.
5
25LSA 352 Summer 2007
Prosodic Modification
Modifying pitch and duration independentlyChanging sample rate modifies both:
Chipmunk speech
Duration: duplicate/remove parts of the signalPitch: resample to change pitch
Text from Alan Black 26LSA 352 Summer 2007
Speech as Short Term signals
Alan Black
27LSA 352 Summer 2007
Duration modification
Duplicate/remove short term signals
Slide from Richard Sproat 28LSA 352 Summer 2007
Duration modification
Duplicate/remove short term signals
29LSA 352 Summer 2007
Pitch Modification
Move short-term signals closer together/further apart
Slide from Richard Sproat 30LSA 352 Summer 2007
Overlap-and-add (OLA)
Huang, Acero and Hon
6
31LSA 352 Summer 2007
Windowing
Multiply value of signal at sample number n by thevalue of a windowing functiony[n] = w[n]s[n]
32LSA 352 Summer 2007
Windowing
y[n] = w[n]s[n]
33LSA 352 Summer 2007
Overlap and Add (OLA)
Hanning windows of length 2N used to multiply theanalysis signalResulting windowed signals are addedAnalysis windows, spaced 2NSynthesis windows, spaced NTime compression is uniform with factor of 2Pitch periodicity somewhat lost around 4th window
Huang, Acero, and Hon 34LSA 352 Summer 2007
TD-PSOLA ™
Time-Domain Pitch Synchronous Overlap and AddPatented by France Telecom (CNET)Very efficient
No FFT (or inverse FFT) required
Can modify Hz up to two times or by half
Slide from Richard Sproat
35LSA 352 Summer 2007
TD-PSOLA ™
WindowedPitch-synchronousOverlap--and-add
36LSA 352 Summer 2007
TD-PSOLA ™
Thierry Dutoit
7
37LSA 352 Summer 2007
Summary: Diphone Synthesis
Well-understood, mature technologyAugmentations
StressOnset/codaDemi-syllables
Problems:Signal processing still necessary for modifying durationsSource data is still not naturalUnits are just not large enough; can’t handle word-specificeffects, etc
38LSA 352 Summer 2007
Problems with diphone synthesis
Signal processing methods like TD-PSOLA leaveartifacts, making the speech sound unnaturalDiphone synthesis only captures local effects
But there are many more global effects (syllablestructure, stress pattern, word-level effects)
39LSA 352 Summer 2007
Unit Selection Synthesis
Generalization of the diphone intuitionLarger units
– From diphones to sentences
Many many copies of each unit– 10 hours of speech instead of 1500 diphones (a few
minutes of speech)
Little or no signal processing applied to each unit– Unlike diphones
40LSA 352 Summer 2007
Why Unit Selection Synthesis
Natural data solves problems with diphonesDiphone databases are carefully designed but:
– Speaker makes errors– Speaker doesn’t speak intended dialect– Require database design to be right
If it’s automatic– Labeled with what the speaker actually said– Coarticulation, schwas, flaps are natural
“There’s no data like more data”Lots of copies of each unit mean you can choose just theright one for the contextLarger units mean you can capture wider effects
41LSA 352 Summer 2007
Unit Selection Intuition
Given a big databaseFor each segment (diphone) that we want to synthesize
Find the unit in the database that is the best to synthesizethis target segment
What does “best” mean?“Target cost”: Closest match to the target description, interms of
– Phonetic context– F0, stress, phrase position
“Join cost”: Best join with neighboring units– Matching formants + other spectral characteristics– Matching energy– Matching F0
!
C(t1
n,u1
n) = C
target(
i=1
n
" ti,ui) + Cjoin(
i= 2
n
" ui#1,ui)
42LSA 352 Summer 2007
Targets and Target Costs
A measure of how well a particular unit in the database matchesthe internal representation produced by the prior stagesFeatures, costs, and weightsExamples:
/ih-t/ from stressed syllable, phrase internal, high F0,content word/n-t/ from unstressed syllable, phrase final, low F0, contentword/dh-ax/ from unstressed syllable, phrase initial, high F0,from function word “the”
Slide from Paul Taylor
8
43LSA 352 Summer 2007
Target Costs
Comprised of k subcostsStressPhrase positionF0Phone durationLexical identity
Target cost for a unit:
!
Ct(ti,ui) = wk
tCk
t(
k=1
p
" ti,ui)
Slide from Paul Taylor 44LSA 352 Summer 2007
How to set target cost weights (1)
What you REALLY want as a target cost is the perceivableacoustic difference between two unitsBut we can’t use this, since the target is NOT ACOUSTIC yet, wehaven’t synthesized it!We have to use features that we get from the TTS upper levels(phones, prosody)But we DO have lots of acoustic units in the database.We could use the acoustic distance between these to help setthe WEIGHTS on the acoustic features.
45LSA 352 Summer 2007
How to set target cost weights (2)
Clever Hunt and Black (1996) idea:Hold out some utterances from the databaseNow synthesize one of these utterances
Compute all the phonetic, prosodic, duration featuresNow for a given unit in the outputFor each possible unit that we COULD have used in itsplaceWe can compute its acoustic distance from the TRUEACTUAL HUMAN utterance.This acoustic distance can tell us how to weight thephonetic/prosodic/duration features
46LSA 352 Summer 2007
How to set target cost weights (3)
Hunt and Black (1996)Database and target units labeled with:
phone context, prosodic context, etc.
Need an acoustic similarity between units tooAcoustic similarity based on perceptual features
MFCC (spectral features) (to be defined next week)F0 (normalized)Duration penalty
!
ACt(ti,ui) = wi
aabs(Pi(un ) "
i=1
p
# Pi(um )
Richard Sproat slide
47LSA 352 Summer 2007
How to set target cost weights (3)
Collect phones in classes of acceptable sizeE.g., stops, nasals, vowel classes, etc
Find AC between all of same phone typeFind Ct between all of same phone typeEstimate w1-j using linear regression
48LSA 352 Summer 2007
How to set target cost weights (4)
Target distance is
For examples in the database, we can measure
Therefore, estimate weights w from all examples of
Use linear regression!
ACt(ti,ui) = wi
aabs(Pi(un ) "
i=1
p
# Pi(um )
Richard Sproat slide
!
Ct(ti,ui) = wk
tCk
t(
k=1
p
" ti,ui)
!
ACt(ti,ui) " wk
tCk
t(
k=1
p
# ti,ui)
9
49LSA 352 Summer 2007
Join (Concatenation) Cost
Measure of smoothness of joinMeasured between two database units (target is irrelevant)Features, costs, and weightsComprised of k subcosts:
Spectral featuresF0Energy
Join cost:
!
Cj(ui"1,ui) = wk
jCk
j(
k=1
p
# ui"1,ui)
Slide from Paul Taylor 50LSA 352 Summer 2007
Join costs
Hunt and Black 1996If ui-1==prev(ui) Cc=0Used
MFCC (mel cepstral features)Local F0Local absolute powerHand tuned weights
51LSA 352 Summer 2007
Join costs
The join cost can be used for more than just part ofsearchCan use the join cost for optimal coupling (Isard andTaylor 1991, Conkie 1996), i.e., finding the best placeto join the two units.
Vary edges within a small amount to find best placefor joinThis allows different joins with different unitsThus labeling of database (or diphones) need not beso accurate
52LSA 352 Summer 2007
Hunt and Black 1996We now have weights (per phone type) for features set betweentarget and database unitsFind best path of units through database that minimize:
Standard problem solvable with Viterbi search with beam widthconstraint for pruning
Total Costs
!
C(t1
n,u1
n) = C
target(
i=1
n
" ti,ui) + Cjoin(
i= 2
n
" ui#1,ui)
!
ˆ u 1n
= argminu1 ,...,u
n
C(t1n,u1
n )
Slide from Paul Taylor
53LSA 352 Summer 2007
Improvements
Taylor and Black 1999: Phonological Structure MatchingLabel whole database as trees:
Words/phrases, syllables, phonesFor target utterance:
Label it as treeTop-down, find subtrees that cover targetRecurse if no subtree found
Produces list of target subtrees:Explicitly longer units than other techniques
Selects on:Phonetic/metrical structureOnly indirectly on prosodyNo acoustic cost
Slide from Richard Sproat 54LSA 352 Summer 2007
Unit Selection Search
Slide from Richard Sproat
10
55LSA 352 Summer 2007 56LSA 352 Summer 2007
Database creation (1)
Good speakerProfessional speakers are always better:
– Consistent style and articulation– Although these databases are carefully labeled
Ideally (according to AT&T experiments):– Record 20 professional speakers (small amounts of data)– Build simple synthesis examples– Get many (200?) people to listen and score them– Take best voices
Correlates for human preferences:– High power in unvoiced speech– High power in higher frequencies– Larger pitch range
Text from Paul Taylor and Richard Sproat
57LSA 352 Summer 2007
Database creation (2)
Good recording conditionsGood script
Application dependent helps– Good word coverage– News data synthesizes as news data– News data is bad for dialog.
Good phonetic coverage, especially wrt contextLow ambiguityEasy to read
Annotate at phone level, with stress, word information, phrasebreaks
Text from Paul Taylor and Richard Sproat 58LSA 352 Summer 2007
Creating database
Unliked diphones, prosodic variation is a good thingAccurate annotation is crucialPitch annotation needs to be very very accuratePhone alignments can be done automatically, asdescribed for diphones
59LSA 352 Summer 2007
Practical System Issues
Size of typical system (Rhetorical rVoice):~300M
Speed:For each diphone, average of 1000 units to choose from, so:1000 target costs1000x1000 join costsEach join cost, say 30x30 float point calculations10-15 diphones per second10 billion floating point calculations per second
But commercial systems must run ~50x faster than real timeHeavy pruning essential: 1000 units -> 25 units
Slide from Paul Taylor 60LSA 352 Summer 2007
Unit Selection Summary
AdvantagesQuality is far superior to diphonesNatural prosody selection sounds better
Disadvantages:Quality can be very bad in places
– HCI problem: mix of very good and very bad is quite annoying
Synthesis is computationally expensiveCan’t synthesize everything you want:
– Diphone technique can move emphasis– Unit selection gives good (but possibly incorrect) result
Slide from Richard Sproat
11
61LSA 352 Summer 2007
Recap: Joining Units (+F0 +duration)
unit selection, just like diphone, need to join the unitsPitch-synchronously
For diphone synthesis, need to modify F0 and durationFor unit selection, in principle also need to modify F0 andduration of selection unitsBut in practice, if unit-selection database is big enough(commercial systems)
– no prosodic modifications (selected targets may already beclose to desired prosody)
Alan Black 62LSA 352 Summer 2007
Joining Units (just like diphones)
Dumb:just joinBetter: at zero crossings
TD-PSOLATime-domain pitch-synchronous overlap-and-addJoin at pitch periods (with windowing)
Alan Black
63LSA 352 Summer 2007
Evaluation of TTS
Intelligibility TestsDiagnostic Rhyme Test (DRT)
– Humans do listening identification choice between two wordsdiffering by a single phonetic feature
Voicing, nasality, sustenation, sibilation– 96 rhyming pairs– Veal/feel, meat/beat, vee/bee, zee/thee, etc
Subject hears “veal”, chooses either “veal or “feel” Subject also hears “feel”, chooses either “veal” or “feel”
– % of right answers is intelligibility score.
Overall Quality TestsHave listeners rate space on a scale from 1 (bad) to 5(excellent) (Mean Opinion Score)
AB Tests (prefer A, prefer B) (preference tests)
Huang, Acero, Hon 64LSA 352 Summer 2007
Recent stuff
Problems with Unit Selection SynthesisCan’t modify signal(mixing modified and unmodified sounds bad)But database often doesn’t have exactly what youwant
Solution: HMM (Hidden Markov Model) SynthesisWon the last TTS bakeoff.Sounds unnatural to researchersBut naïve subjects preferred itHas the potential to improve on both diphone andunit selection.
65LSA 352 Summer 2007
HMM Synthesis
Unit selection (Roger)HMM (Roger)
Unit selection (Nina)HMM (Nina)
66LSA 352 Summer 2007
Summary
Diphone SynthesisUnit Selection Synthesis
Target costUnit cost