Sheep, Goats, Lambs and WolvesAn Analysis of Individual Differences in
Speaker Recognition Performance11/23/98
These are the slides of the NIST coordinated presentation for the ICSLP '98 Conference in Sidney,Australia, November 1998. They are not part of the official Proceedings of this Conference.
Click here to start
Table of Contents
Sheep, Goats, Lambs and Wolves, AnAnalysis of Individual Differences in SpeakerRecognition Performance
The Hypothetical Menagerie
The Question
The Task and the Data
Speaker Recognition Performance
The Goat Test Statistic, nonparametricKruskal-Wallis rank sum test
The Hypothetical Menagerie
Lamb/Wolf Test Statistics
Statistical Significance
Distribution of Errors versus Animal Rankings
Can a Lamb be also a Wolf?
Can a Sheep be also a Wolf?
Author: George R. Doddington
Home Page: http://www.nist.gov/speech
Sheep, Goats, Lambs and Wolves, An Analysis of Individual Differences in Speaker Recognition Performance
http://www.nist.gov/speech/tests/spk/1998/icslp_98/index.htm (1 of 2) [6/23/2002 11:01:32 PM]
Can a Sheep be also a Lamb?
Conclusion: It’s a ZOO out there!
Sheep, Goats, Lambs and Wolves, An Analysis of Individual Differences in Speaker Recognition Performance
http://www.nist.gov/speech/tests/spk/1998/icslp_98/index.htm (2 of 2) [6/23/2002 11:01:32 PM]
Slide 1 of 14
Sheep, Goats, Lambs and Wolves
http://www.nist.gov/speech/tests/spk/1998/icslp_98/sld001.htm [6/23/2002 11:01:41 PM]
Slide 2 of 14
The Hypothetical Menagerie
http://www.nist.gov/speech/tests/spk/1998/icslp_98/sld002.htm [6/23/2002 11:01:43 PM]
Slide 14 of 14
Conclusion: It’s a out there!
http://www.nist.gov/speech/tests/spk/1998/icslp_98/sld014.htm [6/23/2002 11:01:45 PM]
Sheep, Goats, Lambs and Wolves AnAnalysis of Individual Differences inSpeaker Recognition Performance
George Doddington1,2,3,5, Walter Liggett1,Alvin Martin1, Mark Przybocki1, DouglasReynolds3,4
1 National Institute of Standards andTechnology 2 The Johns Hopkins University 3U.S. Department of Defense 4 MIT LincolnLaboratory 5 SRI International
Next slide Back to first slide View graphic version
Sheep, Goats, Lambs and Wolves
http://www.nist.gov/speech/tests/spk/1998/icslp_98/tsld001.htm [6/23/2002 11:01:45 PM]
Slide 3 of 14
The Question
http://www.nist.gov/speech/tests/spk/1998/icslp_98/sld003.htm [6/23/2002 11:01:47 PM]
The Hypothetical Menagerie
Typical speakers: The well-behaved majority.Sheep: Speakers who exhibit good true speaker acceptance.❍
●
Problem speakers: The troublesome minorities.Goats: Speakers who are exceptionally unsuccessful at being accepted.❍
Lambs: Speakers who are exceptionally vulnerable to impersonation by others.❍
Wolves: Speakers who are exceptionally successful at impersonating others.❍
●
Previous slide Next slide Back to first slide View graphic version
The Hypothetical Menagerie
http://www.nist.gov/speech/tests/spk/1998/icslp_98/tsld002.htm [6/23/2002 11:01:48 PM]
Slide 4 of 14
The Task and the Data
http://www.nist.gov/speech/tests/spk/1998/icslp_98/sld004.htm [6/23/2002 11:01:50 PM]
The Question
Do these animals really exist? That is, do thespeakers in the population of speakers trulyexhibit individual differences?
●
– OR –
Are the observed performance differencesmerely a result of statistical variance (due tohandset, message content, acousticenvironment, etc.)?
●
Previous slide Next slide Back to first slide View graphic version
The Question
http://www.nist.gov/speech/tests/spk/1998/icslp_98/tsld003.htm [6/23/2002 11:01:50 PM]
Slide 5 of 14
Speaker Recognition Performance
http://www.nist.gov/speech/tests/spk/1998/icslp_98/sld005.htm [6/23/2002 11:01:52 PM]
The Task and the Data
Text-independent speaker verification: acceptspeaker if score ³ threshold (if x ³ q).
30-second test segments (3400 segments, total)❍
Conversational telephone speech❍
200 men and 220 women (no cross-sex trials)❍
Different handsets used in training and test❍
Data restricted to electret handsets❍
For details, refer to the 1998 Speaker RecognitionEvaluation Plan at URL ftp://jaguar.ncsl.nist.gov/evaluations/speaker/feb98/plans/current_plan.htm
Previous slide Next slide Back to first slide View graphic version
The Task and the Data
http://www.nist.gov/speech/tests/spk/1998/icslp_98/tsld004.htm [6/23/2002 11:01:55 PM]
Slide 6 of 14
The Goat Test Statistic
http://www.nist.gov/speech/tests/spk/1998/icslp_98/sld006.htm [6/23/2002 11:01:57 PM]
Speaker Recognition Performance
Previous slide Next slide Back to first slide View graphic version
Speaker Recognition Performance
http://www.nist.gov/speech/tests/spk/1998/icslp_98/tsld005.htm [6/23/2002 11:01:58 PM]
Slide 7 of 14
The Hypothetical Menagerie
http://www.nist.gov/speech/tests/spk/1998/icslp_98/sld007.htm [6/23/2002 11:01:59 PM]
The Goat Test Statistic nonparametricKruskal-Wallis rank sum test
Each model speaker, Mi, contributes ni testsegments (ni ³ 5).
●
Each of these segments is scored against itsrespective model.
●
Test Procedure
Replace each score by its rank among allscores: xij Þ rij.
●
Sum the segment ranks for each model speaker:●
Compute the Kruskal-Wallis test statistic:●
Previous slide Next slide Back to first slide View graphic version
The Goat Test Statistic
http://www.nist.gov/speech/tests/spk/1998/icslp_98/tsld006.htm [6/23/2002 11:02:00 PM]
Slide 8 of 14
Lamb/Wolf Test Statistics
http://www.nist.gov/speech/tests/spk/1998/icslp_98/sld008.htm [6/23/2002 11:02:01 PM]
The Hypothetical Menagerie
Typical speakers: The well-behaved majority.Sheep: A speaker who exhibits good true speaker acceptance.❍
●
Problem speakers: The troublesome minorities.Goat: A speaker who is exceptionally unsuccessful at being accepted.❍
Lamb: A model speaker who is exceptionally vulnerable to impersonation by others.❍
Wolf: A segment speaker who is exceptionally successful at impersonating others.❍
●
Previous slide Next slide Back to first slide View graphic version
The Hypothetical Menagerie
http://www.nist.gov/speech/tests/spk/1998/icslp_98/tsld007.htm [6/23/2002 11:02:03 PM]
Slide 9 of 14
Statistical Significance
http://www.nist.gov/speech/tests/spk/1998/icslp_98/sld009.htm [6/23/2002 11:02:05 PM]
Lamb/Wolf Test Statistics
Each segment speaker, Si, contributes ni testsegments
●
Each of these segments is scored against allmodels
●
Test Procedure
Compute average impostor scores, xij, for all Siand Mj.
●
Replace each score by its rank among allscores.
●
Sum the ranks for each putative animal:Lamb: Row sum❍
Wolf: Column sum❍
●
Previous slide Next slide Back to first slide View graphic version
Lamb/Wolf Test Statistics
http://www.nist.gov/speech/tests/spk/1998/icslp_98/tsld008.htm [6/23/2002 11:02:06 PM]
Slide 10 of 14
Distribution of Errors
http://www.nist.gov/speech/tests/spk/1998/icslp_98/sld010.htm [6/23/2002 11:02:07 PM]
Statistical Significance
Goatish, lambish and wolfish behaviors were alldemonstrated with confidence > 99%
Previous slide Next slide Back to first slide View graphic version
Statistical Significance
http://www.nist.gov/speech/tests/spk/1998/icslp_98/tsld009.htm [6/23/2002 11:02:08 PM]
Slide 11 of 14
Can a Lamb be also a Wolf?
http://www.nist.gov/speech/tests/spk/1998/icslp_98/sld011.htm [6/23/2002 11:02:09 PM]
Distribution of Errors versus AnimalRankings
Cumulative Errors
Misses for Model Speakers False Alarms for Model Speakers False Alarms for Segment Speakers
Cumulative Trials
ordered by Goat/Lamb/Wolf rank
Previous slide Next slide Back to first slide View graphic version
Distribution of Errors
http://www.nist.gov/speech/tests/spk/1998/icslp_98/tsld010.htm [6/23/2002 11:02:10 PM]
Slide 12 of 14
Can a Sheep be also a Wolf?
http://www.nist.gov/speech/tests/spk/1998/icslp_98/sld012.htm [6/23/2002 11:02:12 PM]
Can a Lamb be also a Wolf?Wolf Scores versus Lamb Scores
Wolf Average
Impostor scores
for a segment speaker
averaged over
the best scoring 10%
of model speakers
Lamb Average
Impostor scores for a model speaker averaged over
the best scoring 10% of segment speakers
R2 = 0.63
R2 = 0.34
Previous slide Next slide Back to first slide View graphic version
Can a Lamb be also a Wolf?
http://www.nist.gov/speech/tests/spk/1998/icslp_98/tsld011.htm [6/23/2002 11:02:12 PM]
Slide 13 of 14
Can a Sheep be also a Lamb?
http://www.nist.gov/speech/tests/spk/1998/icslp_98/sld013.htm [6/23/2002 11:02:14 PM]
Can a Sheep be also a Wolf?Wolf Average
Impostor scores
for a segment speaker averaged over the best scoring 10% of model speakers
Model Average
True Speaker scores for a model speaker
averaged over all model speaker segments
Wolf Scores versus Model Scores
R2 = 0.31
R2 = 0.03
Previous slide Next slide Back to first slide View graphic version
Can a Sheep be also a Wolf?
http://www.nist.gov/speech/tests/spk/1998/icslp_98/tsld012.htm [6/23/2002 11:02:14 PM]
Can a Sheep be also a Lamb?Lamb Average
Impostor scores
for a model speaker averaged over the best scoring 10% of segment speakers
Model Average
True Speaker scores for a model speaker
averaged over all model speaker segments
Lamb Scores versus Model Scores
R2 = 0.01
R2 = 0.09
Previous slide Next slide Back to first slide View graphic version
Can a Sheep be also a Lamb?
http://www.nist.gov/speech/tests/spk/1998/icslp_98/tsld013.htm [6/23/2002 11:02:15 PM]
Conclusion: It’s a out there!
Goats, Lambs and Wolves exist!
Recommendation:
Direct research toward the minimization of theseundesirable animals.
●
Include measurement of speaker differences aspart of performance evaluation.
An html copy of this presentation is available at URL http://www.nist.gov/speech/icslp_98
●
Previous slide Back to first slide View graphic version
Conclusion: It’s a out there!
http://www.nist.gov/speech/tests/spk/1998/icslp_98/tsld014.htm [6/23/2002 11:02:15 PM]