This presentation "reintroduces" watchlist statistics to the biometrics community at the 2004 Biometric Consortium Conference. At this time, many biometric studies and recommendations incorrectly referenced closed set identification statistics for open set applications.
Text of BCC 2003 - Reintroducing Watchlist Statistics
1. Reintroducing Watchlist Statistics Duane Blackburn Federal Bureau of Investigation firstname.lastname@example.orgBiometrics Technology
2. Face Recognition DebateFall 2001Could the hijackers have been spotted?Biometrics Technology 3. Face Recognition DebateFall 2001Can it track me? GregoryTara KayeNancy DOB: 4/21/74 Dorsey Leflore Dorothy Divorced Mother of 2 Shedd2002 Income: $45,328 Matthew Millner Dennis Ricks Joe GrammerGerald ElinorGalbraith PardiniAllanOlliff Wilfred MoeschClayton TaylerWeddell Persky Biometrics TechnologyNicolas Marocco 4. Face Recognition Debate Fall 2001 Or will this happen? Osama bin Laden Biometrics Technology 5. Why the confusion? Confusing Verification vs. Identification incorrect usage of statistics Identification (open set) vs. Identification (closed set) False alarm rate, false accept rate, false match rate, etc.Biometrics Technology 6. Statistics Shortcomings ROC (Verification) Making a claim to your identity CMC (Identification) Each probe is in the database Are measuring closed set identification Neither relates to the problem! Solution: develop statistics that measure this problem. Biometrics Technology 7. Chronologically Face Recognition at a Chokepoint - Scenario Evaluation Results.http://www.dodcounterdrug.com/facialrecognition FRVT 2002.http://www.frvt.org. Both available in the Biometrics Catalog, http://www.biometricscatalog.org.Biometrics Technology 8. First Step Forget what you know! Biometrics Technology 9. Baseline PerformanceFace Recognition VendorTest 2002 (FRVT 2002)- http://www.frvt.org Biometrics Technology 10. Three Tasks Verification Are you who you say you are? Watchlist Are you in my database? If so, who are you? Identification You are in my database, can I find you?Biometrics Technology 11. Verification Threshold = 0.8Claim Response = 0.75 Threshold = 0.7ClaimResponse = 0.65 Threshold = 0.6 Determines if the claimed identity of a face is correct. Metrics: Probability of Correct Verification 1- (False Reject Rate) False AcceptBiometrics Technology 12. Verification Case #1 - Want a low false accept rate False Accept Rate = 0.1%Verification1 0.9 Probability of Correct Verification 0.8 0.7 Cognitec 0.6 0.5 Eyematic 0.4 Identix 0.3 0.2 0.1 0 1False Accept Rate = 0.1%Biometrics Technology 13. Verification Case #2 - Want a high verification rate Probability of Verification = 95%Verification Probability of Correct Verification =Cognitec 1EyematicIdentix 95% 00.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 False Accept RateBiometrics Technology 14. Verification ROCVerification Probability of CorrectVerification = 95%Cognitec1 EyematicIdentix0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 False Accept RateVerification1 Probability of Correct0.8 Verification0.6Cognitec Eyematic0.4Identix 0.20 1False Accept Rate = 0.1%Biometrics Technology 15. WatchlistThreshold = 0.80.75 Threshold = 0.7 0.65 Threshold = 0.62 Database0.600.55 0.25Biometrics Technology 16. WatchlistThreshold = 0.80.75 Threshold = 0.7Database 0.650.700.250.55Biometrics Technology 17. WatchlistCase #1 - Want a high detect and identification rate Detect and Identify = 75% Watchlist Detect and Identification Rate =Cognitec1 Eyematic75%Identix0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1False Alarm RateBiometrics Technology 18. Watchlist Case #2 - Want a low false alarm rate False Alarm Rate = 0.1%Watchlist Detection and Identification Rate 1 0.9 0.8 0.7 Cognitec 0.6 0.5 Eyematic 0.4 Identix 0.3 0.2 0.1 01 False Alarm Rate = 0.1%Biometrics Technology 19. Watchlist WatchlistDetect and Identification Rate = 75%Cognitec1 EyematicIdentix0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1False Alarm Rate Watchlist1 Detection and Identification0.8 0.6 Cognitec RateEyematic0.4 Identix 0.20 1False Alarm Rate = 0.1% Biometrics Technology 20. Identification0.75 Database0.65 0.80 0.70 0.25 0.55Biometrics Technology 21. Identification Database Thats watchlist, not identification!Biometrics Technology 22. Identification I know you are in the database, but who are you? Closed universe test. Compares your picture to all pictures in the database and lists them according to similarity. Metrics: How often is the top answer correct? How often is the answer in the top 2? How often is the answer in the top 3? How often is the answer in the top n?Biometrics Technology 23. Identification Biometrics Technology 24. Three Tasks Verification Are you who you say you are? Watchlist Are you in my database? If so, who are you? Identification You are in my database, can I find you?Make sure you use the correct termsand the correct statistics!Biometrics Technology 25. Government Biometric References Biometrics Catalog - http://www.biometricscatalog.org FRVT - http://www.frvt.org Biometric Consortium http://www.biometrics.org Biometrics Technology