Download pdf - BCC 2003 - Reintroducing Watchlist Statistics

Biometrics Technology

Duane BlackburnFederal Bureau of Investigation

[email protected]

Reintroducing Watchlist Statistics


Face Recognition DebateFall 2001

Could the hijackers have been spotted?

Biometrics TechnologyTayler Persky

Wilfred Moesch

Nicolas Marocco

Allan Olliff

Clayton Weddell

Elinor Pardini

Joe Grammer

Matthew Millner

Gerald Galbraith

Dorothy Shedd

Gregory Dorsey

Nancy Leflore

Tara Kaye DOB: 4/21/74

Divorced Mother of 22002 Income:

$45,328

Dennis Ricks


Can it track me?



Osama bin Laden

Or will this happen?


Why the confusion?

• Confusing– Verification vs. Identification

• incorrect usage of statistics– Identification (open set) vs. Identification

(closed set)– False alarm rate, false accept rate, false

match rate, etc.


Statistics Shortcomings

• ROC (Verification)– Making a claim to your identity

• CMC (Identification)– Each probe is in the database

• Are measuring closed set identification

• Neither relates to the problem!– Solution: develop statistics that measure

this problem.


Chronologically

• Face Recognition at a Chokepoint -Scenario Evaluation Results. http://www.dodcounterdrug.com/facialrecognition

• FRVT 2002. http://www.frvt.org.

• Both available in the Biometrics Catalog, http://www.biometricscatalog.org.


First Step

• Forget what you know!


Baseline Performance

Face Recognition Vendor Test 2002 (FRVT 2002) - http://www.frvt.org


Three Tasks

• Verification – Are you who you say you are?

• Watchlist – Are you in my database? If so, who are you?

• Identification – You are in my database, can I find you?


Verification

• Determines if the claimed identity of a face is correct.

• Metrics:– Probability of Correct Verification

• 1- (False Reject Rate)– False Accept

Claim Response = 0.75

Response = 0.65Claim

Threshold = 0.7

Threshold = 0.6

Threshold = 0.8


Verification Case #1 - Want a low false accept rate

00.10.20.30.40.50.60.70.80.9

1

1

Prob

abili

ty o

f Cor

rect

Ver

ifica

tion

False Accept Rate = 0.1%

Verification

Cognitec

Eyematic

Identix



VerificationCase #2 - Want a high verification rate

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

1

False Accept Rate

Prob

abili

ty o

f Cor

rect

Ver

ifica

tion

= 95

%

Verification

Cognitec

Eyematic

Identix

Probability of Verification = 95%


Verification ROC

Verification

0

0.2

0.4

0.6

0.8

1

1


Prob

abili

ty o

f Cor

rect

Ve

rific

atio

n

CognitecEyematicIdentix

Verification

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

1

Prob

abili

ty o

f Cor

rect

Ve

rific

atio

n =

95%

False Accept Rate



Watchlist

Dat

abas

e

0.75

0.65

0.60

0.25

0.55

Threshold = 0.7

Threshold = 0.8

Threshold = 0.62


Watchlist

Dat

abas

e

0.75

0.65

0.70

0.25

0.55

Threshold = 0.8

Threshold = 0.7


WatchlistCase #1 - Want a high detect and identification rate

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

1

False Alarm Rate

Dete

ct a

nd Id

entif

icat

ion

Rate

=

75%

Watchlist

Cognitec

Eyematic

Identix

Detect and Identify = 75%


Watchlist Case #2 - Want a low false alarm rate

00.10.20.30.40.50.60.70.80.9

1

1

Dete

ctio

n an

d Id

entif

icat

ion

Rate

False Alarm Rate = 0.1%

Watchlist

Cognitec

Eyematic

Identix



Watchlist

Watchlist

0

0.2

0.4

0.6

0.8

1

1


Dete

ctio

n an

d Id

entif

icat

ion

Rate


Watchlist

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

1

Dete

ct a

nd Id

entif

icat

ion

Rate

= 7

5%

False Alarm Rate



Identification

Dat

abas

e

0.75

0.65

0.70

0.25

0.55

0.80


Identification

Dat

abas

eThat’s watchlist, not identification!


Identification

• I know you are in the database, but who are you?– Closed universe test.

• Compares your picture to all pictures in the database and lists them according to similarity.

• Metrics:– How often is the top answer correct?– How often is the answer in the top 2?– How often is the answer in the top 3?– How often is the answer in the top n?


Identification


Three Tasks

• Verification – Are you who you say you are?

• Watchlist – Are you in my database? If so, who are you?

• Identification – You are in my database, can I find you?

Make sure you use the correct terms and the correct statistics!


• Biometrics Catalog -http://www.biometricscatalog.org

• FRVT - http://www.frvt.org• Biometric Consortium – http://www.biometrics.org

Government Biometric References