Embed Size (px)
Citation preview
ICB 2007 - 2007/08/28
Curvewise DET Confidence Regions and Pointwise EER
Confidence IntervalsMichael Schuckers
Yordan MinevSt. Lawrence University, Canton, NY, US
and Center for Identification Technology Research (CITeR)
Andy AdlerCarleton University, Ottawa, Canada
Funding for this work comes from NSF grants CNS-0325640 and CNS-0520990. CNS-0325640 is cooperatively funded by the National Science Foundation and the United States Department of Homeland Security. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.
ICB 2007 - 2007/08/28
Motivation
Automated Classifier
Individual Humans
ICB 2007 - 2007/08/28
Significantly Different?
ICB 2007 - 2007/08/28
Background
• Goals (ROC Confidence Region-CR)
– Truly bivariate in nature (both FAR & FRR)
– Deals with multiplicity
– Yields inference for EER
ICB 2007 - 2007/08/28
Previous Work
• Scores and scores in statistics literature, see Pepe (2004) or Zhou et al (2002) for overviews
• In biometrics/machine learning– Poh et al (2007?)– Dass et al (2007)– Macskassy et al (2005)
ICB 2007 - 2007/08/28
Background
Automated Classifier
Individual HumansEER
ICB 2007 - 2007/08/28
Significantly Different?
EER
ICB 2007 - 2007/08/28
Radial Sweep
ICB 2007 - 2007/08/28
Background
• Goals (ROC/DET Confidence Region(CR))
– Truly bivariate in nature (both FAR & FRR)
– Deals with multiplicity
– Yields inference for EER
ICB 2007 - 2007/08/28
Radial Sweep
• Following choices– DET Center (location) cy=cx=1
For EER chose cy=cx
– Number of angles and uniformityToo few = poor performance
– Number of thresholds and uniformity
Original scores
ICB 2007 - 2007/08/28
Steps in Radial Sweep
1. Determine thresholds
2. Calculate ROC/DET
3. Convert from (x,y) to (r,) at each angle
ICB 2007 - 2007/08/28
Confidence Region (CR) Algorithm1. Calculate ROC/DET in polar coordinates and call it
r = {r: in }.
1. Bootstrap both genuine & imposter (healthy & disease) scores
2. Create new ROC/DET in polar coordinates, call it r* for same ’s
3. Repeat Steps 2 & 3 K times
4. Find bounds at each angle such that (1-)100% of bootstraps curves fall inside bounds.
6. Use bounds in previous step to make CR
ICB 2007 - 2007/08/28
Finding CR bounds
Multiple ways to find bounds:1. r+/-
2. r+ U, r- L
3. r+/- √ (var(r))
4. r+U √ (var(r)), r- L√(var(r))
5. r+/-
6. r+u, r-u
We chose #4 with slight adjustment
ICB 2007 - 2007/08/28
Finding U, L
• For the kth bootstrapped ROC curve, find all normalized difference of rθ* from rθ at each θ
tkθ =
• For the kth bootstrapped ROC curve,
if |min(t(θ))| > |max(t(θ))|
m[k] = min(t(θ))
else m[k] = max(t(θ))
• Repeat this for all K bootstrapped ROC curves
)(
*
θ
θθ
rV
rr
ICB 2007 - 2007/08/28
Confidence Region Algorithm
)V(rθ
ICB 2007 - 2007/08/28
Finding U, L
• Sort all normalized absolute differences m[k], k=1,…,K
• To find (1-)100% confidence region for the Population ROC curve, we need to ensure that (1-)100% of bootstrapped ROC curves fall within the following bounds:
– lower bound:
U = (1 - /2)th percentile of the m[k]’s
– upper bound:
L = (/2)th percentile of the m[k]’s
Hall adjustment on bounds, using =10-4
)V(rη - r U θθ
)V(rη - r L θθ
ICB 2007 - 2007/08/28
Recommendations
– At least 100 ’s (angles)
– At least 100 thresholds
– ~10-4 edge (axes) effects
– Min(sample size)>50 for 95% CR
ICB 2007 - 2007/08/28
Examples: Hand Geometry
ICB 2007 - 2007/08/28
Example: Fingerprint
ICB 2007 - 2007/08/28
Example: Facial Recognition
ICB 2007 - 2007/08/28
Table of Pointwise EER’s
Modality Lower Bound EER Upper Boundh
Hand Geometry 0.1178 0.1494 0.1820
Fingerprint 0.0458 0.0715 0.1000
Face 0.0303 0.0575 0.0766
Note: Hall adjustment to confidence intervals
ICB 2007 - 2007/08/28
Final Comments
CR for entire curve (ROC or DET), not pointwise
CI for EER(angle at 45 degrees)
Generalizes most other sweeping methods vertical cy = -1010, cx= 0.5horizontal cy = 0.5, cx= -1010
