148
Introduction Bob GMM Extension Reproducible Bob’s Biometric Recognition Framework A Hands-on Tutorial with Face Recognition Examples Dr. Manuel G¨ unther Vision and Security Technology Department of Computer Science University of Colorado Colorado Springs United States of America – Colorado http://vast.uccs.edu/ ˜ mgunther [email protected] http://vast.uccs.edu/public-data/IJCB.html http://www.idiap.ch/software/bob http://www.idiap.ch/software/bob/docs/bob/bob.bio.base/stable October 1, 2017 – IJCB in Denver 1 / 140

Bob’s Biometric Recognition FrameworkIntroduction Bob GMM Extension Reproducible Watch List Gallery-based M:N Identification Enroll models of different clients into a watch list

  • Upload
    others

  • View
    6

  • Download
    0

Embed Size (px)

Citation preview

Page 1: Bob’s Biometric Recognition FrameworkIntroduction Bob GMM Extension Reproducible Watch List Gallery-based M:N Identification Enroll models of different clients into a watch list

������������������������Introduction

���������������������������Bob

���������������������������GMM

�������������������Extension

�������Reproducible

Bob’s Biometric Recognition FrameworkA Hands-on Tutorial with Face Recognition Examples

Dr. Manuel GuntherVision and Security Technology

Department of Computer ScienceUniversity of Colorado Colorado Springs

United States of America – Coloradohttp://vast.uccs.edu/˜mgunther

[email protected]://vast.uccs.edu/public-data/IJCB.html

http://www.idiap.ch/software/bobhttp://www.idiap.ch/software/bob/docs/bob/bob.bio.base/stable

October 1, 2017 – IJCB in Denver

1 / 140

Page 2: Bob’s Biometric Recognition FrameworkIntroduction Bob GMM Extension Reproducible Watch List Gallery-based M:N Identification Enroll models of different clients into a watch list

������������������������Introduction

���������������������������Bob

���������������������������GMM

�������������������Extension

�������Reproducible

Biometric Recognition

What we will learn today1 Which steps are required for biometric recognition?2 How to run biometric recognition experiments using

bob.bio?→ Hands-On: Face recognition with eigenfaces

3 What are the advantages of GMM-based systems?→ Hands-On: Face recognition with GMMs

4 How to implement your own biometric recognitionalgorithm?→ Hands-On: Face recognition with DCNNs

5 How to provide reproducible research?2 / 140

Page 3: Bob’s Biometric Recognition FrameworkIntroduction Bob GMM Extension Reproducible Watch List Gallery-based M:N Identification Enroll models of different clients into a watch list

������������������������Introduction

���������������������������Bob

���������������������������GMM

�������������������Extension

�������Reproducible

Contents

1 Biometric Recognition

2 Biometric Recognition using Bob

3 Gaussian Mixture Modeling

4 Extensions of bob.bio

5 Reproducible Research

3 / 140

Page 4: Bob’s Biometric Recognition FrameworkIntroduction Bob GMM Extension Reproducible Watch List Gallery-based M:N Identification Enroll models of different clients into a watch list

Advertisement Special Issue

an Open Access Journal by MDPI

COVERED BY

CLARIVATE ANALYTIC

S

EMERGING SOURCES

CITATION INDEX

Guest Editor:

Dr.-Ing Manuel Günther Vision and SecurityTechnology Lab (VaST),University of ColoradoColorado Springs, P.O. Box7150, 1420 Austin BluffsParkway, CO 80933-7150, USA

[email protected]

Deadline for manuscriptsubmissions:31 March 2018

Message from the Guest Editor

Dear Colleagues,

Automatic face recognition has a long history in both industry and academia.While some automatic face systems work well in constraint environmentswith good illumination, high resolution images and cooperating subjects, facerecognition in uncontrolled scenarios with unknown illumination, facialexpression, face pose and/or occlusion still needs more research.Furthermore, it has been shown lately that open-set face recognition is veryfar from being deployable.

This special issue targets researchers in the area of automatic facerecognition, including 2D and 3D algorithms. While new research in open-setface recognition would be favorable, any of the following topics are welcome:

Open-set face recognitionClosed-set face recognitionFace recognition in the wildFace recognition at a distanceFace clusteringFace detectionFacial landmark localizationFace recognition with convolutional neural networks3D algorithms for face recognitionFace recognition datasets and evaluation protocols

Dr.-Ing. Manuel Günther

Author Benefits

Open Access: - free for readers, free publication for well-preparedmanuscripts submitted in 2017.High visibility: Indexed in the Emerging Sources Citation Index (ESCI -Web of Science), Inspec (IET) from Vol. 3 and in the DBLP Computer ScienceBibliography.Rapid publication: manuscripts are peer-reviewed and a first decisionprovided to authors approximately 36 days a er submission; acceptance topublication is undertaken in 7 days (median values for papers published inthis journal in first half of 2017).

Special IssueAutomatic Face Recognition

www.mdpi.com

J. Imaging Editorial Office, [email protected]. Alban-Anlage 66, 4052 Basel, Switzerland Tel. +41 61 683 77 34, Fax: +41 61 302 89 18 mdpi.com/si/11394

http://www.mdpi.com/journal/jimaging/special_issues/face_recognition

Page 5: Bob’s Biometric Recognition FrameworkIntroduction Bob GMM Extension Reproducible Watch List Gallery-based M:N Identification Enroll models of different clients into a watch list

������������������������Introduction

���������������������������Bob

���������������������������GMM

�������������������Extension

�������Reproducible

Outline

1 Biometric RecognitionApplications of Biometric RecognitionBiometric Recognition Tool Chain

Database Interfaces and ProtocolsDetection and AlignmentPhotometric EnhancementFeature ExtractionFeature ProjectionModel EnrollmentProbing/Scoring

Evaluation of Biometric Systems

5 / 140

Page 6: Bob’s Biometric Recognition FrameworkIntroduction Bob GMM Extension Reproducible Watch List Gallery-based M:N Identification Enroll models of different clients into a watch list

•�����������������������Introduction

���������������������������Bob

���������������������������GMM

�������������������Extension

�������Reproducible

Goal of Biometric RecognitionSimilarity Score for Pair of Samples

6 / 140

Page 7: Bob’s Biometric Recognition FrameworkIntroduction Bob GMM Extension Reproducible Watch List Gallery-based M:N Identification Enroll models of different clients into a watch list

�•����������������������Introduction

���������������������������Bob

���������������������������GMM

�������������������Extension

�������Reproducible

Biometric Recognition

Applications of Biometric RecognitionAuthentication: Unlock your phoneVerification: Border controlIdentification: Who is in the data sample?Watch List: Unwanted personsSearch: Give me all samples with XXXClustering: Group samples of subjectsForensics: Who committed the crime?

7 / 140

Page 8: Bob’s Biometric Recognition FrameworkIntroduction Bob GMM Extension Reproducible Watch List Gallery-based M:N Identification Enroll models of different clients into a watch list

��•���������������������Introduction

���������������������������Bob

���������������������������GMM

�������������������Extension

�������Reproducible

Authentication and Verification

Token-based 1:1 ComparisonEnroll a model (token) from one or more samplesTake one or several probe sample(s)Compute similarity between model and probe(s)Threshold similarity to accept or reject

CharacteristicsEnvironment is controlledSubjects cooperate with the system

8 / 140

Page 9: Bob’s Biometric Recognition FrameworkIntroduction Bob GMM Extension Reproducible Watch List Gallery-based M:N Identification Enroll models of different clients into a watch list

���•��������������������Introduction

���������������������������Bob

���������������������������GMM

�������������������Extension

�������Reproducible

Identification

Gallery-based 1:N IdentificationEnroll models of different clients into a galleryTake one or several probe image(s)Compute similarities between models and the probe(s)Return one or several most similar clients

VariationsClosed-set: probe always has a match in the galleryOpen-set: probe might not be in gallery⇒ Threshold similarities

9 / 140

Page 10: Bob’s Biometric Recognition FrameworkIntroduction Bob GMM Extension Reproducible Watch List Gallery-based M:N Identification Enroll models of different clients into a watch list

����•�������������������Introduction

���������������������������Bob

���������������������������GMM

�������������������Extension

�������Reproducible

Watch ListGallery-based M:N Identification

Enroll models of different clients into a watch listDetect subjects in probe sampleCompute similarities between models and all probesDecide, which subjects are on the watch list

CharacteristicsUncontrolled samples with uncooperative subjectsMany more unknown than known subjectsMust handle false alarms of the detector

10 / 140

Page 11: Bob’s Biometric Recognition FrameworkIntroduction Bob GMM Extension Reproducible Watch List Gallery-based M:N Identification Enroll models of different clients into a watch list

�����•������������������Introduction

���������������������������Bob

���������������������������GMM

�������������������Extension

�������Reproducible

Biometric RecognitionTool Chain Steps

Database→ Assures comparability

Preprocessing→ Detection→ Alignment→ Enhancement

Features→ Local vs. global→ Feature projection

Algorithms→ Model enrollment→ Scoring

Evaluation 11 / 140

Page 12: Bob’s Biometric Recognition FrameworkIntroduction Bob GMM Extension Reproducible Watch List Gallery-based M:N Identification Enroll models of different clients into a watch list

�����•������������������Introduction

���������������������������Bob

���������������������������GMM

�������������������Extension

�������Reproducible

Biometric RecognitionTool Chain Steps

Database→ Assures comparability

Preprocessing→ Detection→ Alignment→ Enhancement

Features→ Local vs. global→ Feature projection

Algorithms→ Model enrollment→ Scoring

Evaluation 11 / 140

Page 13: Bob’s Biometric Recognition FrameworkIntroduction Bob GMM Extension Reproducible Watch List Gallery-based M:N Identification Enroll models of different clients into a watch list

������•�����������������Introduction

���������������������������Bob

���������������������������GMM

�������������������Extension

�������Reproducible

Database InterfacesTool Chain Database

Provides data→ Face image, video→ Voice sample, fingerprint→ (Annotations)

Protocol→ Training set→ Enrollment data→ Probe data (per model)→ Score normalization

Protocol Types→ Biased / Unbiased→ Lausanne (development

and evaluation set)12 / 140

Page 14: Bob’s Biometric Recognition FrameworkIntroduction Bob GMM Extension Reproducible Watch List Gallery-based M:N Identification Enroll models of different clients into a watch list

������������������������Introduction

���������������������������Bob

���������������������������GMM

�������������������Extension

�������Reproducible

Biometric RecognitionTool Chain Steps

Database→ Assures comparability

Preprocessing→ Detection→ Alignment→ Enhancement

Features→ Local vs. global→ Feature projection

Algorithms→ Model enrollment→ Scoring

Evaluation 13 / 140

Page 15: Bob’s Biometric Recognition FrameworkIntroduction Bob GMM Extension Reproducible Watch List Gallery-based M:N Identification Enroll models of different clients into a watch list

�������•����������������Introduction

���������������������������Bob

���������������������������GMM

�������������������Extension

�������Reproducible

Face Alignment

SourcesFace detector→ bounding box

Landmark detector→ eyes

Hand-labeledannotations→ eyes

Example

14 / 140

Page 16: Bob’s Biometric Recognition FrameworkIntroduction Bob GMM Extension Reproducible Watch List Gallery-based M:N Identification Enroll models of different clients into a watch list

��������•���������������Introduction

���������������������������Bob

���������������������������GMM

�������������������Extension

�������Reproducible

Photometric EnhancementColor

Grayscale

Hist. Eq.

Self-Quot.

Tan&Triggs

LBP

15 / 140

Page 17: Bob’s Biometric Recognition FrameworkIntroduction Bob GMM Extension Reproducible Watch List Gallery-based M:N Identification Enroll models of different clients into a watch list

������������������������Introduction

���������������������������Bob

���������������������������GMM

�������������������Extension

�������Reproducible

Biometric RecognitionTool Chain Steps

Database→ Assures comparability

Preprocessing→ Detection→ Alignment→ Enhancement

Features→ Local vs. global→ Feature projection

Algorithms→ Model enrollment→ Scoring

Evaluation 16 / 140

Page 18: Bob’s Biometric Recognition FrameworkIntroduction Bob GMM Extension Reproducible Watch List Gallery-based M:N Identification Enroll models of different clients into a watch list

���������•��������������Introduction

���������������������������Bob

���������������������������GMM

�������������������Extension

�������Reproducible

Global Features

Concatenate Pixels (link)

17 / 140

Page 19: Bob’s Biometric Recognition FrameworkIntroduction Bob GMM Extension Reproducible Watch List Gallery-based M:N Identification Enroll models of different clients into a watch list

����������•�������������Introduction

���������������������������Bob

���������������������������GMM

�������������������Extension

�������Reproducible

Global FeaturesLBP Equation

LBP =7∑

i=0s(pi − pc) 2i

s(x) =

0 if x < 01 else

Binary Pattern Example

83 75 126

99 95 141

91 91 100

0 0

0 10

1

1

1binary: 00111001

decimal: 57 comparison

with the center

binary

intensity

Round LBP LBP Histogram Sequences

...

concatened histogram

local histogram

local histogram

LBP code

...

18 / 140

Page 20: Bob’s Biometric Recognition FrameworkIntroduction Bob GMM Extension Reproducible Watch List Gallery-based M:N Identification Enroll models of different clients into a watch list

�����������•������������Introduction

���������������������������Bob

���������������������������GMM

�������������������Extension

�������Reproducible

Global Features

Neural Network (link)

19 / 140

Page 21: Bob’s Biometric Recognition FrameworkIntroduction Bob GMM Extension Reproducible Watch List Gallery-based M:N Identification Enroll models of different clients into a watch list

������������•�����������Introduction

���������������������������Bob

���������������������������GMM

�������������������Extension

�������Reproducible

Local FeaturesGabor Graphs

−50 0 50

−50

0

50

Gabor Wavelet

ψ~kj(~t) =

~k2jσ2 e−

~k2j ~t

2

2σ2[ei ~kT

j ~t − e−σ22

]20 / 140

Page 22: Bob’s Biometric Recognition FrameworkIntroduction Bob GMM Extension Reproducible Watch List Gallery-based M:N Identification Enroll models of different clients into a watch list

�������������•����������Introduction

���������������������������Bob

���������������������������GMM

�������������������Extension

�������Reproducible

Local Features

DCT Blocks

21 / 140

Page 23: Bob’s Biometric Recognition FrameworkIntroduction Bob GMM Extension Reproducible Watch List Gallery-based M:N Identification Enroll models of different clients into a watch list

��������������•���������Introduction

���������������������������Bob

���������������������������GMM

�������������������Extension

�������Reproducible

Local FeaturesMel-Frequency Cepstral Coeff. (MFCC) (link)

22 / 140

Page 24: Bob’s Biometric Recognition FrameworkIntroduction Bob GMM Extension Reproducible Watch List Gallery-based M:N Identification Enroll models of different clients into a watch list

���������������•��������Introduction

���������������������������Bob

���������������������������GMM

�������������������Extension

�������Reproducible

Local FeaturesFingerprint Minutiae (link)

23 / 140

Page 25: Bob’s Biometric Recognition FrameworkIntroduction Bob GMM Extension Reproducible Watch List Gallery-based M:N Identification Enroll models of different clients into a watch list

������������������������Introduction

���������������������������Bob

���������������������������GMM

�������������������Extension

�������Reproducible

Biometric RecognitionTool Chain Steps

Database→ Assures comparability

Preprocessing→ Detection→ Alignment→ Enhancement

Features→ Local vs. global→ Feature projection

Algorithms→ Model enrollment→ Scoring

Evaluation 24 / 140

Page 26: Bob’s Biometric Recognition FrameworkIntroduction Bob GMM Extension Reproducible Watch List Gallery-based M:N Identification Enroll models of different clients into a watch list

����������������•�������Introduction

���������������������������Bob

���������������������������GMM

�������������������Extension

�������Reproducible

Feature ProjectionLinear Subspaces

~z = W(~x − ~µ)

with ~x , ~µ ∈ RD, ~z ∈ RM and W ∈ RM×D

Dimensionality Reduction to 5 Dimensions~x ~w1 ~w2 ~w3 ~w4 ~w5 ~µ

~z = (−6.4, −0.9, −6.8, +0.9, +2.4) (1)

25 / 140

Page 27: Bob’s Biometric Recognition FrameworkIntroduction Bob GMM Extension Reproducible Watch List Gallery-based M:N Identification Enroll models of different clients into a watch list

�����������������•������Introduction

���������������������������Bob

���������������������������GMM

�������������������Extension

�������Reproducible

Feature Projection ExamplesEigenfaces from PCA

Fisherfaces from PCA+LDA

26 / 140

Page 28: Bob’s Biometric Recognition FrameworkIntroduction Bob GMM Extension Reproducible Watch List Gallery-based M:N Identification Enroll models of different clients into a watch list

������������������������Introduction

���������������������������Bob

���������������������������GMM

�������������������Extension

�������Reproducible

Biometric RecognitionTool Chain Steps

Database→ Assures comparability

Preprocessing→ Detection→ Alignment→ Enhancement

Features→ Local vs. global→ Feature projection

Algorithms→ Model enrollment→ Scoring

Evaluation 27 / 140

Page 29: Bob’s Biometric Recognition FrameworkIntroduction Bob GMM Extension Reproducible Watch List Gallery-based M:N Identification Enroll models of different clients into a watch list

������������������•�����Introduction

���������������������������Bob

���������������������������GMM

�������������������Extension

�������Reproducible

Model Enrollment

IdeaGiven one or more features of the same person

(client, subject, identity),combine them into a representation of the person

(model, template, token)

Simple SolutionsStore all featuresCompute average feature

Complex SolutionsTrain classifier→ SVM

MAP adaptation→ GMM

28 / 140

Page 30: Bob’s Biometric Recognition FrameworkIntroduction Bob GMM Extension Reproducible Watch List Gallery-based M:N Identification Enroll models of different clients into a watch list

������������������������Introduction

���������������������������Bob

���������������������������GMM

�������������������Extension

�������Reproducible

Biometric RecognitionTool Chain Steps

Database→ Assures comparability

Preprocessing→ Detection→ Alignment→ Enhancement

Features→ Local vs. global→ Feature projection

Algorithms→ Model enrollment→ Scoring

Evaluation 29 / 140

Page 31: Bob’s Biometric Recognition FrameworkIntroduction Bob GMM Extension Reproducible Watch List Gallery-based M:N Identification Enroll models of different clients into a watch list

�������������������•����Introduction

���������������������������Bob

���������������������������GMM

�������������������Extension

�������Reproducible

Probing/ScoringIdea

Given one model and one probe feature,compute a similarity score

Feature ComparisonsCosine similarityHistogram similarityGabor Jet similarity

Complex SolutionsClassification result→ SVM

Log-likelihood→ GMM

VariationSeveral probe features for one comparison

30 / 140

Page 32: Bob’s Biometric Recognition FrameworkIntroduction Bob GMM Extension Reproducible Watch List Gallery-based M:N Identification Enroll models of different clients into a watch list

������������������������Introduction

���������������������������Bob

���������������������������GMM

�������������������Extension

�������Reproducible

Biometric RecognitionTool Chain Steps

Database→ Assures comparability

Preprocessing→ Detection→ Alignment→ Enhancement

Features→ Local vs. global→ Feature projection

Algorithms→ Model enrollment→ Scoring

Evaluation 31 / 140

Page 33: Bob’s Biometric Recognition FrameworkIntroduction Bob GMM Extension Reproducible Watch List Gallery-based M:N Identification Enroll models of different clients into a watch list

��������������������•���Introduction

���������������������������Bob

���������������������������GMM

�������������������Extension

�������Reproducible

Closed-Set Identification

Rank rk for Probe f ∈ Prk(f ) =

∣∣{m | s(m, f ) ≥ s(m∗, f ),m ∈ G}∣∣

Rank r Recognition RateRR(r) =

∣∣{f | rk(f )≤r}∣∣

|P|

Cumulative Match Char.

1 10Rank

0.0

0.2

0.4

0.6

0.8

1.0

Prob

abilit

y

CMC

Algorithm 1Algorithm 2Algorithm 3

32 / 140

Page 34: Bob’s Biometric Recognition FrameworkIntroduction Bob GMM Extension Reproducible Watch List Gallery-based M:N Identification Enroll models of different clients into a watch list

���������������������•��Introduction

���������������������������Bob

���������������������������GMM

�������������������Extension

�������Reproducible

Open-Set Identification – Watch List

Detection & IdentificationKnown probes PK :

DIRr(θ) =∣∣{f | rk(f )≤r ∧ s(m∗,f )≥ θ}

∣∣|PK |

False Alarm RateUnknown probes PU :

FAR(θ) =

∣∣{f |maxm∈G

s(m,f )>θ}∣∣

|PU |

Detection & Identification

10 2 10 1 100

FAR

0.0

0.2

0.4

0.6

0.8

1.0

DIR

DIR

Algorithm 1Algorithm 2Algorithm 3

33 / 140

Page 35: Bob’s Biometric Recognition FrameworkIntroduction Bob GMM Extension Reproducible Watch List Gallery-based M:N Identification Enroll models of different clients into a watch list

����������������������•�Introduction

���������������������������Bob

���������������������������GMM

�������������������Extension

�������Reproducible

Verification – Single Test SetFalse Non-Match Rate

Genuine scores sg:

FNMR(θ) = |{sg|sg <θ}||{sg}|

Receiver Operating Char.

10 4 10 3 10 2 10 1 100

FMR

0.0

0.2

0.4

0.6

0.8

1.0

1 - F

NMR

ROC

Algorithm 1Algorithm 2Algorithm 3

False Match RateImpostor scores si :

FMR(θ) = |{si |si ≥ θ}||{si}|

Detection Error Tradeoff

0.001 0.01 0.05 0.2 0.7 0.95FMR

0.00020.001

0.0050.010.020.05

0.10.2

0.5

0.7

0.90.95

FNM

R

DETAlgorithm 1Algorithm 2Algorithm 3

34 / 140

Page 36: Bob’s Biometric Recognition FrameworkIntroduction Bob GMM Extension Reproducible Watch List Gallery-based M:N Identification Enroll models of different clients into a watch list

�����������������������•Introduction

���������������������������Bob

���������������������������GMM

�������������������Extension

�������Reproducible

Verification – Lausanne Protocol

Expected Performanceθ∗ = arg min

θ

α · FMRdev(θ) + (1− α) · FNMRdev(θ)

Equal Error RateEER = FMRdev(θ∗)+FNMRdev(θ∗)

2

Half Total Error RateHTER = FMReval(θ∗)+FNMReval(θ∗)

2

Expected Performance Curve

0.0 0.2 0.4 0.6 0.8 1.0alpha

0.0

0.1

0.2

0.3

0.4

0.5

HTER

EPCAlgorithm 1Algorithm 2Algorithm 3

35 / 140

Page 37: Bob’s Biometric Recognition FrameworkIntroduction Bob GMM Extension Reproducible Watch List Gallery-based M:N Identification Enroll models of different clients into a watch list

������������������������Introduction

���������������������������Bob

���������������������������GMM

�������������������Extension

�������Reproducible

Outline

2 Biometric Recognition using BobOpen Source Toolboxesbob.bio

Bob’s Biometric Recognition ToolsBiometric Recognition with bob.bioEvaluation

Hands on: Eigenfaces/FisherfacesDatasetExperiment

Bob’s tools

36 / 140

Page 38: Bob’s Biometric Recognition FrameworkIntroduction Bob GMM Extension Reproducible Watch List Gallery-based M:N Identification Enroll models of different clients into a watch list

������������������������Introduction

•��������������������������Bob

���������������������������GMM

�������������������Extension

�������Reproducible

Open Source Biometric RecognitionOpenBR: Application-Oriented

C++ only (thin Python wrapper)Only a few algorithms(Relatively) complicated command line

CSU Face Recognition Resources: Example-OrientedPython only ⇒ slowTwo face recognition algorithms, one datasetNot extensible

BEAT Platform Comparison-OrientedEvaluates algorithms onlineNot designed for algorithm development

37 / 140

Page 39: Bob’s Biometric Recognition FrameworkIntroduction Bob GMM Extension Reproducible Watch List Gallery-based M:N Identification Enroll models of different clients into a watch list

������������������������Introduction

�•�������������������������Bob

���������������������������GMM

�������������������Extension

�������Reproducible

Bob

Signal-Processing and Machine Learning toolboxhttp://www.idiap.ch/software/bob

Mixture of C++ (Blitz++) and Python (NumPy)Integrates well into NumPy, SciPy, scikit-learn, . . .Large signal and image processing libraryMachine learning (Machines and Trainers) ⇒ GMMImage, video and audio I/O (most formats)Binary I/O with HDF5Simple interface for running biometric experimentsUnified interfaces for biometric databases

38 / 140

Page 40: Bob’s Biometric Recognition FrameworkIntroduction Bob GMM Extension Reproducible Watch List Gallery-based M:N Identification Enroll models of different clients into a watch list

������������������������Introduction

��•������������������������Bob

���������������������������GMM

�������������������Extension

�������Reproducible

BobBob Package Dependencies

39 / 140

Page 41: Bob’s Biometric Recognition FrameworkIntroduction Bob GMM Extension Reproducible Watch List Gallery-based M:N Identification Enroll models of different clients into a watch list

������������������������Introduction

���•�����������������������Bob

���������������������������GMM

�������������������Extension

�������Reproducible

Bob’s Biometric Recognition ToolsBob’s Biometrics Packages: Research-Oriented

Designed to be modular, extensible and configurableSupports different biometrics (face, speech, . . . )Enables more than 10 different biometric databasesImplements several data preprocessorsImplements several types of featuresImplements several different biometric algorithmsHandles several features for enrollment and probingCharacteristics

Python interface to C++ implementationsParallelism by design (through GridTK)

40 / 140

Page 42: Bob’s Biometric Recognition FrameworkIntroduction Bob GMM Extension Reproducible Watch List Gallery-based M:N Identification Enroll models of different clients into a watch list

������������������������Introduction

����•����������������������Bob

���������������������������GMM

�������������������Extension

�������Reproducible

Bob’s Biometric Recognition ToolsTool Chain Steps File-based

Database (database)→ Assures comparability

Preproc. (preprocessor)→ Face image, video→ Voice sample, fingerprint→ (Annotations)

Features (extractor)→ Local vs. global

Algorithms (algorithm)→ Feature projection→ Model enrollment→ Scoring

Evaluation (evaluate.py)41 / 140

Page 43: Bob’s Biometric Recognition FrameworkIntroduction Bob GMM Extension Reproducible Watch List Gallery-based M:N Identification Enroll models of different clients into a watch list

������������������������Introduction

�����•���������������������Bob

���������������������������GMM

�������������������Extension

�������Reproducible

Bob’s Biometric Recognition ToolsDatabaseclass bob.bio.base.database.BioFile:

__init__(id, client_id, path)

class bob.bio.base.database.BioDatabase:__init__(original_directory, protocol, ...)

all_files() -> [BioFile]training_files() -> [BioFile]model_ids() -> [model_id]enroll_files(model_id) -> [BioFile]probe_files(model_id) -> [BioFile]client_id_from_model_id(model_id) -> client_id

original_file_names([BioFile]) -> [filename]annotations(BioFile) -> annotations 42 / 140

Page 44: Bob’s Biometric Recognition FrameworkIntroduction Bob GMM Extension Reproducible Watch List Gallery-based M:N Identification Enroll models of different clients into a watch list

������������������������Introduction

������•��������������������Bob

���������������������������GMM

�������������������Extension

�������Reproducible

Bob’s Biometric Recognition Tools

Preprocessorclass bob.bio.base.preprocessor.Preprocessor:

__init__(parameters)

read_original_data(BioFile, dir, ext) -> data

__call__(data, annotations) -> preprocessedwrite_data(preprocessed, filename)read_data(filename) -> preprocessed

43 / 140

Page 45: Bob’s Biometric Recognition FrameworkIntroduction Bob GMM Extension Reproducible Watch List Gallery-based M:N Identification Enroll models of different clients into a watch list

������������������������Introduction

�������•�������������������Bob

���������������������������GMM

�������������������Extension

�������Reproducible

Bob’s Biometric Recognition ToolsImplemented Preprocessorsbob.bio.face.preprocessor.Basebob.bio.face.preprocessor.FaceCropbob.bio.face.preprocessor.FaceDetect

bob.bio.face.preprocessor.HistogramEqualizationbob.bio.face.preprocessor.SelfQuotientImagebob.bio.face.preprocessor.TanTriggsbob.bio.face.preprocessor.INormLBP

bob.bio.spear.preprocessor.Mod_4Hzbob.bio.spear.preprocessor.Energy_2Gaussbob.bio.spear.preprocessor.Energy_Thr

44 / 140

Page 46: Bob’s Biometric Recognition FrameworkIntroduction Bob GMM Extension Reproducible Watch List Gallery-based M:N Identification Enroll models of different clients into a watch list

������������������������Introduction

���������������������������Bob

���������������������������GMM

�������������������Extension

�������Reproducible

Photometric EnhancementFaceCrop

FaceCrop

HistogramEqualization

SelfQuotientImage

TanTriggs

INormLBP

45 / 140

Page 47: Bob’s Biometric Recognition FrameworkIntroduction Bob GMM Extension Reproducible Watch List Gallery-based M:N Identification Enroll models of different clients into a watch list

������������������������Introduction

��������•������������������Bob

���������������������������GMM

�������������������Extension

�������Reproducible

Bob’s Biometric Recognition Tools

Extractorclass bob.bio.base.extractor.Extractor:

__init__(parameters)

train(training_features, extractor_file)load(extractor_file)

__call__(preprocessed) -> featurewrite_feature(feature, filename)read_feature(filename) -> feature

46 / 140

Page 48: Bob’s Biometric Recognition FrameworkIntroduction Bob GMM Extension Reproducible Watch List Gallery-based M:N Identification Enroll models of different clients into a watch list

������������������������Introduction

���������•�����������������Bob

���������������������������GMM

�������������������Extension

�������Reproducible

Bob’s Biometric Recognition Tools

Implemented Extractorsbob.bio.base.extractor.Linearize

bob.bio.face.extractor.GridGraphbob.bio.face.extractor.LGBPHSbob.bio.face.extractor.DCTBlocksbob.bio.face.extractor.Eigenface

bob.bio.spear.extractor.Cepstralbob.bio.spear.extractor.HTKFeaturesbob.bio.spear.extractor.SPROFeatures

47 / 140

Page 49: Bob’s Biometric Recognition FrameworkIntroduction Bob GMM Extension Reproducible Watch List Gallery-based M:N Identification Enroll models of different clients into a watch list

������������������������Introduction

���������������������������Bob

���������������������������GMM

�������������������Extension

�������Reproducible

Local Features

DCT Blocks

48 / 140

Page 50: Bob’s Biometric Recognition FrameworkIntroduction Bob GMM Extension Reproducible Watch List Gallery-based M:N Identification Enroll models of different clients into a watch list

������������������������Introduction

����������•����������������Bob

���������������������������GMM

�������������������Extension

�������Reproducible

Bob’s Biometric Recognition ToolsAlgorithmclass bob.bio.base.algorithm.Algorithm:

__init__(parameters)

train_projector(training_features, projector_file)load_projector(projector_file)project(feature) -> projectedwrite_feature(projected, filename)read_feature(filename) -> projected

train_enroller(training_features, enroller_file)load_enroller(enroller_file)enroll(features) -> modelwrite_model(model, model_file)read_model(model_file) -> model

score(model, probe) -> similarityscore_for_multiple_probes(model, probes) -> similarity 49 / 140

Page 51: Bob’s Biometric Recognition FrameworkIntroduction Bob GMM Extension Reproducible Watch List Gallery-based M:N Identification Enroll models of different clients into a watch list

������������������������Introduction

�����������•���������������Bob

���������������������������GMM

�������������������Extension

�������Reproducible

Bob’s Biometric Recognition ToolsImplemented Algorithmsbob.bio.base.algorithm.Distancebob.bio.base.algorithm.PCAbob.bio.base.algorithm.LDAbob.bio.base.algorithm.PLDAbob.bio.base.algorithm.BIC

bob.bio.face.algorithm.GaborJetbob.bio.face.algorithm.Histogram

bob.bio.gmm.algorithm.GMMbob.bio.gmm.algorithm.JFAbob.bio.gmm.algorithm.ISVbob.bio.gmm.algorithm.IVector

50 / 140

Page 52: Bob’s Biometric Recognition FrameworkIntroduction Bob GMM Extension Reproducible Watch List Gallery-based M:N Identification Enroll models of different clients into a watch list

������������������������Introduction

���������������������������Bob

���������������������������GMM

�������������������Extension

�������Reproducible

Outline

2 Biometric Recognition using BobOpen Source Toolboxesbob.bio

Bob’s Biometric Recognition ToolsBiometric Recognition with bob.bioEvaluation

Hands on: Eigenfaces/FisherfacesDatasetExperiment

Bob’s tools

51 / 140

Page 53: Bob’s Biometric Recognition FrameworkIntroduction Bob GMM Extension Reproducible Watch List Gallery-based M:N Identification Enroll models of different clients into a watch list

������������������������Introduction

������������•��������������Bob

���������������������������GMM

�������������������Extension

�������Reproducible

Excursion: condaInstallation Through conda

Local environments using condaProvides different environmentsBob is on conda!Environment defined in bob.yml

Installation via conda# create new environment$ conda env create -f bob.yml# activate the environment$ source activate bob# test the installation$ python -c "import caffe; import bob.bio.base"

52 / 140

Page 54: Bob’s Biometric Recognition FrameworkIntroduction Bob GMM Extension Reproducible Watch List Gallery-based M:N Identification Enroll models of different clients into a watch list

������������������������Introduction

�������������•�������������Bob

���������������������������GMM

�������������������Extension

�������Reproducible

Biometric Recognition with Bob

One Command Line to Run Experimentsverify.py

And one More to Evaluateevaluate.py

53 / 140

Page 55: Bob’s Biometric Recognition FrameworkIntroduction Bob GMM Extension Reproducible Watch List Gallery-based M:N Identification Enroll models of different clients into a watch list

������������������������Introduction

�������������•�������������Bob

���������������������������GMM

�������������������Extension

�������Reproducible

Biometric Recognition with Bob

One Command Line to Run Experimentsverify.py

And one More to Evaluateevaluate.py

53 / 140

Page 56: Bob’s Biometric Recognition FrameworkIntroduction Bob GMM Extension Reproducible Watch List Gallery-based M:N Identification Enroll models of different clients into a watch list

������������������������Introduction

��������������•������������Bob

���������������������������GMM

�������������������Extension

�������Reproducible

Biometric Recognition with bob.bioConfiguration File (see Documentation)$ verify.py --create-config-file experiment.py

Requireddatabasepreprocessorextractoralgorithmsub_directory

Optionalprotocolgroupsparallelverbosedry_runforceskip_...

Files and Directoriestemp_directoryresult_directoryextractor_fileprojector_fileenroller_fileextracted_directory...

Running an Experiment$ verify.py experiment.py 54 / 140

Page 57: Bob’s Biometric Recognition FrameworkIntroduction Bob GMM Extension Reproducible Watch List Gallery-based M:N Identification Enroll models of different clients into a watch list

������������������������Introduction

���������������•�����������Bob

���������������������������GMM

�������������������Extension

�������Reproducible

Biometric Recognition with bob.bioWriting Configuration Files

Instantiating a classimport bob.bio.baseimport scipy.spatialalgorithm = bob.bio.base.algorithm.PCA(

subspace_dimension=10,distance_function=scipy.spatial.distance.cosine

)

Including other configuration filespreprocessor = "path/to/other/config.py"

Using registered resourcesdatabase = "atnt"

55 / 140

Page 58: Bob’s Biometric Recognition FrameworkIntroduction Bob GMM Extension Reproducible Watch List Gallery-based M:N Identification Enroll models of different clients into a watch list

������������������������Introduction

����������������•����������Bob

���������������������������GMM

�������������������Extension

�������Reproducible

Biometric Recognition with bob.bioResources

Defined in certain bob.bio packageLinks to configuration fileSpecifies default configurationCan be listed with resources.py

Listing Resources$ resources.py --details --types algorithmList of registered algorithms:...- bob.bio.base 2.0.9 @ /opt/conda/envs/bob/lib/python2.7/site-packages:...

+ pca --> bob.bio.base.config.algorithm.pca: algorithm==> bob.bio.base.algorithm.PCA.PCA(multiple_probe_scoring=’average’, multiple_model_scoring=’average’,

subspace_dimension=0.95, is_distance_function=True, uses_variances=False,distance_function=’<function euclidean at 0x7f649a4e55f0>’)

Actual Configuration File/opt/conda/envs/bob/lib/python2.7/site-packages/bob/bio/base/config/algorithm/pca.py

56 / 140

Page 59: Bob’s Biometric Recognition FrameworkIntroduction Bob GMM Extension Reproducible Watch List Gallery-based M:N Identification Enroll models of different clients into a watch list

������������������������Introduction

�����������������•���������Bob

���������������������������GMM

�������������������Extension

�������Reproducible

EvaluationInformation about Experiment: Experiment.info

Inside results/[sub directory]Contains (complete) configuration of all toolsContains (exact) command line to repeat experiment

Score file: scores-devInside results/[sub directory]/[protocol]/nonormContains 4 values per line→ client id of model→ client id of probe→ probe path→ score

57 / 140

Page 60: Bob’s Biometric Recognition FrameworkIntroduction Bob GMM Extension Reproducible Watch List Gallery-based M:N Identification Enroll models of different clients into a watch list

������������������������Introduction

������������������•��������Bob

���������������������������GMM

�������������������Extension

�������Reproducible

EvaluationEvaluation Command Line$ evaluate.py -vv --dev-files \

results/[sub_directory1]/[protocol]/nonorm/scores-dev \results/[sub_directory2]/[protocol]/nonorm/scores-dev \

--legends legend1 legend2 \...

Different Ways to EvaluateRecognition rate --rrEqual error rate --criterion EERCMC curve --cmc cmc.pdfROC curve --roc roc.pdfDET plot --det det.pdfDIR curve --dir dir.pdf

58 / 140

Page 61: Bob’s Biometric Recognition FrameworkIntroduction Bob GMM Extension Reproducible Watch List Gallery-based M:N Identification Enroll models of different clients into a watch list

������������������������Introduction

���������������������������Bob

���������������������������GMM

�������������������Extension

�������Reproducible

Outline

2 Biometric Recognition using BobOpen Source Toolboxesbob.bio

Bob’s Biometric Recognition ToolsBiometric Recognition with bob.bioEvaluation

Hands on: Eigenfaces/FisherfacesDatasetExperiment

Bob’s tools

59 / 140

Page 62: Bob’s Biometric Recognition FrameworkIntroduction Bob GMM Extension Reproducible Watch List Gallery-based M:N Identification Enroll models of different clients into a watch list

������������������������Introduction

�������������������•�������Bob

���������������������������GMM

�������������������Extension

�������Reproducible

Hands on: Eigenfaces

Example Dataset: AT&T Database of Faces (ORL)40 identities, 10 images eachTraining set: 20 identitiesTest set: 20 identities

Enrollment: 1 model from 5 images per identityProbes: 5 images per identity → 5 probes

Gray images, roughly aligned, 92× 112Small dataset, do not use for publications

60 / 140

Page 63: Bob’s Biometric Recognition FrameworkIntroduction Bob GMM Extension Reproducible Watch List Gallery-based M:N Identification Enroll models of different clients into a watch list

������������������������Introduction

��������������������•������Bob

���������������������������GMM

�������������������Extension

�������Reproducible

Hands on: EigenfacesExamples of the AT&T Dataset

61 / 140

Page 64: Bob’s Biometric Recognition FrameworkIntroduction Bob GMM Extension Reproducible Watch List Gallery-based M:N Identification Enroll models of different clients into a watch list

������������������������Introduction

���������������������•�����Bob

���������������������������GMM

�������������������Extension

�������Reproducible

Hands on: EigenfacesFirst Face Recognition Experiment

Database: AT&T database of faces "atnt"Preprocessor: None (faces are aligned) "base"Extractor: Concatenate pixels "linearize"Algorithm: Principal component analysis "pca"

Experiment PreparationWrite configuration file atnt pca.pyOpen terminal in same directoryActive conda environment: $ source activate bob

Running the Experiment$ verify.py atnt pca.py

62 / 140

Page 65: Bob’s Biometric Recognition FrameworkIntroduction Bob GMM Extension Reproducible Watch List Gallery-based M:N Identification Enroll models of different clients into a watch list

������������������������Introduction

����������������������•����Bob

���������������������������GMM

�������������������Extension

�������Reproducible

Hands on: EigenfacesBasic Experiments

Use a face detector "face-detect"Change photometric enhancement "tan-triggs"Compare PCA with PCA+LDA "pca+lda"

Advanced ExperimentsSelect different subspace dimensions for PCAChange distance function to cosineChange image resolution

NoteEach experiment needs a separate sub directory

63 / 140

Page 66: Bob’s Biometric Recognition FrameworkIntroduction Bob GMM Extension Reproducible Watch List Gallery-based M:N Identification Enroll models of different clients into a watch list

������������������������Introduction

�����������������������•���Bob

���������������������������GMM

�������������������Extension

�������Reproducible

Hands on: Eigenfaces

Expected CMC

1 10Rank

0.0

0.2

0.4

0.6

0.8

1.0

Prob

abilit

y

CMC curve for development set

PCALDA

Expected ROC

10 4 10 3 10 2 10 1 100

FMR

0.0

0.2

0.4

0.6

0.8

1.0

1 - F

NMR

ROC for development set

PCALDA

64 / 140

Page 67: Bob’s Biometric Recognition FrameworkIntroduction Bob GMM Extension Reproducible Watch List Gallery-based M:N Identification Enroll models of different clients into a watch list

������������������������Introduction

������������������������•��Bob

���������������������������GMM

�������������������Extension

�������Reproducible

Bob’s Biometric Recognition Toolsbob.bio.base

Experimentinfrastructure→ verify.py→ evaluate.py→ Toolchain→ Parallelism→ Command line→ Resource handling→ Interfacing databases→ Base classes

Implemented tools→ Filelist database→ Basic algorithms

bob.bio.faceFaceRec databasesFace detectionFacial featuresFaceRec algorithms

bob.bio.spearSpeaker databasesVoice activity detectionSpeech features (MFCC)

bob.bio.gmmGMM-based algorithms

65 / 140

Page 68: Bob’s Biometric Recognition FrameworkIntroduction Bob GMM Extension Reproducible Watch List Gallery-based M:N Identification Enroll models of different clients into a watch list

������������������������Introduction

�������������������������•�Bob

���������������������������GMM

�������������������Extension

�������Reproducible

How it Works in the BackgroundConnecting Parts of the Experiments

Generic I/O interface bob.io.base.loadBinary I/O support bob.io.base.HDF5FileImage I/O support bob.io.imageAudio I/O support bob.io.audioVideo I/O support bob.io.video

Image ProcessingFace detection bob.ip.facedetectFace alignment bob.ip.base.FaceEyesNormPhotometric enhancement bob.ip.base.TanTriggsLinearization numpy.ndarray.flatten

66 / 140

Page 69: Bob’s Biometric Recognition FrameworkIntroduction Bob GMM Extension Reproducible Watch List Gallery-based M:N Identification Enroll models of different clients into a watch list

������������������������Introduction

��������������������������•Bob

���������������������������GMM

�������������������Extension

�������Reproducible

How it works in the backgroundLearning

PCA bob.learn.linear.PCATrainerLDA bob.learn.linear.FisherLDATrainer

Database InterfaceBase interface bob.bio.base.databaseImplemented List of all interfacesAT&T database interface bob.db.atnt

EvaluationLoading score files bob.measure.loadCurves bob.measurePlots matplotlib.pyplot.plot

67 / 140

Page 70: Bob’s Biometric Recognition FrameworkIntroduction Bob GMM Extension Reproducible Watch List Gallery-based M:N Identification Enroll models of different clients into a watch list

������������������������Introduction

���������������������������Bob

���������������������������GMM

�������������������Extension

�������Reproducible

Outline

3 Gaussian Mixture ModelingGaussian Mixture Models

TheoryThe EM AlgorithmBiometric Recognition with GMMs

Hands on: Face Recognition with GMMsParallelizationFeature SelectionCommand Line

GMM ExtensionsInter-Session Variability ModelingTotal Variability Modeling

68 / 140

Page 71: Bob’s Biometric Recognition FrameworkIntroduction Bob GMM Extension Reproducible Watch List Gallery-based M:N Identification Enroll models of different clients into a watch list

������������������������Introduction

���������������������������Bob

•��������������������������GMM

�������������������Extension

�������Reproducible

Biometric Recognition with GMMsGMMs for Biometric Recognition?

Widely used for speech processingMostly unknown in other biometrics

Basic IdeaExtract several local features from data sampleTreat features as independent observationsEnrollment: Model distribution of observationsfrom enrollment samplesScoring: Compute probability that probeobservations were generated by model

69 / 140

Page 72: Bob’s Biometric Recognition FrameworkIntroduction Bob GMM Extension Reproducible Watch List Gallery-based M:N Identification Enroll models of different clients into a watch list

������������������������Introduction

���������������������������Bob

�•�������������������������GMM

�������������������Extension

�������Reproducible

Local FeaturesSpeech Signal Extraction (link)

70 / 140

Page 73: Bob’s Biometric Recognition FrameworkIntroduction Bob GMM Extension Reproducible Watch List Gallery-based M:N Identification Enroll models of different clients into a watch list

������������������������Introduction

���������������������������Bob

���������������������������GMM

�������������������Extension

�������Reproducible

Local Features

DCT Blocks

71 / 140

Page 74: Bob’s Biometric Recognition FrameworkIntroduction Bob GMM Extension Reproducible Watch List Gallery-based M:N Identification Enroll models of different clients into a watch list

������������������������Introduction

���������������������������Bob

���������������������������GMM

�������������������Extension

�������Reproducible

Outline

3 Gaussian Mixture ModelingGaussian Mixture Models

TheoryThe EM AlgorithmBiometric Recognition with GMMs

Hands on: Face Recognition with GMMsParallelizationFeature SelectionCommand Line

GMM ExtensionsInter-Session Variability ModelingTotal Variability Modeling

72 / 140

Page 75: Bob’s Biometric Recognition FrameworkIntroduction Bob GMM Extension Reproducible Watch List Gallery-based M:N Identification Enroll models of different clients into a watch list

������������������������Introduction

���������������������������Bob

��•������������������������GMM

�������������������Extension

�������Reproducible

Gaussian DistributionsGaussian Distribution (Normal Distribution)

Nµ,σ(x) = 1√2πσ2 e−

(x−µ)2

2σ2

x : the input variable ∈ Rµ: the mean ∈ Rσ2: the variance ∈ R

Mean

0

0.02

0.04

0.06

0.08

0.1

0.12

-10 -5 0 5 10

mu = -5 mu = 3

Variance

0

0.02

0.04

0.06

0.08

0.1

0.12

0.14

0.16

-10 -5 0 5 10

sigma^2 = 2.6sigma^2 = 1.0

73 / 140

Page 76: Bob’s Biometric Recognition FrameworkIntroduction Bob GMM Extension Reproducible Watch List Gallery-based M:N Identification Enroll models of different clients into a watch list

������������������������Introduction

���������������������������Bob

���•�����������������������GMM

�������������������Extension

�������Reproducible

Gaussian DistributionsEstimating Gaussian Parameters from Data

Given: dataset D = {x1, . . . , xN}; xn ∈ RGoal: Compute optimal µ and σ2

Mean

µ = 1N

N∑n=1

xn

Variance

σ2 = 1N−1

N∑n=1

(xn − µ)2

74 / 140

Page 77: Bob’s Biometric Recognition FrameworkIntroduction Bob GMM Extension Reproducible Watch List Gallery-based M:N Identification Enroll models of different clients into a watch list

������������������������Introduction

���������������������������Bob

����•����������������������GMM

�������������������Extension

�������Reproducible

Multivatiate GaussiansMultivariate Gaussian (Normal) DistributionN~µ,Σ(~x) = 1√

(2π)d |Σ|e− 1

2 (~x−~µ)TΣ−1(~x−~µ)

~x : the multivariate input ∈ Rd

~µ: the mean ∈ Rd

Σ: the covariance matrix ∈ Rd×d

2D Gaussian (link)

75 / 140

Page 78: Bob’s Biometric Recognition FrameworkIntroduction Bob GMM Extension Reproducible Watch List Gallery-based M:N Identification Enroll models of different clients into a watch list

������������������������Introduction

���������������������������Bob

�����•���������������������GMM

�������������������Extension

�������Reproducible

Multivatiate GaussiansEstimating Multivariate Parameters from Data

Given: dataset D = {~x1, . . . ,~xN};~xn ∈ Rd

Goal: Compute optimal ~µ and Σ

Mean

~µ = 1N

N∑n=1

~xn

Covariance

Σ = 1N−1

N∑n=1

(~xn − ~µ)(~xn − ~µ)T

76 / 140

Page 79: Bob’s Biometric Recognition FrameworkIntroduction Bob GMM Extension Reproducible Watch List Gallery-based M:N Identification Enroll models of different clients into a watch list

������������������������Introduction

���������������������������Bob

������•��������������������GMM

�������������������Extension

�������Reproducible

Gaussian Mixture ModelsMultivariate Gaussian Mixture Models

Collection of M Gaussian components:PΘ(~x) =

M∑i=1

wi N~µi ,Σi(~x)

~x : the multivariate input ∈ Rd

Θ = {wi , ~µi ,Σi | ∀i = 1, . . . ,M}wi : mixing coefficient (weight) ∈ R

→M∑

i=1wi = 1

~µi : mean ∈ Rd of component iΣi : covariance matrix of component i→ full rank ∈ Rd×d

→ or diagonal ∈ Rd with elements ~σi77 / 140

Page 80: Bob’s Biometric Recognition FrameworkIntroduction Bob GMM Extension Reproducible Watch List Gallery-based M:N Identification Enroll models of different clients into a watch list

������������������������Introduction

���������������������������Bob

�������•�������������������GMM

�������������������Extension

�������Reproducible

Gaussian Mixture ModelsExample GMM with d = 2 and M = 3 (link)

78 / 140

Page 81: Bob’s Biometric Recognition FrameworkIntroduction Bob GMM Extension Reproducible Watch List Gallery-based M:N Identification Enroll models of different clients into a watch list

������������������������Introduction

���������������������������Bob

���������������������������GMM

�������������������Extension

�������Reproducible

Outline

3 Gaussian Mixture ModelingGaussian Mixture Models

TheoryThe EM AlgorithmBiometric Recognition with GMMs

Hands on: Face Recognition with GMMsParallelizationFeature SelectionCommand Line

GMM ExtensionsInter-Session Variability ModelingTotal Variability Modeling

79 / 140

Page 82: Bob’s Biometric Recognition FrameworkIntroduction Bob GMM Extension Reproducible Watch List Gallery-based M:N Identification Enroll models of different clients into a watch list

������������������������Introduction

���������������������������Bob

��������•������������������GMM

�������������������Extension

�������Reproducible

The EM Algorithm for GMMs

How to Obtain the GMM Parameters Θ?Given: dataset D = {~x1, . . . ,~xN};~xn ∈ Rd

Goal: Θ∗ that maximizes likelihood L(Θ | D)Assuming independence of training examples ~xn

Θ∗ = arg maxΘ

L(Θ | D)

= arg maxΘ

N∏n=1PΘ(~xn)

= arg maxΘ

N∑n=1

logPΘ(~xn)

80 / 140

Page 83: Bob’s Biometric Recognition FrameworkIntroduction Bob GMM Extension Reproducible Watch List Gallery-based M:N Identification Enroll models of different clients into a watch list

������������������������Introduction

���������������������������Bob

���������•�����������������GMM

�������������������Extension

�������Reproducible

The EM Algorithm for GMMsLog Likelihood

logL(Θ | D) =N∑

n=1log

{M∑

i=1wiN~µi ,Σi(~xn)

}

Idea of Expectation Maximization for GMMsInitialize parameters Θ(0)

Iterate (here: round r):1 E-Step: estimate relevance PΘ(r)(i | ~xn) of each

component i for each data point ~xn2 M-Step: estimate new Gaussian parameters Θ(r+1)

using new relevancesUntil logL(Θ(r+1) | D)− logL(Θ(r) | D) < ε

81 / 140

Page 84: Bob’s Biometric Recognition FrameworkIntroduction Bob GMM Extension Reproducible Watch List Gallery-based M:N Identification Enroll models of different clients into a watch list

������������������������Introduction

���������������������������Bob

����������•����������������GMM

�������������������Extension

�������Reproducible

The EM Algorithm for GMMsE-step: Relevance of Component i

PΘ(i | ~xn) = wi N~µi ,Σi (~xn)M∑

j=1wj N~µj ,Σj (~xn)

M-step: Weights

wi = 1N

N∑n=1

PΘ(i | ~xn)

M-step: Means

~µi =N∑

n=1PΘ(i |~xn)~xn

N∑n=1

PΘ(i |~xn)

M-step: Variances (Dull or Diagonal)

Σi =N∑

n=1PΘ(i |~xn)(~xn−~µi)(~xn−~µi)T

N∑n=1

PΘ(i |~xn)or ~σ2

i =N∑

n=1PΘ(i |~xn) (~xn−~µi)2

N∑n=1

PΘ(i |~xn)82 / 140

Page 85: Bob’s Biometric Recognition FrameworkIntroduction Bob GMM Extension Reproducible Watch List Gallery-based M:N Identification Enroll models of different clients into a watch list

������������������������Introduction

���������������������������Bob

�����������•���������������GMM

�������������������Extension

�������Reproducible

The EM Algorithm for GMMsParallelization of the E-step over Training set D

1 Load GMM model from previous step2 Compute relevances (posterior probabilities)3 Pre-compute weights, means and variances4 Store weights, means and variances into GMMStats

M-step (not Parallelized)1 Load all GMMStats2 Aggregate weights, means and variances3 Store new GMM model

83 / 140

Page 86: Bob’s Biometric Recognition FrameworkIntroduction Bob GMM Extension Reproducible Watch List Gallery-based M:N Identification Enroll models of different clients into a watch list

������������������������Introduction

���������������������������Bob

���������������������������GMM

�������������������Extension

�������Reproducible

Outline

3 Gaussian Mixture ModelingGaussian Mixture Models

TheoryThe EM AlgorithmBiometric Recognition with GMMs

Hands on: Face Recognition with GMMsParallelizationFeature SelectionCommand Line

GMM ExtensionsInter-Session Variability ModelingTotal Variability Modeling

84 / 140

Page 87: Bob’s Biometric Recognition FrameworkIntroduction Bob GMM Extension Reproducible Watch List Gallery-based M:N Identification Enroll models of different clients into a watch list

������������������������Introduction

���������������������������Bob

������������•��������������GMM

�������������������Extension

�������Reproducible

Biometric Recognition with GMMs

Practical ConsiderationsSelect initial Θ(0)

→ Randomly ⇒ slow and unreliable→ Based on initial K-Means clustering on D

Small sample size problem during enrollment→ Ignore ⇒ models overfit→ Compute Universal Background Model (UBM)

+ Adapt model to enrollment data

How to compute the score for model and probe→ Log likelihood using only model→ Log likelihood ratio using model and UBM

85 / 140

Page 88: Bob’s Biometric Recognition FrameworkIntroduction Bob GMM Extension Reproducible Watch List Gallery-based M:N Identification Enroll models of different clients into a watch list

������������������������Introduction

���������������������������Bob

�������������•�������������GMM

�������������������Extension

�������Reproducible

Initialization of Θ(0)

K-MeansGiven: dataset D = {~x1, . . . ,~xN};~xn ∈ Rd

Goal: Cluster input data into K = M clustersImplementation: EM-Algorithm

EM for K-MeansInitialize M means ~µi randomly or at data points ~xnIterate:

1 Assign each data point to the nearest cluster2 Compute new mean ~µi from cluster

Until clusters do not change any more86 / 140

Page 89: Bob’s Biometric Recognition FrameworkIntroduction Bob GMM Extension Reproducible Watch List Gallery-based M:N Identification Enroll models of different clients into a watch list

������������������������Introduction

���������������������������Bob

��������������•������������GMM

�������������������Extension

�������Reproducible

K-MeansRandom Initialization

87 / 140

Page 90: Bob’s Biometric Recognition FrameworkIntroduction Bob GMM Extension Reproducible Watch List Gallery-based M:N Identification Enroll models of different clients into a watch list

������������������������Introduction

���������������������������Bob

��������������•������������GMM

�������������������Extension

�������Reproducible

K-Means

Random Initialization E-Step 1: Assign Clusters

87 / 140

Page 91: Bob’s Biometric Recognition FrameworkIntroduction Bob GMM Extension Reproducible Watch List Gallery-based M:N Identification Enroll models of different clients into a watch list

������������������������Introduction

���������������������������Bob

��������������•������������GMM

�������������������Extension

�������Reproducible

K-MeansE-Step 1: Assign Clusters M-Step 1: Update Means

87 / 140

Page 92: Bob’s Biometric Recognition FrameworkIntroduction Bob GMM Extension Reproducible Watch List Gallery-based M:N Identification Enroll models of different clients into a watch list

������������������������Introduction

���������������������������Bob

��������������•������������GMM

�������������������Extension

�������Reproducible

K-MeansM-Step 1: Update Means E-Step 2: Assign Clusters

87 / 140

Page 93: Bob’s Biometric Recognition FrameworkIntroduction Bob GMM Extension Reproducible Watch List Gallery-based M:N Identification Enroll models of different clients into a watch list

������������������������Introduction

���������������������������Bob

��������������•������������GMM

�������������������Extension

�������Reproducible

K-MeansE-Step 2: Assign Clusters M-Step 2: Update Means

87 / 140

Page 94: Bob’s Biometric Recognition FrameworkIntroduction Bob GMM Extension Reproducible Watch List Gallery-based M:N Identification Enroll models of different clients into a watch list

������������������������Introduction

���������������������������Bob

��������������•������������GMM

�������������������Extension

�������Reproducible

K-MeansM-Step 2: Update Means E-Step 3: Assign Clusters

87 / 140

Page 95: Bob’s Biometric Recognition FrameworkIntroduction Bob GMM Extension Reproducible Watch List Gallery-based M:N Identification Enroll models of different clients into a watch list

������������������������Introduction

���������������������������Bob

��������������•������������GMM

�������������������Extension

�������Reproducible

K-MeansE-Step 3: Assign Clusters M-Step 3: Update Means

87 / 140

Page 96: Bob’s Biometric Recognition FrameworkIntroduction Bob GMM Extension Reproducible Watch List Gallery-based M:N Identification Enroll models of different clients into a watch list

������������������������Introduction

���������������������������Bob

���������������������������GMM

�������������������Extension

�������Reproducible

Biometric Recognition with GMMs

Practical ConsiderationsSelect initial Θ(0)

→ Randomly ⇒ slow and unreliable→ Based on initial K-Means clustering on D

Small sample size problem during enrollment→ Ignore ⇒ models overfit→ Compute Universal Background Model (UBM)

+ Adapt model to enrollment data

How to compute the score for model and probe→ Log likelihood using only model→ Log likelihood ratio using model and UBM

88 / 140

Page 97: Bob’s Biometric Recognition FrameworkIntroduction Bob GMM Extension Reproducible Watch List Gallery-based M:N Identification Enroll models of different clients into a watch list

������������������������Introduction

���������������������������Bob

���������������•�����������GMM

�������������������Extension

�������Reproducible

Universal Background ModelUBM/GMM enrollment

Compute Universal Background Model (UBM) Θ→ Utilize features from training set

Adapt UBM to client-specific model Θc

→ Use enrollment data {~xc1 , . . . ,~xc

Nc} of client c

From Reynolds et al. (2000)

89 / 140

Page 98: Bob’s Biometric Recognition FrameworkIntroduction Bob GMM Extension Reproducible Watch List Gallery-based M:N Identification Enroll models of different clients into a watch list

������������������������Introduction

���������������������������Bob

����������������•����������GMM

�������������������Extension

�������Reproducible

Mean-Only MAP AdaptationRelevance for Component i (E-Step)

PΘ(i | ~xcn) = wi N~µi ,Σi (~x

cn)

M∑j=1

wj N~µj ,Σj (~xcn)

Mean Adaptation (M-Step)

∆~µci =

Nc∑n=1

PΘ(i |~xcn)~xc

n

Nc∑n=1

PΘ(i |~xcn)

Relevance Factor τ

αi =Nc∑n=1

PΘ(i |~xcn)

τ+Nc∑n=1

PΘ(i |~xcn)

Weighted Mean Adaptation~µc

i = αi ∆~µci + (1− αi) ~µi

90 / 140

Page 99: Bob’s Biometric Recognition FrameworkIntroduction Bob GMM Extension Reproducible Watch List Gallery-based M:N Identification Enroll models of different clients into a watch list

������������������������Introduction

���������������������������Bob

���������������������������GMM

�������������������Extension

�������Reproducible

Biometric Recognition with GMMs

Practical ConsiderationsSelect initial Θ(0)

→ Randomly ⇒ slow and unreliable→ Based on initial K-Means clustering on D

Small sample size problem during enrollment→ Ignore ⇒ models overfit→ Compute Universal Background Model (UBM)

+ Adapt model to enrollment data

How to compute the score for model and probe→ Log likelihood using only model→ Log likelihood ratio using model and UBM

91 / 140

Page 100: Bob’s Biometric Recognition FrameworkIntroduction Bob GMM Extension Reproducible Watch List Gallery-based M:N Identification Enroll models of different clients into a watch list

������������������������Introduction

���������������������������Bob

�����������������•���������GMM

�������������������Extension

�������Reproducible

Log Likelihood Ratio ScoringTest Probe Features {~xp

1 , . . . ,~xpN p} with Model Θc

s =N p∑n=1

log PΘc (~xpn)

PΘ(~xpn)

AdvantagesConcentrates on client-specific features (e.g. scars)→ Can be explained by Θc but not by Θ

Ignores features that are common between identities→ Can be explained by Θc and by Θ

Ignores features from background or occlusions→ Cannot be explained by either Θc or Θ

Denominator can be precomputed (projection)92 / 140

Page 101: Bob’s Biometric Recognition FrameworkIntroduction Bob GMM Extension Reproducible Watch List Gallery-based M:N Identification Enroll models of different clients into a watch list

������������������������Introduction

���������������������������Bob

������������������•��������GMM

�������������������Extension

�������Reproducible

Implementation in bob.bio.gmm

Algorithm bob.bio.gmm.algorithm.GMMDefine parameters initCompute UBM Θ train projectorPre-compute PΘ(~xp

n) projectClient-specific model Θc enrollLog-likelihood ratio score

Parallelizing UBM TrainingK-Means for GMM initialization (E-Step)UBM training (E-Step)Use specific command line verify gmm.py

93 / 140

Page 102: Bob’s Biometric Recognition FrameworkIntroduction Bob GMM Extension Reproducible Watch List Gallery-based M:N Identification Enroll models of different clients into a watch list

������������������������Introduction

���������������������������Bob

���������������������������GMM

�������������������Extension

�������Reproducible

Outline

3 Gaussian Mixture ModelingGaussian Mixture Models

TheoryThe EM AlgorithmBiometric Recognition with GMMs

Hands on: Face Recognition with GMMsParallelizationFeature SelectionCommand Line

GMM ExtensionsInter-Session Variability ModelingTotal Variability Modeling

94 / 140

Page 103: Bob’s Biometric Recognition FrameworkIntroduction Bob GMM Extension Reproducible Watch List Gallery-based M:N Identification Enroll models of different clients into a watch list

������������������������Introduction

���������������������������Bob

�������������������•�������GMM

�������������������Extension

�������Reproducible

Excursion GridTK

bob.bio Speaks GridTKDeveloped to submit and monitor jobs with SGEImplements a local scheduler with dependenciesverify.py will submit and execute jobs locally

Monitoring Jobs with the Job ManagerList current jobs and their dependencies: jman list -xl

List parallel jobs jman list -j 1-4 -a

Check logs of processes jman report -j 1-2

Delete jobs jman del -j 1-20

95 / 140

Page 104: Bob’s Biometric Recognition FrameworkIntroduction Bob GMM Extension Reproducible Watch List Gallery-based M:N Identification Enroll models of different clients into a watch list

������������������������Introduction

���������������������������Bob

���������������������������GMM

�������������������Extension

�������Reproducible

Local Features

DCT Blocks

96 / 140

Page 105: Bob’s Biometric Recognition FrameworkIntroduction Bob GMM Extension Reproducible Watch List Gallery-based M:N Identification Enroll models of different clients into a watch list

������������������������Introduction

���������������������������Bob

��������������������•������GMM

�������������������Extension

�������Reproducible

Feature Selection

GMM Configurationimport bob.bio.gmmalgorithm = bob.bio.gmm.algorithm.GMM(

number_of_gaussians = 20,kmeans_training_iterations = 5,gmm_training_iterations = 5

)

Resource (not Recommended in this Tutorial)algorithm = "gmm"

97 / 140

Page 106: Bob’s Biometric Recognition FrameworkIntroduction Bob GMM Extension Reproducible Watch List Gallery-based M:N Identification Enroll models of different clients into a watch list

������������������������Introduction

���������������������������Bob

���������������������•�����GMM

�������������������Extension

�������Reproducible

Hands on: Gaussian Mixture ModelsGMM-based Face Recognition

Database: AT&T database of faces "atnt"Preprocessor: Tan&Triggs "tan-triggs"Features: DCT Blocks "dct-blocks"Algorithm: Gaussian Mixture Models (see above)

Experiment PreparationWrite configuration file atnt gmm.pyOpen terminal in Desktop/TutorialActive conda environment: $ source activate bobRunning the Experiment in Parallel$ verify gmm.py atnt gmm.py --parallel 4

98 / 140

Page 107: Bob’s Biometric Recognition FrameworkIntroduction Bob GMM Extension Reproducible Watch List Gallery-based M:N Identification Enroll models of different clients into a watch list

������������������������Introduction

���������������������������Bob

����������������������•����GMM

�������������������Extension

�������Reproducible

Hands on: Gaussian Mixture Models

Basic ExperimentsCompare different numbers of Gaussians→ Warning! More Gaussians require more memory

Compare different numbers of training iterations→ Warning! More iterations require more time

Advanced ExperimentsCompare different feature parametersChange photometric enhancementChange image resolutionTry out ISV or JFA extensions

99 / 140

Page 108: Bob’s Biometric Recognition FrameworkIntroduction Bob GMM Extension Reproducible Watch List Gallery-based M:N Identification Enroll models of different clients into a watch list

������������������������Introduction

���������������������������Bob

�����������������������•���GMM

�������������������Extension

�������Reproducible

Hands on: Gaussian Mixture Models

Possible CMC

1 10Rank

0.0

0.2

0.4

0.6

0.8

1.0

Prob

abilit

y

CMC curve for development set

PCAGMM-baseGMM-advanced

Possible ROC

10 4 10 3 10 2 10 1 100

FMR

0.0

0.2

0.4

0.6

0.8

1.0

1 - F

NMR

ROC for development set

PCAGMM-baseGMM-advanced

100 / 140

Page 109: Bob’s Biometric Recognition FrameworkIntroduction Bob GMM Extension Reproducible Watch List Gallery-based M:N Identification Enroll models of different clients into a watch list

������������������������Introduction

���������������������������Bob

���������������������������GMM

�������������������Extension

�������Reproducible

Outline

3 Gaussian Mixture ModelingGaussian Mixture Models

TheoryThe EM AlgorithmBiometric Recognition with GMMs

Hands on: Face Recognition with GMMsParallelizationFeature SelectionCommand Line

GMM ExtensionsInter-Session Variability ModelingTotal Variability Modeling

101 / 140

Page 110: Bob’s Biometric Recognition FrameworkIntroduction Bob GMM Extension Reproducible Watch List Gallery-based M:N Identification Enroll models of different clients into a watch list

������������������������Introduction

���������������������������Bob

������������������������•��GMM

�������������������Extension

�������Reproducible

Session Variability Modeling

GMM Mean Super-Vector Notation

~m =

~µ1~µ2...~µM

∈ RM ·d Σ =

Σ1 0 · · · 00 Σ2 0... . . . ...0 0 · · · ΣM

∈ RM ·d×M ·d

Mean-only MAP Adaptation~mc = ~m + ~dc ∈ RM ·d

102 / 140

Page 111: Bob’s Biometric Recognition FrameworkIntroduction Bob GMM Extension Reproducible Watch List Gallery-based M:N Identification Enroll models of different clients into a watch list

������������������������Introduction

���������������������������Bob

�������������������������•�GMM

�������������������Extension

�������Reproducible

Session Variability ModelingInter-Session Variability Modeling (ISV)Suppress session variations (illumination, channels, . . . )

Estimate Session Offset per Sample j~mc = ~m + ~uc

j + ~dc = ~m + U~hcj + ~dc

Training of Session Variability Matrix U using Class LabelsExpectation Maximization (EM)

Test Probe Features {~xp1 , . . . ,~xp

N p} with Model ~mc

Estimate latent variables ~hcn and ~hn and:

s =N p∑n=1

log P(~xpn | ~mc+U~hc

n)P(~xp

n | ~m+U~hn)

103 / 140

Page 112: Bob’s Biometric Recognition FrameworkIntroduction Bob GMM Extension Reproducible Watch List Gallery-based M:N Identification Enroll models of different clients into a watch list

������������������������Introduction

���������������������������Bob

��������������������������•GMM

�������������������Extension

�������Reproducible

Session Variability ModelingTotal Variability Modeling (TV)

Estimate GMM ν that best describes {~x1, . . . ,~xN}:ν = ~m + T~w

TV-Matrix T ∈ RM ·d×r : Linear subspace of variationsi-vector ~w ∈ Rr : Estimated variation

Training of TV-Matrix T w/o Class LabelsExpectation Maximization (EM)

TV-Matrix as Feature ExtractorDistance (e.g. cosine) between i-vectors ~wc and ~wp

104 / 140

Page 113: Bob’s Biometric Recognition FrameworkIntroduction Bob GMM Extension Reproducible Watch List Gallery-based M:N Identification Enroll models of different clients into a watch list

������������������������Introduction

���������������������������Bob

���������������������������GMM

�������������������Extension

�������Reproducible

Outline

4 Extensions of bob.bioWriting new Tools

CSU Face Recognition ResourcesVideo Face Recognition

Deep Convolutional Neural NetworksTraining DCNNsUsing a Pretrained DCNN in bob.bio

Hands on: DCNN FeaturesCommand LineEvaluation

105 / 140

Page 114: Bob’s Biometric Recognition FrameworkIntroduction Bob GMM Extension Reproducible Watch List Gallery-based M:N Identification Enroll models of different clients into a watch list

������������������������Introduction

���������������������������Bob

���������������������������GMM

•������������������Extension

�������Reproducible

More AlgorithmsWriting new Tools

Databasebob.bio.base.BioDatabase

FileList databasebob.bio.base.database.FileListBioDatabase

Preprocessorbob.bio.base.preprocessor.Preprocessor

Extractorbob.bio.base.extractor.Extractor

Algorithmbob.bio.base.algorithm.Algorithm

106 / 140

Page 115: Bob’s Biometric Recognition FrameworkIntroduction Bob GMM Extension Reproducible Watch List Gallery-based M:N Identification Enroll models of different clients into a watch list

������������������������Introduction

���������������������������Bob

���������������������������GMM

�•�����������������Extension

�������Reproducible

LRPCA and LDA-IRCSU Face Recognition Resources

Provides two baseline algorithms→ Local Region PCA (LRPCA)→ LDA on color channels (LDA-IR) aka. CohortLDA

Evaluated on Good, Bad & Ugly (GBU) databasebob.bio.csu

Installation instructions bob.bio.csuImplements wrappers for all tools→ Preprocessor bob.bio.csu.preprocessor→ Extractor bob.bio.csu.extractor→ Algorithm bob.bio.csu.algorithm

Applicable to all implemented databases107 / 140

Page 116: Bob’s Biometric Recognition FrameworkIntroduction Bob GMM Extension Reproducible Watch List Gallery-based M:N Identification Enroll models of different clients into a watch list

������������������������Introduction

���������������������������Bob

���������������������������GMM

��•����������������Extension

�������Reproducible

CSU Face Recognition ResourcesGunther et al. (2012)“An Open Source Framework for StandardizedComparisons of Face Recognition Algorithms”

GBU – Good

0.01 0.1 1 10 100FAR (%)

0

20

40

60

80

100

CA

R (

%)

LDA-IR

LGBPHS

Graphs

ISV

GBU – Bad

0.01 0.1 1 10 100FAR (%)

0

20

40

60

80

100

CA

R (

%)

LDA-IR

LGBPHS

Graphs

ISV

GBU – Ugly

0.01 0.1 1 10 100FAR (%)

0

20

40

60

80

100

CA

R (

%)

LDA-IR

LGBPHS

Graphs

ISV

BANCALDA-IR LGBPHS Graphs ISV

EER 26.2% 13.2% 11.7% 10.0%HTER 27.2% 16.1% 12.4% 10.9%

108 / 140

Page 117: Bob’s Biometric Recognition FrameworkIntroduction Bob GMM Extension Reproducible Watch List Gallery-based M:N Identification Enroll models of different clients into a watch list

������������������������Introduction

���������������������������Bob

���������������������������GMM

�������������������Extension

�������Reproducible

Outline

4 Extensions of bob.bioWriting new Tools

CSU Face Recognition ResourcesVideo Face Recognition

Deep Convolutional Neural NetworksTraining DCNNsUsing a Pretrained DCNN in bob.bio

Hands on: DCNN FeaturesCommand LineEvaluation

109 / 140

Page 118: Bob’s Biometric Recognition FrameworkIntroduction Bob GMM Extension Reproducible Watch List Gallery-based M:N Identification Enroll models of different clients into a watch list

������������������������Introduction

���������������������������Bob

���������������������������GMM

���•���������������Extension

�������Reproducible

Video Face Recognition

Use Face Recognition Algorithms in Videosbob.bio.video.FrameSelector→ Extract frames from videos

bob.bio.video.FrameContainer→ Data I/O using HDF5

bob.bio.video.preproecssor.Wrapper→ Preprocess frames individually

bob.bio.video.extractor.Wrapper→ Extract features for each frame independently

bob.bio.video.algorithm.Wrapper→ Enroll from several frames (of several videos)→ Probe with several frames of one video

110 / 140

Page 119: Bob’s Biometric Recognition FrameworkIntroduction Bob GMM Extension Reproducible Watch List Gallery-based M:N Identification Enroll models of different clients into a watch list

������������������������Introduction

���������������������������Bob

���������������������������GMM

����•��������������Extension

�������Reproducible

Video Face RecognitionExample Configuration File video gmm.pyimport bob.bio.faceimport bob.bio.video

database = "mobio-video"protocol = "male"

preprocessor = bob.bio.video.preprocessor.Wrapper(preprocessor = ’face-detect’,frame_selector = bob.bio.video.FrameSelector(20, ’spread’)

)

extractor = bob.bio.video.extractor.Wrapper(extractor = bob.bio.face.extractor.DCTBlocks(12,0,45),frame_selector = bob.bio.video.FrameSelector(selection_style=’all’)

)

algorithm = bob.bio.video.algorithm.Wrapper(algorithm = ’gmm’,frame_selector = bob.bio.video.FrameSelector(selection_style=’all’)

)

sub_directory = "gmm_MOBIO_video"

Example Command Line$ verify gmm.py video gmm.py --parallel 4 -vv

111 / 140

Page 120: Bob’s Biometric Recognition FrameworkIntroduction Bob GMM Extension Reproducible Watch List Gallery-based M:N Identification Enroll models of different clients into a watch list

������������������������Introduction

���������������������������Bob

���������������������������GMM

�����•�������������Extension

�������Reproducible

The MOBIO Dataset

Examples of the MOBIO Dataset

112 / 140

Page 121: Bob’s Biometric Recognition FrameworkIntroduction Bob GMM Extension Reproducible Watch List Gallery-based M:N Identification Enroll models of different clients into a watch list

������������������������Introduction

���������������������������Bob

���������������������������GMM

������•������������Extension

�������Reproducible

Video FR (Gunther et al., 2016)Algorithm Performance with Frames

Graphs LGBPHS ISV LDA-IR COTS0

5

10

15

20

25

30

HT

ER

onM

OB

IOev

al.i

n%

1 Frame3 Frames

10 Frames20 Frames

1 Image

113 / 140

Page 122: Bob’s Biometric Recognition FrameworkIntroduction Bob GMM Extension Reproducible Watch List Gallery-based M:N Identification Enroll models of different clients into a watch list

������������������������Introduction

���������������������������Bob

���������������������������GMM

�������������������Extension

�������Reproducible

Outline

4 Extensions of bob.bioWriting new Tools

CSU Face Recognition ResourcesVideo Face Recognition

Deep Convolutional Neural NetworksTraining DCNNsUsing a Pretrained DCNN in bob.bio

Hands on: DCNN FeaturesCommand LineEvaluation

114 / 140

Page 123: Bob’s Biometric Recognition FrameworkIntroduction Bob GMM Extension Reproducible Watch List Gallery-based M:N Identification Enroll models of different clients into a watch list

������������������������Introduction

���������������������������Bob

���������������������������GMM

�������•�����������Extension

�������Reproducible

Deep Convolutional Neural NetworksLayered Network TopologyM(I) =M(L)(M(L−1)(· · · (M(2)(M(1)(I))) · · · ))

Feature Extraction using DCNN (link)

115 / 140

Page 124: Bob’s Biometric Recognition FrameworkIntroduction Bob GMM Extension Reproducible Watch List Gallery-based M:N Identification Enroll models of different clients into a watch list

������������������������Introduction

���������������������������Bob

���������������������������GMM

��������•����������Extension

�������Reproducible

Deep Convolutional Neural NetworksNetwork Layer Types

Convolution layers: 2D ⇒ 2DM(l)(x , y) = ∑

u,vw(l)

uv · M(l−1)(u − x , v − y)Fully connected layer: 1D ⇒ 1DM(l)(k) = ∑

jw(l)

kj · M(l−1)(j)Activation function: 1D ⇒ 1D; no weightsM(l)(k) = φ(M(l−1)(k))

Conv. (link) Fully (link) Activation (link)

116 / 140

Page 125: Bob’s Biometric Recognition FrameworkIntroduction Bob GMM Extension Reproducible Watch List Gallery-based M:N Identification Enroll models of different clients into a watch list

������������������������Introduction

���������������������������Bob

���������������������������GMM

���������•���������Extension

�������Reproducible

Deep Convolutional Neural Networks

Feature Extraction: VGG Network (Parkhi et al., 2015)

117 / 140

Page 126: Bob’s Biometric Recognition FrameworkIntroduction Bob GMM Extension Reproducible Watch List Gallery-based M:N Identification Enroll models of different clients into a watch list

������������������������Introduction

���������������������������Bob

���������������������������GMM

����������•��������Extension

�������Reproducible

Deep Convolutional Neural Networks

Network TrainingTraining images: D = {I1, . . . , IN}; In ∈ RW×H

According labels: T = {t1, . . . , tN}; tn ∈ {1, . . . ,C}

Loss function, e.g., J =N∑

n=1(M(In)− tn)2

Unary Label RepresentationLast layer M(L) with C outputsRepresent each class by C − 1 zeros and 1 one

118 / 140

Page 127: Bob’s Biometric Recognition FrameworkIntroduction Bob GMM Extension Reproducible Watch List Gallery-based M:N Identification Enroll models of different clients into a watch list

������������������������Introduction

���������������������������Bob

���������������������������GMM

�����������•�������Extension

�������Reproducible

Deep Convolutional Neural NetworksNetwork Training via Gradient Descent

Compute gradient ∇(L)kj for image/label pair

Chain rule (Backpropagation)∇(L)

kj = ∂J∂w(L)

kj= ∂J

∂M(L) · ∂M(L)

∂w(L)kj

∇(L−1)kj = ∂J

∂w(L−1)kj

= ∂J∂M(L) · ∂M(L)

∂M(L−1) · ∂M(L−1)

∂w(L−1)kj

Example: Fully connected

∂M(l)

∂w(l)kj

=∑j′

w(l)kj′ ·M

(l−1)(j ′)

∂w(l)kj

=M(l−1)(j)

Update weights: w(l)kj −= η∇(l)

kj119 / 140

Page 128: Bob’s Biometric Recognition FrameworkIntroduction Bob GMM Extension Reproducible Watch List Gallery-based M:N Identification Enroll models of different clients into a watch list

������������������������Introduction

���������������������������Bob

���������������������������GMM

������������•������Extension

�������Reproducible

Deep Convolutional Neural Networks

How to Train a Deep Network1 Design network topology2 Label tons of training data3 Buy 1-100 powerful GPUs4 Prepare training data5 Start network training6 Go for a trip to Hawaii

120 / 140

Page 129: Bob’s Biometric Recognition FrameworkIntroduction Bob GMM Extension Reproducible Watch List Gallery-based M:N Identification Enroll models of different clients into a watch list

������������������������Introduction

���������������������������Bob

���������������������������GMM

�������������������Extension

�������Reproducible

Outline

4 Extensions of bob.bioWriting new Tools

CSU Face Recognition ResourcesVideo Face Recognition

Deep Convolutional Neural NetworksTraining DCNNsUsing a Pretrained DCNN in bob.bio

Hands on: DCNN FeaturesCommand LineEvaluation

121 / 140

Page 130: Bob’s Biometric Recognition FrameworkIntroduction Bob GMM Extension Reproducible Watch List Gallery-based M:N Identification Enroll models of different clients into a watch list

������������������������Introduction

���������������������������Bob

���������������������������GMM

�������������������Extension

�������Reproducible

Deep Convolutional Neural Networks

CharacteristicsParkhi et al. (2015)

Image resolution:224× 224 RGBFeature vector @fc7:4096 floatTraining set:982,803 images of2,622 identities

VGG Network

Triplet LossminW

∑(a,p,n)∈T

max{0, α− ‖~za −~zn‖2 + ‖~za −~zp‖2}

~z = W VGG(I)‖VGG(I)‖

122 / 140

Page 131: Bob’s Biometric Recognition FrameworkIntroduction Bob GMM Extension Reproducible Watch List Gallery-based M:N Identification Enroll models of different clients into a watch list

������������������������Introduction

���������������������������Bob

���������������������������GMM

�������������•�����Extension

�������Reproducible

Using a Pretrained DCNN in bob.bio

Implementation DetailsFramework: Caffehttp://caffe.berkeleyvision.orgCan use GPU (here CPU only)Two files:→ .prototxt: Network description (text)→ .caffemodel: Network parameters (binary)

Provides Python interface:caffe.Net(".caffemodel", ".prototxt", caffe.TEST)→ Not very well documented

123 / 140

Page 132: Bob’s Biometric Recognition FrameworkIntroduction Bob GMM Extension Reproducible Watch List Gallery-based M:N Identification Enroll models of different clients into a watch list

������������������������Introduction

���������������������������Bob

���������������������������GMM

��������������•����Extension

�������Reproducible

Using a Pretrained DCNN in bob.bio

Shallow Network Wrapper Class (VGG.py)class Network (bob.bio.base.extractor.Extractor):

__init__(network_file,network_proto,output_layer,input_layer = ’data’,mean_values = None

)

load()__call__(data) -> feature

124 / 140

Page 133: Bob’s Biometric Recognition FrameworkIntroduction Bob GMM Extension Reproducible Watch List Gallery-based M:N Identification Enroll models of different clients into a watch list

������������������������Introduction

���������������������������Bob

���������������������������GMM

�������������������Extension

�������Reproducible

Outline

4 Extensions of bob.bioWriting new Tools

CSU Face Recognition ResourcesVideo Face Recognition

Deep Convolutional Neural NetworksTraining DCNNsUsing a Pretrained DCNN in bob.bio

Hands on: DCNN FeaturesCommand LineEvaluation

125 / 140

Page 134: Bob’s Biometric Recognition FrameworkIntroduction Bob GMM Extension Reproducible Watch List Gallery-based M:N Identification Enroll models of different clients into a watch list

������������������������Introduction

���������������������������Bob

���������������������������GMM

���������������•���Extension

�������Reproducible

Hands on: DCNN Features

VGG Configuration (atnt vgg.py)import bob.bio.facefrom .VGG import Networkimport numpy

preprocessor = bob.bio.face.preprocessor.FaceCrop(cropped_image_size = (224,224),cropped_positions = {"reye" : (112, 86), "leye" : (112, 138)},fixed_positions = {"reye" : (50, 26), "leye" : (50, 64)},dtype = numpy.float32,color_channel = "rgb"

)

extractor = Network(network_caffemodel = "./vgg_face_caffe/VGG_FACE.caffemodel",network_prototxt = "./vgg_face_caffe/VGG_FACE_deploy.prototxt",output_layer = "fc7",mean_values = [129.1863, 104.7624, 93.5940]

)

126 / 140

Page 135: Bob’s Biometric Recognition FrameworkIntroduction Bob GMM Extension Reproducible Watch List Gallery-based M:N Identification Enroll models of different clients into a watch list

������������������������Introduction

���������������������������Bob

���������������������������GMM

����������������•��Extension

�������Reproducible

Hands on: DCNN Features

Face Recognition with DCNN featuresDatabase: AT&T database of faces "atnt"Preprocessor: Color images in 224× 224 (see above)Features: VGG deep features 4096-D (see above)Algorithm: Distance "distance-euclidean"

Experiment Command Line$ verify.py atnt vgg.py --parallel 4

127 / 140

Page 136: Bob’s Biometric Recognition FrameworkIntroduction Bob GMM Extension Reproducible Watch List Gallery-based M:N Identification Enroll models of different clients into a watch list

������������������������Introduction

���������������������������Bob

���������������������������GMM

�����������������•�Extension

�������Reproducible

Hands on: DCNN FeaturesBasic Experiments

Test different distance functionsTry different output layers (see .prototxt)

Advanced ExperimentsUse different algorithm (e.g., PCA)→ Re-use extracted features from previous experiments

Optimize face alignment (see atnt vgg.py)Expert Experiments

Use facial landmark detector for alignmentImplement your own algorithm→ Enroll by averaging deep features

128 / 140

Page 137: Bob’s Biometric Recognition FrameworkIntroduction Bob GMM Extension Reproducible Watch List Gallery-based M:N Identification Enroll models of different clients into a watch list

������������������������Introduction

���������������������������Bob

���������������������������GMM

������������������•Extension

�������Reproducible

Hands on: DCNN Features

Expected CMC

1 10Rank

0.0

0.2

0.4

0.6

0.8

1.0

Prob

abilit

y

CMC curve for development set

PCAGMM-baseVGG

Expected ROC

10 4 10 3 10 2 10 1 100

FMR

0.0

0.2

0.4

0.6

0.8

1.0

1 - F

NMR

ROC for development set

PCAGMM-baseVGG

129 / 140

Page 138: Bob’s Biometric Recognition FrameworkIntroduction Bob GMM Extension Reproducible Watch List Gallery-based M:N Identification Enroll models of different clients into a watch list

������������������������Introduction

���������������������������Bob

���������������������������GMM

�������������������Extension

�������Reproducible

Parenthesis: Classification of the ResultsAmount of Training Data and Training Time

PCA and GMM trained on AT&T training set→ 200 images of 20 identities, some minutes

VGG trained on external dataset→ 982,803 images of 2,622 identities, several days

Dataset and Evaluation CriteriaAT&T database small, outdated (Jain and Li, 2005)→ Easy recognition of 20 identities, no low FMR

Only small frontal images with little variations→ Frontal face recognition is considered solved

Enrollment from 5 images, scoring with one image→ No image/image or template/template comparison

Closed-set recognition, pre-cropped faces 130 / 140

Page 139: Bob’s Biometric Recognition FrameworkIntroduction Bob GMM Extension Reproducible Watch List Gallery-based M:N Identification Enroll models of different clients into a watch list

������������������������Introduction

���������������������������Bob

���������������������������GMM

�������������������Extension

�������Reproducible

Outline

5 Reproducible ResearchWhy Reproducible Research?How to Provide Reproducible Research

131 / 140

Page 140: Bob’s Biometric Recognition FrameworkIntroduction Bob GMM Extension Reproducible Watch List Gallery-based M:N Identification Enroll models of different clients into a watch list

������������������������Introduction

���������������������������Bob

���������������������������GMM

�������������������Extension

•������Reproducible

Reproducible Research

What is Required for Reproducible ResearchA paper that describes your work in allrelevant detailsSome code to reproduce all resultsThe data required to reproduce the resultsDetailed instructions on how to apply the codeon the data to replicate the results of the paper

132 / 140

Page 141: Bob’s Biometric Recognition FrameworkIntroduction Bob GMM Extension Reproducible Watch List Gallery-based M:N Identification Enroll models of different clients into a watch list

������������������������Introduction

���������������������������Bob

���������������������������GMM

�������������������Extension

�•�����Reproducible

Reproducible Research

Levels of Reproducibility (Vandewalle 2009)0 Irreproducible1 Cannot seem to reproduce2 Reproducible, with extreme effort (> 1 month)3 Reproducible, with considerable effort (> 1 week)4 Easily reproducible (∼ 15 min.), but requires

proprietary software (e.g. Matlab)5 Easily reproducible (∼ 15 min.), only free

software

133 / 140

Page 142: Bob’s Biometric Recognition FrameworkIntroduction Bob GMM Extension Reproducible Watch List Gallery-based M:N Identification Enroll models of different clients into a watch list

������������������������Introduction

���������������������������Bob

���������������������������GMM

�������������������Extension

��•����Reproducible

Reproducible Research

Why Reproducible Research?Providing source code and data requires some time . . .

Boost your Research Impact! (Vandewalle 2012)Lower entrance barrier to your publicationsStand out current trends:→ Only 10 % of TIP papers provide source code

(reproducibility level ≥ 2)Statistically, RR work is more valuable:→ 13 / 15 most cited TPAMI or TIP papers provide code→ Multiply number of citations by a factor of 7

134 / 140

Page 143: Bob’s Biometric Recognition FrameworkIntroduction Bob GMM Extension Reproducible Watch List Gallery-based M:N Identification Enroll models of different clients into a watch list

������������������������Introduction

���������������������������Bob

���������������������������GMM

�������������������Extension

���•���Reproducible

Why Reproducible Research?Productivity Boost of your Research Team

Start from the state-of-the-artStart to work fastCollaborate with external partnersCreate something durableAvoid re-doing thingsYou or your team members:→ Improve source code→ Test it→ Document it→ Keep it operational

For the benefit of all team members 135 / 140

Page 144: Bob’s Biometric Recognition FrameworkIntroduction Bob GMM Extension Reproducible Watch List Gallery-based M:N Identification Enroll models of different clients into a watch list

������������������������Introduction

���������������������������Bob

���������������������������GMM

�������������������Extension

����•��Reproducible

How to Provide Reproducible Research

Requirements of Reproducible Research (Anjos et al., 2017)1 Repeatable: Re-run experiments, obtain same results

→ Configuration file2 Shareable: Possible to share data and code

→ Open-source code and data3 Extensible: Easy to implement new research

→ Bob’s biometric recognition framework4 Stable: Guaranteed repeatability through time

→ Local environments with conda and pin-pointed versions

⇒ Mostly fulfilled by Bob and bob.bio

136 / 140

Page 145: Bob’s Biometric Recognition FrameworkIntroduction Bob GMM Extension Reproducible Watch List Gallery-based M:N Identification Enroll models of different clients into a watch list

������������������������Introduction

���������������������������Bob

���������������������������GMM

�������������������Extension

�����•�Reproducible

How to Publish Reproducible ResearchSteps to Publish your Research (Anjos et al., 2017)

1 Design your experiment reproducible→ Avoid proprietary software (Matlab), use free libraries→ One experiment = one environment = one code package→ Keep track of required software, see bob.yml

2 Use a version control system (SVN, git, . . . )→ Add tags, e.g., submitted to conference X→ Track issues and share with your team

3 Implement unit tests, run them for each change4 Provide documentation

→ Installation instructions→ Copy-and-paste command lines

5 Version, package and publish→ The Python Package Index (PyPI) 137 / 140

Page 146: Bob’s Biometric Recognition FrameworkIntroduction Bob GMM Extension Reproducible Watch List Gallery-based M:N Identification Enroll models of different clients into a watch list

������������������������Introduction

���������������������������Bob

���������������������������GMM

�������������������Extension

������•Reproducible

How to Publish Reproducible ResearchBiometric Evaluation And Testing (BEAT)

Web-based platform to evaluate biometric algorithmsOpen-source toolchains and algorithmsAttestation of results→ Link paper with implementation

Toolchain Interface Implementation

138 / 140

Page 147: Bob’s Biometric Recognition FrameworkIntroduction Bob GMM Extension Reproducible Watch List Gallery-based M:N Identification Enroll models of different clients into a watch list

������������������������Introduction

���������������������������Bob

���������������������������GMM

�������������������Extension

�������Reproducible

Escape the Vicious Circle! (link)

139 / 140

Page 148: Bob’s Biometric Recognition FrameworkIntroduction Bob GMM Extension Reproducible Watch List Gallery-based M:N Identification Enroll models of different clients into a watch list

������������������������Introduction

���������������������������Bob

���������������������������GMM

�������������������Extension

�������Reproducible

The EndTopics

PCA:Turk and Pentland (1991)

GMM, EM:Bishop (2006)

UBM-GMM:Reynolds et al. (2000)

ISV:Vogt and Sridharan (2008)

DCNN:Bishop (2006); Parkhi et al. (2015)

Evaluation:Jain and Li (2005)

Bob:Anjos et al. (2012)

bob.bio:Gunther et al. (2012, 2016)

Reproducible Research:Vandewalle et al. (2009);Vandewalle (2012); Anjos et al.(2017)

ReferencesAnjos, A., El Shafey, L., Wallace, R., Gunther, M., McCool, C., and Marcel, S.

(2012). Bob: a free signal processing and machine learning toolbox forresearchers. In ACMMM.

Anjos, A., Gunther, M., de Freitas Pereira, T., Korshunov, P., Mohammadi, A.,and Marcel, S. (2017). Continuously reproducing toolchains in patternrecognition and machine learning experiments. In ICML.

Bishop, C. M. (2006). Pattern Recognition and Machine Learning (InformationScience and Statistics). Springer.

Gunther, M., El Shafey, L., and Marcel, S. (2016). Face recognition inchallenging environments: An experimental and reproducible researchsurvey. In Face Recognition Across the Imaging Spectrum. Springer.

Gunther, M., Wallace, R., and Marcel, S. (2012). An open source framework forstandardized comparisons of face recognition algorithms. In ECCVWorkshops and Demonstrations.

Jain, A. K. and Li, S. Z. (2005). Handbook of Face Recognition. Springer.Parkhi, O. M., Vedaldi, A., and Zisserman, A. (2015). Deep face recognition. In

BMVC.Reynolds, D. A., Quatieri, T. F., and Dunn, R. B. (2000). Speaker verification

using adapted Gaussian mixture models. Digital Signal Processing.Turk, M. and Pentland, A. (1991). Eigenfaces for recognition. Journal of

Cognitive Neuroscience.Vandewalle, P. (2012). Code sharing is associated with research impact in image

processing. Computing in Science Engineering.Vandewalle, P., Kovacevic, J., and Vetterli, M. (2009). Reproducible Research in

Signal Processing - What, why, and how. IEEE Signal Processing Magazine.Vogt, R. and Sridharan, S. (2008). Explicit modelling of session variability for

speaker verification. Computer Speech and Language. 140 / 140