Dynamic Footprint-based Person Recognition Method …web.science.mq.edu.au/~isvr/Documents/s... · · 2005-08-23Dynamic Footprint-based Person Recognition Method and Application

대한전자공학회 2002 하계종합학술대회Dynamic Footprint-based Person Recognition Method-1/59

Dynamic Footprint-basedPerson Recognition Method and

Application to Intelligent Residential Space

August 25, 2004

Jin-Woo Jung

Human-friendly Welfare Robot System Research CenterKAIST, KOREA

1st UNSW-KAIST joint seminar

Dynamic Footprint-based User Identification System-2/59

ContentsI. Introduction

II. Literature Review: Gait vs. Footprint1. Gait vs. Footprint2. Problem Statement

III. Dynamic Footprint1. Definition of dynamic footprint2. Measurement of dynamic footprint3. Footprint database using MAT sensor (MAT-DB)

IV. Dynamic Footprint-based User Identification System1. User Identification Method via Overlapped Footprint Shape2. User Identification Method via COP Trajectory3. Information fusion

V. Concluding Remarks


I. Introduction


Customer’sBehaviors

Observation

Modeling

Left-handedor

Right-handed

Personalized Service : A veteran waitress in a restaurant

Introduction


Personalized Service : A short-cut from the movie “Minority Report”

• A Person comes in the GAP store.• The system tells him with his biometrics and his

previous purchase data as follows:– Hello, Mr. Yakamodo.– Welcome back to the GAP.– How was the tank-tops of GAP for you?

Introduction


Realize a personalized service for multi-user living environmentby human-friendly user identification method !

User1User2

…UserN

UserIdentification

Multi agentLearningSystem

IntentionReading

VariousEnvironmental

SensorsVariousHome

Appliances

Motivation: Personalized serviceIntroduction


Historical Trends of User Identification

• “something that you possess”– Lost, stolen, forgotten, misplaced, or abused

• “something that you know” & “something that you possess”– Difficult to remember, easily recallable password

• “something that you are” (biometrics)– Cannot be misplaced or forgotten

[Jain 1999] A. Jain, R. Bolle, and S. Pankanti, Biometrics: Personal Identification in Networked Society, KAP press, 1999

Introduction

To enhance security

To enhance convenience(human-friendliness)


Remark: What is biometrics?• A science involving the statistical analysis of

biological characteristics (wide-sense)

• The automated method of physiological or behavioral characteristics to determine or verify identity (narrow-sense)

– Universality– Uniqueness– Permanence– Collectability– Performance– Acceptability (Human-friendliness)– Circumvention

[Zhang 2000] D. D. Zhang , Automated Biometrics – Technologies and Systems, KAP press, 2000

Introduction


Comparison of biometrics


0.5101000.5BehavioralGait

0100.5000.5BehavioralVoice

0101000BehavioralSignature

0.50.500.5000BehavioralKeystrokes

0.510.50.510.50.5PhysiologicalEar

0010111PhysiologicalDNA

00.500111PhysiologicalOdor

110.51011PhysiologicalThermograms

10100.511PhysiologicalRetina

1010.5111PhysiologicalIris

10.50.50.50.50.50.5PhysiologicalHand Vein

0.50.50.510.50.50.5PhysiologicalHand

Geometry

10.510.5110.5PhysiologicalFingerprint

01010.501PhysiologicalFace

Circumvention

Acceptability

Performance

Collectability

Permanence

Uniqueness

Universality

CharacteristicBiometrics

1: High, 0.5: Medium, 0: Low

Introduction


0

0.2

0.4

0.6

0.8

1

1.2

Unive

rsa l

ityUni

quen

ess

Perm

anen

ceCol

lect

abilit

yPe

rform

ance

Acce

ptab

il ity

Circum

vent

ion

Fingerprint

Iris

0

0.2

0.4

0.6

0.8

1

1.2

Unive

rsa l

ityUni

quen

ess

Perm

anen

ceCol

lect

abilit

yPe

rform

ance

Acce

ptab

il ity

Circum

vent

ion

S ignature

Voice

Gait


Physiological vs. Behavioral

Note: behaviroal/physiological distinction is slightly artificial.An important characteristic is that behavioral biometrics has time series data.

High accuracy-oriented application

Human-friendlyapplication

Introduction


Behavioral biometrics

Speaker Recognition

Signature System

Keystroke biometrics

Gait biometrics

[Zhang 2000] D. D. Zhang , Automated Biometrics – Technologies and Systems, KAP press, 2000

Microphone

Electric Pad like PDA

Keyboard

Camera (Indoor/Outdoor)

Gait information is one of the most appropriate methodfor the general living environment including futuristic home environment !

Introduction

Requirement onthe environment

Behavioral biometrics[Zhang 2000]

Intentionality

Collectability Acceptability

Predeterminedsentence

Predeterminedwriting

Natural stroking

Natural walking


II-1. Gait vs. FootprintII-2. Problem Statement

II. Literature Review andProblem Statement


Gait as “a total walking cycle”• M. P. Murray et al. (1967) suggests that gait could be considered as a

total walking cycle (periodic) and, if all gait movements were considered, gait is unique.

• Recent research trends on gait recognition

Literature Review: Gait vs. Footprint

Template-based approach [Huang 1999]Model-based approach [Jain 1999]

[Jain 1999] A. Jain, R. Bolle, and S. Pankanti, Biometrics: Personal Identification in Networked Society, KAP press, 1999[Huang 1999] P. S. Huang, C. J. Harris and M. S. Nixon, “Human gait recognition in canonical space using temporal

templates”, IEE Proc. of Vision, Image and Signal Processing, Vol. 146, No. 2, 1999


[Yam 2001] C.-Y. Yam, M. S. Nixon, and J. N. Carter, “Extended Model-Based Automatic Gait Recognition of Walkingand Running”, 3rd Proc. Audio- and Video-based Biometric Person Authentication, pp.284-294, 2001

[Yam 2002] C.-Y. Yam, M. S. Nixon, and J. N. Carter, “Performance Analysis on New Biometric Gait Motion Model”,Proc. of 5th IEEE Southwest Symposium on Image Analysis and Interpretation, pp. 31-34,, 2002

[Yam 2002]

[Yam 2001]

Report

91.7%84.2%20

92%96%5

Recog. Rate of running person

Recog. rate of walking person

Num. of person

Issues in gait recognition: different speed



Issues in gait recognition: different viewing direction

[BenAbdelkader 2002] C. BenAbdelkader, R. Cutler, and L. Davis, „View-invariant Estimation of Height andStride for Gait Recognition“, Lecture notes in computer science, No. 2338, pp.155-167, 2002

65 %17Non-fronto parallel

47 %41Fronto-parallel

Recognition ratesNumber of personsDirection of view

Fronto-parallelview

Non-frontoParallel view



• Multi-modal biometrics

[Shakhnarovich 2001] G. Shakhnarovich, L. Lee, T. Darrell, “Integrated Face and Gait Recognition from Multiple Views”, Proc. of IEEE Computer Society Conference on Computer Vision and Pattern Recognition, Vol.1, pp.439-446 , 2001

[Shakhnarovich 2001]

Issues in gait recognition: Low performance

Face + Gait

Using multiple views



Age, gender, race, shoes andage at which wearing shoes begins

Different foot shape

[Razeghi 2002] M. Razeghi and M. E. Batt, “Foot type classification: a critical review of current methods”,Gait and Posture, Vol. 15, pp. 282-291, 2002



Human footprint as a biometrics

[Kennedy 1996] R. B. Kennedy, “Uniqueness of bare feet and its use as a possible means of identification”,Forensic Science International, Vol. 82, Issue 1, pp.81-87, 1996

• R. B. Kennedy suggests that footprint is unique in each person by comparing 38 measurements from 3000 people’s 6000 inked barefoot impression data.

• With 5mm margin, there was no same foot.



Footprint feature used by Kennedy’s work


bordercenterLMT



Automated footprint recognition

Assumption: Same body posture at every measurement time

Result: 85 % (10 men, 10 training images per individual)

[Nakajima 2000] K. Nakajima, Y. Mizukami, K. Tanaka, T. Tamura, “Footprint-based Personal Recognition”, IEEE Tr. on Biomedical Engineering, Vol. 47, No. 11, Nov. 2000

• K. Nakajima suggests the first version of automated footprint recognition system based on template matching, COP distance and foot angle.



Features used by Nakajima

[Nakajima 2000] K. Nakajima, Y. Mizukami, K. Tanaka, T. Tamura, “Footprint-based Personal Recognition”, IEEE Tr. on Biomedical Engineering, Vol. 47, No. 11, Nov. 2000

Normalized footfor template matching



Human footprint as a biometrics

Kennedy’s work (1996) Nakajima’s work (2001)

• R. B. Kennedy suggests that footprint is unique in each person by comparing 38 measurements from 3000 people’s 6000 inked barefootimpression data.

• Result:, no same foot with 5mm margin (3000 people)

• K. Nakajima suggests the first version of automated footprint recognition system based on template matching, COP distance and foot angle.

• Result: 86.55 % (10 men, 11 training images per individual)


[Nakajima 2001] K. Nakajima, Y. Mizukami, K. Tanaka, and T. Tamura, “A New Biometrics Using Footprint”,Transactions-Institute of Electrical Engineers of Japan D, Vol.121, No.7, pp.770-776, 2001



Comparison of Kennedy’s and Nakajima’s work: spatial resolution

Kennedy’s data Nakajima’s dataSensor size = 10 x 10 mm2



Comparison: overlapped footprint during walking vs. standing footprint

Kennedy’s work Nakajima’s work

• Footprints were taken under controller conditions ensuring that each person steps onto the paper in the same manner.

• Footprints were taken under controller conditions ensuring that each person stood straightat the center of mat and looked toward a mark on the wall. And then averaging data during 5 sec.



Comparison: local feature vs. global feature (template matching)

Kennedy’s work Nakajima’s work



Problem Statement

Fusion of footprint and gait information

Vision(lateral-view)

Pressure sensing mat

MeasurementDevice

84.2% (walking)91.7% (running)

(20 person, 6 samples)

86.55%(10 person, 11 samples)

RecognitionResult

Disadvantages

View-dependency(Non-lateral view: 65% in 17 person,

7 samples [BenAbdelKader 2002])

Lighting, Clothing,Occlusion, Privacy

Gait-based[Yam 2002]

No shoes,High price (more than $15000),

Weight-loading

Footprint-based

[Nakajima 2001]

[Nakajima 2001] K. Nakajima, Y. Mizukami, K. Tanaka, and T. Tamura, “A New Biometrics Using Footprint”,Transactions-Institute of Electrical Engineers of Japan D, Vol.121, No.7, pp.770-776, 2001

[Yam 2002] C.-Y. Yam, M. S. Nixon, and J. N. Carter, “Performance Analysis on New Biometric Gait Motion Model”,Proc. of 5th IEEE Southwest Symposium on Image Analysis and Interpretation, pp. 31-34,, 2002

[BenAbdelKader 2002] C. BenAbdelkader, R. Cutler, and L. Davis, “View-invariant Estimation of Height and Stridefor Gait Recognition”, Lecture notes in computer science, No. 2338, pp.155-167, 20021

Problem Statement


Problem Statement“Given N person’s sequential walking footprints and a new sequential walking footprint M, find the person whose walking footprint is most similar to the walking footprint M.”

Problem Statement

Sequential walking footprint (my idea !)= 2D projected data of 3D human gait

= Dynamic information (gait)Static information (foot shape)

COP/COA trajectoryOverlapped footprint

Human walking Sequential walking footprint (dynamic footprint)

Foot shape

COPtrajectory


III. Dynamic Footprint

III-1. Definition of dynamic footprintIII-2. Measurement of dynamic footprintIII-3. Footprint database using MAT sensor


Characteristics of sequential footprints from human walking

I. Repeatability

“Gait could be considered as a periodic walking cycle.” [Jain 1999]Footprints during walking are 2D projected information of human gait.

Therefore, footprints during walking also have a repeatability.

II. Asymmetry

“The shape of the left foot is typically different fromthat of the right foot.” [Sforza 1998]

Footprint patterns of each foot during walking are also asymmetric.

Dynamic footprint

[Jain 1999] A. Jain, R. Bolle, and S. Pankanti, Biometrics: Personal Identification in Networked Society, KAP press, 199[Sforza 1998] C. Sforza, G. Michielon, N. Fragnito, and V. F. Ferrario, "Foot Asymmetry in Healthy Adults:

Elliptic Fourier Analysis of Standardized Footprints", Jr. of orthopedic research, Vol.16, No.6, pp.758-765, 1998


Definition of dynamic footprint

Dynamic footprint

A (discrete-)time sequence of footprint images FK , K=0, 1,···, T-1

is called a dynamic footprint if the following conditions are satisfied:

1) ∃ m ∈ N s. t. FK ≈ Fm+K for ∀ K=0,…,T-1

(“A ≈ B” means A=R (T (B)) for some rotation R and Translation T)

2) F 0 is the image at the right or left heel

contact time in a single gait cycle.

For K=0, 1,···, T-1, FK : [0≤x≤Wx, 0≤y≤Wy] [0≤p≤Pmax](Wx/Wy: spatial width/height of sensor, Pmax: normalized max. pressure value)

F 0 F 0

F 0 F T-1


Definition of dynamic footprint

Dynamic footprint

Overlapped Footprint (Shape): F (x, y) = 1 if F k (x, y) ≥ Pthreshold

0 otherwise

where Pthreshold is the threshold for noise elimination

Center-Of-Pressure Point in F k: COPk = (xK*, yK*)

where xK*= , yK*=

Center-Of-Pressure Trajectory: COPT = {COP0, COP1, COP2,…,COPT-1}

For K=0, 1,···, T-1, FK : [0≤x≤Wx, 0≤y≤Wy] [0≤p≤Pmax](Wx/Wy: spatial width/height of sensor, Pmax: normalized max. pressure value)

U1

0

−

=

T

K

∑∑ ∑

yxK

x yK

yxF

yxFx

,

),(

),(

∑∑ ∑

yxK

y xK

yxF

yxFy

,

),(

),(

F 0 F T-1


Measurement of dynamic footprint

Dynamic footprint

humanfoot

floor

Location of sensor

MAT sensor Shoe sensor

Noise sensitivity

Human’s burden

Action segmentation

Feature extraction

MAT sensor

Low

Low

Easy

Difficult

Shoe sensor

High

High

Difficult

Easy

ResponseTime

Sensitivity

Pressuresensor

Fast

High

Temperaturesensor

Slow

High

Mat-type pressure sensor[REF] J.-W. Jung, T. Sato and Z. Bien, "Dynamic Footprint-based Person Recognition Method using Hidden Markov Model

and Neural Network", Accepted for publication in Int. Jr. of Intelligent Systems


Assumptions on measurement

Dynamic footprint

(A-1) There is no other object on the MAT sensor except the walking person.

(A-2) There is only one walking person on the MAT sensor in each trial.

(A-3) During one’s walking on the MAT sensor, at least one element ofthe MAT sensor is fired.

(A-4) Normally, user always walks along the designated direction(for example, outdoor-to-indoor) on the MAT sensor.

To find the starting time of dynamic footprint

To find the ending time of dynamic footprint

To exclude the change of walking direction


Assumptions on measurement (cont’d)

Dynamic footprint

(A-5) User’s step length (the distance between the COA point of the first footand that of the second foot) is greater than a predefined maximum footlength, LMAX_FOOT.

(A-6) The ‘two’ footprints are always acquired after a single gait cycle.

(A-7) The user has to walk on the MAT sensor without wearing any shoes(that is, barefoot walking is assumed).

To discriminate the first foot and second foot parts

To exclude the imperfect footprint

To exclude the effect of the sole of shoes


Assumptions on walking person

Dynamic footprint

(A-8) The person who may sometimes drag their foot after another isexcluded from the subjects.

(A-9) Users are in normal mood when they walk.

(A-10) Users have not any additional loading weight except their bodyand clothes.

To exclude abnormally walking users (users with foot pain)

To exclude the emotional effect

To exclude the effect of weight loading


Footprint database, MAT-DB

Footprint DB usingmat-type sensor

• Number of person = 11 (M:9, F:2)• Number of data per each person

= 40• Period of data gathering = 40 days• Contents of data = dynamic footprint• Sensor = FOOT ANALZER• Frame rate = 30 Hz

Dynamic footprint

FOOT ANALZER (TechStorm, Inc.)

-Total size: 400 x 800 mm2- Spatial resolution: 10 x 10 mm- Number of sensors: 40 x 80- Max. sampling speed = 30 Hz- FSR-based mat-type pressure sensor


IV-1. User Identification Method via Overlapped Footprint ShapeIV-2. User Identification Method via Center-Of-Pressure TrajectoryIV-3. Information fusion

IV. Dynamic Footprint-based User Identification System


Feature Extraction

WalkingFootprint

OverlappedFootprint

ShapeExtraction

User Identification

Footprint ShapeMatching(Type I)

UserID

RegisteredFootprintTemplates

Degreesof ShapeDissimilarity

Overlapped Footprint Shape

Person Recognition Method via Overlapped Footprint Shape


Overlapped Footprint ShapesOverlapped Footprint Shape


[Procedures for extracting aligned footprint]

When a dynamic footprint is given,Step 1. Labeling of each blobStep 2. Finding COA points of each blobStep 3. Determine first foot / second footStep 4. Updating overlapped footprint imageStep 5. Creation of overlapped footprint imageStep 6. Finding the rotation angle using the principal axes of

overlapped footprint imageStep 7. Creation of aligned footprint,


),(* yxF

),(*

yxF



[User Identification Method]

When a dynamic footprint and ,i=1,..,U are given, (Num. of User = U)

[Type I: Dissimilarity]User ID =

Deg. of Dissimilarity =

[Type II. Similarity]User ID = Deg. of Dissimilarity =


),(*

yxF ),( yxFi

∑

−

yx

i

iyxFyxF

,

2*),(),(minarg

)()(

)(maxarg*

*

i

i

i FPFP

FFP ∩

∑

−

yx

iyxFyxF

,

2**),(),(

)()(

)(**

**

i

i

FPFP

FFP ∩

*i

*i

where P(A) is total number of pixels which has bigger value than threshold value

Considering the foot size

Considering the normalization



0.90919.09090.86368.6364average

0.501011

0.5150.51510

1.5152159

0.5100.558

1101.557

0.5101106

0.55055

1.5152204

250.553

1150.5102

0.50051

FAR(%)FRR(%)FAR(%)FRR(%)

Method for normalization

Nakajima methoduser

Similarity value =

where P(A) is total number of pixels which hasbigger value than threshold value

P ( A B ) P ( A ) P ( B )

∩

Given two images A and B,

∑ −yx

BA yxIyxI,

2)},(),({Dissimilarity value =

Nakajima method (Type I)

Method for normalization (Type II)




Discussion on dissimilarity value(Type I)

1 2 3 4 5 6 7 8 9 10 11

0

2

4

6

8

10

12

14

16

18

20

user ID

diss

imila

rity

(pix

els)

Result with the data of user ID = 1

1 2 3 4 5 6 7 8 9 10 11

0

2

4

6

8

10

12

14

16

18

20

user ID

diss

imila

rity

(pix

els)


Changing range: 3 pixels Changing range: 4 pixels

Sensitive to noise

USER 1 USER 2



Feature Extraction

WalkingFootprint

Quantized COPT

Extraction

User Identification

First Foot’sCOPT Matching

UserID

RegisteredHMMs

Second Foot’sCOPT Matching

Degreesof COPTSimilarity

weightedsum

COPT: COP TrajectoryCenter-Of-Pressure Trajectory

Person Recognition Method viaCOP Trajectory


COP trajectories (COPTs)Center-Of-Pressure Trajectory


[Procedures for extracting aligned footprint]Step 1. Labeling of each blobStep 2. Finding COA points of each blobStep 3. Determine first foot / second footStep 4. Check whether current time is the ending time of first foot

or the start time of second footStep 5. Updating overlapped footprint imageStep 6. Updating COP trajectory of each footStep 7. Creation of overlapped footprint imageStep 8. Finding the rotation angle using the principal axes of

overlapped footprint imageStep 9. Creation of aligned COP trajectory

Center-Of-Pressure Trajectory

Person Recognition Method viaCOP Trajectory


Characteristics of COP trajectory

• Varying length time-series data with noise (by quantization)

• Different COP trajectories between Left and right feet

• Different curvatures as users

Hidden Markov Model

Separate model for each foot

Direction-based quantization(Relative position-based quantization)



Period of second foot COP trajectory

Period of first foot COP trajectory

COPtrajectory

Relative position-basedquantization

HMM-basedfootprint recognizer


[User Identification Method]

Person Recognition Method via COPT


• Element of HMM– Set of hidden states: S={S1,S2,…,SN}

where N is the number of states in the model

– State transition probability: A={aij} for 1 ≤ i,j ≤ Nwhere aij=P[qt+1=Sj | qt=Si], 0 ≤ aij, ∑aij=1

– Set of observation symbols: V={v1,v2,…,vM}where M is the number of observation symbols per state

– Observation symbol probability: B={bj(k)} for 1≤j≤ N, 1≤k≤ Mwhere bj(k) = P[vk at t | qt=Si], 0 ≤ bj(k), ∑ bj(k)=1

– Initial state probability: ∏={∏i}, 1 ≤ i ≤ NWhere ∏i=P[q1=Si], 0 ≤ ∏i, ∑ ∏i =1

[1] X. Li, M. Parizeau, and R. Plamondon, “Training Hidden Markov Models with Multiple Observations-A Combinatorial Method”,IEEE Tr. On PAMI, Vol.22, No.4, Apr. 2000

Registration of HMMs: Baum-Welch Algorithm


[REF]


∑∈

++

++

+

=

=

===

FSlT

ttjijt

ttjijt

ttt

ljObai

OpjObai

Ojsispji

)()()()(

)()()()(

),,(),(

11

11

1

αβα

λβα

λγ

∑

∑

∑∑

∑−

=

−

=−

=

−

= == 1

1

1

11

1

1

1

)(

),(

),(

),(ˆ T

tt

T

tt

T

tj t

T

tt

ij

i

ji

ji

jia

γ

γ

γ

γ)(ˆ 1 ii γπ =

ελλ ≤− )()ˆ( OpOp

∑∈

=

==

FSlT

tt

tt

liiOispi

)()()(

),()(

αβα

λγ

∑

∑

=

=∈= T

tt

vOtt

i

i

ikb kt

1)(

)()(ˆ

γ

γ

Registration of HMMs: Baum-Welch Algorithm


[REF]

[REF] C. M. Bishop, Neural Networks for Pattern Recognition, Oxford University Press Inc., 1995

),,( πλ BA=1) Initialization of HMMs : Equal prob. Dist.2) Calculation of prob. at each node (state)

3) Re-estimation of model parameters

4) If , stop. Else, continue to step 2.

∑=

=N

iT iOp

1)()( αλObservation evaluation


User Identification Method

1( ) ( )T Tnew old oldw w wλ ε−= − Ζ Ζ + Ι Ζ

2 21 ( ) ( )2 old new old new oldE w w w w wε λ= + Ζ − + −%

Error function :

Weight-updating rule :

( )n

n thni

i

where is the error for n patternwε ε∂Ζ ≡

∂

λSmall : Newton methodLarge : Gradient descent method

Used : Adaptive updating with initial value 0.01

λλ

[REF] C. M. Bishop, Neural Networks for Pattern Recognition, Oxford University Press Inc., 1995

[REF]


iλ)(ii OpP λ=Given the prob. from U user’s HMMs ,i=1,…,2U (O: quantized COPT)

jj

SmaxargUser ID = , Deg. of Similarity = *j

LM(Levenberg-Marquart) Learning Method

)(i

iijj OpwS λ∑= *jS


4.502.0550.0020.4520

6.502.5070.0025.0018

6.503.0970.0030.9116

5.503.0975.0030.9114

6.003.2755.0032.7312

9.504.3665.0043.6410

12.005.1480.0051.368

20.506.0085.0060.006

58.009.27100.092.734

23.509.32100.093.182

Max. (%)Average (%)Max. (%)Average (%)

FARFRR# of learning dataAs time goes on,

User identification via COP trajectoryCenter-Of-Pressure Trajectory


Comparison of foot shape and COPT

1 2 3 4 5 6 7 8 9 10 11

0

2

4

6

8

10

12

14

16

18

20

user ID

diss

imila

rity

(pix

els)


Min. FRR error = 8.64 %Changing range in dissimilarity = 3 pixels

(Sensitive to noise)

Min. FRR error = 20.45 %

Result by overlapped footprint shape Result by COP trajectory

Combination of two independent information

Information fusion


Information Fusion

Feature Extraction User Identification

WalkingFootprint

OverlappedFootprint

ShapeExtraction

Quantized COPT

Extraction

Footprint ShapeMatching(Type-II)

RegisteredFootprintTemplates

Degreesof ShapeSimilarity

First Foot’sCOPT Matching

RegisteredHMMs

Second Foot’sCOPT Matching

Degreesof COPTSimilarity

weightedsum

COPT: COP Trajectory

UserID

weightedSum

RegisteredWeight

RegisteredWeight

Information fusion


Unified algorithm for the extraction ofaligned overlapped foot shape and COP trajectory

[Procedures for extracting aligned footprint]Step 1. Labeling of each blobStep 2. Finding COA points of each blobStep 3. Determine first foot / second footStep 4. Check whether current time is the ending time of first foot

or the start time of second footStep 5. Updating overlapped footprint imageStep 6. Updating COP trajectory of each footStep 7. Creation of overlapped footprint imageStep 8. Finding the rotation angle using the principal axes of

overlapped footprint imageStep 9. Creation of aligned footprintStep 10. Creation of aligned COP trajectory Common step

Step for foot shapeStep for COP

trajectory

Information fusion


Result of Information Fusion

0.141.36TOTAL

1.000.00USER 11

0.005.00USER 10

0.005.00USER 9

0.000.00USER 8

0.005.00USER 7

0.000.00USER 6

0.000.00USER 5

0.500.00USER 4

0.000.00USER 3

0.000.00USER 2

0.000.00USER 1

FAR (%)FRR (%)User ID

Result of the fusion of footprint shape info. & COP trajectory info.

20 40 60 80 100 120 140 160 180 200 2200

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

Each person’s test data

Out

put

Recognition result of a user

Static + Dynamic Footprints

Static Footprint

Dynamic Footprint

u1 u2 u3 u4 u5 u6 u7 u8 u9 u10u11

One sample from u1 (user 1)is compared with all other data.

COPT Foot ShapeFoot Shape + COPT

Information fusion


Walking RailSystem

- Example of Personalized Service

- Person Recognition byOverlapped Footprint Shape

- Greeting/Messaging Service

Automatic Door-Opening Service SystemApplication Example


Concluding Remark

1. Analysis of previous footprint-based user identification method

2. Performance enhancement of previous footprint-based user identification methodby using the overlapped footprint shape (from 86.55% to 91.36%)

3. Proposal of COP trajectory-based user identification (FRR error = 20.45%)

6. Proposal of multi-modal identification using both aligned overlapped foot shapeand aligned COP trajectory

8. Successful fusion of previous gait and footprint method (FRR error = 1.36%)


Thank youfor your attention