Finger Gesture Recognition through Sweep Sensor

Finger Gesture Recognition through

Sweep SensorPong C Yuen1, W W Zou1, S B Zhang1, Kelvin K F Wong2 and Hoson H S

Lam2

1Department of Computer ScienceHong Kong Baptist University

2World Fair International Ltd

OutlineMotivations

Design Criteria

Proposed Method

Experimental results

Conclusions

MotivationsVision-based

interfaceInsert some images using face, expression, body movement…

Sensor-based interface

http://www.blogcdn.com/www.tuaw.com/media/2008/11/mac-101_-multi-touch-tips.jpg

http://www.fitbuff.com/wp-content/uploads/2007/10/wii-fitness.jpg

There should be a video about the body movement interface

Common objective: natural input to replace traditional physical input devices

MotivationsWhile many sensor-based gesture input have been developed, there is no algorithm/system using sweep sensor

Why Sweep Sensor?low costNo latency problem (fingerprint recognition)popularity

Design CriteriaUser friendliness

easily performed by a user. intuitive and easy to understand.

User independentGeneric for all users.

Robustnessdiversity of patterns captured.

Efficiency Real-time applicationmobile devices

Classification

t > t0

left rightNo

left tick right tick

Yes

feature vector

D > 0.5 left

right

D > 1.3 left tick

right tickD > 1/1.3

D < -0.5

Feature Extractionnoise

reduction

envelope enhanceme

ntinput image

direction estimation

direction index D = Dleft

/Dright

envelope

0 40 80 120 160 200

0

20

40

60

80

100

120

140

160

0 40 80 120 160 200

0

20

40

60

80

100

120

140

160

yi

i

y

0

0

miiright

miileft

yD

yD

SegmentationInput image

CharacteristicsFormulate the noise

Proposed MethodInput image

CharacteristicsFormulate the noise

Segmentation

Feature Extractionnoise

reduction

envelope enhanceme

ntinput image

direction estimation

direction index D = Dleft

/Dright

envelope

0 40 80 120 160 200

0

20

40

60

80

100

120

140

160

0 40 80 120 160 200

0

20

40

60

80

100

120

140

160

yi

i

y

0

0

miiright

miileft

yD

yD

Classification

t > t0

left rightNo

left tick right tick

Yes

feature vector

D > 0.5 left

right

D > 1.3 left tick

right tickD > 1/1.3

D < -0.5

Input image

CharacteristicsFormulate the noise

Segmentation

Feature Extractionnoise

reduction

envelope enhanceme

ntinput image

direction estimation

direction index D = Dleft

/Dright

envelope

0 40 80 120 160 200

0

20

40

60

80

100

120

140

160

0 40 80 120 160 200

0

20

40

60

80

100

120

140

160

yi

i

y

0

0

miiright

miileft

yD

yD

Classification

t > t0

left rightNo

left tick right tick

Yes

feature vector

D > 0.5 left

right

D > 1.3 left tick

right tickD > 1/1.3

D < -0.5

Input Image Characteristics

Different sensor characteristicsNoise level

Figure 2. The block diagram of feature extraction

SegmentationOwing to different sensor characteristics, the gesture images obtained, even the gesture is the same, will be different

Segmentation by estimating the sweeping time

noise reduction

vertical gradient thresholding horizontal

projection

Segmentation (cont.)

blank partsw

eeping part

noise reduction

vertical gradient

THthresholdin

g

horizontal projection

sth1t 't

] 0[on )( ), ))((1 ( 1)(0

2

1

1

ttEdttstcth ts

t

s

Feature ExtractionTime information t (sweeping time)

Finger motion information d (direction)Left and rightLeft diagonal and right diagonal

Feature Extraction (left / right)

noise reduction

Left

Right

direction

enhancement

input image

direction estimation

direction index D = Pleft - Pright

i-th fingerprint texture

A B

C D )(

)(

CBDAP

CBDAP

right

left

Feature Extraction (left tick / right tick)

noise reduction

envelope enhancement

input image

direction estimation

direction index D = Dleft /Dright

0 40 80 120 160 2000

20

40

60

80

100

120

140

160

0 40 80 120 160 2000

20

40

60

80

100

120

140

160

0

0

miiright

miileft

yD

yD

envelope

yi

i

y

ClassificationA very simple rule based on a combination of movement

Classification tree (decision tree)

left rightNo

left tick right tick

Yes

feature vector t > t0

D > 1.3 left tick

right tickD > 1/1.3

D > 0.5 left

rightD < -0.5

Designed GesturesLeft and Right Left tick and Right

tick

Experimental Results2 testing groups

3 technical users – Engineers, and technical managers, research staff (95.0%)3 Non-technical users – secretary, clerk (86.87%)

Test on different sensors4 different sensors manufacture at different period of time

Experimental ResultsEvaluation interface

There should be a video here

Experimental Results

Sensor1 Sensor2 Sensor3 Sensor40

102030405060708090

100

Results of fingerprint recognition using different sensors

success fail%

Results by 3 non-technical staff with 4 different sensors

Experimental ResultsIntegrated application with an image viewer

There should be a video here

Conclusions

Thank You