Upload
others
View
5
Download
0
Embed Size (px)
Citation preview
Shape Recognitionfor Multi-Touch Table
Yehoraz Kasher Project Competition10/06/2012
Students: Itay Hubara
Amit Nishry
Supervisor: Yair Moshe
Introduction
System
Algorithm
Implementation
Demo Game
Conclusion
Introduction
System
Algorithm
Implementation
Demo Game
Conclusion
Outline
2
Background
• Multi-touch is starting to appear in every day
computers
• New User Interfaces (NUI) are extensively
pursued
3
Motivation
• Existing multi-touch surfaces are expensive
• Current technology is unable to recognize
shapes
• Shape recognition enhances user interface
4
Goals
• Design a shape recognition algorithm
• Track and recognize combined shapes
• Integrate into open-source environment
• Real-Time performance
5
Challenges
• Noisy input image
• Real-time constrains
• Implementing into open-source environment
• Creating a complete system solution
• Shape combinations and movement
6
• Real-world view• IR input
Challenges (cont.)
7
Outline
Introduction
System
Algorithm
Implementation
Demo Game
Conclusion8
System
• Hardwareo Multi-Touch Table
o FTIR Technology
• Softwareo Community Core Vision
o Client / Server Model
• Algorithmo Image Processing
o Shape Detection
o Tracking
• Applicationo Game
Hardware
9
Software
Algorithm
Application
FTIR - Frustrated Total Internal Reflection
Frustrated Total Internal Reflection
10
Multi-Touch Table
11
Outline
Introduction
System
Algorithm
Implementation
Demo Game
Conclusion12
Shape Recognition
Pre-Processing
VideoFeature
ExtractionClassification Tracking
LearningShape
ModelApplication
13
Learning Phase
On-line Phase
• Global thresholding leads to
inconsistent performance
• Each frame has a different
dynamic range
• Solution: Adaptive thresholding
Pre-Processing
Cropping
Stitching
Backgroundremoval
Noise filtering
Thresholding
Segmentation
14
Pre-processing (cont.)
IR input Background removed
15
Pre-processing (cont.)
Global threshold Adaptive threshold
16
Shape Recognition
Pre-Processing
VideoFeature
ExtractionClassification
LearningShape
Model
17
Contour RegionStructural Global
Feature Extraction
Shape
features
[Zhang & Lu, 2004] 18
Shape
Region
Global
moments area axis …
• Area
• Major & minor axis
• Orientation
• Eccentricity
• 6 Hu moments
• 7 complex moments
Candidate Features
19
Image Moments
• Complex moments:
• Moments are centralized for translation
invariance
( , ) - image indicator function
,p q
pq
x y
c x iy x iy x y dxdy
20
• In polar coordinates:
• Rotation by
• In general
• Which leads to
2
1
0 0
,i p qp q
pqc r e r drd
Rotation Invariance
i p q
pq pqc e c
0
0n
j j jjk p q
1
j
j j
Nk
p q
j
I c
21
• Seven rotation and translation invariant
moments
• Useful for visual pattern recognition
• Rotation invariance via complex moments
1 11
2 20 02
c
c c
[Hu, 1962][Flusser, Suk & Zitová, 2009]
Hu Moments
22
Shape Recognition
Pre-Processing
VideoFeature
ExtractionClassification
LearningShape
Model
23
Learning
Feature
selection
Shape
model
• Many features high computation
• Dimensionality reduction required
• Extract features at various locations
• Approx. 50 samples per shape
• Independent Gaussian model
24
Feature Selection• Exhaustive algorithm based on cross-validation
25
Features
First 10% of samples
Second 10% of samples
Tenth 10% of samples
Class
Third 10% of samples
Error 1
Error 2
Learning set
Test set
Test set
Learning set
Error 3
Error 10
Result: 1 2 10, , ,totale mean e e e
Feature Selection (cont.)
26
• Initialization o Set n=1 feature(s) out of a total of N features
• Stepo Calculate for all possible combinations
o Set n=n+1
• Stopping condition
o When
n
totaleN
n
1n n
total totale e
Feature Selection (cont.)
• Chosen features:o Area
o Major & minor axis
o Eccentricity
o 4 Hu moments:
• Empirical selection
• Similar to other shape recognition works
1 4 6 7, , ,
27[Hu, 1962][Sarfraz, 1993]
Shape Recognition
Pre-Processing
VideoFeature
ExtractionClassification
LearningShape
Model
28
Classification
• Independent Gaussian distributions
• Naïve Bayes Classifier
•
29
2
1
1
1
1
( | ) ,
( | ,..., ) ( ( | )
ˆ classify( , , ) argmax ( | ,...,
)
)j
j i
j
m
j j
i i i
m i
m
j
j m
i
s
s f
s
s
p f N
p f f p p f
s f f
s
p
s
f fs
For each shape and feature
assume :
Naïve Bayes Classifier
• Advantages:o Generic
o Fast
o Allows thresholding
o Good performance
• Disadvantages:o Long offline learning process
30
Shape Recognition
Pre-Processing
VideoFeature
ExtractionClassification Tracking
Application
31
Tracking
NN tracking
Combined
shapes
Internal
movement
False
positive
detection
• Nearest neighbors tracking
• Simplifies data handling
• Prevents false detections
• Allows complex situationso Shape combinations
o Internal movement in a combination
32
Complex Situations
• Shape combinations can appear as one
(legitimate) shape
• Possible classification mistakes:
33
Internal Shape Movement
• Complex shape modification
• Combined shape structure is maintained
34
Tracking
Segmentation
Classification
False Positive
Detection
Combined Shapes
Detection
Internal Movement Detection
Status Update
Blobs
Shapes
CombinedShapes
Blobs
Shapes
CombinedShapes
Current framePrevious frame
35
Video
36
Complex Situations
Shape Recognition
Pre-Processing
VideoFeature
ExtractionClassification Tracking
Application
37
Application
Client /
Server
Data
transfer
protocol
• Many possible applications
• Client / Server model established
• Community Core Vision
• Tangible User Interface Objects (TUIO) protocol for data transfer
38
Community Core Vision
• Open-source / cross-platform solution for
computer vision and machine sensing
• Supports multi-touch input
• Supports multi-camera feed
• Commonly used by developers
39
TUIO Protocol
• Open framework communication protocol and API for tangible multi-touch surfaces
• Supports various multi-touch informationo Location
o Orientation
o Acceleration
o …
• Commonly used with Community Core Vision
40
Outline
Introduction
System
Algorithm
Implementation
Demo Game
Conclusion41
Implementation
42
• Simulations (Matlab)
• Implemented into CCV (C++)o Adaptive thresholding
o Learning process
o Classification
o Shape tracking
o Import / export classifiers
o Additions to GUI
• Implemented into TUIO (C++ & C#)o Sending shape information
o Distinguishing between touch and shape
Graphical User Interface
43
Learning
44
Outline
Introduction
System
Algorithm
Implementation
Demo Game
Conclusion45
Tangram
• Chinese dissection puzzle with seven shapes
• The objective is to fill a given silhouette
• Demonstrates the algorithm capabilities
46
SIPL Tangram
47
Outline
Introduction
System
Algorithm
Implementation
Demo Game
Conclusion48
Demo
49
Performance Analysis
• Resourceso Intel i7 processor, 8 GB of memory
o Three IR cameras, 60 fps
• Real-Time performance
• Computing time < 2 miliseconds per frame
50
Summary
Multi-Touch table
Feature Extraction
Shape Classification
Tracking
Complex situations
CCV & TUIO
Tangram51
Shapes
Future Work
• Extending shape recognitiono Additional shapes
o Shape color
• Improving camera registration
• Cross-Platform implementations
• Finalize implementation of the demo game
52
Thanks
53
AviYairKobi
References
• D. Zhang and G. Lu, "Review of shape representation and
description techniques“, Pattern Recognition, vol. 37, pp. 1–19,
2004.
• M. K. Hu, "Visual Pattern Recognition by Moment Invariants“, IRE Trans. Information Theory, vol. 8, pp. 179–187, 1962.
• J. Flusser, T. Suk and B. Zitová, “Moments and Moment
Invariants”, John Wiley & Sons Ltd., 2009.
• C. M. Bishop, “Pattern Recognition and Machine Learning”,
Springer, 2007.
• "CCV, Community Core Vision“, [Online],
http://ccv.nuigroup.com/.
• "TUIO.org“, [Online], http://www.tuio.org/
54