Embed Size (px)
Citation preview
Multi Hand Pose Recognition System using Kinect Depth Sensor
O. Lopes1, M. Pousa1, S. Escalera2 and J. Gonzalez3
Computer Vision Center, Campus UAB, Edifici O, 08193, Bellaterra, Spain1{oscar.pino.lopes, miguelpousafreire}@gmail.com, [email protected], [email protected]
Abstract
Hand pose recognition is a hard problem due to theinherent structural complexity of the hand that canshow a great variety of dynamic configurations andself occlusions. This work presents a hand pose recog-nition pipeline that takes advantage of RGB-Depthdata stream, including hand detection and segmen-tation, hand point cloud description using the novelSpherical Blurred Shape Model (SBSM) descriptor,and hand classification using OvO Support VectorMachines. We have recorded a hand pose dataset ofmultiple hand poses, and show the high performanceand fast computation of the proposed methodology.
1 Introduction
In recent years a great interest and progress inHuman-Computer Interaction (HCI) have been per-formed with the objective of improving the overalluser experience [1].
When considering vision-based interfaces and in-teraction tools, user hand gesture interaction opensa wide range of possibilities for innovative applica-tions. It provides a natural and intuitive languageparadigm to interact with computer virtual objectsthat are inspired in how humans interact with real-world objects. However hand pose recognition is anextremely difficult problem due to the highly dynamicrange of configurations the hand can show, and to theassociated limitation of traditional approaches basedsolely on RGB data.
More recently with the appearance of affordabledepth sensors such as KinectTM, a new spectrumof approaches were opened. Kinect provided a newsource of information that dealt closely with the 3Dnature of the objects of interest. This way it is nowfeasible to extract 3D information from the hand poseusing a gloveless input source and, from then on, anal-
yse and estimate the observed pose. This proposalcomprises a system for hand pose recognition froma live feed from the Kinect device. This is essen-tially a classification problem, however the overall re-sults are boosted due to the novel Spherical BlurredShape Model descriptor created specifically for thistask. This descriptor both fullfills temporal complex-ity requirements and a high discriminative power thatcan take full advantage from the depth information.
2 Live Hand Pose RecognitionPipeline
The proposed hand pose recognition pipeline (Fig-ure 1) comprises the following main components:
• Hand pose detection and segmentation module,that consists in a two hand scale-invariant blobdetector.
• Rotation invariant Spherical Blurred ShapeModel descriptor, that performs a point cloudpartitioning that harnesses the highly discrimi-native information provided by the depth data.
• Hand pose classification based on One-versus-One Support Vector Machines (SVM), that wereintegrated in the pipeline in order to perform on-line classification of the live data stream.
3 Hand Pose Dataset
The system’s overall performance, accuracy wise, de-pends heavily on its correct training with a represen-tative hand pose dataset. Since currently there wasno point cloud hand pose dataset available, it wasrequired to craft one. This was performed using thehand pose detector modification to perform the data
1
Figure 1: System architecture.
persistence. The dataset comprises 6 classes, 2000samples each, including both hands.
Figure 2: Hand pose dataset.
In order to improve the system robustness, andsince the problem is treated as a multi-classificationtask, a no pose class was added, which includes sam-ples captured by the device referring to non-hand el-ements.
4 Descriptor
The description of the 3D hand is performed usingthe Spherical Blurred Shape Model (SBSM) descrip-tor (Figure 3). The SBSM descriptor is based onits 2D counterpart the CSBM descriptor [2]. SBSMdecompose a 3D shape with an spherical quantiza-tion and perform an assignment based on the distancefrom the points to the bins centroids. The result is afeature vector that codifies the input shape in a com-pact manner and has a high discriminative power.
Figure 3: SBSM descriptor.
5 Classification
The classification step is based on SVM. The prob-lem is tackled as a multi-classification task, consid-ering C-SVM using the RBF kernel, combined witha descriptor configured with 257 features, that ac-cordingly to the experimental analysis offers the bestbalance between performance and final results relia-bility.
6 Demonstration
The pipeline is demonstrated by visualizing a livefeed from the Kinect device, where a single user ex-poses his hands and the system applies a on-screenlabel accordingly to the visualized hand pose (see Fig-ure 4). This initial prototype could be included in amore rich HCI environment, using both the hand poseand its spacial location, in order to execute commandand control actions, object manipulation, or interfacenavigation.
Figure 4: Prototype of the system.
References
[1] F. Periverzov and H. Ilie, “3d imag-ing for hand gesture recognition: Explor-ing the software-hardware interaction of cur-rent technologies,” 3D Research, vol. 3,pp. 1–15, 2012. [Online]. Available: http://dx.doi.org/10.1007/3DRes.03%282012%291
[2] S. Escalera, A. Fornes ands, O. Pujol,J. Llado ands, and P. Radeva, “Circularblurred shape model for multiclass symbol recog-nition,” Systems, Man, and Cybernetics, PartB: Cybernetics, IEEE Transactions on, vol. 41,no. 2, pp. 497 –506, april 2011.
2
