Simulating Bipedal Character Motions for Kinect Games

8/11/2019 Simulating Bipedal Character Motions for Kinect Games

http://slidepdf.com/reader/full/simulating-bipedal-character-motions-for-kinect-games 1/6

The SIJ Transactions on Computer Science Engineering & its Applications (CSEA), Vol. 2, No. 3, May 2014

ISSN: 2321-2381 © 2014 | Published by The Standard International Journals (The SIJ) 93

Abstract — Most Kinect games have only few reacting motions, and may not feel realistic enough for the

reactions, so we propose an interactive system to simulate real-time motions of bipedal characters for one-on-

one fighting games. The primary contribution of the proposed system is that we dynamically generate

believable reacting motions according to different fighting actions and hit parts of a body instead of solely

playing pre-captured or hand-crafted motions from an example motion dataset. First, Kinect skeletal tracking is

employed to capture a user’s fighting motion for attacking his \her game opponent. Second, we apply the principal component analysis to reduce the dimensionality of the fighting motions such that we can obtain a

feature vector for determining what a fighting pattern is and how to attack the game opponent. In addition, we

apply Mahalanobis distances to increase the accuracy of Kinect motions comparison. In the last step, a

proportional-derivative control is exerted to track the pre-prepared example motion for reacting properly when

the opponent is hit by the fighting pattern. Accordingly, a support vector machine is employed offline to

classify the example motions automatically. Experimental results show that our system is feasible and flexible

to provide dynamic reacting motions in fighting games.

Keywords — Character Animation; Feature Extraction; Motion Capture; Motion Classification; Skeletal

Tracking.

Abbreviations — Principal Component Analysis (PCA); Proportional-Derivative (PD); Self-Organizing Map

(SOM); Support Vector Machine (SVM).

I. INTRODUCTION

game is a type of playing activity; theoretically,

people like to play all types of games due to the

inherent human desire. According to the common

definition, playing and pretending are two essential elements

of play games. High reactions and good hand-eye

coordination are fundamental features that an action game

needs to possess. Fighting games are a type of action game

where game characters fight each other through a hand-to-

hand combat. Most design innovation in fighting gamesfocuses on developments in game characters’ actions and

reactions. In particular, the strategy to control attacking and

defending players is an interesting issue for most game

developers over the past few years. The straightforward

method is to employ a one-on-one strategy in which two

opposite fighters take their turns to be attacking and

defending actions for taking place several rounds.

In this paper, we devise a novel framework and make use

of the proposed framework to implement a 3D one-on-one

fighting game with Kinect. Kinect skeletal tracking, principal

component analysis (PCA), support vector machine (SVM),

and Proportional-Derivative (PD) control are exerted to build

the game framework. Regarding the conventional motion

sensing games, they take advantage of Kinect device to track

a real player’s skeleton for directly animating the captured

skeletal motions on the screen without accomplishing motion

recognition. On the other hand, even though Kinect is capable

of tracking two humanoid skeletons simultaneously, it will be

an amazing phenomenon to see that two players fight each

other in real-time while directly using Kinect to design a

fighting game without any design adaptation. Moreover, most

fighting games adopt pre-captured motion data to be served

as the reacting motions of a player. The reacting motions of

the character cannot vary with the attacking power and the hit

part of the body due to the fact that game developers could

not prepare all possible reacting motions of a character while

designing a game. Obviously, we can see that all reacting

motions of the characters are restricted no matter how a

player attacks his\her opponent.

Kinect is an easy-to-use motion sensing device with the

affordable price. In addition to adopt Kinect to track a

A

*Associate Professor, Department of Computer Science and Information Engineering, National Chiayi University, Chiayi, TAIWAN.

E-Mail: tclu{at}mail{dot}ncyu{dot}edu{dot}tw

**Graduate Student, Department of Computer Science and Information Engineering, National Chiayi University, Chiayi, TAIWAN.

***Graduate Student, Department of Computer Science and Information Engineering, National Chiayi University, Chiayi, TAIWAN.

****Ph.D. Student, Department of Computer Science and Information Engineering, National Chiayi University, Chiayi, TAIWAN.

Tainchi Lu*, Yuchen Chen**, Jiayi Li*** & Minchih Tsai****

Simulating Bipedal Character Motions

for Kinect Games





player’s attacking motion, we designate a standard attacking

motion offline and then use PCA to extract a feature vector

from the attacking motion data for comparing with each other

in real time. Consequently, we can confirm whether the

attacking motion is valid or not at a time. This is the

difference between the proposed system and the conventional

interactive games with Kinect. For the purpose of providing

dynamic and believable reacting motions, we do not just

animate the attacked character by using the designated pre-captured reacting motions. Automatic motion classification

works are essential. SVM is adopted to classify the numerous

pre-captured reacting motions that are captured by utilizing

the motion capture system into different motion categories.

The motion classification is an automatic process without any

user tedious intervention. Therefore, the proposed system can

provide a one-to-one motion correspondence with the

opponent’s attack. This is another innovation of the proposed

system. After confirming which type of the attack is and how

to hit the attacked character, we tackle the physics-based

reacting problem by applying a PD control to joints of a

character. Relying on the PD control, the attacked character

can physically react to the attacking perturbation and

gradually recovery from the perturbation by tracking the

designated reference motion. The main contribution of the

proposed system is to combine kinematics with physics

models to simulate the natural-looking reacting motions.

The remainder of the paper is organized as follows.

Section 2 addresses the related studies about Kinect, PCA,

SVM, and PD control servo. In Section 3, we present the

adopted technologies in detail. Experimental results are

shown in Section 4 for demonstrating the effectiveness of the

proposed method. Section 5 concludes the paper and

mentions some future directions.

II. RELATED WORKS

With the rapid growth of entertainment technologies, a wide

variety of wearable, wired or wireless controllers have been

developed to be worked well for facilitating the game play.

Kinect is a motion sensing input device that is developed by

Microsoft for Xbox 360 game consoles. Game users need not

to hold any body control device for interacting with a game;

in contrast, they simply use their body motions and voice to

control the game. Kinect comprises 3D depth sensors, RGB

cameras, and multi-array microphones that can provide the

full-body motion capture and voice recognition capabilities.

In this paper, we take advantage of Kinect to capture depthand skeleton information of a bipedal character.

Principal Component Analysis (PCA) is an effective

technique to perform dimensionality reduction [Wheatland et

al., 2013]. Lim et al., (2005) proposed a framework for robot

movement coordination and learning, and they adopted PCA

to extract the joint trajectory basis functions from human

motion capture data. Tang et al., (2006) presented a method

to simulate interactive falling behaviors and they used PCA

on captured motions to reduce the joint dimensionality in

distance calculation.

Support Vector Machine (SVM) is a part of machine

learning algorithm and it is a useful technique to quickly

classify online the physical motions for various purposes.

SVM needs a set of training data to build a model for

predicting the expected class of motion data. Zordan et al.,

(2007) proposed an approach that computes fast dynamic

response to unanticipated interactions and they used SVM for

data classification by selecting among a set of possible

reaction strategies.Self-Organizing Map (SOM) is a type of Artificial

Neural Network (ANN) that is trained using unsupervised

learning to produce a low-dimensional (typically two-

dimensional), and this makes SOM useful for visualizing

low-dimensional views of high-dimensional data, and

convenient for observing and analyzing data.

Proportional-Derivative (PD) control has been used

successfully for the purpose of tracking reference motions

stored in a database while simulating characters’ current

motions in different environments [Yin et al., 2007]; in

addition, a PD controller has been also applied to work on

characters with disturbances for keeping them in balance

[Abe & Popović, 2006; Sok et al., 2007]. Conventionally, this

controller is used to employ in robotics, and Zordan &

Hodgins (1999) further took the controller into account to

properly implement it in the field of computer graphics and

animation. Now most of the studies about response motions

take good advantage of a PD controller with an optimization

method to search for target motions in the next time step [Yin

et al., 2003; Abe et al., 2007; Silva et al., 2008]. In particular,

using a PD controller needs to specify exact parameters that

could be manually configured by using biometrics or

experimental experience.

III.

SYSTEM OVERVIEW

We show the block diagram of the proposed system in figure

1 and then give an overview for each system component in

this subsection. In the part of Kinect skeletal tracking, a game

player poses the body and waves the hands to pretend to fight

his\her opponent; meanwhile, Kinect will track and extract

his\her continuous motion data. PCA is adopted to

accomplish the feature extraction such that we can obtain a

feature vector to stand for the high-dimensional motion data.

A standard fighting pose has been defined in advance to

compare with the tracked motion data. As a result, we can

confirm that the fighting action is a valid attacking motion or

not. Concerning the part of the reference motionclassification, the reference motion data are captured by using

a motion capture system from a real actor and are stored in a

motion dataset. The reference motion data will be

automatically segmented into different motion categories by

means of applying SVM. After we verify the attack invoking

by the player’s opponent is a valid fighting action, the motion

correspondence will be carried out to pick up the most

correlative pre-captured motion to serve as the target

reference motion. In the case of a successful hit, a PD control

will be employed to track the target reference motion for





properly reacting; otherwise, the player will be still animated

by using the original reference motion, such as idle or

defensive motion. Because the system is a one-on-one

fighting game, the player gets his\her turn to attack his\her

opponent in the next round.

Figure 1: The Block Diagram of the Proposed System

IV. SKELETAL TRACKING AND FEATURE

E XTRACTION

In this subsection, we first describe how to use Kinect to

accomplish a skeletal tracking and then apply PCA to obtain

a feature extraction. As shown in Figure 2, we specify a

skeletal model of a character to be corresponded to the

default skeleton format defined by Kinect. First, image,

depth, and skeleton data streams of a game player are

continuously captured by using Kinect where the frame rate

was often set to a value of 30 frames per second. In addition,

a standard attacking motion will be specified in advance in

order to compare with the on-line captured motion. As a

result, the game player will be demanded to imitate the

specified attacking motion; for example, the game player lifts

up his\her hands with his\her face facing upwards and he\she

puts down his\her hands as if throwing an energy ball. After

calculating standard deviations between the standard motion

and the captured motion, we can allow a valid attacking

motion of the game player to fight his\her opponent in terms

of the different orientation, strength, velocity, and

acceleration of an attacking action.

Figure 2: A Skeletal Structure of a Character

The second step is to apply PCA to extract a low-

dimensional feature vector from the on-line captured motion

to avoid high-dimensional data computation and to further

verify the motion accuracy. We refer to Smith (2002) to build

a PCA model that is applied to accomplish the dimensionality

reduction for the sake of extracting a motion feature vector.

Suppose that there is a configuration vector X of thecharacter T

n x x x X ),,,( 21 . The mean vector X is

calculated and we must subtract the mean from each of the

data dimension. The formula of calculating a variance in one

dimension is:

)1(

)(1

2

2

n

X X s

n

i i

(1)

The next step is to calculate the covariance matrix xC

between i Dim and j Dim . It is defined as:

)),cov(,( ,, ji ji ji x Dim DimccC ,

)1(

))((),cov( 1

n

X X X X Dim Dim

n

i j jii

ji

(2)

where jic , represents the covariance between i X and j X .

After obtaining the covariance matrix, we calculate the

eigenvectors and eigenvalues of the covariance matrix.

vv A ,

0)( I A Det 0 ,(3)

where is an eigenvalue of a square matrix. From the

eigenvectors and eigenvalues, we can choose principal

components and form a feature vector to stand for the original

high-dimensional motion data with or without leading to lossof some information.

, = − −1 − (4)

, = − Σ−1 −

=1 (5)

Based on Lu et al., (2014), our new dimension space is

composed of eigenvalues and eigenvectors. In comparison

with the above study, we use Mahalanobis distances in (4) to

determine the similarity between two covariance matrices,

where x is the observation, c is the sample data and Σ is the





covariance matrix of the sample data, and then we sum up

each observation values in (5), to determine the dissimilarity

of the sample data, where there are k observations.

V. REACTION DETERMINATION AND

MOTION TRACKING

In order to pursue approaching trajectories from referencemotion data in real time, a PD controller was applied here to

yield appropriate joint torque on a character. In each time

step, we generate extra torque by means of a PD controller to

be as closely to the example motion as possible. An

illustration of applying an external force to a rigid body is

shown in figure 3. The formula of a PD control servo is

defined as:

= − + − (6)

In (6), and are the proportional gain and derivative

gain, and are the joint angles and joint velocities, and

are the desired joint angles and joint velocities, and is

the joint control torque. The differences,

− and

− ,

will give the tracking directions and magnitudes from thecurrent poses to the desired poses. An illustration of applying

an external force to a rigid body is shown in Figure 3. The

gains will control the weights for the produced torque.

Concerning an articulated rigid body, we need further take

moment of inertia c I into account and the equation is defined

as:

= − − 2 − (7)

where c I is the total accumulated moment of inertia with

respect to all linked joints. In practice, we suppose that d K is

about pk 2 and then we add c I 2 to the second term of

(7).

Figure 3: An Illustration of Applying an External Force to a Rigid

Body

VI. IMPLEMENTATION AND RESULTS

In this section we describe how to systematically implement

the proposed system and show the experimental results to

verify the framework. We use Unity to serve as our game

development system and take advantage of Kinect to extract

game players’ attacking motions. With respect to example

motions, META GypsyGyro-18motion capture suit was used

to capture real motions from a real actor for building areacting motion data set. We make use of Libsvm [Chang &

Lin, 2011] and Scilab [Scilab Enterprises, 2013] to

accomplish off-line SVM classification and PCA verification.

On the other hand, singular value decomposition (SVD) is

employed to provide the on-line eigenvalues and eigenvectors

calculation for dynamically obtain a feature vector after

capturing a player’s attacking motion. Figure 4 shows the

result the SOM weighting position of a 418 * 57 matrix

sample data, which includes three motions for this type of

motion. Figure 5 shows the sample hits of the data.

While adopting Kinect to track the skeletal motions of

the game player, we initially set the frame rate to a value of

30 frames per second and capture the motions for twoseconds. Consequently, there are total 60 frames within the

captured motion. We then sample the 60-frames motion every

four frames and acquire 15 samples. In addition, our scheme

is to ignore some of the joints that have less importance in

motion tracking and the number of joint dimensions which is

from 46 to 64. After applying PCA, we reduce the number of

joint dimensionality from 46 to 4 and 64 to 5 dimensions with

about 5% of data loss for our motion information. Table 1

lists the parameters used in the tracking examples. Figure 6

shows the result of performing PCA, and we choose the four

highest eigenvalues for the left figure, and five highest

eigenvalues for the right figure to standard for the originalcaptured motion. A screenshot of the proposed game system

is shown in figure 7. One game player tries to attack another

player through an energy ball.

Table 1: Parameters used in the Tracking Examples

Body Head Left Upper Arm Left Fore Arm Left Hand Right Upper Arm Right Fore Arm Right Hand Spine

K p 150 300 200 50 300 200 50 3000

K d 15 30 20 5 30 20 5 300





Figure 4: Weight Positions of SOM Results

Figure 5: Sample Hits of the SOM Results

Figure 6: The Result of Performing PCA for Obtaining the Feature

Vector

Figure 7: A Screenshot of the Proposed Game System. One Game

Player Tries to Attack another Player through an Energy Ball

VII. CONCLUSIONS AND FUTURE WORK

In this paper, we have presented a method that adopts Kinect,

PCA, SVM, and PD control to simulate real-time reacting

motions for one-on-one fighting game. The key insight behind our approach is that we adopt physical-based model to

incorporate with kinematics-based model. However, we did

not come up with optimization criteria but give an easy-to-

implement and practical solution for Kinect games. The idea

that we give is very intuitive and efficient; that is, PCA was

used to reduce motion dimensionality for obtain a low-

dimensional feature vector, SVM was applied to perform

motion classification automatically and accurately, and PD

control was employed to track an example motion by yielding

appropriate joint torque on a character.

For future research, we would like to explore more

attacking motions to provide combo attacks and define a

strategy to try to efficiently defend an opponent’s fightingaction. Moreover, PD servos typically produce unstable

control forces, and we will aim to use more stable PD

controllers to replace the current one.

ACKNOWLEDGMENT

This work is supported in part by the National Science

Council at Taiwan, R.O.C., under the project grant numbers

NSC102-2221-E-415-021.





REFERENCES

[1] V.B. Zordan & J.K. Hodgins (1999), “Tracking and Modifying

Upper-Body Human Motion Data with Dynamic Simulation”,

Proceedings of the Computer Animation and Simulation, Pp.

13 – 22.

[2] I.L. Smith (2002), “A Tutorial on Principal Component

Analysis”, Retrieved Nov. 2013, from

http://www.cs.otago.ac.nz/cosc453/student_tutorials/principal_ components.pdf.

[3] K.K. Yin, M.B. Cline & D.K. Pai (2003), “Motion Perturbation

based on Simple Neuromotor Control Models”, Proceedings of

the 11th Pacific Conference on Computer Graphics and

Applications, Pp. 445 – 449.

[4] B. Lim, S. Ra & F.C. Park (2005), “Movement Primitives,

Principal Component Analysis, and the Efficient Generation of

Natural Motions”, Proceedings of the 2005 IEEE International

Conference Robotics and Automation, Pp. 4630 – 4635.

[5] N. Tang, Z. Pan, L. Zheng & M. Zhang (2006), “Interactive

Generation of Falling Motions”, Journal of Visualization and

Computer Animation, Vol. 17, No. 3 – 4, Pp. 271 – 279.

[6]

Y. Abe & J. Popović (2006), “Interactive Animation ofDynamic Manipulation”, Proceedings of the 2006 ACM

SIGGRAPH/Eurographics Symposium on Computer Animation,

Pp. 195 – 204.

[7] V. Zordan, A. Macchietto, M. Soriano & C.C. Wu (2007),

“Interactive Dynamic Response for Games”, Proceedings ofthe 2007 ACM SIGGRAPH Symposium on Video Games, Pp.9 – 14.

[8] K.W. Sok, M. Kim & J. Lee (2007), “Simulating Biped

Behaviors from Human Motion Data”, ACM Transactions on

Graphics, Vol. 26, No. 3.

[9] K.K. Yin, K. Loken & M.v.d. Panne (2007), “SIMBICON:

Simple Biped Locomotion Control”, ACM Transactions on

Graphics, Vol. 26, No. 3, Pp. 1 – 10.

[10] Y. Abe, M.d. Silva & J. Popović (2007), “Multiobjective

Control with Frictional Contacts”, Proceedings of the 2007

ACM SIGGRAPH/Eurographics Symposium on Computer

Animation, Pp. 249 – 258.

[11] M.d. Silva, Y. Abe & J. Popović (2008), “Interactive

Simulation of Stylized Human Locomotion”, ACM

Transactions on Graphics, Vol. 27, No. 3, Pp. 1 – 10.

[12] C.C. Chang & C.J. Lin (2011), “A Library for Support Vector

Machines”, ACM Transactions on Intelligent Systems and

Technology, Vol. 2, No. 3.

[13] Scilab Enterprises (2013), “Scilab for Very Beginners”,

Retrieved Nov. 2013, from

http://www.scilab.org/content/view/full/849.

[14] N. Wheatland, S. Jörg & V. Zordan (2013), “Automatic Hand-

Over Animation using Principle Component Analysis” ,

Proceedings of the Motion on Games, Pp. 175 – 180.

[15] T.C. Lu, Y.C. Chen, C.Y. Li & M.C. Tsai (2014), “Real-time

Reacting Motion Simulation for One-on-One Fighting Games”,

Proceedings of 2014 Annual Conference on Engineering&Information Technology (ACEAIT).

Tainchi Lu received his Ph.D. degree incomputer science and engineering from

National Sun Yat-sen University, Kaohsiung,Taiwan, R.O.C., in 1999. Since 2006, he has

been an associate professor in the Dept. ofComputer Science and Information

Engineering at National Chiayi University,Chiayi, Taiwan, R.O.C. His current researchinterests are in the areas of computer graphics,

computer animation, and interactive multimedia.

Yuchen Chen received his B.S. degree from

National Chiayi University (NCYU) of

computer science of information engineering.He is a graduate student in computer scienceof information engineering at NCYU. Hisresearch interest includes motion clusteringand pattern recognition.

Jiayi Li received his B.S. degree from I-Shou

University of computer science of informationengineering. Now, he is a graduate student incomputer science of information engineering

at National Chiayi University. His researchinterest includes human and machineinteraction and motion data analysis.

Minchih Tsai received his master ’s degreefrom National Chiayi University (NCYU) ofcomputer science of information engineering.He is a Ph.D. student in computer science ofinformation engineering at NCYU. His

research includes computer graphics and physics-based character animation.

Documents

Simulating Bipedal Character Motions for Kinect Games