Accelerometer-based User Interfaces forthe Control of a Physically Simulated Character

Takaaki ShiratoriJessica K. Hodgins

Carnegie Mellon University

Physical Simulation in Games

Controllable interface for physically simulated character

Jurassic Park: Trespasser, 19982

Simulation for everything.- Very limited.

Little Big Planet, 20081

Simulation ONLY for passive objects.- Effective!

1 http://www.littlebigplanet.com/2 http://www.mobygames.com/game/trespasser-jurassic-park

Half speed

Physically Simulated Character• Natural-looking motion.• Natural responses to environment/disturbance.

• Natural-looking motion.• Natural responses to environment/disturbance.• Hard to control due to delayed response (anticipation).

Physically Simulated Character

Walk Jump

– Our Hypothesis –Performing similar actions might make the delay seem intuitive.

0 1.0 2.0 3.0 0

1

2

3

4

5

Speed [m/s]

Time [sec]

Walk Jump

Anticipation

Our Approach• Performance interface:

User Imitates character’s motion with Wiimotes.

–Wrist interface–Arm interface–Leg interface

• Test our hypothesis about delay with user study.Leg interface

• Accelerometer• IR sensor• $40 / unit• 30 million copies

WiimoteTM

1 http://www.gizmag.com/go/6773/2 http://www.cs.cmu.edu/~johnny/projects/wii/

1 2

Related Work• Interface for controlling a character

– Physically simulated character in 2D.

• Virtual navigation– Control “view point”.

• Wii games– Either dynamic or static measurement.

[Johnson et al. 1999]

[van de Panne and Lee 2003]

[Slyper and Hodgins 2008]

[Chai and Hodgins 2005]

[Templeman et al. 1999][Razzaque et al. 2002][Nintendo 2006] [Nintendo 2007] [Ubisoft 2007]

Overview

2-3 Wiimotes

Physically simulated motion

Mapping Controllerselection

Parameterchange

Amp., Mean,Inclination

Moving or not,Freq., Phase

Motion controller(walk, run, jump, step)

Physical Simulation

Overview

2-3 Wiimotes

Physically simulated motion

Mapping Controllerselection

Parameterchange

Amp., Mean,Inclination

Moving or not,Freq., Phase

Motion controller(walk, run, jump, step)

Physical Simulation

• Focus on periodicity of character’s motion.

User Input

Walking Running Jumping

Basic command: swing Wiimotes

In-phaseOut-of-phase

Acceleration Analysis

FeaturesFeatures

YesNo

Raw acceleration

Frequency

Amplitude

Inclination Phase diff.L

R

Mean

Kalman filter

Moving?

Variance

Frequency

• Auto-correlation function

a: acceleration dataT: current timetp: window size

Acceleration Analysis

FeaturesFeatures

YesNo

Raw acceleration

Frequency

Amplitude

Inclination Phase diff.L

R

Mean

Kalman filter

Moving?

Phase Difference

• Cross-correlation function

a: acceleration dataa: mean accelerationT: current timetp: window size

Acceleration Analysis

FeaturesFeatures

YesNo

Raw acceleration

Frequency

Amplitude

Inclination Phase diff.L

R

Mean

Kalman filter

Moving?

Inclination Estimation• If Wiimote is not moving,

• If Wiimote is moving, Wiimote’s local coordinate.

xy

z

g

t

acc.

x

acc.

x t

mean

Overview

2-3 Wiimotes

Physically simulated motion

Mapping Controllerselection

Parameterchange

Amp., Mean,Inclination

Moving or not,Freq., Phase

Hopper Model

Hip: Ball joint (3 DoFs)Knee: Slider joint (1 DoF)

Simulation Rendering

Basic Motions

Stepping in place

(Stopping)

RunningWalking Jumping

• Consists of 3 contact states with Proportional-Derivative (PD) controller.

Support foot passes

under hip.

Rear legleaves ground.

Swing legcontacts ground.

Fall Rise

DoubleSupport

Walking Controller

[Raibert and Hodgins, 1991]

Stepping-in-place controller: target velocity = 0

• Consists of 4 contact states with PD controller.

Running Controller

Front swing legcontacts ground.

Support foot passes

under hip.

Support legleaves ground.

Vertical hipvelocity

less than zero. Upward

FlightDownward

Flight

Extension Compression

[Raibert and Hodgins, 1991]Jumping controller: both legs in phase.

• Based on the robustness of motion.

Gait Transition

JumpingStepping in place

Walking RunningWalking RunningSlow Fast

Overview

2-3 Wiimotes

Physically simulated motion

Amp., Mean,Inclination

Moving or not,Freq., Phase

Motion controller(walk, run, jump, step)

Physical Simulation

Mapping Wiimotes to Physical Simulation

Wiimotesnot moving.

Wiimotesout of phase.

Wiimotesin phase.

Height: Wiimote amplitude

JumpingStepping in place

Walking RunningWalking RunningSlow FastWii frequency

Wrist Interface• Imitate character’s leg motions with user’s wrists.

Wrist Interface

• Imitate character’s leg motions with user’s wrists.

Walking(Slow swing)

Running(Fast swing)

Jumping Turning

Arm Interface• Imitate arm motion of human’s biped motion.

(though the character doesn’t have upper body)

Arm Interface• Imitate arm motion of human’s biped motion.

(though the character doesn’t have upper body)

Walking Running Jumping Turning

Leg Interface• Imitate character’s leg motion with user’s legs.

Leg Interface

• Imitate character’s leg motion with user’s legs.

Walking(Slow step)

Running(Fast step)

Jumping Turning

• Typical usage.

Joystick Interface

RightLeft

Forward

Run

Walk

Turn in place

Step in place

Locomotion Jumping

User Study• 15 subjects.• Tasks

– Straight track completion.– Test track completion.

• Questionnaire– Fun, ease of use, stress, familiarity, immersion, how much

they liked it?– Free-form questions.

Courses:

• Task1. Motion transition at line.2. Keep straight walking/running until line.

Straight Track Completion

Average failure count

Straight walking: All of our interfaces < Joystick

Failure for Straight Walking

Approximate motions

RightLeftForward

Run

WalkTurn in placeStep in place

Precise manipulation

Wrist Arm Leg Joystick

Straight walk

Test Track Completion

Simulation failure Jump failure Curve failure

+Time to completion

Time to failure

Result of Test Track CompletionAverage count Time [sec]

Curve failure: Leg interface < joystick

Our interfaces are easier to control than joystick.

Result of Questionnaire

“Fun” “Ease of use” “Stress” “Familiarity” “Immersion” “Like”

Score

Free-form questions:• Most subjects did not complain about the delay.• A few subjects complained about the delay of all

interfaces (including joystick).

Questionnaire

Rating score:• “Immersion”: Wrist, Leg > Joystick• “Like”: Wrist, Leg > Joystick

Insights from User Study• Delay factors:

– Acceleration analysis: 100 – 500 ms (Not included in joystick interface)

– Physical simulation (anticipation): 200 – 500 ms

• Task completion:Easy to control: our interfaces > joystick.

• Questionnaire:“Immersive” and “Like”: our interfaces > joystick.

Immersive performance interface might help with the delay issue.

Competitive Game

ConclusionSummary• Performance interfaces for controlling a physically simulated

character.• User study mainly focusing on delay issues.

– Performance interface might be able to help with delay issue.

Future Work• Reduce delay.

– Improve acceleration analysis.• Further user study.

– Delay issue: Compare with data-driven control.– Other scenarios (e.g. fighting game, FPS).