DIY Motion Capture

Kinect Motion Capture for Filmmaking3/23/13

Why DIY MoCap?Motion capture is now cheap and accessibleNo longer need a giant studioIts FAST.Ideal use cases: short-turnaround animation, rapid prototyping etc.1

How does the Kinect work?

Photo by y3rdua

Getting Started: HardwareYou will need:A Kinect, or generic equivalent such as the Asus XtionUSB/AC adapter cable, if you have a Kinect that was bundled with the Xbox (dont buy the Microsoft adapter, the cheap ones work fine)Enough space to perform

Getting Started: Software

OpenNI driver bundle:simple-openni.googlecode.com

KinectToPin:kinecttopin.com

Microsoft SDK vs. OpenNI4

Two competing SDKsOpenNIFree and open sourceWorks with all models of Kinect as well as other devices such as the Asus Xtion.Cross platform even supports ARM processors and Android

Microsoft (official) SDKMore tracking pointsEstimates occluded joints and hand/foot rotationSpecial case capture modes (seated figure etc.)Only works with Kinect For Windows on Windows

KinectToPinKinect Motion Capture for After Effects

6

What is KinectToPin?

A standalone motion capture applicationA UI panel for After Effects+

Rig 2D character models in 3D

Working BackwardsTraditional AnimationStart with scene already storyboarded and timed outCreate sketches of key poses for each characterAdd in-between drawings for movementColor and composite characters in final sceneMotion Capture WorkflowStart with full color, full-motion charactersRefine blocking and movement; add facial animationChoose shots and timing of edit, add camera moves(Optional) Re-record any performances that arent working

Working Backwards

Think of it like working with raw footage.

Character Design2D models in 3D space

Why multi-layered characters?Single-layer model:

Multi-layered:

Design Tips

Break your model at every joint.Build in overlap so joints have room to rotate without disconnecting.Parts that fall in front of and behind another layer (e.g. the hair of the model at left) should be two layers.The hands, feet and head can each have multiple elements.

CutoutsRaster-based characters drawn or cut out of existing images or photographs.

Work big if you want to be able to zoom in on your charactersWhen cutting out layers, use Refine Edge to smooth your edges and clean up any extra junk that got selectedClone/heal/paint in portions that will be missing or problematic when the model changes pose

Vector-Based CharactersDrawn with math, not pixels. Easier to break into layers but watch out.

Can be built in Illustrator, Photoshop or with AE Shape LayersVector-based characters are not infinitely scalable as youd expect: rigging with AEs Puppet Tool generates a mesh at a set resolution.Building puppets with vector layers adds an extra step: they need to be precomposed before rigging.

Motion CaptureRecording with KinectToPin

Setting up your space to record

6 10 feet

17

CapturingRecord 15-point 3D skeletal tracking dataCapture in sync to dialogueRemote control with a standard presentation remoteOpen source and cross-platform: runs on OpenNI, built in Processing, also works with generic sensors like the Asus Xtion

Acting for KinectThe Kinect cant record what it cant see. Keep your entire body in the cameras rangeFace the camera when capturingWatch out for occluded joints (sitting down, putting your hands behind your back etc. can cause it to lose the track)Its surprisingly good at picking up subtle posture shifts, but not smaller motions like laughter. Use a different performer for each character if possible

Data formats for exportKinectToPin can output quite a few data, but the ones well be using today are:XML (Native recording format, no export required)After Effects 3D Point Controls (CS5.5+)After Effects 2D Point Controls (CS4+)

Tip: Edit KinectToPins settings.txt file to choose your output formats.

Lunch.

Importing tracking dataNative import or copy & paste

Meet the KinectToPin UI Panel.Create character templates2D and 3D setupsNative XML import (SLOW)Automatically add expressions to smooth tracking dataAutomate rigging

The Puppet ToolTHREE OR MORE PINS ON ONE LAYER: Elbows dont bend this way. Ow.

TWO PINS EACH, MULTIPLE LAYERS:Rotation with stretch and squash!

AE Puppet best practicesWork BIG. Start high-res, scale down in the puppet comp, then scale back up in the project comp (enable Collapse Transformations).Precompose if youre using vectors or shape layersKeep multiple angles of the same character in syncEnd big.Start big.

Character RiggingUsing the KinectToPin UI panel

Retargeting for arbitrary character shapes

Can add multiple motion tracks and duplicate nullsPin names need to match control nulls

28

Its easy!Just grab a red control null and move it to the appropriate position.

Quick auto-lip syncSelect audio layer and choose Convert Audio to Keyframes.

Quick auto-lip sync2. Alt-click the stopwatch to add an expression. Use the pickwhip to connect the audio keyframes to the mouth animation.

Quick auto-lip syncSize the audio data to fit. Use this expression:

audio = [your audio keyframes];linear(audio,sourceMin,sourceMax,targetMin,targetMax)Lowest value of source dataClosed mouth (probably 0)Highest value of source dataWidest possible open mouth

Embrace the limitations of the format.One last thought:

2D vs. 3D: Avoiding the Uncanny ValleyAwesome glitches

33

Further non-K2P toolsFREEBrekel KinectOpen-source capture for 3D animation (Windows-only)www.brekel.com

DuikInverse kinematics plugin for AEwww.duduf.netCOMMERCIALiPi SoftUses two Kinects to recordwww.ipisoft.com

NI MateKinect for OSC/MIDI/Blenderwww.ni-mate.com

34

Get in TouchVictoria Necewww.victorianece.com@FakeGreenDress