Kinect

C. Andjar

Contents

Introduction

Hardware

Demos

Software Software

Freenect

OpenNI + NITE + SensorKinect

More demos

Projects using Kinect

Upcoming sensors

INTRODUCTION

3D input competitors

Kinect origins

PrimeSense Israeli company (2005) focused on vision and natural interaction

Natural interface for controlling living-room devices (e.g. TVs)

Gesture-based browsing of images, video

Full-body interaction for games

PrimeSensor low-cost device for TV screens providing real-time

depth, color and audio data of the living room.

The user is not required to wear or hold anything

Insensitive to lighting conditions

It does not require calibration

Microsoft Kinect

New controller for Microsofts Xbox 360

Full-body tracking, face and voice recognition

Low cost (99 Euros as of April 2011)

HARDWARE

Main components

Main components

Video

Color CMOS camera

Infrared (IR) CMOS camera

Infrared projector - 830nm, 60mW laser diode.

AudioAudio

Four microphones

Tilt control

Motor

Accelerometer (3-axes)

Processors & memory

PrimeSense chip PS1080-A2

64 MB DDR2 SDRAM

Image sensors

Color camera: 640x480 sensor, 640x480@30fps output

IR camera: 1280x1024 sensor, 640x480@30fps output

Operation range (depth sensor) = 0.8m - 3.5m

FOV = 58 H, 45 V, 70 D

Spatial resolution (@ 2m distance) = 3mm

Depth resolution (@ 2m distance) = 1cm

Image sensors specs

Depth sensing

The IR emitter projects an irregular pattern of IR dots of

varying intensities.

The IR camera reconstructs a depth image by recognizing the

distortion in this pattern.

Connecting the Kinect to a PC

Trivial for old Kinect models (those sold

separately): USB-2 standard connector

Further info on Kinects hardware

http://www.ifixit.com/Teardown/Microsoft-Kinect-Teardown/4066/1

http://openkinect.org/

http://www.freepatentsonline.com/7433024.pdf

DEMOS

Viewer (color map, IR map, depth map); skeletor

SOFTWARE

Drivers

libfreenect

www.openkinect.org

Unofficial, open source driver (GLP2)

Linux, Windows, OS X

Raw data (color map, IR map, depth map) Raw data (color map, IR map, depth map)

Installation: trivial

OpenNI + NITE + SensorKinect

OpenNI

SDK for natural interfaces; open source

www.openni.org

NITE

OpenNI Plugin for gesture/pose recognition; closed source OpenNI Plugin for gesture/pose recognition; closed source

www.openni.org

SensorKinect

Driver for the Kinect; fork from the PrimeSensor driver (open source)

github.com/avin2/SensorKinect

OpenNI

Framework for modules/sensors providing depth maps, color

maps, scene maps, gesture recognition, user pose (skeleton)

www.openni.org

OpenNI

OpenNI aims at abstracting developers from sensor hardware

(Kinect, PrimeSensor) and CV algorithms (scene analysis,

gesture recognition, pose recognition)

OpenNI

OpenNI basic concepts

Production node: device/map/CV algorithm abstraction

Production graph: a particular combination of nodes

Module: plug-in (.dll, .so file) implementing nodes

OpenNI nodes

Audio generates an audio stream

Depth generates depth-maps

Image generates color maps

IR generates IR maps

SceneAnalyzer generates a label map (e.g. segmenting objects)

Gestures recognizes gestures and calls specific callbacksGestures recognizes gestures and calls specific callbacks

Hands recognizes hands and calls specific callbacks when a handappears/is moved/disappears

User computes user 3D info (including center of mass, pose detection,skeleton) and calls specific callbacks

Recorder saves node data to a file

Player read data from a file and replays it

Codec used for compression and decompression of data in recordings

OpenNI nodes

Audio generates an audio stream

Depth generates depth-maps

Image generates color maps

IR generates IR maps

SceneAnalyzer generates a label map (e.g. segmenting objects)

Gestures recognizes gestures and calls specific callbacks

Hands recognizes hands and calls specific callbacks when a handappears/is moved/disappears

User computes user 3D info (including center of mass, pose detection,skeleton) and calls specific callbacks

Recorder saves node data to a file

Player read data from a file and replays it

Codec used for compression and decompression of data in recordings

OpenNI nodes - Audio

OpenNI nodes

Audio generates an audio stream

Depth generates depth-maps

Image generates color maps

IR generates IR maps

SceneAnalyzer generates a label map (e.g. segmenting objects)

Gestures recognizes gestures and calls specific callbacks

Hands recognizes hands and calls specific callbacks when a handappears/is moved/disappears

User computes user 3D info (including center of mass, pose detection,skeleton) and calls specific callbacks

Recorder saves node data to a file

Player read data from a file and replays it

Codec used for compression and decompression of data in recordings

OpenNI nodes - Maps

OpenNI nodes - Maps

OpenNI nodes

Audio generates an audio stream

Depth generates depth-maps

Image generates color maps

IR generates IR maps

SceneAnalyzer generates a label map (e.g. segmenting objects)

Gestures recognizes gestures and calls specific callbacks

Hands recognizes hands and calls specific callbacks when a handappears/is moved/disappears

User computes user 3D info (including center of mass, pose detection,skeleton) and calls specific callbacks

Recorder saves node data to a file

Player read data from a file and replays it

Codec used for compression and decompression of data in recordings

OpenNI nodes - Recognition

OpenNI nodes - User node

OpenNI nodes - Skeleton

XN_SKEL_HEAD

XN_SKEL_NECK

XN_SKEL_TORSO

XN_SKEL_WAIST

XN_SKEL_LEFT_COLLAR

XN_SKEL_LEFT_SHOULDER

XN_SKEL_LEFT_ELBOW

XN_SKEL_LEFT_WRIST

XN_SKEL_LEFT_HAND

XN_SKEL_LEFT_FINGERTIP

XN_SKEL_RIGHT_COLLAR

XN_SKEL_RIGHT_SHOULDER

XN_SKEL_RIGHT_ELBOW

XN_SKEL_RIGHT_WRIST

XN_SKEL_RIGHT_HAND

XN_SKEL_RIGHT_FINGERTIP

XN_SKEL_LEFT_HIP

XN_SKEL_LEFT_KNEE

XN_SKEL_LEFT_ANKLE

XN_SKEL_LEFT_FOOT

XN_SKEL_RIGHT_HIP

XN_SKEL_RIGHT_KNEE

XN_SKEL_RIGHT_ANKLE

XN_SKEL_RIGHT_FOOT

OpenNI example

// Initialize context object

xn::Context context;

context.Init();

// Create a DepthGenerator node

xn::DepthGenerator depth;

depth.Create(context);

// Make it start generating data

context.StartGeneratingAll();

while (true)

{

// Wait for new data to be available

XnStatus status = context.WaitOneUpdateAll(depth);

if (status == XN_STATUS_OK) {

const XnDepthPixel* pDepthMap = depth.GetDepthMap();

// process depth map

.

}

OpenNI - Initializing nodes

// Create a DepthGenerator node

xn::DepthGenerator depth;

depth.Create(context);

// Configure node

XnMapOutputMode outputMode;

outputMode.nXRes = 640; outputMode.nXRes = 640;

outputMode.nYRes = 480;

outputMode.nFPS = 30;

status = depth.SetMapOutputMode(outputMode);

Sample applications

NiSimpleRead takes a depth generator node from an

XML config file and prints out the depth value of the

middle pixel.

NiSimpleViewer - OpenGL application which draws the

depth maps and the color maps to the screen. depth maps and the color maps to the screen.

NiSampleModule - sample for writing a module

implementing a depth node with mirror capability.

NiViewer displays depth, image and IR maps, plays

audio, etc.

NITE

Middleware for pose/gesture recognition

NITE

Joint positions: given in real world coords (mm)

Joint orientations: 3x3 rotation matrix representing the rotation of joints

local coords w.r.t world coords (the first column is the direction of the

joints +X axis in world coordinates, and so on).

Kinect quick start (Windows)

Install OpenNI binary

Install SensorKinect binary

Install NITE key 0KOIk2JeIBYClPWVnMoRKn5cdY4=

Connect Kinect to PC Connect Kinect to PC

Try demos

MORE DEMOS

Sample-PointViewer, reconstructor, Miku2

Projects using Kinect

About 420 projects in www.kinecthacks.net

Upcoming sensors