KINECT – THE HOW, WHEN, AND WHERE OF DEVELOPING WITH IT

PHILIP WHEAT

MANAGER, CHAOTIC MOON LABS

PHIL@CHAOTICMOON.COM

WHO I AM

Philip Wheat – Currently managing the Labs Division of Chaotic Moon (http://www.chaoticmoon.com/Labs)

Former Evangelist with Microsoft – Architecture and Startups.

Active in the Maker Movement – it’s about Bits AND Atoms, not Bits or Atoms.

My neglected blog can be found at http://PhilWheat.net

I can be found on Twitter as @pwheat

WHY THIS TALK?

There has been a lot of talk about next generation interfaces – but most interface talk today still seems to be around HTML5/Native.

User interaction is very focused today on touch interfaces – but this limits various scenarios and usage models.

We’ll look at the components of interacting with the user through Kinect today and help you start looking at new scenarios.

THE PROJECTS

THE HOW

DEVELOPING WITH KINECT

HARDWARE VERSIONS

Remember, there are two versions of Kinect Hardware – Kinect 360 and Kinect for Windows.

• Kinect for 360

• Can be used for SDK development• Some Kinect for Windows Functionality not supported• Not supported for Microsoft SDK production

• Kinect for Windows

• Can be used for development and deploymen• Full Kinect for Windows Functionality (Near Mode,

Additional resolutions)• Supported for production

HARDWARE CAPABILITIES• Cameras

• RGB Camera – 320x240, 640x480, 1024x768 (KFW)• Depth Camera – 320x240, 640x480 • Depth Camera – .8M – 4M (standard Mode)

– .5M – 3M (near Mode)• Audio

• Microphone Array• 4 Microphones• Audio directional detection• Audio detection steering

• Tilt

• Tilt from -27 to 27 degrees from horizontal (accelerometer)

SOFTWARE CAPABILITIES• Cameras

• RGB Frames – event driven and polled• Depth Frames – event driven and polled• Skeleton Tracking – event driven and polled• Seated Skeleton Tracking (Announced for KFW 1.5)

• Audio

• Voice Recognition in English• Voice Recognition – French, Spanish, Italian, Japanese

(Announced for KFW 1.5)

OTHER HARDWARE ITEMS• Kinect needs 12V for operations – this requires a non-

standard connector. For use with a PC, Kinect requires a supplemental power supply which is included for the Kinect for Windows hardware but is not included with Kinect 360’s that are bundled with Xbox slim models.

• Kinect tilt motor is update limited to prevent overheating.

• Lenses are available to reduce focus range – but produce distortion and are not supported.

• Kinect has a fan to prevent overheating – be careful of enclosed areas.

• Kinect contains a USB Hub internally – connecting it through another USB Hub is not supported and will only work with certain Hubs (through practical testing.)

CONNECTING TO YOUR KINECT

Some Key code –

• KinectSensor.KinectSensors.Count - # of Kinects.

• kinectSensor.Status – Enumerator of

• Connected (it’s on and ready)• Device Not Genuine (not a Kinect)• Device Not Supported (Xbox 360 in Production mode)• Disconnected (has been removed after being inited)• Error (duh)• Initializing (can be in this state for seconds)• InsufficientBandwidth (USB Bus controller contention)• Not Powered (5V power is good, 12V power problems)• NotReady (Some component is still initing)

WHEN

DEVELOPING WITH KINECT

VIDEO FRAMES

SIMPLE VIDEOKinect can be used to simply get RGB FramesEnable the stream

kinectSensor.ColorStream.Enable(ColorImageFormat.RgbResolution640x480Fps30);

Start the Kinect -

kinectSensor.Start();

Set up your event handler (if doing events – if not you’ll likely drop frames) -

kinectSensor.ColorFrameReady += new EventHandler<Microsoft.Kinect.ColorImageFrameReadyEventArgs>(kinect_VideoFrameReady);

Then the handler –

void kinect_VideoFrameReady(object sender,Microsoft.Kinect.ColorImageFrameReadyEventArgs e)

{ using (ColorImageFrame image = e.OpenColorImageFrame())

{ if (image != null)

{ if (colorPixelData == null)

{ colorPixelData = new byte[image.PixelDataLength]; }

else

{ image.CopyPixelDataTo(colorPixelData);

BitmapSource source = BitmapSource.Create(image.Width, image.Height, 96, 96, PixelFormats.Bgr32, null, colorPixelData, image.Width * image.BytesPerPixel);

videoImage.Source = source;

...

DEPTH FRAMES

DEPTH FRAMESAdditionally you can use Depth frames to give you more information about your environment.

kinectSensor.DepthStream.Enable(DepthImageFormat.Resolution320x240Fps30);

kinectSensor.Start();

kinectSensor.DepthFrameReady += new EventHandler<Microsoft.Kinect.DepthImageFrameReadyEventArgs>(kinect_DepthImageFrameReady);

void kinect_DepthImageFrameReady(object sender, Microsoft.Kinect.DepthImageFrameReadyEventArgs e)

{

using (DepthImageFrame imageFrame = e.OpenDepthImageFrame())

{

if (depthPixelData == null)

{

depthPixelData = new short[imageFrame.PixelDataLength];

}

if (imageFrame != null)

{

imageFrame.CopyPixelDataTo(this.depthPixelData);

…

But!

int depth = depthData[x + width * y] >> DepthImageFrame.PlayerIndexBitmaskWidth;

SKELETON TRACKING

SKELETON TRACKINGAnd one of the most interesting items is skeleton tracking –

(This should start looking familiar)

kinectSensor.SkeletonStream.Enable();

kinectSensor.Start();

kinectSensor.SkeletonFrameReady += new EventHandler<Microsoft.Kinect.SkeletonFrameReadyEventArgs>(kinect_SkeletonFrameReady);

void kinect_SkeletonFrameReady(object sender, SkeletonFrameReadyEventArgs e)

{ using (SkeletonFrame skeletonFrame = e.OpenSkeletonFrame())

{ if (skeletonFrame != null)

{ if ((this.skeletonData == null) || (this.skeletonData.Length != skeletonFrame.SkeletonArrayLength))

{ this.skeletonData = new Skeleton[skeletonFrame.SkeletonArrayLength]; }

skeletonFrame.CopySkeletonDataTo(this.skeletonData);

Skeleton thisSkeleton = null;

foreach (Skeleton skeleton in this.skeletonData)

{ if ((SkeletonTrackingState.Tracked == skeleton.TrackingState)

&& (thisSkeleton == null))

{ thisSkeleton = skeleton; }

}

if (thisSkeleton != null)

{

thisSkeleton.Position

}

SKELETON TRACKING (CONT)Key items for Skeleton Tracking –

SkeletonPoint – X, Y, Z

JointType enumeration – AnkleLeft, AnkleRight, ElbowLeft, ElbowRight, FootLeft, FootRight, HandLeft, HandRight, Head, HipCenter, HipLeft,HipRight,KneeLeft,KneeRight, ShoulderCenter, ShoulderLeft, ShoulderRight,Spine, WristLeft, WristRight

JointTrackingState enumeration – Inferred, NotTracked, Tracked

These together tell you not just where joints are, but how confident the system is. Even so – remember that these are always estimates – you’ll need to be prepared to handle jittery data.

SPEECH RECOGNITION

Building a grammar is very accessible and relatively pain free.

(A bit more code than the others.)

Key items – it takes up to 4 seconds for the recognizer to be ready.

Each recognition has a confidence level of 0-1.

Results are text and match text you send to a Grammar Builder (“yes”, “no”, “launch”)

Multiple word commands are helpful for disambiguation, but chained commands work much better. The recognition engine can have nested grammars to help you with this.

WHERE

DEVELOPING WITH KINECT

LOCATION• Human Interface

• Kinect skeleton tracking is optimized for full body imaging and waist to head height camera position.

• Kinect 1.5 software update (est May 2012) is scheduled to support sitting skeleton tracking.

• Speech Recognition

• Depth or Skeleton tracking can enable recognition vectoring to increase confidence.

• Confidence level can be misleading if grammar items are close together (false matching.)

• If possible, use multiple word commands for validation. Build a command grammar and use it for error/confidence checks.

LOCATION (CONT)• Depth Frames

• Sunlight/IR can affect results.• Items in depth frame have a shadow – be prepared for

interactions in those areas. • Depth Frames are not required to be the same resolution

as associated Video frames.• Environment

• Kinect is surprisingly robust for environment• IR washout is the biggest factor• Camera angle biggest factor for Skeleton Tracking.

FURTHER INFORMATION

Kinect for Windows information:

http://www.KinectForWindows.com

Team blog:

http://blogs.msdn.com/b/kinectforwindows

Channel 9

http://channel9.msdn.com/Tags/kinect

And of course, our projects pages

http://ChaoticMoon.com/Labs !

OR CONTACT ME AT:@PWHEATPHILWHEAT.NET

Q&A