EEC-693/793 Applied Computer Vision with Depth Cameras

EEC-693/793EEC-693/793Applied Computer Vision Applied Computer Vision

with Depth Cameraswith Depth Cameras

Lecture 7Lecture 7

Wenbing ZhaoWenbing Zhaowenbing@ieee.orgwenbing@ieee.org

OutlineOutline Depth data stream How depth measurement is done Capture and process depth data Classes related to depth data Getting the Distance from a Particular Pixel

Depth Data Stream Depth data stream consists of a sequence of depth image frames Each depth image frame is in the form of a 16-bit grey scale image

Viewable range: 43 degrees vertical, 57 degrees horizontal A depth pixel contains the following

(x,y) coordinates represent the location of the object in the depth sensor view

The value of the pixel at (x,y) contains the distance between the object and the Kinect sensor in millimeters

Depth Data Stream

Depth data stream supports 640x480 at 30fps 320x240 at 30fps 80x60 at 30fps

Kinect depth sensing range Up to 13.1 feet (4 meters) As close as 40cm

How the Depth Data is Obtained

Stereo Triangulation Stereo triangulation is an analysis algorithm for

computing the 3D position of points in an image frame

In general stereo triangulation, two images are used to obtain the two different views on a scene, in a similar fashion to human binocular vision. By comparing these two images, the relative depth

information is calculated Kinect stereo triangulation uses one real (IR) image

and a virtual image (more accurately, the line of each projected point) http://www.youtube.com/watch?v=uq9SEJxZiUg

Depth Measurement When there is a pixel in any of the diagonal views of

the sensor, it internally draws a line that is perpendicular to the sensor and then calculates the distance directly from there

Capturing and Processing Depth Data Enable the depth stream channel with the type of depth image

format

Attach the event handler to the stream channel

Process the incoming depth frames

Render the frames on UI

this.sensor = KinectSensor.KinectSensors[0];this.sensor.DepthStream.Enable();

sensor.DepthFrameReady += depthFrameReady;

void depthFrameReady(object sender, DepthImageFrameReadyEventArgs e){}

Processing Depth Data void depthFrameReady(object sender, DepthImageFrameReadyEventArgs e){ using (DepthImageFrame imageFrame = e.OpenDepthImageFrame()) { if (null == imageFrame) {

return; } imageFrame.CopyPixelDataTo(depthPixels);

// Write the pixel data into our bitmap this.depthBitmap.WritePixels(

new Int32Rect(0, 0, this.depthBitmap.PixelWidth, this.depthBitmap.PixelHeight),

this.depthPixels, stride, 0);

}}

Very similar to the code for ColorImage Processing: color => depth

Classes Related to Depth Data

Classes Related to Depth Data

Depth Data and Distance The DepthImageStream and DepthImageFrame

classes have the MaxDepth and MinDepth properties, which return the maximum and minimum depth ranges for that particular stream or captured image frame, in millimeters

This range value returns the best range where distance can be measured properly

These values will change automatically based on the selection of DepthRange for the stream

Depth Range The Range property sets the viewable range for the Kinect

sensor, which is a type of DepthRange enumeration A DepthRange enumeration has the following two values:

Default Near sensor.DepthStream.Range = DepthRange.Near;

Special Depth Range Values DepthImageStream defines three additional read-only

properties TooFarDepth: It provides the depth point where the object is

beyond the range of the sensor TooNearDepth: It provides the depth point where the object is too

close to the sensor UnknownDepth: There are instances when a depth is completely

indeterminate, and the value will be zero

Default Depth Range

Depth Data Representation The Kinect sensor returns 16-bit raw depth frame data

The first three bits are used to represent the identified players and the remaining 13 bits give you the measured distance in millimeters

To retrieve the 13-bit depth data, perform a bitwise shift operation >> to move the bits to the beginning

int depth = depthFrame[depthIndex] >> DepthImageFrame.PlayerIndexBitmaskWidth;

Depth Data Representation

Getting the Distance from a Particular Pixel Add feature to our DepthCam app

Very important: make the image control exactly the same size as the resolution chosen Image should be 320x240 if Resolution320x240Fps30 Image should be 640x480 if Resolution640x480Fps30

Use mouse to select a pixel in the image => display the depth for that pixel Need to add event handler for MouseDown

Point currentPoint = e.GetPosition(depthImageControl);

int pixelIndex = (int)(currentPoint.X + ((int)currentPoint.Y * this.frame.Width));

int distancemm = this.pixelData[pixelIndex] >> DepthImageFrame.PlayerIndexBitmaskWidth;

Build the DepthCam App Create a new WFP project named DepthCam Draw the GUI for the app Modify xaml file to add WindowLoaded and

WindowClosing as before Add Microsoft.Kinect reference Add member variables for displaying depth images Start Kinect sensor, enable depth data stream,

register depth stream handler, connect image control to the bitmap for display

Process each depth image Add code for displaying depth data for pixel clicked

GUI Design Image control Group control for depth information

Two sets of labels and two sets of textboxes Use meaningful names for the textboxes

Group control for pixel information Four sets of labels and four sets of textboxes

Adding Code Add member variables:

WindowLoade method (WindowClosing() same as before):KinectSensor sensor;private WriteableBitmap depthBitmap;private short[] depthPixels;private int frameWidth; // to calculate depthIndex

private void WindowLoaded(object sender, RoutedEventArgs e) { if (KinectSensor.KinectSensors.Count > 0) { this.sensor = KinectSensor.KinectSensors[0]; if (this.sensor != null && !this.sensor.IsRunning) {

this.sensor.DepthStream.Enable(DepthImageFormat.Resolution320x240Fps30);this.depthPixels = new short[this.sensor.DepthStream.FramePixelDataLength];this.depthBitmap = new WriteableBitmap(this.sensor.DepthStream.FrameWidth, this.sensor.DepthStream.FrameHeight, 96.0, 96.0, PixelFormats.Gray16, null);

this.image1.Source = this.depthBitmap;this.sensor.DepthFrameReady += this.depthFrameReady;this.sensor.Start();

} else { MessageBox.Show("No device is connected!"); this.Close(); } } }

Adding Code Event handler for depth frames:

void depthFrameReady(object sender, DepthImageFrameReadyEventArgs e) { using (DepthImageFrame imageFrame = e.OpenDepthImageFrame()) { if (null == imageFrame) {

return; } this.frameWidth = imageFrame.Width; this.maxDepthField.Text = "" + imageFrame.MaxDepth; this.minDepthField.Text = "" + imageFrame.MinDepth;

imageFrame.CopyPixelDataTo(depthPixels); int stride = imageFrame.Width * imageFrame.BytesPerPixel; this.depthBitmap.WritePixels( new Int32Rect(0, 0, this.depthBitmap.PixelWidth, this.depthBitmap.PixelHeight), this.depthPixels, stride, 0); }}

Adding Code Add mouse down event on

image control Find the MouseDown line, double

click it to get event handler template

Event handler for mouse click:

private void image1_MouseDown(object sender, MouseButtonEventArgs e) { Point currentPoint = e.GetPosition(image1); this.pixelXField.Text = currentPoint.X.ToString(); this.pixelYField.Text = currentPoint.Y.ToString(); int pixelIndex = (int)(currentPoint.X + ((int)currentPoint.Y * this.frameWidth)); this.depthIndexField.Text = "" + pixelIndex; int distancemm = this.depthPixels[pixelIndex] >> depthImageFrame.PlayerIndexBitmaskWidth; this.depthField.Text = "" + distancemm;}