05/04/23EEC492/693/793 - iPhone Application

Development 1

EEC-693/793EEC-693/793Applied Computer Vision Applied Computer Vision

with Depth Cameraswith Depth Cameras

Lecture 4Lecture 4

Wenbing ZhaoWenbing Zhaowenbing@ieee.orgwenbing@ieee.org

2

OutlineOutline Working with Kinect color stream Types of color images Retrieving color images: event & polling models Steps on handling of color images Event handler: C# delegate Build the KinectCam app

3

Working with Kinect Streams Kinect SDK supports two image stream formats

Color image stream Depth image stream Both are children of ImageStream class

The image frames are stored into a buffer for use by the application. If there is any delay in reading the buffer data and rendering it as images, the buffer will be overwritten => old frame lost

Main steps in handling image streams Enabling the stream Capturing the stream frame by frame Processing the image frames

Types of Color Images RGB: read-green-blue color space (RGB)

Each pixel is an array of four: Blue, Green, Red, Alpha Alpha: transparency degree 32bits per pixel at 640x480 at 30FPS or 1280x960 at 12FPS

YUV: Y: luminance channel; U: blue channel, V: red channel 16 bits per pixel at 640x480 at 15 FPS Uses less memory than RGB

Bayer: raw Bayer color image format with a Bayer color filter array (or Bayer filter): 50% green, 25% red, 25% blue Resolution at 640x480 at 30 FPS or 1280x960 at 12 FPS

Two ways: event model and polling model Event model

Kinect sends the frame to the app whenever a new frame is captured by the sensor

Need to register a event handler for the event

Retrieving Color Image Stream

Polling Model

The application sends a request to the sensor whenever there is a need to get an image frame

Pass the time interval after which the sensor will return the image frame

Capturing Color Images

Main steps Starting Kinect sensor Enable color stream channel with desirable image

format Subscribe (register) an event handler (a method)

with the stream channel Process the image frame Render the image frame on the user interface

Enable Color Stream Channel In class ColorImageStream, two methods public void Enable();

By default, use RgbResolution640x480Fps30 public void Enable(ColorImageFormat format);

RgbResolution640x480Fps30 RgbResolution1280x960Fps12 YuvResolution640x480Fps15 RawYuvResolution640x480Fps15 InfraredResolution640x480Fps30 RawBayerResolution640x480Fps30 RawBayerResolution1280x960Fps12 Undefined

You can enable only one type of color stream at a time

Registering an Event Handler What is colorFrameReady?!

It is the event handler that you will implement You can use any method name as you wish The method must have the following signature:

void colorFrameReady(object sender, ColorImageFrameReadyEventArgs e)

Sender: the object that fires the event e: the object that holds data to be retrieved (i.e., image frame)

this.sensor.ColorFrameReady += colorFrameReady;

Event Handler Event handler is similar to a function pointer in C/C++ In C# (and objective C), the event handler is a delegate with

pedefined signature General format:

public delegate void MyEventHandler(object sender, MyEventArgs e); Sender: files the event e: holds the data to be handled

A delegate allows you to pass methods of one class to objects of other classes that can call those methods

Delegate in C#(http://www.developerfusion.com/article/2137/event-handling-in-net-using-c/2/ )using System; // Step 1. Declare a delegate with the signature of the encapsulated method  public delegate void MyDelegate(string input); 

//Step 2. Define methods that match with signature of delegate declaration class MyClass1{ 

public void delegateMethod1(string input){        Console.WriteLine("delegateMethod1 and input to the method is {0}",input);    } 

   public void delegateMethod2(string input){        Console.WriteLine("delegateMethod2 and input to the method is {0}",input);    } } 

}

Delegate in C#(http://www.developerfusion.com/article/2137/event-handling-in-net-using-c/2/ )

//Step 3. Create delegate object and plug in the methods class MyClass2 {    public MyDelegate createDelegate(){        MyClass1 c2=new MyClass1();        MyDelegate d1 = new MyDelegate(c2.delegateMethod1);        MyDelegate d2 = new MyDelegate(c2.delegateMethod2);        MyDelegate d3 = d1 + d2;        return d3;    } } 

Delegate in C#(http://www.developerfusion.com/article/2137/event-handling-in-net-using-c/2/ )

//Step 4. Call the encapsulated methods through the delegate class MyClass3{    public void callDelegate(MyDelegate d, string input){        d(input);    } } class Driver{    static void Main(string[] args){        MyClass2 c2 = new MyClass2();        MyDelegate d = c2.createDelegate();        MyClass3 c3 = new MyClass3();        c3.callDelegate(d,"Calling the delegate");    } }

Event Handler Delegate for ColorImage

In Kinect.Sensor class

That is why we can this:

namespace System {public delegate void EventHandler<TEventArgs>

(object sender, TEventArgs e);}

public event EventHandler<ColorImageFrameReadyEventArgs> ColorFrameReady;

this.sensor.ColorFrameReady += colorFrameReady;

Retrieve the Color Image Frame

What is using(a new object) { }? It defines the scope of the new object Once the last statement in the {} block is executed, the object is

garbage deleted

void colorFrameReady(object sender,ColorImageFrameReadyEventArgs e){

using (ColorImageFrame imageFrame = e.OpenColorImageFrame()) { ……. }}

Handling/Displaying Color Image// copy image to a byte arrayimageFrame.CopyPixelDataTo(colorPixels);

// width in bytes of a single row of pixel data including padding.int stride = imageFrame.Width * imageFrame.BytesPerPixel;// Write the pixel data into our bitmapthis.colorBitmap.WritePixels( new Int32Rect(0, 0, this.colorBitmap.PixelWidth,

this.colorBitmap.PixelHeight),this.colorPixels,stride,

0);

// Don’t do this for stride calculation// If format is changed, it may not work!this.colorBitmap.PixelWidth * sizeof(int);

Color Image Handling

Updates the pixels in the specified region of the bitmap. srcRect: The rectangle of the

System.Windows.Media.Imaging.WriteableBitmap to update pixels: The pixel array used to update the bitmap. stride: The stride of the update region in pixels. offset: The input buffer offset

public void WritePixels(Int32Rect srcRect, Array pixels, int stride, int offset);

WriteableBitmap Class Inherits from BitmapSource Constructor

pixelWidth:width of the bitmap pixelHeight: height of the bitmap dpiX: horizontal dots per inch (dpi) dpiY: vertical dots per inch pixelFormat: pixel format palette: finite set of colors

public WriteableBitmap(int pixelWidth, int pixelHeight, double dpiX, double dpiY, PixelFormat

pixelFormat, BitmapPalette palette);

System.Windows.Media.PixelFormats Bgr32 property: blue, green, red channels (3 bytes),

last byte is set to 0 http://msdn.microsoft.com/en-us/magazine/cc534995.aspx

Bgra32 Bgr24 Gray2 Gray16 Etc.

Build KinectCam App Create a new WFP project named KinectCam Draw the GUI for the app Modify xaml file to add WindowLoaded and

WindowClosing as before Add member variables for displaying color images Start Kinect sensor, enable color stream, register

color stream handler, connect image control to the bitmap for display

Process each color image

Design GUI Add label title Add image control

Add Member Variables

KinectSensor sensor;

// Bitmap that will hold color information private WriteableBitmap colorBitmap;

// Intermediate storage for the color data received from the camera

private byte[] colorPixels;

Initialization (in WindowLoaded())if (KinectSensor.KinectSensors.Count > 0) {

this.sensor = KinectSensor.KinectSensors[0]; if (this.sensor != null && !this.sensor.IsRunning) {

this.sensor.ColorStream.Enable();this.colorPixels = new byte[this.sensor.ColorStream.FramePixelDataLength];this.colorBitmap = new WriteableBitmap(this.sensor.ColorStream.FrameWidth, this.sensor.ColorStream.FrameHeight, 96.0, 96.0, PixelFormats.Bgr32, null);this.image1.Source = this.colorBitmap;

this.sensor.ColorFrameReady += this.colorFrameReady;this.sensor.Start();

} else { MessageBox.Show("No device is connected!"); this.Close(); } }}

Handling/Displaying Color Images void colorFrameReady(object sender, ColorImageFrameReadyEventArgs e) {

using (ColorImageFrame imageFrame = e.OpenColorImageFrame()) { if (null == imageFrame) return;

imageFrame.CopyPixelDataTo(colorPixels); int stride = imageFrame.Width * imageFrame.BytesPerPixel;

// Write the pixel data into our bitmap this.colorBitmap.WritePixels( new Int32Rect(0, 0, this.colorBitmap.PixelWidth,

this.colorBitmap.PixelHeight), this.colorPixels, stride, 0); }}