EEC-693/793EEC-693/793Applied Computer Vision Applied Computer Vision

with Depth Cameraswith Depth Cameras

Lecture 10Lecture 10

Wenbing ZhaoWenbing Zhaowenbing@ieee.orgwenbing@ieee.org

OutlineOutline Human skeleton tracking (part III)

Skeleton Smoothing Skeleton data can be smoothed by registering a set of

smooth parameters As a parameter to SkeletonStream.Enable(….);

// create the smooth parametersvar smoothParameters = new TransformSmoothParameters{ Correction = 0.1f, JitterRadius = 0.05f, MaxDeviationRadius = 0.05f, Prediction = 0.1f, Smoothing = 0.5f};// Enable the skeleton stream with smooth parametersthis.sensor.SkeletonStream.Enable(smoothParameters);

Smoothing Parameters Correction: specifies the amount of correction

needed for the raw data. Range [0.0, 1.0] The value must be within the range of 0 to 1.0 and the

default value is 0.5. Lower values are slower to correct towards the raw data and

appear smoother, while higher values correct toward the raw data more quickly

Smoothing: determines the amount of smoothing applied while processing. Range [0.0, 1.0] The value must be within the range of 0 to 1.0 and the

default value is 0.5. The larger value, the smoother skeleton data, however, it

increases the latency Zero value => you will get the raw skeleton data

Smoothing Parameters JitterRadius: limit the radius value for jittery data, >=0

Measured in meters and the default value is 0.05. If the position of a jitter is outside the set radius, it is

corrected to be positioned at the radius MaxDeviationRadius: The maximum radius in meters that

filtered positions are allowed to deviate from raw data, >=0 Filtered values that would exceed the radius from the raw data are

clamped at this distance, in the direction of the filtered value default value is 0.04 meters

Prediction: number of frames predicted into the future, >=0 default value: 0.5 A value greater than 0.5 will likely lead to overshoot when the data

changes quickly

Kinect SDK Smoothing Algorithm Holt double exponential smoothing algorithm is used to reduce the jitters from skeletal joint data The smoothing algorithm applies to each set of data and

calculates a moving average based on the previous set of data

During the calculation of moving average, it uses the values passed by the smoothing parameter.

Getting Data Frames Together We have covered three (color, depth, skeleton) types of data

streams that are returned by the sensor For a real application, we often need all three types of data

streams We can use a single event AllFramesReady, which will do the

job for all three of them The AllFramesReady event fires when new frames are available

for the color, depth, and skeleton streams

this.sensor.AllFramesReady+=sensor_AllFramesReady;

void sensor_AllFramesReady(object sender, AllFramesReadyEventArgs e){}

Getting Data Frames Together

Floor Determination Each skeleton frame contains a floor-clipping-plane

vector, which contains the coefficients of an estimated floor-plane equation The equation is normalized so that the physical

interpretation of D is the height of the camera from the floor, in meters

The skeleton tracking system updates this estimate for each frame and uses it as a clipping plane for removing the background and segmenting players

Ax + By + Cz + D = 0

How to Use FloorClipPlane In Your App FloorClipPlane can be used to calculate the height

of each joint with respect to the floor Can be used for validation study, as well as fall

detection How to get FloorClipPlane parameters:

How to calculate the height of a joint:

float A = frame.FloorClipPlane.Item1;float B = frame.FloorClipPlane.Item2;float C = frame.FloorClipPlane.Item3;float D = frame.FloorClipPlane.Item4;

float height = A*joint.Position.X+B*joint.Position.Y+C*joint.Position.Z+D;

Joint Orientation The bone orientation is provided in two forms:

A hierarchical rotation based on a bone relationship defined on the skeleton joint structure

An absolute orientation in Kinect camera coordinates The orientation information is provided in form of quaternions

and rotation matrices for use in different animation scenarios BoneOrientation Class, public properties:

StartJoint: Gets the skeleton joint where the bone starts EndJoint: Gets the skeleton joint where the bone ends HierarchicalRotation: Gets or sets the rotation of a bone relative to its

parent bone, of type BoneRotation AbsoluteRotation: Gets or sets the rotation of the bone relative to

camera coordinates, of type BoneRotation

BoneRotation Class BoneRotation has two properties

Matrix: Gets or sets a matrix representation of the bone rotation Quaternion: Gets or sets a quaternion representation of the bone

rotation

Vector4 struct: has four properties W, X, Y, Z

Matrix4 struct: has 17 properties Identity, of type Matrix4 M11, M12, M13, M14, M21, M22, M23, M24, M31, M32, M33,

M34, M41, M42, M43, M44

public Matrix4 Matrix { get; set; } public Vector4 Quaternion { get; set; }

Access Joint Orientationprivate void DrawSkeletonsWithOrientations() { foreach (Skeleton skeleton in this.skeletonData) { if (skeleton.TrackingState == SkeletonTrackingState.Tracked) { foreach (BoneOrientation orientation in skeleton.BoneOrientations) { // Display bone with Rotation using quaternion

DrawBonewithRotation(orientation.StartJoint, orientation.EndJoint,

orientation.AbsoluteRotation.Quaternion);

// Display hierarchical rotation using matrix DrawHierarchicalRotation(orientation.StartJoint, orientation.HierarchicalRotation.Matrix) } } } }

Build ShapeGame App Create a new C# WPF project with name ShapeGame

It is a much simplified app from the Kinect ShapeGame Only one shape, i.e., a ball is used You are limited to use your righthand to hit the ball Ball only drops from a designated spot down, i.e., no X velocity

Add Microsoft.Kinect reference Design GUI Added WindowLoaded() method in xaml file Adding code

GUI Design

Image control

Canvas

Adding Code Add member variables:

WindowLoaded(): Enable both ColorImageStream and SkeletonStream Register event handler for both ColorFrameReady and SkeletonFrameReady events Initialize the shape (i.e., ball)

KinectSensor sensor;Skeleton[] totalSkeleton = new Skeleton[6];WriteableBitmap colorBitmap;byte[] colorPixels;Skeleton skeleton;Thing thing = new Thing(); // a struct for balldouble gravity = 0.017;

Adding Code The Thing Struct. Make it a private struct inside the MainWindow class

private struct Thing { public System.Windows.Point Center; public double YVelocity; public double XVelocity; public Ellipse Shape; public bool Hit(System.Windows.Point joint) { double minDxSquared = this.Shape.RenderSize.Width; minDxSquared *= minDxSquared; double dist = SquaredDistance(Center.X, Center.Y, joint.X, joint.Y); if (dist<= minDxSquared) { return true; } else return false; }}

Adding Codeprivate static double SquaredDistance(double x1, double y1, double x2, double y2){ return ((x2 - x1) * (x2 - x1)) + ((y2 - y1) * (y2 - y1));}

private void WindowLoaded(object sender, RoutedEventArgs e){ // same as before …. // new code for ball initialization thing.Shape = new Ellipse(); thing.Shape.Width = 30; thing.Shape.Height = 30; thing.Shape.Fill = new SolidColorBrush(Color.FromRgb(0, 255, 255)); thing.Center.X = 300; thing.Center.Y = 0; thing.Shape.SetValue(Canvas.LeftProperty, thing.Center.X - thing.Shape.Width); thing.Shape.SetValue(Canvas.TopProperty, thing.Center.Y - thing.Shape.Width); canvas1.Children.Add(thing.Shape);}

Canvas.LeftProperty: A Double that represents the offset position from the left side of a parent Canvas.

Modify Event Handler for Skeleton Framesvoid skeletonFrameReady(object sender, SkeletonFrameReadyEventArgs e){ canvas1.Children.Clear(); advanceThingPosition(); canvas1.Children.Add(thing.Shape);

// remaining code same as before …..

Point handPt = ScalePosition(skeleton.Joints[JointType.HandRight].Position); if(thing.Hit(handPt)) { this.thing.YVelocity = -1.0*this.thing.YVelocity; }}

Shape Animation Code void advanceThingPosition() { thing.Center.Offset(thing.XVelocity, thing.YVelocity); thing.YVelocity += this.gravity; thing.Shape.SetValue(Canvas.LeftProperty, thing.Center.X - thing.Shape.Width); thing.Shape.SetValue(Canvas.TopProperty, thing.Center.Y - thing.Shape.Width);

// if goes out of bound, reset position, as well as velocity if (thing.Center.Y >= canvas1.Height) { thing.Center.Y = 0; thing.XVelocity = 0; thing.YVelocity = 0; }}

Challenge Task For advanced students, improve the shape game in

the following ways: Keep the hit count, count should reset if ball falls to the

bottom Display the hit count at the ball Consider the angle of the hit to the ball, and adjust the x

velocity accordingly (you must consider physics here) Allow other joints to hit the ball too, such as left hand,

shoulder, etc. Add smoothing to skeleton data Add user guidance on his/her position based ClippedEdges

05/06/23EEC492/693/793 - iPhone Application

Development 21