EEC-492/592EEC-492/592Kinect Application Kinect Application

DevelopmentDevelopmentLecture 16Lecture 16

Wenbing ZhaoWenbing Zhao

wenbing@ieee.orgwenbing@ieee.org

OutlineOutline Algorithmic gesture recognition overview Swipe to left gesture Object-oriented new design for recognition

engine Implementation details Build an app to test

Algorithmic Gesture Recognition Uses a set of predefined conditions and parameters to

detect and validate a gesture against each of them Validate a gesture as it is being performed, by ensuring the

start points, constraints, parameters, and the end points are always valid

The algorithmic approach not only recognizes the gestures, but it also tracks if the gesture is performed correctly or not

Examples gestures which can be considered as algorithmic Hand moving in the same direction A swipe to the right or the left Zooming in and out Waving hands

Algorithmic Gesture Recognition StartStart ConditionCondition ValidationValidation FinishFinish

SwipeToLeft Example

Start: The left hand joint should be below the left elbow and the spine

joint The right hand joint should be below the right shoulder joint and

above the right elbow joint

Condition & validation The user should move the hand from right to left while maintaining

the right hand and left hand joint positions

Finish: After a specific number of frames when the gesture reaches to

validate the last condition, it will check if the distance between the right hand joint and the left shoulder has reduced from the starting point

Implementation of Algorithmic Gesture Recognition

GestureType:

Event argument

public enum GestureType{ SwipeToRight, SwipeToLeft, ZoomIn, ZoomOut}

public class GestureEventArgs : EventArgs{ public RecognitionResult Result { get ; internal set; } public GestureType GestureType { get; internal set; } public GestureEventArgs(RecognitionResult result, GestureType type) { this.Result = result; this.GestureType = type; }}

Implementation of Algorithmic Gesture Recognition

Implementation of Algorithmic Gesture Recognition

Implementation of Algorithmic Gesture Recognition Add a new C# file called GestureBase.csAdd a new C# file called GestureBase.cs

Common operation for all types of gestures suitable Common operation for all types of gestures suitable for algorithmic gesture recognitionfor algorithmic gesture recognition

using Microsoft.Kinect;public abstract class GestureBase { public GestureBase(GestureType type) { this.CurrentFrameCount = 0; this.GestureType = type; } public bool IsRecognitionStarted { get; set; } private int CurrentFrameCount { get; set; } public GestureType GestureType { get; set; } protected virtual int MaximumNumberOfFrameToProcess { get { return 15; } }

GestureBase.cspublic long GestureTimeStamp { get; set; }protected abstract bool ValidateGestureStartCondition(Skeleton skeleton);protected abstract bool ValidateGestureEndCondition(Skeleton skeleton);protected abstract bool ValidateBaseCondition(Skeleton skeleton);protected abstract bool IsGestureValid(Skeleton skeleton);

GestureBase.cspublic virtual bool CheckForGesture(Skeleton skeleton) { if (this.IsRecognitionStarted == false) { if (this.ValidateGestureStartCondition(skeleton)) { this.IsRecognitionStarted = true; this.CurrentFrameCount = 0; } } else { if (this.CurrentFrameCount == this.MaximumNumberOfFrameToProcess) { this.IsRecognitionStarted = false; if (ValidateBaseCondition(skeleton) && ValidateGestureEndCondition(skeleton)) { return true; } } this.CurrentFrameCount++; if (!IsGestureValid(skeleton) && ! ValidateBaseCondition(skeleton)) { this.IsRecognitionStarted = false; } } return false;}

SwipeToLeftGesture.csusing Microsoft.Kinect;public class SwipeToLeftGesture : GestureBase { // intermediate right hand position used for validation of the gesture private SkeletonPoint validatePosition;

// starting right hand position when the gesture start condition is met (starting pose) private SkeletonPoint startingPostion;

// distance between the hand right and the left shoulder private float shoulderDiff;

// constructor public SwipeToLeftGesture() : base(GestureType.SwipeToLeft) { }

// check to see if the starting pose is seen // called for every skeleton frame received protected override bool ValidateGestureStartCondition(Skeleton skeleton) { }

// …}

SwipeToLeftGesture.csprotected override bool ValidateGestureStartCondition(Skeleton skeleton) { var handRightPoisition = skeleton.Joints[JointType.HandRight].Position; var handLeftPosition = skeleton.Joints[JointType.HandLeft].Position; var shoulderRightPosition = skeleton.Joints[JointType.ShoulderRight].Position; var spinePosition = skeleton.Joints[JointType.Spine].Position;

// Starting pose: // right hand lower than right should && right hand higher than right elbow // && left hand lower than spine if ((handRightPoisition.Y < shoulderRightPosition.Y) && (handRightPoisition.Y > skeleton.Joints[JointType.ElbowRight].Position.Y) && handLeftPosition.Y < spinePosition.Y) { shoulderDiff = GestureHelper.GetJointDistance(skeleton.Joints[JointType.HandRight],

skeleton.Joints[JointType.ShoulderLeft]); validatePosition = skeleton.Joints[JointType.HandRight].Position; startingPostion = skeleton.Joints[JointType.HandRight].Position; return true; } return false; }

SwipeToLeftGesture.cs// called for every skeleton frameprotected override bool IsGestureValid(Skeleton skeletonData){ // current right hand position var currentHandRightPoisition = skeletonData.Joints[JointType.HandRight].Position; // current right hand should be on the left of the previous right hand position, // i.e., the right hand is moving to the left if (validatePosition.X < currentHandRightPoisition.X) { // if the right hand is moving to the right, stop doing gesture recognition return false; } // update the validatePosition using the current right hand position validatePosition = currentHandRightPoisition;

// gesture so far so good return true;}

SwipeToLeftGesture.cs// check if the final pose has reachedprotected override bool ValidateGestureEndCondition(Skeleton skeleton){ // distance between the staring right hand position and // the last right hand position double distance = Math.Abs(startingPostion.X - validatePosition.X); // the distance between the current right hand and the left shoulder float currentshoulderDiff = GestureHelper.GetJointDistance(skeleton.Joints[JointType.HandRight], skeleton.Joints[JointType.ShoulderLeft]);

// the right hand has moved for 0.1m since its starting position and // the right hand is getting closer to the left shoulder => we are done! if (distance > 0.1 && currentshoulderDiff < shoulderDiff) return true;

// otherwise, the right hand has not moved enough distance yet return false;}

SwipeToLeftGesture.csprotected override bool ValidateBaseCondition(Skeleton skeleton){ var handRightPoisition = skeleton.Joints[JointType.HandRight].Position; var handLeftPosition = skeleton.Joints[JointType.HandLeft].Position; var shoulderRightPosition = skeleton.Joints[JointType.ShoulderRight].Position; var spinePosition = skeleton.Joints[JointType.Spine].Position;

// right hand is to the left of the right shoulder, and // right hand is higher than right elbow, and // left hand is lower than spine if ((handRightPoisition.Y < shoulderRightPosition.Y) && (handRightPoisition.Y > skeleton.Joints[JointType.ElbowRight].Position.Y) && (handLeftPosition.Y < spinePosition.Y)) { // swipe to the left is ongoing, so far so good return true; } // condition is not met, terminate return false;}

GestureRecognitionEngine.cs

class GestureRecognitionEngine{ int SkipFramesAfterGestureIsDetected = 0; public event EventHandler<GestureEventArgs> GestureRecognized; public GestureType GestureType { get; set; } public Skeleton Skeleton { get; set; } public bool IsGestureDetected { get; set; } // list of gestures to be detected private List<GestureBase> gestureCollection = null; public GestureRecognitionEngine() { this.InitilizeGesture(); } …..}

Add a new C# file called GestureRecognitionEgine.cs to the project Resembles the previous one, but uses inheritance Add GestureType, RecogntionResult, GestureEventArgs to the new

file, or three separate files

GestureRecognitionEngine.csprivate void InitilizeGesture(){ this.gestureCollection = new List<GestureBase>(); //this.gestureCollection.Add(new ZoomInGesture()); //this.gestureCollection.Add(new ZoomOutGesture()); //this.gestureCollection.Add(new SwipeToRightGesture()); // add SwipeToLeftGesture recognizer to the list this.gestureCollection.Add(new SwipeToLeftGesture());}

// reset data structures for a new round of gesture recognitionprivate void RestGesture(){ this.gestureCollection = null; this.InitilizeGesture(); this.SkipFramesAfterGestureIsDetected = 0; this.IsGestureDetected = false;}

GestureRecognitionEngine.cspublic void StartRecognize() { if (this.IsGestureDetected) { // create a short break when we are done one round of gesture recognition while (this.SkipFramesAfterGestureIsDetected <= 30) { this.SkipFramesAfterGestureIsDetected++; } // reset our data structures for a new round of gesture recognition this.RestGesture(); return; } // perform gesture recognition for every gesture recognizer in our list foreach (var item in this.gestureCollection) { if (item.CheckForGesture(this.Skeleton)) { if (this.GestureRecognized != null) { // fire a gesture event when a gesture is recognized this.GestureRecognized(this, new GestureEventArgs(RecognitionResult.Success, item.GestureType)); this.IsGestureDetected = true; } }}

GestureHelper.csGestureHelper.cs

public static float GetJointDistance(Joint firstJoint, Joint secondJoint){ float distanceX = firstJoint.Position.X - secondJoint.Position.X; float distanceY = firstJoint.Position.Y - secondJoint.Position.Y; float distanceZ = firstJoint.Position.Z - secondJoint.Position.Z; return (float)Math.Sqrt(Math.Pow(distanceX, 2) + Math.Pow(distanceY, 2) + Math.Pow(distanceZ, 2));}

Added a GestureHelper.cs file to your projectAdded a GestureHelper.cs file to your project The class has only the following static method

Build a Gesture Recognition Build a Gesture Recognition App App for SwipeToLeft Gesturefor SwipeToLeft Gesture User interfaceUser interface

Canvas

TextBox

Image

Build a Gesture Recognition Build a Gesture Recognition AppApp Add member variablesAdd member variables

Modify constructorModify constructor

KinectSensor sensor;private WriteableBitmap colorBitmap;private byte[] colorPixels;Skeleton[] totalSkeleton = new Skeleton[6];Skeleton skeleton;

GestureRecognitionEngine recognitionEngine;

public MainWindow(){ InitializeComponent(); Loaded += new RoutedEventHandler(WindowLoaded);}

Build a Gesture Recognition Build a Gesture Recognition AppAppprivate void WindowLoaded(object sender, RoutedEventArgs e){ if (KinectSensor.KinectSensors.Count > 0) { this.sensor = KinectSensor.KinectSensors[0]; if (this.sensor != null && !this.sensor.IsRunning) { this.sensor.Start(); this.sensor.ColorStream.Enable(); this.colorPixels = new byte[this.sensor.ColorStream.FramePixelDataLength]; this.colorPixels = new WriteableBitmap(this.sensor.ColorStream.FrameWidth, this.sensor.ColorStream.FrameHeight, 96.0, 96.0, PixelFormats.Bgr32, null); this.image1.Source = this.colorBitmap; this.sensor.ColorFrameReady += this.colorFrameReady; this.sensor.SkeletonStream.Enable(); this.sensor.SkeletonFrameReady += skeletonFrameReady; recognitionEngine = new GestureRecognitionEngine(); recognitionEngine.GestureRecognized += gestureRecognized; } } }

Build a Gesture Recognition Build a Gesture Recognition AppApp Gesture recognized event handlerGesture recognized event handler

colorFrameRead(), DrawSkeleton(), drawBone(), colorFrameRead(), DrawSkeleton(), drawBone(), ScalePosition() same as beforeScalePosition() same as before

void gestureRecognized(object sender, GestureEventArgs e){ textBox1.Text = e.GestureType.ToString();}

Build a Gesture Recognition Build a Gesture Recognition AppApp Handle skeleton frame ready eventHandle skeleton frame ready eventvoid skeletonFrameReady(object sender, SkeletonFrameReadyEventArgs e) { canvas1.Children.Clear(); using (SkeletonFrame skeletonFrame = e.OpenSkeletonFrame()) { if (skeletonFrame == null) { return; } skeletonFrame.CopySkeletonDataTo(totalSkeleton); skeleton = (from trackskeleton in totalSkeleton where trackskeleton.TrackingState == SkeletonTrackingState.Tracked select trackskeleton).FirstOrDefault(); if (skeleton == null) return;

DrawSkeleton(skeleton); recognitionEngine.Skeleton = skeleton; recognitionEngine.StartRecognize(); }}

Challenge TasksChallenge Tasks Add recognition of zoom-in gestureAdd recognition of zoom-in gesture Start conditionStart condition

Right hand lower than right shoulder, left hand lower than right shoulderRight hand lower than right shoulder, left hand lower than right shoulder Right hand higher than hip centerRight hand higher than hip center Left hand higher than hip centerLeft hand higher than hip center Distance between two hands smaller than 0.5m. This distance is recorded in a variable Distance between two hands smaller than 0.5m. This distance is recorded in a variable

for comparison laterfor comparison later Check gesture for validityCheck gesture for validity

Current distance between two hands must be bigger than initial valueCurrent distance between two hands must be bigger than initial value End gesture conditionEnd gesture condition

Distance between two hands exceed 1.0mDistance between two hands exceed 1.0m

Base conditionBase condition Right hand lower than right shoulderRight hand lower than right shoulder Left hand lower than right shoulderLeft hand lower than right shoulder Right hand higher than hip centerRight hand higher than hip center Left hand higher than hip centerLeft hand higher than hip center