EEC-492/592EEC-492/592Kinect Application Kinect Application

DevelopmentDevelopmentLecture 14Lecture 14

Wenbing ZhaoWenbing Zhao

wenbing@ieee.orgwenbing@ieee.org

OutlineOutline Unity3D + ZDK

Creating a New Unity ProjectCreating a New Unity Project Open Unity, create a new

project, choose a location folder of your choice

Important ZDK plugin Go to Assets menu, select

“Import Package”, then “Custom Package…”

In the dialogue, find and select the ZDK plugin and click OK: ZDK_Unity40_1.1_trial.unitypackage

In the “Import package” dialogue, keep default selection and click Import button

Project Layout after ZDK is Project Layout after ZDK is importedimported

Scene ViewScene View Where you plan and execute your ideasWhere you plan and execute your ideas

To show the grid, toggle off the game overlay buttonTo show the grid, toggle off the game overlay button

Hierarchy ViewHierarchy View Shows what’s in the currently active scene. GameObjects that are dynamically added and removed from

the scene during runtime will appear here when they are active in the scene

By default, the maincamera is added

Constructing SceneConstructing Scene Add the following items to the scene:

Avatar: Dana (from ZDK) In Assets from Project panel, go to ZigFu folder, then

_Data folder Drag and drop Dana@t-pose_3 to the Hierarchy panel Rotation 180 degree so that it faces front

Floor Add a Cube game object, change its size to a thin layer

Scale: x=30, y=0.1, z=30 Change material to floor (imported from ZDK) Use Inspector, to go Mesh Renderer section, then materials

=> Element 0 => browse => choose Floor Directional light Empty object: to enable bootstrapping with ZDK

From GameObject menu, choose Create Empty Rename it to InitZDK (right click the object, then Rename)

Adjust main camera so that Dana almost fill the entire game view

Connecting Existing ScriptsConnecting Existing Scripts Change Project panel to one column Expand ZigFu folder, then expand Scripts folder and all subfolders Locate Zig script, drag and drop it to InitZDK in the Hierarchy panel Locate ZigDepthViewer script under the Viewers subfoler to InitZDK Locate ZigUsersRadar script under the Viewers subfoler to InitZDK Locate ZigEngageSingleUser script under the UserEngagers

subfoler to InitZDK Drag and drop Dana from the Hierarchy panel to Engaged Users

section under Zig Engage Singler User in the inspector Locate kinectSpecific script under the _Internal subfoler to InitZDK Locate ZigSkeleton script under the UserControls subfoler to Dana

After Connecting Existing After Connecting Existing ScriptsScripts

Connecting Joints for Dana the Connecting Joints for Dana the AvatarAvatar Expand the entire hierarchy of Dana mesh

Drag and drop matching joints from the Dana mesh in the hierarchy panel to the appropriate joints under Zig Skeleton section in the inspector Head => Head Left Hip => LeftUpLeg Neck => Neck Left Knee=> LeftLeg Torso => Spine1 Left Ankle => LeftFoot Waist => Spine Right Hip => RightUpLeg Left Shoulder => LeftArm Right Knee => RightLeg Left Elbow => LeftForeArmRight Ankle => RightFoot Left Wrist => Left Hand Right Shoulder => RightArm Right Elbow => RightForeArm Right Wrist => RightHand

Check the Mirror checkbox in the Dana inspector

Connecting Joints for Dana the Connecting Joints for Dana the AvatarAvatar

What Do the Scripts Do?What Do the Scripts Do? Zig.cs

Take user settings on what to update, smoothing, and smoothing parameters

Public member variables can be set via the inspector

public class Zig : MonoBehaviour { public ZigInputType inputType = ZigInputType.Auto; public ZigInputSettings settings = new ZigInputSettings(); public List<GameObject> listeners = new List<GameObject>(); public bool Verbose = true;

ZigDepthViewer.csZigDepthViewer.cs OnGUI(): called for rendering and handling GUI events static void DrawTexture(Rect position, Texture image,

ScaleMode scaleMode = ScaleMode.StretchToFill, bool alphaBlend = true, float imageAspect = 0); Position: Rectangle on the screen to draw the texture within. Image: Texture to display. scaleMode: How to scale the image when the aspect ratio of it

doesn't fit the aspect ratio to be drawn within. alphaBlend: Whether to apply alpha blending when drawing the

image (enabled by default). imageAspect: Aspect ratio to use for the source image

void OnGUI() { if (null == target) { GUI.DrawTexture(new Rect(Screen.width - texture.width - 10,

Screen.height - texture.height - 10, texture.width, texture.height), texture); }}

ZigDepthViewer.csZigDepthViewer.cs Texture: the visual and especially tactile quality of a

surface. Parent class for Texture2D Texture2D reference

http://docs.unity3d.com/Documentation/ScriptReference/Texture2D.html

You can also add a button

void OnGUI() { if (GUI.Button(new Rect(10, 10, 150, 100), "I am a button")) print("You clicked the button!"); }

ZigDepthViewer.csZigDepthViewer.cs Getting input from Kinect

void Zig_Update(ZigInput input) { if (UseHistogram) { UpdateHistogram(ZigInput.Depth); // ZigInput.Depth contains the depth data } else { depthToColor[0] = Color.black; for (int i = 1; i < MaxDepth; i++) { float intensity = 1.0f - (i/(float)MaxDepth); depthToColor[i].r = (byte)(BaseColor.r * intensity); depthToColor[i].g = (byte)(BaseColor.g * intensity); depthToColor[i].b = (byte)(BaseColor.b * intensity); depthToColor[i].a = 255; } } UpdateTexture(ZigInput.Depth);}

ZigDepthViewer.csZigDepthViewer.cs Update Depth Image

void UpdateTexture(ZigDepth depth){ short[] rawDepthMap = depth.data; int depthIndex = 0; int factorX = depth.xres / textureSize.Width; int factorY = ((depth.yres / textureSize.Height) - 1) * depth.xres; // invert Y axis while doing the update for (int y = textureSize.Height - 1; y >= 0 ; --y, depthIndex += factorY) { int outputIndex = y * textureSize.Width; for (int x = 0; x < textureSize.Width; ++x, depthIndex += factorX, ++outputIndex) { outputPixels[outputIndex] = depthToColor[rawDepthMap[depthIndex]]; } } texture.SetPixels32(outputPixels); texture.Apply();}

ZigUserRadar.cs: ZigUserRadar.cs: Track where the user isvoid OnGUI () {

if (!ZigInput.Instance.ReaderInited) return; int width = (int)((float)PixelsPerMeter * (RadarRealWorldDimensions.x / 1000.0f)); int height = (int)((float)PixelsPerMeter * (RadarRealWorldDimensions.y / 1000.0f)); GUI.BeginGroup (new Rect (Screen.width - width - 20, 20, width, height)); Color oldColor = GUI.color; GUI.color = boxColor; GUI.Box(new Rect(0, 0, width, height), "Users Radar", style); GUI.color = oldColor; foreach (ZigTrackedUser currentUser in ZigInput.Instance.TrackedUsers.Values) { // normalize the center of mass to radar dimensions Vector3 com = currentUser.Position; Vector2 radarPosition = new Vector2(com.x / RadarRealWorldDimensions.x, -com.z / RadarRealWorldDimensions.y); // X axis: 0 in real world is actually 0.5 in radar units (middle of field of view) radarPosition.x += 0.5f; radarPosition.x = Mathf.Clamp(radarPosition.x, 0.0f, 1.0f); radarPosition.y = Mathf.Clamp(radarPosition.y, 0.0f, 1.0f); Color orig = GUI.color; GUI.color = (currentUser.SkeletonTracked) ? Color.blue : Color.red; GUI.Box(new Rect(radarPosition.x * width - 10, radarPosition.y * height - 10, 20, 20), currentUser.Id.ToString()); GUI.color = orig; } GUI.EndGroup();}

ZigEngageSingleUser.csZigEngageSingleUser.cs Connect ZigInput to the avatar: that is why you must drag and

drop Dana to EngagedUsers field

public class ZigEngageSingleUser : MonoBehaviour { public bool SkeletonTracked = true; public bool RaiseHand; public List<GameObject> EngagedUsers;

void Start() { // make sure we get zig events ZigInput.Instance.AddListener(gameObject);}

void Zig_Update(ZigInput zig) { if (SkeletonTracked && null == engagedTrackedUser) { foreach (ZigTrackedUser trackedUser in zig.TrackedUsers.Values) { if (trackedUser.SkeletonTracked) { EngageUser(trackedUser); } } }}

ZigEngageSingleUser.csZigEngageSingleUser.csvoid EngageUser(ZigTrackedUser user) { if (null == engagedTrackedUser) { engagedTrackedUser = user; foreach (GameObject go in EngagedUsers) user.AddListener(go); SendMessage("UserEngaged", this, SendMessageOptions.DontRequireReceiver); }}

Component.SendMessage: void SendMessage(string methodName, object value = null,

SendMessageOptions options = SendMessageOptions.RequireReceiver);// Calls the method named methodName on every MonoBehaviour in this game object

kinectSpecific.cs: kinectSpecific.cs: Kinect specific settingsvoid OnGUI() {

longWord = GUI.TextField(new Rect(10, 10, 200, 30), readingAngle ? getAngle().ToString() : longWord, 20);

if (GUI.Button(new Rect(10, 40, 200, 30), "SetElevation")) { angle = int.Parse(longWord); NuiWrapper.NuiCameraElevationSetAngle(angle); t = new Thread(setAngle); //attempted a Paramaterized Thread to no avail t.Start(); Thread.Sleep(0); } readingAngle = GUI.Toggle(new Rect(10, 80, 200, 30), readingAngle, "Read Angle"); bool nNearMode = GUI.Toggle(new Rect(10, 160, 200, 20), NearMode, "Near Mode"); if (nNearMode != NearMode) { NearMode = nNearMode; ZigInput.Instance.SetNearMode(NearMode); } bool nSeatedMode = GUI.Toggle(new Rect(10, 190, 200, 20), SeatedMode, "Seated Mode"); bool nTrackSkeletonInNearMode = GUI.Toggle(new Rect(10, 220, 200, 20),

TrackSkeletonInNearMode, "Track Skeleton In NearMode"); if ((nSeatedMode != SeatedMode) || (TrackSkeletonInNearMode != nTrackSkeletonInNearMode)) { SeatedMode = nSeatedMode; TrackSkeletonInNearMode = nTrackSkeletonInNearMode; ZigInput.Instance.SetSkeletonTrackingSettings(SeatedMode, TrackSkeletonInNearMode); }}

ZigSkeleton.csZigSkeleton.cspublic class ZigSkeleton : MonoBehaviour{ public Transform Head; public Transform Neck; public Transform Torso; public Transform Waist; public Transform LeftCollar; public Transform LeftShoulder; public Transform LeftElbow; public Transform LeftWrist; public Transform LeftHand; public Transform LeftFingertip; public Transform RightCollar; public Transform RightShoulder; public Transform RightElbow; public Transform RightWrist; public Transform RightHand; public Transform RightFingertip; public Transform LeftHip; public Transform LeftKnee; public Transform LeftAnkle; public Transform LeftFoot; public Transform RightHip; public Transform RightKnee; public Transform RightAnkle; public Transform RightFoot;

public bool mirror = false; public bool UpdateJointPositions = false; public bool UpdateRootPosition = false; public bool UpdateOrientation = true; public bool RotateToPsiPose = false; public float RotationDamping = 30.0f; public float Damping = 30.0f; public Vector3 Scale = new Vector3(0.001f, 0.001f, 0.001f);

public Vector3 PositionBias = Vector3.zero;

private Transform[] transforms; private Quaternion[] initialRotations; private Vector3 rootPosition;

Quaternions are used to represent rotations, they are based on complex numbers and are not easy to understand intuitively

ZigSkeleton.csZigSkeleton.csZigJointId mirrorJoint(ZigJointId joint) { switch (joint) { case ZigJointId.LeftCollar: return ZigJointId.RightCollar; case ZigJointId.LeftShoulder: return ZigJointId.RightShoulder; case ZigJointId.LeftElbow: return ZigJointId.RightElbow; case ZigJointId.LeftWrist: return ZigJointId.RightWrist; case ZigJointId.LeftHand: return ZigJointId.RightHand; case ZigJointId.LeftFingertip: return ZigJointId.RightFingertip; case ZigJointId.LeftHip: return ZigJointId.RightHip; ….. // map right to left default: return joint; }}

ZigSkeleton.csZigSkeleton.cspublic void Awake() { int jointCount = Enum.GetNames(typeof(ZigJointId)).Length; transforms = new Transform[jointCount]; initialRotations = new Quaternion[jointCount];

transforms[(int)ZigJointId.Head] = Head; transforms[(int)ZigJointId.Neck] = Neck; transforms[(int)ZigJointId.Torso] = Torso; transforms[(int)ZigJointId.Waist] = Waist; …..

// save all initial rotations // NOTE: Assumes skeleton model is in "T" pose since all rotations are relative to that pose foreach (ZigJointId j in Enum.GetValues(typeof(ZigJointId))) { if (transforms[(int)j]) { // we will store the relative rotation of each joint from the gameobject rotation // we need this since we will be setting the joint's rotation (not localRotation) but we // still want the rotations to be relative to our game object initialRotations[(int)j] = Quaternion.Inverse(transform.rotation) * transforms[(int)j].rotation; } }}

ZigSkeleton.csZigSkeleton.csvoid Zig_UpdateUser(ZigTrackedUser user) { UpdateRoot(user.Position); if (user.SkeletonTracked) { foreach (ZigInputJoint joint in user.Skeleton) { if (joint.GoodPosition) UpdatePosition(joint.Id, joint.Position); if (joint.GoodRotation) UpdateRotation(joint.Id, joint.Rotation); } }}

void UpdateRoot(Vector3 skelRoot) { // +Z is backwards in OpenNI coordinates, so reverse it rootPosition = Vector3.Scale(new Vector3(skelRoot.x, skelRoot.y, skelRoot.z), doMirror(Scale))

+ PositionBias; if (UpdateRootPosition) { transform.localPosition = (transform.rotation * rootPosition); }}

Vector3.Scale(Vector3 a, Vector3 b): Multiplies two vectors component-wise.

ZigSkeleton.csZigSkeleton.csvoid UpdateRotation(ZigJointId joint, Quaternion orientation) { joint = mirror ? mirrorJoint(joint) : joint; // make sure something is hooked up to this joint if (!transforms[(int)joint]) { return; } if (UpdateOrientation) { Quaternion newRotation = transform.rotation * orientation * initialRotations[(int)joint]; if (mirror) { newRotation.y = -newRotation.y; newRotation.z = -newRotation.z; } transforms[(int)joint].rotation = Quaternion.Slerp(transforms[(int)joint].rotation,

newRotation, Time.deltaTime * RotationDamping); }}

void UpdatePosition(ZigJointId joint, Vector3 position) { joint = mirror ? mirrorJoint(joint) : joint; if (!transforms[(int)joint]) { return; } if (UpdateJointPositions) { Vector3 dest = Vector3.Scale(position, doMirror(Scale)) - rootPosition;

//Vector3.Lerp: Linearly interpolates between two vectors. transforms[(int)joint].localPosition = Vector3.Lerp(transforms[(int)joint].localPosition,

dest, Time.deltaTime * Damping); }}

static Quaternion Slerp(Quaternion from, Quaternion to, float t);// Spherically interpolates between from and to by t.