Smart Homes Made Smarter

With JSJoel LordGuidebook app:

NEPHP2016

www.spiria.com

Robocalypse: Controlling Nodebots with a Kinect

Presented ByJOEL LORD

North East PHP 2016, Charlottetown, PE

August 5th, 2016

@joel__lord#NEPHP

JOEL LORDAbout me, eh?

• Javascript Junkie• Technology enthusiast• Tinkerer

@joel__lord#NEPHP

AGENDA WhyThe GoalsWhat ExistsWhat I wantedGestures 101Demo

@joel__lord#NEPHP

MY NODEBOTSay hello to my little friend

Why?CONTROLLING A ROBOT WITH A KINECT

@joel__lord#NEPHP

Why

• Worked with Nodebots for a while now

@joel__lord

@joel__lord#NEPHP

Why

• Worked with Nodebots for a while now• Wanted to explore other possibilities of NodeJs

@joel__lord

@joel__lord#NEPHP

Why

• Worked with Nodebots for a while now• Wanted to explore other possibilities of NodeJs• Because controlling a robot with a Kinect is pretty cool

@joel__lord

@joel__lord#NEPHP

Why

• Worked with Nodebots for a while now• Wanted to explore other possibilities of NodeJs• Because controlling a robot with a Kinect is pretty cool• But mostly… for fame !

@joel__lord

GoalsCONTROLLING A ROBOT WITH A KINECT

@joel__lord#NEPHP

Goals

• Needed a way to use events to tell the robot to perform an action

@joel__lord

@joel__lord#NEPHP

Goals

• Needed a way to use events to tell the robot to perform an action

• Gestures seemed the best way to achieve this

@joel__lord

What already existsCONTROLLING A ROBOT WITH A KINECT

@joel__lord#NEPHP

The Nodebot Stack

• NodeJs (https://nodejs.org/)

@joel__lord

@joel__lord#NEPHP

The Nodebot Stack

• NodeJs (https://nodejs.org/)• Socket.io (http://socket.io/)

@joel__lord

@joel__lord#NEPHP

The Nodebot Stack

• NodeJs (https://nodejs.org/)• Socket.io (http://socket.io/)• Johnny-Five (http://johnny-five.io/)

@joel__lord

A little parenthesisJOHNNY-FIVE?

@joel__lord#NEPHP

Johnny-Five

• Uses Javascript to control an Arduino (or other boards) via Standard Firmata

• Also works with a bunch of platforms like Tessel, Raspberry Pi, Particle

• Uses a standard API for all the platforms

@joel__lord

@joel__lord#NEPHP

Johnny-Five’s Hello World

var five = require("johnny-five");var board = new five.Board();

board.on("ready", function() {

// Create a standard `led` // component instance var led = new five.Led(13);

// "blink" the led in 500ms // on-off phase periods led.blink(500);});

@joel__lord

Back to our regular programming

@joel__lord#NEPHP

OpenNI and node-openni

• Connects the Kinect to NodeJs using OpenNI

@joel__lord

@joel__lord#NEPHP

OpenNI and node-openni

• Connects the Kinect to NodeJs using OpenNI• Very basic event for almost everything

@joel__lord

@joel__lord#NEPHP

OpenNI and node-openni

• Connects the Kinect to NodeJs using OpenNI• Very basic event for almost everything• No gestures !

@joel__lord

What I neededCONTROLLING A ROBOT WITH A KINECT

@joel__lord#NEPHP

What I needed

• Support for gestures

@joel__lord

@joel__lord#NEPHP

What I needed

• Support for gestures• Support for a full « skeleton »

@joel__lord

Gestures 101CONTROLLING A ROBOT WITH A KINECT

@joel__lord#NEPHP

Gestures 101

• Is the initial condition met?

@joel__lord

@joel__lord#NEPHP

Gestures 101

• Is the initial condition met?• Is the condition still met?

@joel__lord

@joel__lord#NEPHP

Gestures 101

• Is the initial condition met?• Is the condition still met?• Is the final condition met?

@joel__lord

@joel__lord#NEPHP

Gestures 101 - Needs

• A skeleton with an event when it changes

@joel__lord

@joel__lord#NEPHP

Gestures 101 - Needs

• A skeleton with an event when it changes• Know the position of the « COM »

@joel__lord

@joel__lord#NEPHP

Gestures 101 - Needs

• A skeleton with an event when it changes• Know the position of the « COM »• Have a « base unit »

@joel__lord

@joel__lord#NEPHP

Gestures 101 – Introducing kinect-gestures

• Gesture detection using a base class

@joel__lord

@joel__lord#NEPHP

Gestures 101 – Introducing kinect-gestures

• Gesture detection using a base class• Tracks the skeleton for an initial condition

@joel__lord

@joel__lord#NEPHP

Gestures 101 – Introducing kinect-gestures

• Gesture detection using a base class• Tracks the skeleton for an initial condition• Checks every 100ms to see if the condition is still met

@joel__lord

@joel__lord#NEPHP

Gestures 101 – Introducing kinect-gestures

• Gesture detection using a base class• Tracks the skeleton for an initial condition• Checks every 100ms to see if the condition is still met• If the final condition is met, trigger an event

@joel__lord

@joel__lord#NEPHP

Gestures 101 – Introducing kinect-gestures

• Gesture detection using a base class• Tracks the skeleton for an initial condition• Checks every 100ms to see if the condition is still met• If the final condition is met, trigger an event• Other classes are used to define the actual gestures

@joel__lord

IT’S CODING TIME!LET’S GET SERIOUS

IT’S CODING TIME!LET’S GET SERIOUS

@joel__lord#NEPHP

Next few steps

• 05/01/2023

@joel__lord

• 42

• Return a more friendly coordinate system for skeleton• Add more information to the returned event (velocity?)• Add more gestures (jump, wave, kick)

DOCUMENT CONFIDENTIEL, TOUT DROIT RÉSERVÉ

PRESENTED BY

The End !

Questions ?

JOEL LORDAugust 5th 2016

TWITTER: @JOEL__LORDGITHUB: HTTP://GITHUB.COM/JOELLORD