Multimodal Interfaces

Multimodal InterfacesNovember, 2008Thomas Winningham

Multi ah-wha?

๏ Computer vision, mostly

๏ The Wiimote 100hz camera

๏ Capacitive surfaces (or lots of buttons)- IPhone & Mac touchpad- HP TouchSmart

๏ Tin Foil & an Arduino

๏ Force transducers, the Wii Fit

๏ Teaching new dogs old tricks

Regions of interest

๏ At 30 fps or faster:- Find a finger smudge (blobs)- Dots of light- Triangulate based on closest data

- Color keying- LEDs as two-way- Use IR for spectrum separation- Isolate features with laser- Map in 3D using Lidar

Better Software & Hardware

๏ DARPA Challenge- Intel OpenCV

๏ Multiprocessor Machines

๏ Home clusters of machines

๏ GPU processors on videocards

๏ Better web cams

๏ Cheaper, more widely available materials

๏ Easier access to information

Fresh Ways to Interface with Computers

It is not a panacea, but at least everyone is mostly invited.

Leave it to the big boys?

๏ IPhone's - And knock-offs

๏ Microsoft Surface & Windows 7 & Tablets- Microsoft Research

๏ HP TouchSmart

๏ Perceptive Pixel

๏ NUI Group - step 1. cut a hole in a box

Gesturing

๏ Combos!

๏ Changes over time

๏ Inferring about the placement of input in orientation to the real world- Surface- Wii fit- You're holding a nail near tinfoil

Simple!

๏ Or at least, reasonably simple given the complexity of what is going on.

๏ Small blocks of code in repition

๏ Again, faster computers

Web Cam in a Box

๏ Step #2- Put your junk in that box

Double true!

๏ Stackable cubes

๏ Large walls

๏ Small hobby projects

๏ Multiple cameras

๏ GPUs

๏ Networked machines

๏ Store Windows, floors , buildings

Mostly flash

๏ (needs a lot of help from c)

๏ But also:- Python- Processing- Max- VVVV- Pure Data- .NET- Firefox / Javascript (Lily)

Rethinking a lot of things

๏ Gesture navigation

๏ User Interface

๏ Feedback

Guerrilla Arm

All kinds of things I tell ya!

๏ Sphere

๏ Gesturing with your hands above the home row

๏ Or just using your hands

๏ Or moving your head, or eyes, or thoughts

๏ This is an extra bullet point. Ha!

Meanwhile, back at the bat cave...๏ Home clustering

๏ Semistructured web and public data

๏ HDTV's, projectors, err... iphones

๏ Social connectivity- Except in the workplace where we'd generally

like everyone to politely sod off

๏ Advanced modeling, free software for fluid dynamics and physics simulation

Lack of standards?

lack of patent free ideas, maybe

TUIO / OSC for ROI or Fiducials

Romper room

๏ I see:- NUI Group- Open Frameworks- Processing- Apple Developer forums- IPhone jailbreakers- Multi-pointer X-Windows, Compiz, and the

whole open-source world.

Feedback loop

๏ Proliferation? Adoption?

๏ Social implications

๏ This will all spawn something else

๏ Everybody's coming to the net

๏ Voluntary mass hallucination (Gibson)

Come out and play!If yer under eighteen ya won't be doin any ty-ee-eye-ee-ime

There's a place for us!

๏ More people playing!- == better software

๏ Not as hard as it sounds, just steal code ;p

๏ Who's getting laid anyway?

๏ It is amazing what you can get done if you don't care who gets the credit. (jets ftw!)