Upload
jun-iio
View
181
Download
0
Embed Size (px)
Citation preview
An Introduction to NUI
NUI programming seminar
Jun Iio, Ph.D.Chuo University
An introduction to NUI Copyright © Jun Iio 2
Self-introduction● Prof. Jun Iio, Ph.D.
Dept. of Socio-Informatics, Faculty ofLetters, Chuo University
● 1994-2013, Working for Mitsubishi ResearchInstitute
● 2013-2014, Assoc. Prof. of Chuo University● 2014-Current, Prof. of Chuo University● Also, 2009-, Guest Assoc. Prof. of Tokyo University
of Agriculture and Technology● My interests are
– Software Engineering, Image Processing,Open-Source Software, User Interface, etc.
An introduction to NUI Copyright © Jun Iio 3
What is NUI?● NUI stands for Natural User Interface
● Improvement of the User Interface?– CUI -> GUI -> NUI?
● Let's look back on the history of UI...
An introduction to NUI Copyright © Jun Iio 4
The first step of data input methodsfor computers
source: Wikimedia
An introduction to NUI Copyright © Jun Iio 5
Emerging personal computers
source: Wikimedia (CC BY-SA 2.0 fr)
Commodore PET 2001 computer. On display at the Musée Bolo, EPFL, Lausanne. The making of this document was supported by Wikimedia CH.
An introduction to NUI Copyright © Jun Iio 6
CUI(Character User Interface)
An introduction to NUI Copyright © Jun Iio 7
GUI(Graphical User Interface)
An introduction to NUI Copyright © Jun Iio 8
WIMP
W … WindowsI … IconsM … MenusP … Pointers
An introduction to NUI Copyright © Jun Iio 9
Physical peripherals● Keyboard● Pointing devices
– Mouse– Track-point– Track-ball– Touch pad– Touch panel
An introduction to NUI Copyright © Jun Iio 10
Novel operations● Tap
– Double tap– Tap and hold
● Flick, drag● Pinch
– Pinch in– Pinch out
● Shake, bump(!)– (Grab) source: http://bump.softonic.jp/iphone
An introduction to NUI Copyright © Jun Iio 11
Elemental technologies● Contact-type
– Touch interface● Multi touch, touch display, pressure-sensitive input● (Image input, acceleration sensor)
● Contactless-type– Space devices
● Image input, three dimensional measurement– Gesture interface
● Human body modeling, inverse kinematics● Acceleration sensor (motion, direction, inclination)● Muscle potential measurement
An introduction to NUI Copyright © Jun Iio 12
An impact of Windows 8
source: http://fesoku.net/archives/6029501.html
An introduction to NUI Copyright © Jun Iio 13
Multi-touch● http://kivy.org/
An introduction to NUI Copyright © Jun Iio 14
TUIO● Tangible User Interface Objects
An introduction to NUI Copyright © Jun Iio 15
CCV(Community Core Vision)● Poorman's multi-touch device? Video
An introduction to NUI Copyright © Jun Iio 16
3D mouse
An introduction to NUI Copyright © Jun Iio 17
Kinect
An introduction to NUI Copyright © Jun Iio 18
OpenNI
An introduction to NUI Copyright © Jun Iio 19
Leap MotionVideo
An introduction to NUI Copyright © Jun Iio 20
MyoVideo
An introduction to NUI Copyright © Jun Iio 21
RingVideo
An introduction to NUI Copyright © Jun Iio 22
Gesture interface● Maximum benefit
– “Direct manipulation”– It enables to control the target directly
source: ”Make It So”
An introduction to NUI Copyright © Jun Iio 23
Note for the NUI designs● Natural User Interface development is still
in progress● Considerable features:
1. How to realize input methods which can notmake users tired?
2. “Mode” problem of gesture input3. Haptic information (feedbacks of user's
operation)● Learning from SciFi movies
– What are applications using NUI efficiently?
An introduction to NUI Copyright © Jun Iio 24
References“Make It So” (Japanese Translation)
An introduction to NUI Copyright © Jun Iio 25
“Minority Report”● A “prototype meter” for the gesture
interface● Lessons learned:
– A grate demo hide many flaws– A gesture interface should understand intent
source: ”Make It So”
An introduction to NUI Copyright © Jun Iio 26
Enthusiasts for keyboards
source: http://www.pfu.fujitsu.com/hhkeyboard/hhkbpro/black.htmlsource: http://www.kmecs.com/products/detail_267.html
An introduction to NUI Copyright © Jun Iio 27
Haptic devices● An input was completed? or not?● Why does this keyboard make user feel
unpleasant?
source: http://www.itmedia.co.jp/pcuser/articles/1204/06/news033.html
An introduction to NUI Copyright © Jun Iio 28
Haptic devices● It enables the
feedbacks of user'sinput by touchinformation such asvibration, resistantforce, and/ormotion.
● Cf. “Myo uses apattern of vibrationas the notice foruser's input”
source: http://lab.sekaimon.com/index/index/c/58058/ym/201201/p/2/ajax/1
An introduction to NUI Copyright © Jun Iio 29
Grammar for the gesture input
source: Leap Motion SDK API document
An introduction to NUI Copyright © Jun Iio 30
Grammar for the gesture input
source: https://support.getmyo.com/hc/en-us/articles/201135025-Controlling-media-with-your-Myo-armband
An introduction to NUI Copyright © Jun Iio 31
Hollywood's pidgin● Wave to activate● Push to move● Turn to rotate● Swipe to dismiss● Point or touch to select● Extend the hand to shoot● Pinch and spread to scale
An introduction to NUI Copyright © Jun Iio 32
Wave to activate
source: ”Make It So”
An introduction to NUI Copyright © Jun Iio 33
Push to move
source: ”Make It So”
An introduction to NUI Copyright © Jun Iio 34
Turn to rotate
source: ”Make It So”
An introduction to NUI Copyright © Jun Iio 35
Swipe to dismiss
source: ”Make It So”
An introduction to NUI Copyright © Jun Iio 36
Point or touch to select
source: ”Make It So”
An introduction to NUI Copyright © Jun Iio 37
Extend the hand to shoot
source: ”Make It So”
An introduction to NUI Copyright © Jun Iio 38
Pinch and spread to scale
source: ”Make It So”
An introduction to NUI Copyright © Jun Iio 39
Be careful if you propose a novelgesture out of the seven patterns
● Another patter canlead usersmisunderstandingor confusing
● Considering globalrules and local rule– Gestures in
Japanese culture– Gestures in another
country's culture
source: Fuji television
An introduction to NUI Copyright © Jun Iio 40
Dependency of local cultures
source: http://matome.naver.jp/odai/2133536386030621501
An introduction to NUI Copyright © Jun Iio 41
Dependency of local cultures
source: http://matome.naver.jp/odai/2133536386030621501
An introduction to NUI Copyright © Jun Iio 42
An application of Kinect● Y. Nakano, T.
Tohyama, and S.Kanemune,“Comparison ofJapanese InputMethods for Kinect” Inproc. of IPSJ-SSS2013, 173-175.– https://www.faceboo
k.com/media/set/?set=a.183589891823257.1073741846.172555706260009
An introduction to NUI Copyright © Jun Iio 43
Applications using NUI● It should be considered what scene
requires NUI fundamentally?
● In the case of promotion videos for Myo:– Controlling drone, sphero, and UMV...
“It would be enough to use a controller!”
● It is suitable for the situations where auser cannot use his/her own hands– Controlling recipe video when cooking– tele-surgery
An introduction to NUI Copyright © Jun Iio 44
Future of interfaces● NUI does not replace {CUI, GUI}
– Programming using NUI? -> a distant idea
Practical
Abstract
Professionaluse
Generalpurpose
CUI
GUI
NUI
An introduction to NUI Copyright © Jun Iio 45
Conclusions● Progress of interacting user and computer:
CUI -> GUI -> NUI– NUI study is the blue ocean and in progress
● Elemental technologies for NUI– multi-touch, 3D measurement, acceleration
sensors, muscle potential measurement● How to solve some problems in gesture UI
– Tiredness, mode-problem, haptic interface● How to make a grammar for gesture UI
– Not only hardware but also cultural constraintsshould be considered!