Upload
denis-perevalov
View
220
Download
0
Embed Size (px)
Citation preview
8/8/2019 Computer Vision as universal controller for interactive media
1/50
Computer Vision as universal controller for interactive media
Denis Perevalov
[email protected]: www.youtube.com/perevalovds#g/u
lections: www.uralvision.blogspot.comInstitute of Mathematics and Mechanics, Ural Branch of Russian Academy of Sciences
http://www.instablogsimages.com/images/2008/01/04/aperture-interactive-display_48.jpg
http://www.uralvision.blogspot.com/http://www.youtube.com/perevalovds#g/umailto:[email protected]8/8/2019 Computer Vision as universal controller for interactive media
2/50
Contents
1. What is controller2. Computer vision as universal controller3. Video cameras for computer vision4. Controllers based on computer vision I
5. Controllers based on computer vision II6. Programming technologies7. The prospects
8/8/2019 Computer Vision as universal controller for interactive media
3/50
1. What is controller
8/8/2019 Computer Vision as universal controller for interactive media
4/50
Definition of controller
Controller is any sensor, any source of data, which can be converted into digitalsignal.
Proximity sensor
8/8/2019 Computer Vision as universal controller for interactive media
5/50
Standard musical controllers
Midi padMidi keyboard
Midi track control
Standard instrumentswith MIDI outputBreath controller
Actually, these are musical instruments or tools for intruments, which output digidal dataabout the sound instead of sound.
8/8/2019 Computer Vision as universal controller for interactive media
6/50
Controllers of human motion
Biosensors
Motion Capture
http://neurocenter.unige.ch/groups/pun.php
Multitouch
8/8/2019 Computer Vision as universal controller for interactive media
7/50
Controllers of physical phenomenas
Waves on the waterAleatoric water musical instrumenthttp://www.youtube.com/watch?v=CZ_KijiwQHE
8/8/2019 Computer Vision as universal controller for interactive media
8/50
Stochastical controllers
Internet, data frommarket
Insects trajectories
"Debug - art by insects"http://vimeo.com/12645870
8/8/2019 Computer Vision as universal controller for interactive media
9/50
2. Computer vision asuniversal controller
8/8/2019 Computer Vision as universal controller for interactive media
10/50
What is computer vision
Computer vision is a branch of Computer Sciences, which studyautomatic image analysis, with images received from digital video andphoto cameras.
8/8/2019 Computer Vision as universal controller for interactive media
11/50
Examples of tasks, which can be solved using computer vision:
SegmentationAnalysis of motion(optical flow)
What is computer vision
8/8/2019 Computer Vision as universal controller for interactive media
12/50
Examples of tasks, which can be solved using computer vision:
Object tracking andmeasurement
Motion Capture
http://armi.kaist.ac.kr/korean/UserFiles/File/MMPC.JPG
What is computer vision
8/8/2019 Computer Vision as universal controller for interactive media
13/50
Computer vision as universal controller
Any parameters of the physical processes, which is realized in mechanicalmotion, shape or color or transparency change- can be digitized using computer vision.
8/8/2019 Computer Vision as universal controller for interactive media
14/50
Difference of computer vision from other sensor types:1) possibility to use huge amount of data (pixel array).
2) possibility to structurize this data, by extracting from the
image the needed parameters (for example, position and sizeof the objects).
Computer vision as universal controller
8/8/2019 Computer Vision as universal controller for interactive media
15/50
Limits of the controllers based oncomputer vision
1. Because of huge amount of data, such controllers normallywork with big latency.
2. Because of cameras see objects at a distance, a precisionof measuremens can be not enough.
3. The light is needed for cameras. (Visible or infrared, butneeded anyway).
(We'll see it in more details below)
8/8/2019 Computer Vision as universal controller for interactive media
16/50
3. Video cameras forcomputer vision
8/8/2019 Computer Vision as universal controller for interactive media
17/50
Camera's basic characteristics
For tasks of image analysis in realtimedifferent kinds of cameras can be needed.
The basic characteristics are:
1. Resolution2. Number of frames per second
3. Type of output data (visible, IR, 3d)
8/8/2019 Computer Vision as universal controller for interactive media
18/50
Resolution
This is dimensions of the camera image in pixels.
https://reader009.{domain}/reader009/html5/0
320 x 240precision for object size 1
meter:3.13
size of 30 frames:6.6 Mb
640 x 480
1.56
26.4
1280 x 1024
0.97
112.5
8/8/2019 Computer Vision as universal controller for interactive media
19/50
Number of frame per second
30 FPS
time between two frames33 msec
60 FPS
16 msec
150 FPS
6 msecCan be used for musical
instrument
http://www.youtube.com/watch?v=7iEvQIvbn8o
8/8/2019 Computer Vision as universal controller for interactive media
20/50
Type of output data
Color or graytone imageof visible light
Infrared image
Using invisible IR-light, suchcamera wil see during dark
performance conditions
Color image + depthinformation
8/8/2019 Computer Vision as universal controller for interactive media
21/50
Examples
Sony PS3 Eye
320 x 240 : 150 FPS640 x 480 : 60 FPS
Data type:visible light,IR (you need to remove IR-filter)
Price: 50$.
USB, CCD
8/8/2019 Computer Vision as universal controller for interactive media
22/50
Examples
Point Grey Flea3
648 x 488 : 120 FPS
Data type:- visible light,
- IR (?)
Price: 600$.
Model FL3-FW-03S1C-C
IEEE 1394b, CCD
8/8/2019 Computer Vision as universal controller for interactive media
23/50
Examples
Microsoft Kinect
640 x 480 : 30 FPS
Data type:color image + depth
Price: 150$.(depth is received using special IR pattern,so will not work in direct sunlight)USB, CMOS
8/8/2019 Computer Vision as universal controller for interactive media
24/50
Examples
Point Grey BumbleBee2
640 x 480 : 48 FPS
Data type:color image + depth
Price: 2000$.
(Depth is receiving using stereovision from two cameras)IEEE 1394b, CCD
8/8/2019 Computer Vision as universal controller for interactive media
25/50
4. Controllers based oncomputer vision I
8/8/2019 Computer Vision as universal controller for interactive media
26/50
Sliders and buttons - by camera
- Camera can read slider position without electronics.- Your finger can be slider or button, camera will read itsposition.
8/8/2019 Computer Vision as universal controller for interactive media
27/50
Additive synthesizer using optical measurement of
brightness
8/8/2019 Computer Vision as universal controller for interactive media
28/50
Using two color markers for creating virtual picture
8/8/2019 Computer Vision as universal controller for interactive media
29/50
Giroscope using camera: AR
Augmented Reality with Markers6-dimensions: X, Y, Z, 3 axes on rotation
http://www.edhv.nl/edhv/wp-content/uploads/2009/12/aug_Picture-10_no-border-450x337.jpg
8/8/2019 Computer Vision as universal controller for interactive media
30/50
Multitouch
FTIR multitouch
http://www.touchuserinterface.com/2010/02/lcd-multi-touch-using-inverted-ftir.htmlhttp://sites.google.com/site/ideolabsdocumentation/images/multitouchdiagram.png
8/8/2019 Computer Vision as universal controller for interactive media
31/50
Virtual reality gloves
Color gloves using one webcamera only, in MIT (prototype)
http://www.csail.mit.edu/videoarchive/research/gv/hand-tracking
8/8/2019 Computer Vision as universal controller for interactive media
32/50
Motion Capture
Microsoft Kinect for XBox(Motion capture is available only for XBox developers yet. But using OpenKinect library inMac/Windows you can get color and depth images - it is really good for many types of
projects. For official opinion of Microsoft on OpenKinect see http://www.thinq.co.uk/2010/11/22/microsoft-declares-openkinect-safe/)
http://www.thinq.co.uk/2010/11/22/microsoft-declares-openkinect-safe/http://www.thinq.co.uk/2010/11/22/microsoft-declares-openkinect-safe/8/8/2019 Computer Vision as universal controller for interactive media
33/50
Conclusion
Not-camera sensors are more precise and rapid.
Computer vision merits:- fast implementstion for prototype
building- universal (one camera can be used for differents kinds ofsensors)
8/8/2019 Computer Vision as universal controller for interactive media
34/50
5. Controllers based oncoputer vision II
8/8/2019 Computer Vision as universal controller for interactive media
35/50
Motion areas detection
Result is coordinates of the areas with motion.
8/8/2019 Computer Vision as universal controller for interactive media
36/50
Optical flow calculation
Result is 2D-vector field of motion vectors.
8/8/2019 Computer Vision as universal controller for interactive media
37/50
Objects of interest detection
Result is coordinates and sizes of detected objects.
8/8/2019 Computer Vision as universal controller for interactive media
38/50
Non-visual structures on the image
- Sum of brightness of some special pixels on the image.
- Vector of special components of image Fast Fourier Transform.Such characteristics are "nonvisual", because have no direct relationwith areas on the image or some objects or its characteristics. Butthese are non-random and sometimes are useful.
8/8/2019 Computer Vision as universal controller for interactive media
39/50
Conclusion
Implementetion of such controllers using not-camera sensors isexpensive, and sometimes specific device constructing isneeded.
8/8/2019 Computer Vision as universal controller for interactive media
40/50
6. Programming technologies
8/8/2019 Computer Vision as universal controller for interactive media
41/50
8/8/2019 Computer Vision as universal controller for interactive media
42/50
Low-level libraries
"Open Graphics Library"
Open library for radip graphics, C/++.
8/8/2019 Computer Vision as universal controller for interactive media
43/50
Low-level libraries
"Open Computing Language"Open library for parallel computing, with GPU too, C/++.
Allows to speedup computings very much. OpenCV + OpenCL is preparinindevelopment stage now (by Intel).
8/8/2019 Computer Vision as universal controller for interactive media
44/50
Middle-level platforms
"Creative coding" platforms with large amount of functions forconvenient programming.
openFrameworksC/C++
ProcessingJavais too slow for computervision
CinderC/C++is young, but popularity isincreasing
f
8/8/2019 Computer Vision as universal controller for interactive media
45/50
High-level platforms
Visual programming platforms for musucians and designers.Almost no text programming needed. If needed, it can beextended by custom plugins, developed by programmers inlower-level platforms.
Max/MSP/JitterAudio-oriented
VVVVVideo effects -oriented
Quest3DHigh quality 3D -oriented
8/8/2019 Computer Vision as universal controller for interactive media
46/50
The prospects
T h l i l t
8/8/2019 Computer Vision as universal controller for interactive media
47/50
Technological prospects
- Manufacturing more models of cameras with FPS > 100 forusing it in live performance.
- Increasing resolution and processing speed for increasing
measurement precise in space and time scales.
A ith i l t
8/8/2019 Computer Vision as universal controller for interactive media
48/50
Agorithmical prospects
- Stable emotions and mimics recognition.
- 3D scenes with many objects recognition (using the cameraswith depth).
http://susiemander.files.wordpress.com/2010/10/facial-expression.jpg
Id t
8/8/2019 Computer Vision as universal controller for interactive media
49/50
Ideas prospects
1. Mutltitouch is not used as it should to be.2. How to use moton capture technology for effectiveperformance?
Id t
8/8/2019 Computer Vision as universal controller for interactive media
50/50
Ideas prospects
- Searching new and unusual processes, which can be visible
by camera.
- Searching new interesting structures in images and ordinalphenomenas.