Upload
yuan-kai-wang
View
284
Download
1
Embed Size (px)
DESCRIPTION
This is an academic talk for professors and graduate students. In addition to introducing recent trends in embedded computer vision (ECV), I also present our research experience in ECV.
Citation preview
Wang, Yuan-Kai(王元凱) Electrical Engineering Department, Fu Jen University (輔仁大學電機工程系) Email: [email protected],
URL: http://www.ykwang.tw Slideshare: Search "Towards Embedded Computer Vision"
Towards Embedded Computer Vision
1
王元凱 Towards Embedded Computer Vision p.
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai
Context Awareness 2
人
時
物
地
事 Situation
High-Level Context Low-Level Contexts
王元凱 Towards Embedded Computer Vision p.
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai
3
MIT Oxygen
http://oxygen.lcs.mit.edu/ 1999
王元凱 Towards Embedded Computer Vision p.
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai
4
The Aware Home
http://awarehome.imtc.gatech.edu/
王元凱 Towards Embedded Computer Vision p.
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai
Ambient Intelligence v.s. Object Intelligence
Embedded system + Sensors
5
Wearable Computing Smart Home, Smart …
王元凱 Towards Embedded Computer Vision p.
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai
6 6
Embedded Systems "Without" Sensors
資料來源:資策會MIC ITIS計畫整理
王元凱 Towards Embedded Computer Vision p.
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai
7 7
Embedded Systems "With" Sensors
GPS Navigation
Wii Roomba
Exoskeleton
王元凱 Towards Embedded Computer Vision p.
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai
8
Driver assist Front view
Night vision
Blind corner
Blind spot
Rear view
Embedded Systems "With" Image Sensors
王元凱 Towards Embedded Computer Vision p.
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai
9 9
Embedded Systems "With" Image Sensors
Google Glass
王元凱 Towards Embedded Computer Vision p.
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai
Smart Watch 10
王元凱 Towards Embedded Computer Vision p.
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai
Microsoft SenseCam
Embedded Systems "With" Image Sensors
11
王元凱 Towards Embedded Computer Vision p.
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai
Observation Miniaturization
High Resolution
3D Capability
12
王元凱 Towards Embedded Computer Vision p.
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai
13
Embedded Computer Vision Embedded System + Camera
+ Computer Vision Algorithm
Image Processing
Image Recognition
Image Capturing
王元凱 Towards Embedded Computer Vision p.
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai
14
Embedded Computer Vision Embedded compute vision
Implements computer vision algorithms on low-cost, low-power, constrained hardware
Constrained hardware Low-speed CPU Low capacity memory No floating-point (FPU) Low-resolution image sensor
王元凱 Towards Embedded Computer Vision p.
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai
A B
Research Status (1/3) 15
王元凱 Towards Embedded Computer Vision p.
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai
Research Status (2/3) H K K
16
王元凱 Towards Embedded Computer Vision p.
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai
17 17
Research Status (3/3)
DARPA Grand
Challenge
Google Car
Google Taxi
王元凱 Towards Embedded Computer Vision p.
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai
18
International Activities (1/2) Special conferences
IEEE Int. Workshop on Embedded Computer Vision
IEEE Int. Conf. Distributed Smart Cameras Special journal issues
IEEE Journal of Selected Topics in Signal Processing, vol. 2, no. 4, Aug. 2008
EURASIP Journal on Embedded Systems Short courses in important CV conferences
CVPR07&08: Distributed vision processing in smart camera networks
ESC07: Embedded CV and smart cameras ICASSP09: Distributed processing in smart
cameras
王元凱 Towards Embedded Computer Vision p.
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai
19
International Activities (2/2) Research projects and Lab.
Princeton Univ./Georgia Tech.: Embedded Systems Lab., Wayne Wolf
Stanford Univ. Wireless sensor networks Lab.
UCLA, CMU, MIT Delft Univ. of Technology
SmartCam Project Graz Univ. of Technology
王元凱 Towards Embedded Computer Vision p.
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai
20
Three Ways for ECV CPU (Central Processing Unit)
ARM, PowerPC DSP (Digital Signal Processor)
TI, ADI, NXP FPGA (Field Programmable Gate Array)
Altera, Xilinx
王元凱 Towards Embedded Computer Vision p.
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai
21
CPU Stanford MeshEye
ARM 7 (55MHz), ZigBee node 3 image sensors
30x30 grayscale x 2 640x480 color x 1
Object detection
(http://wsnl.stanford.edu/smartcam.html)
"MeshEye:a hybrid-resolution smart camera mote for applications in distributed intelligent surveillance", IPSN-SPOTS, 2007
王元凱 Towards Embedded Computer Vision p.
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai
22
DSP "Visual surveillance on DSP-based
embedded platform," Graz Univ. of Technology, 2008(Phd. dissertation) TI C6414 (600MHz) Applications: Adaboost face detection, vehicle
detection, license plate detection
王元凱 Towards Embedded Computer Vision p.
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai
23
FPGA (2/3)
"Hardware, Design and Implementation Issues on a
FPGA-Based Smart Camera," ICDSC, 2007
王元凱 Towards Embedded Computer Vision p.
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai
24
Hybrid: CPU+DSP "Distributed Embedded Smart Cameras
for Surveillance Applications," IEEE Computer, 2006.
Developed for traffic surveillance
王元凱 Towards Embedded Computer Vision p.
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai
Hybrid: CPU+DSP by SoC TI OMAP
25
王元凱 Towards Embedded Computer Vision p.
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai
26
Hybrid: CPU+FPGA UCLA Cyclops
MCU: Atmega128 CPLD: image processing ECV app.: Hand gesture recognition
http://research.cens.ucla.edu/projects/2007/Multiscaled_Actuated_Sensing/Cyclops/
王元凱 Towards Embedded Computer Vision p.
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai
Hybrid: CPU+FPGA by SoC
Xilinx Zynq
27
王元凱 Towards Embedded Computer Vision p.
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai
28
Challenges for ECV Algorithm refinement
Parallel computation Function partition, Multi-threading
Stream processing Memory hierarchy optimization
Hardware design Pipeline, SIMD, board design
Optimized programming skills Fixed-point arithmetic Memory management Intrinsic commands
王元凱 Towards Embedded Computer Vision p.
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai
29
Our Experiences
Entertainment Robot (CPU)
Vision Sensor Network (CPU)
Smart Video Surveillance (DSP)
Wearable Camera (DSP)
王元凱 Towards Embedded Computer Vision p.
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai
30
Case 1 Entertainment
Robot CPU
王元凱 Towards Embedded Computer Vision p.
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai
31
Sony AIBO
王元凱 Towards Embedded Computer Vision p.
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai
Rolling Dice 32
王元凱 Towards Embedded Computer Vision p.
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai
33
Face Detection
王元凱 Towards Embedded Computer Vision p.
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai
34
Face Recognition
王元凱 Towards Embedded Computer Vision p.
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai
35
Facial Expression Recognition
王元凱 Towards Embedded Computer Vision p.
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai
36
Case 2 Embedded Vision
Sensor CPU
王元凱 Towards Embedded Computer Vision p.
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai
37
FJUCam
• Power consumption: 1W
• Power sources: • 5V DC current • 5V Cell Battery
• Weight: 35gm • Size: 6 x 4.5 x 5 (cm) (W x H x D)
Three Modules 1. Main board, 2. Lens module
3. Storage module
王元凱 Towards Embedded Computer Vision p.
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai
38
Face Detection
FJUCam Face Detection
Image Display
王元凱 Towards Embedded Computer Vision p.
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai
39
Cyclops Robot Color tracking
王元凱 Towards Embedded Computer Vision p.
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai
40
Case 3 Video Surveillance
FPGA
王元凱 Towards Embedded Computer Vision p.
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai
41
Mobile Video Surveillance
FPGA
行動通訊網路(3G/GPRS)
網際網路
網路攝影機
監控伺服器
行動裝置(3G行動電話)GSM
數據機
Wap Push
傳遞視訊串流
PCBackground subtraction, ... • 2.8 GHz Intel CPU • Software: C/C++ • Frame rate: 10 fps
王元凱 Towards Embedded Computer Vision p.
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai
42
Background Subtraction Current Frame
-
BackgroundImage Update
Post Processing
Motion Object Image
Background Image
1+kB
kB
1+kP
1+kM),(),(
),(
1
1
yxByxPyxM
kk
k
−= +
+
),()1(),(),(
1
1
yxPyxByxB
kk
k
+
+
−+= αα
Speed up by (1) Circuit design, (2) Paralization
王元凱 Towards Embedded Computer Vision p.
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai
43
FPGA Speed Up Comparisons
PC: 2.8GHz CPU, C implementation FPGA can speed up 500 times
Clock(Hz) FPS
CPU
FPGA
2.8G
25M
51
10
王元凱 Towards Embedded Computer Vision p.
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai
44
Case 4 Wearable Camera
X-EYE ARM + DSP
王元凱 Towards Embedded Computer Vision p.
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai
45
Goal
45
王元凱 Towards Embedded Computer Vision p.
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai
Demo Video 46
王元凱 Towards Embedded Computer Vision p.
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai
47
Development
滑鼠
鍵盤
SD卡 USB-WIFI卡
讀卡機 USB-RS232
控制線
微投影機
自製 外殼 USB
連接線
顯示器
觸控面板
筆電
移動電源
BeagleBoard
Camera USB
Hub
王元凱 Towards Embedded Computer Vision p.
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai
48
Hardware
48 48
Peripheral I/O
• USB Host
• JTAG
• DVI-D video out
• S-Video out
• SD/MMC+
• Stereo in/out
• RS-232 serial1
• Alternate power
• USB 2.0 HS OTG
USER
OMAP3530 Processor • 600MHz Cortex-A8
• NEON+VFPv3 • 16KB/16KB L1 • 256KB L2 • 430MHz C64x+ DSP • 32K/32K L1 • 48K L1D • 32K L2
• Power VR SGX GPU • 64K on-chip RAM POP Memory • 256MB LPDDR RAM • 256MB NAND flash
RESET
7.6 cm
王元凱 Towards Embedded Computer Vision p.
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai
49
System
2010.04.25 49
王元凱 Towards Embedded Computer Vision p.
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai
50
Software
王元凱 Towards Embedded Computer Vision p.
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai
Next : UAV
51
王元凱 Towards Embedded Computer Vision p.
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai
Next : Sleep Monitoring 52
王元凱 Towards Embedded Computer Vision p.
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai
Next : Vehicular Object Detection
53
王元凱 Towards Embedded Computer Vision p.
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai
54
Future of Embedded Vision
王元凱 Towards Embedded Computer Vision p.
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai
55
Summary Embedded CPU is not appropriate for
ECV Pros
Great flexibility Fast development
Cons No float-point arithmatics Interrupt-based architecture
Smart phone with 4/8 cores is OK?
王元凱 Towards Embedded Computer Vision p.
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai
56
I Think … Embedded compute vision
Low-cost, low-power, minimal hardware
High-Performance computer vision Fast speed without cost,
power, and hardware constraints
From contradiction to convergence ?!
Constrained Resource
Abundant Resource
Yes by multicore
Contradiction
王元凱 Towards Embedded Computer Vision p.
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai
57
Challenges (1/2) Algorithm decomposition
Function decomposition Partition serial part and parallel part Data flow analysis and
data dependency analysis Parallelism
Loop unrolling Multithreading Pipeline
王元凱 Towards Embedded Computer Vision p.
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai
58
Challenges (2/2) Performance analysis method
For efficiency improvement Implementation efforts
Choose a good embedded platform for computer vision Software issues Hardware issues
Programming skills Multi-threading
王元凱 Towards Embedded Computer Vision p.
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai
59
Future Research Directions
Heterogeneous Multicore
ARM + DSP
ARM + FPGA
ARM + GPGPU
王元凱 Towards Embedded Computer Vision p.
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai
I Think Again …
Is there a development platform for embedded computer vision with Built-in camera
AE, AF, AWB High-performance processor
Heterogeneous multicore Software development environment:
C, Matlab OpenCV Sample codes Labs
60
王元凱 Towards Embedded Computer Vision p.
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai
Embedded Vision Alliance 61
王元凱 Towards Embedded Computer Vision p.
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai
Ramesh Raskar
MIT Platform
62
王元凱 Towards Embedded Computer Vision p.
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai
Andrew Adams
Stanford Platform
63
王元凱 Towards Embedded Computer Vision p.
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai
Taiwan Platform
for Embedded
Vision ?
64
王元凱 Towards Embedded Computer Vision p.
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai
本簡報授權聲明 此簡報內容採用
Creative Commons 「姓名標示 - 非商業性台灣 3.0 版」授權條款
歡迎非商業目的的重製、散布或修改本簡報的內容,但請標明: (1)原作者姓名:王元凱; (2)圖標示:
簡報中所取用的部份圖形創作乃截取自網際網路,僅供演講者於自由軟體推廣演講時主張合理使用,請讀者不得對其再行取用,除非您本身自忖亦符合主張合理使用之情狀,且自負相關法律責任。
66