Embed Size (px)
Citation preview
Non-photorealistic Video Effects in the Compressed Domain
Dept. of Computer ScienceNational Chengchi UniversityStudent : Fu-Liang HsuAdvisor : Wen-Hung Liao
2005/7/18
2
Outline
Motivation and introduction Issues in non-photorealistic rendering
(NPR) Objectives of this research NPR in the spatial domain NPR in the compressed domain Conclusion and future work
3
Motivation
Sony EyeToy Apple toySight ImageTech
(台灣夢工場科技 )
4
NPR: Introduction
Non-Photorealistic Rendering Photorealistic Rendering
5
NPR Example: Images
Oil paint effect using Ulead PhotoImpact® Image size=450x315. Took 4 seconds on machines with P4 2.4G CPU.
Source Image Oil paint Image
6
NPR Example: 3D Models
Applying NPR to 3D model Can be done in real-time.
7
NPR Effects of Interest
Static NPR algorithm NPR animation Real-time NPR
SIGGRAPH 1997 SIGGRAPH 2004
8
Drawbacks of Frame-by-Frame NPR Generation
Processing time is demanding Coherence problem Flickering
Oil paint Image
9
Reducing Processing Time
Post-processing Modifying existing NPR algorithms Developing hierarchical NPR algorithms Applying NPR to regions of interest (ROI) in
the video
10
Dealing with the Coherence Problem
Coherence Stroke-based NPR Optical flow
Flickering Paint-over
11
Objectives of this Research
Develop near real-time NPR algorithms (frame rate >= 10 fps) to facilitate interactive applications.
Attempt to employ existing NPR algorithms and generate similar effects.
Try to devise methods that are applicable to most NPR algorithms.
12
Possible Enhancements
In the spatial domain In the compressed domain
13
NPR in the Spatial Domain
Most NPR algorithms’ complexities are dependent on image size.
Apply NPR to regions-of-interest (ROIs) can effectively reduce the processing time.
14
Combine with Source Video
SkinFace
NPR in the Spatial Domain: Framework
Full Frame
Edge
Detected Area
Motion Background Random
Image Filter
Source Video
NPR Video
15
NPR Algorithm
Oil paint
Time Complexity K= 7
For color = 1:3 For y= 1: height For x=1:width find most frequent value M in (x,y)'s n*n neighborhood oil_image(x,y)= M;
nmknmkT 223
16
Development Environment
CPU:Pentium-4 2.4G Hz with 1GB of RAM
Visual C++ 6.0 Intel OpenCV Library Asiamajor V-Gear MaxCam1300
USB 2.0 Frame rate:15 frame/sec
17
Frame rate : 0.75 ~ 1.0 fps
Full Frame NPR Demo
18
Edge Image NPR Demo
Frame rate : 1.5 ~2.0 fps
• Edge ImageCanny edge detection + Dilation• Frame rate : 15 fps
19
Moving Region
Foreground vs. background region
Motion filter: Frame rate :15 fps
Dt(x)=Difference(t,t+1);If(| Dt(x) | > Threshold)
Mt(x)= 1 else Mt(x)=0
B(t+1)= Bt + [ a1*( 1-Mt ) + a2*Mt ]*Dt
// ax: 降低改變大的區域對背景的影響 , a1+a2=1, a1>a2
20
Moving Region NPR Demo
Frame rate : 1.5~ 8.0 fps
21
Background Region NPR Demo
Frame rate : 1.0~ 2.0 fps
22
Random Region NPR Demo
Frame rate : 1.8~4.0 fps
Random region selection: 15 fps
23
Possible Enhancements
In the spatial domain: image filter Edge Moving region Background Random
Detected area Face Skin
24
NPR in the Spatial Domain
Face detection based on Viola and Jones’ algorithm proposed in ”Rapid object detection using a boosted cascade of simple features”
25
Face NPR Demo
Frame rate: 3.8~7.5 fps
Region of interest: face
Frame rate :15 fps
26
Color-Based Skin Detection
Hue value 0.3~1.5 Frame rate : 15 fps
27
Skin NPR demo
Frame rate : 3.2~4.5 fps
28
Summary of Spatial Domain Processing
Frame rate NPR 效果 影響速度關鍵
Full frame 1~1.2 frame/sec 最佳，速度最慢 整張影像，負擔過重
邊緣 2.5~3 frame/sec 邊緣區域較少，視覺效果較差，可搭配其他層次使用
偵測邊緣演算法的門檻值，影響邊緣數量的多寡
移動區塊 2.5~4.0 frame/sec 使用移動區塊可以和使用者互動，有額外的效果
移動區塊大小
背景區塊 1.8~4.0 frame/sec 使用背景區塊可以和使用者互動
背景區塊大小
隨機選取 3.9~6.0 frame/sec 搭配其他的方式會有比較好的效果
隨機選取區塊大小
臉部區塊 3.8~7.5 frame/sec 針對人體的部分套用 NPR 演算法，互動效果佳
臉部偵測函式及臉部區塊大小
膚色區塊 3.2~4.5 frame/sec 針對人體的部分套用 NPR 演算法，互動效果佳
膚色區塊大小
29
Possible Enhancements
In the spatial domain Edge Moving region Background Random Face Skin
In the compressed domain
30
MPEG-I Compression Format
Forward prediction of P-frameForward prediction of B-frameBackward prediction of B-frame
MPEG Display Order
31
Applying NPR in the Compressed Domain
32
NPR in the Compressed Domain
Encode
Decode all frames to spatial domain
Decode I-frame to Spatial domain
Change AC in the
Compressed domain
Compressed Video Image
Compressed NPR Video
Decode I-frame and large difference P,
B-frame to Spatial domain
33
Development Environment
CPU Pentium-4 2.4G Hz Memory 1GB Visual C++ 6.0 Intel OpenCV library Dali library for video compression MPEG-1 standard video
320x240 419 frames, 13sec GOP:IBBPBBPBBPBBPBB Source Video
Encoding source image captured from Webcam to MPEG-I using hardware.
Source Video
34
Macro Block
320 x 240 Image
…….
.
.AC
Macro block 8x8
DC
35
NPR in the Compressed Domain: Changing the DC Coefficient
Making changes to DC value in the compressed domain is equivalent to adding/subtracting a constant to every pixel in the spatial domain.
36
Changing the AC Coefficients: Frequency Domain Filtering Model
Butterworth Lowpass Filter
nDvuDvuH
20/,1
1,
vuFvuHvuG ,,,
37
Butterworth Lowpass Filter Demo
I-frame only
D0=1,n=2 1.071 sec / 419 frames
Source I-frame BLPF
38
Butterworth Lowpass Filter Demo
All I,P,B frames
D0=1,n=2 1.375 sec / 419 frames
Source IPB-frame BLPF
39
NPR in the Compressed Domain
Gaussian Lowpass Filter
20
2
2,
, DvuD
evuH
40
Gaussian Lowpass Filter Demo
I-frame only
D0=2 1.219 sec / 419 frames
Source I-frame GLPF
41
Gaussian Lowpass Filter Demo
All I,P,B frames
D0=2 1.968 sec / 419 frames
Source IPB-frame GLPF
42
NPR in the Compressed Domain: Highpass Filtering
Butterworth Highpass Filter
vuHvuH lowpasshighpass ,1,
43
Butterworth Highpass Filter Demo
I-frame only
D0=4,n=4 1.109 sec / 419 frames
Source I-frame BHPF
44
Butterworth Highpass Filter Demo
All I,P,B frames
D0=1,n=2 1.328 sec / 419frames
Source IPB-frame BHPF
45
Gaussian Highpass Filter
I-frame only
D0=4 1.125 sec / 419 frames
Source I-frame GHPF
46
Gaussian Highpass Filter Demo
All I,P,B frames
D0=4 1.813 sec / 419 frames
Source IPB-frame GHPF
47
Summary of DCT Domain Filtering
Butterworth Lowpass Filter
Gaussian Lowpass Filter
Butterworth Highpass Filter
Gaussian Highpass Filter
I-frame 套用
1.071sec /419 Frames
1.219sec /419 Frames
1.109sec /419 Frames
1.125sec /419 Frames
IPB-frame 套用
1.375sec /419 Frames
1.968sec /419 Frames
1.328sec /419 Frames
1.813sec /419 Frames
視覺效果 smear smear mosaic mosaic
48
Possible Enhancements
In the compressed domain Changing DC,AC coefficients Apply NPR to I-frame Applying NPR to I frames and to P,B-
frames with discontinuities
49
Frame-by-Frame Oil Paint
399.734 sec / 419 frames Frame rate : 1.048 fps Flickering
Source frame by frame NPR
50
28.407 sec / 419 frames frame rate : 14.749 fps Lost frame
Applying NPR to I-frame Only
frame by frame I-frame NPR
51
Applying NPR to I Frames and P,B-Frames with Discontinuities
52
Computing Image Differences
Can be done in DCT or spatial domain. Spatial domain approach:
For all pixels if | ( Dt +1(i,j) - Dt(i,j) )| > constant
diff++
if( diff > percentage * pixels )ApplyNPR( Dt+1 )
53
Order of Computation
54
I-and discontinuous P,B-Frames NPR
Difference>5 Percentage=60%
I-and Discontinuous P,B-Frames NPR
I-frame NPR I-and discontinuous
P,B- frames NPR
55
I-and Discontinuous P,B- frames NPR: Performance
I-and discontinuous P,B-frames NPR Difference+NPR
54.469 sec/419 frames Frame rate : 7.65 fps
NPR 49.437 sec/419 frames Frame rate : 8.475 fps
56
Summary of I,P,B-frame NPR
IPB-frame NPR I-frame NPRI-frame & Difference
NPR
花費時間 399.734sec 28.407sec 54.469sec
Frame rate 1.048 frame/sec 14.749 frame/sec 7.65 frame/sec
相對於逐張frame 套用 NPR增進效率百分
比0% 93%
87% ( Difference threshold
= 60% )
效果每張皆有 NPR
效果，但會有閃爍情形，無法即時
場景差異過大時P,B-frame 無 NP
R效果，達到即時
有適當的差異門檻值可兼顧效果並接近即時
57
Summary of Compressed Domain NPR
改變 DCT 係數 僅套用於 I-frame PB-frame 補強
NPR 套用速度
速度最快，可以達到即時的效果 可達到即時效果 近似即時
額外負擔 無 解壓縮 I-frame解壓縮 I,PB-frame,計算 Difference
特點 速度快但效果有限可以達到即時，可套用空間上的 NP
R演算法
效果最好，但是否達到即時視差異值
而定
58
Optimize Pixel-Based NPR
Calculate variance in the compressed domain for pixel-based NPR
Sum of AC2 is equal to calculate variance in the spatial domain
If the macro block is quite uniform, do not apply the NPR effect.
ACMacro block
8x8
DC
59
Applying NPR to Selective Macro Blocks
Variance =sum of all AC2 in the macro block
if( Variance > threshold ) ApplyNPR(macro block)else macro block = macro block of source image
60
Threshold =100
61
Threshold =200
62
Summary of Applying NPR to Selective Macro Blocks
門檻值 0 100 200 1000 1500
花費時間 1.09sec 0.875sec 0.719sec 0.65sec 0.65sec
減少區塊比例 0% 21.5% 34.5% 45% 47%
相對於整張影像套用 NPR 加速時間比例
0% 20% 33% 40% 40%
63
Conclusions
Applying NPR to regions-of-interest can indeed reduce processing time, making interactive applications feasible.
Compressed domain processing has proven to be effective.
64
Future Work
Encoding source image captured from Webcam to MPEG-I using hardware.
Incorporating information in motion vectors to avoid the need to perform optical flow analysis.
65
Future Work (cont’d)
MPEG-2 Resolution is higher Difficult to achieve real-time performance. Hardware acceleration is required.
MPEG-4 Video Object (VO)
66
Q & A
67
Calculate Variance in the Compressed
Domain for pixel-based NPR
1
0
1
0
22
1
0
1
0
21
0
1
0
2
21
0
1
0
2
221
0
1
0
2
1
0
1
0
21
0
1
0
2
1
0
1
0
1
0
1
0
1
0
1
0
221
0
21
0
0,0,
0,0,
,
2,
0,00,02,
,2,,
M
u
N
v
N
u
M
v
N
u
M
v
N
u
M
v
N
u
M
v
N
u
M
v
N
u
M
v
N
x
M
y
N
x
M
y
N
x
M
y
N
x
M
y
FvuF
FvuF
MNmvuF
MNmMNmvuF
FmMNFvuF
myxfmyxfmyxf
Sum of AC2
DFT Domain
Parseval's Theorem
68
NPR in the Compressed Domain
Model
Change DC value
vuFvuHvuG ,,,
otherwise
vuifvuH
1
0 ,
69
NPR in the Compressed Domain
Change DC value
else
vuvuwhere
vuFvuvuG
,0
0 ,1,
,,11,
70
NPR in the Compressed Domain
Change DC value
α> 1 increase, α < 1 decrease
1
0
1
0
1
0
1
0
2
,1
,
0,01,
exp,,1,,
M
x
N
y
M
u
N
v
Nvy
Muxj
yxfMN
yxf
Fyxf
vuFvuyxfyxg
