Upload
clementine-giles
View
39
Download
2
Tags:
Embed Size (px)
DESCRIPTION
Video Coding with Spatio -temporal Texture Synthesis and Edge-based inpainting. Chunbo Zhu , Xiaoyan Sun , Feng Wu , and Houqiang Li. ICME 2008. Outline. Introduction Framework Previous work Encoder Decoder Experiment Result Conclusion. Introduction. Texture synthesis - PowerPoint PPT Presentation
Citation preview
Video Coding with Spatio-temporal Texture Synthesis and Edge-based inpainting
Chunbo Zhu , Xiaoyan Sun , Feng Wu , and Houqiang Li
ICME 2008
Outline
•Introduction•Framework
▫Previous work▫Encoder▫Decoder
•Experiment Result•Conclusion
Introduction
•Texture synthesis•Image/video inpainting
Introduction
Encoder
Decoder
[1] D. Liu, X. Sun, F. Wu, S. Li, and Y.-Q. Zhang, “Image compression with edge-based inpainting,” IEEE Transactions on Circuits and Systems for Video Technology, Oct. 2007.
Original image
Edge map Removed blocks
Structure propagation
Texture synthesis
JPEG
Introduction
•Texture regions can be well synthesized.▫Selectively removed during encoding.▫Restored by texture synthesis and edge-
based inpainting in the decoder.•Assistant information can be extracted to
guide restoration.▫Region types, edge information, motion
parameters. Keep spatio-temporal consistency
Framework—previous work
[2] C. Zhu, X. Sun, F. Wu, and H. Li, “Video coding with spatio-temporal texture synthesis,” IEEE International Conference on Multimedia and Expo, 2007.
I B PBB …
Framework—encoder
I B PBB …
Local Motion Region
Global Motion Region
Same as previous work
Framework—encoder
Structure blocks
Textureblocks
Block Categorization
Edge detection
Block categorizati
on
Exemplar selection
examplars
Check connective degreeremoved blocks do not constitute a large region
Framework—encoder
Motion Threading
[3] L. Luo, F. Wu, S. Li, Z. Xiong, and Z.Q. Zhuang, “Advanced motion threading for 3D wavelet video coding,” Signal Processing: Image Communication, Vol. 19, Issue 7, pp. 601-616, Aug. 2004.
•Spatial and temporal consistency▫Block-based backward motion estimation▫Motion threads in texture regions are pruned
B0 B2 B3 B4 B5B1
Exemplar Selection• All the motion threads are 3-D exemplar
candidates• Spatio-temporal variation(SVT)
N : length of a thread which consists of N blocks Bi(i=1…N) : spatio-temporal 6-neighboring blocks of each block : variance of a blockE() : mean pixel value of a block
Choose the threads with higher variation
Output a sequence of binary masks indicating which blocks are removed
Framework—encoder
Sprite Generation•GMR
▫pixel motion can be described by a parametric geometrical model
•Sprite (background mosaic)▫A sprite for a GOP
[4] Y. Lu, F. Wu, S. Li, and Y.-Q. Zhang, “Efficient background video coding with static sprite generation and arbitrary-shape spatial prediction techniques,” IEEE Transactions on Circuits and Systems for Video Technology, May 2003.
(x, y) and (x’,y’) are coordinates of pixels in original frame and reference image
Framework—encoder
removable blocksunremovabl
e blocks
Region removal in GMR
• Mapping into original frame inverse warping
• if one 8x8 block in an original frame contains more than half of the removable pixels, it is selected as a removable block and then skipped when encoding. Otherwise, it is selected as an exemplar and coded with H.264 encoder.
Edge Detectio
n
Region Classificatio
n
Mapping
Exemplar SelectionR
SRF
SPT
RS: Removable blocks in spriteRF: Removable blocks in original frame
Framework—decoderMask1: region mapMask2: removing map
Decoder—Spatio-temporal texture synthesis in LMR•Texture synthesis module
▫Arbitrary-shaped regions▫Perform frame by frame▫Square patch▫Confidence map
Similarity of candidate patch and target patch𝑆=𝑆𝑆𝐷 (𝑊 𝑡 ,𝑊𝑐 )+𝛼 ∙𝑆𝑆𝐷 (𝑊 𝑡 ′ ,𝑊 𝑐)
Squared difference of known pixels
Framework—decoder
Decoder—Edge-based inpainting in GMR
Warping
Blending
Edge inpaintin
g
Texture Synthesis
GMR
GMR parameter
SPT edges
SPTH
SPT
Experimental Results• Implemented into JM 10.2 of H.264• YUV 4:2:0 sequence format with CIF
resolution(352x288)• rate distortion optimization (RDO) and CABAC were
turned on
Experimental Results
Conclusion• In this paper, we present a video coding
scheme in which some regions are removed in the encoder and restored in the decoder by spatio-temporal texture synthesis and edge-based image inpainting.
• To keep temporal consistency, different types of motion have been considered in region removal and restoration for both textural and structural regions.