Resampling

Recent Advance in Video Resampling

Tejus Adiga MDepartment of Electronics and Communication, NMAMIT, Nitte.

Presented By:

May 1, 2023 Department of Electronics and Communications, NMAMIT, Nitte. 2

Resampling

• Ideal resampling: Discrete -> Continuous -> Discrete.

• Practical resampling: Done entirely in discrete domain.

• Types of Resampling: • Downsampling: Decrease size by M. • Upsampling: Increase size by N.• Fractional Resampling: Increase size my M and decrease by N (M/N).

• Traditional Methods:• Blind Resampling: 2D Convolution. Eg Kernels Nearest Neighbor, Bilinear,

Bicubic, Bspline.

• Content Aware Resampling: Seam Carving, Edge Directed Interpolation (EDI), Super Resolution.

• Seperability: 2D filtering = Performing 1D filtering two times in each dimension one after another.


Downsampling

• General Approach: Anti Alias Filter

(LPF)Downsampler

Image Downsampled

Image

• Practical Approach

𝑦𝑚= ∑𝑘=− 𝑁 /2

𝑁 /2

𝐶𝑘 𝑥2𝑚−𝑘− 10 ≤𝑚≤𝑊 ,𝐻 .𝑁 𝑖𝑠 𝑓𝑖𝑙𝑡𝑒𝑟 h𝑙𝑒𝑛𝑔𝑡

Downsampling Kernel

Downsampled Image Image

• Convolution

m-2 m-1 m m+1 m+2 m+3Downscaled Image

Original Image

Fig 1: Downsampling process

xCk1Ck1 Ck2Ck2

hd(x)

0

1 Pixel Distance

Fig 2: Downsampling Kernel


Upsampling

• General Approach: Upsampler

Low Pass FilterImage Upsampled Image

• Practical Approach

𝑦 2𝑚+1= ∑𝑘=−𝑁 /2

𝑁 /2

𝐶𝑘𝑥𝑚−𝑘−10≤𝑚≤𝑊 ,𝐻 .𝑁 𝑖𝑠 𝑓𝑖𝑙𝑡𝑒𝑟 h𝑙𝑒𝑛𝑔𝑡

Upsampling Kernel Upsampled ImageImage

• Convolution

Upscaled Image

m-2 m-1 m m+1 m+2 m+3Original Image

m-3 m+4

Fig 3: Upsampling process

hu(k)

x0

Fig 4: Upsampling Kernel


Blind Resampling – Nearest Neighbor

Fig 5: Spatial kernel and Frequency Response

• Downsampling: Discard Every alternate pixel.

• Upsampling: Replicate the Nearest Pixel.

• Artifacts: Aliasing-Increase 4 times for two fold resample.

• Kernel: Rectangular spatial kernel. Infinite frequency contents.


Blind Resampling – Nearest Neighbor

Downsampled by 4

Downsampled by 2

Captured Image

Upsampled by 4

Fig 6: Downsampled and Upsampled by factor of 2 and 4

Upsampled by 2


Blind Resampling – Bilinear


• Downsampling and Upsampling: Average of two pixels. (4 pixels in 2D)

• Artifacts: Aliasing, Blurring.

• Filter Coefficients:

• Kernel: Triangular or Tent Spatial kernel.

• Frequency response: Stop band attenuation better than Nearest Neighbor.

• Aliasing is reduced when compared to nearest Neighbor.


Blind Resampling – Bilinear

Captured Image

Upsampled by 4


Upsampled by 2

Downsampled by 4

Downsampled by 2


Blind Resampling – Bicubic


• Downsampling and Upsampling: Weighted average of 4 pixels.

• Artifacts: Blurring.• Filter:

• Frequency response: Stop band attenuation better than Bilinear.

• The 1st negative side lobe introduce controlled Ringing effect which makes image

appear sharper than they actually are.


Blind Resampling – Bicubic

Captured Image

Upsampled by 4


Upsampled by 2

Downsampled by 4

Downsampled by 2


Blind Resampling – Windowed Sinc

Fig 11: Windowed Sinc Kernel

• Artifacts: Ringing, Blurring.• Filter: Truncated Sinc function.

• Side lobes significantly contributes to Ringing.


Blind Resampling – Lanczos

Fig 12: Lanczos Kernel for a=2 and a=3

• Weighted Average of 4 pixels.

• Artifacts: Blurring.• Filter:

• ‘a’ indicates number of lobes in one half of the filter.• Effect of side lobes is decreased by multiplying another scaled sinc function.

But stronger enough to make image look sharper.

• Upsampled image sharper than Bicubic, Bilinear.


Blind Resampling – Lanczos

Captured Image

Upsampled by 4


Upsampled by 2

Downsampled by 4

Downsampled by 2


Artifacts in Image Resampling• Aliasing:

• Jagged Edges.

• Introduced in Downsampling and Enhanced in Upsampling.

• Priority: Lanczos, Bicubic, Bilinear, nearest Neighbor.

• Ringing:

• Side lobes of lengthy filter contribute to false edges.

• Optimal filter length 4.

• Windowed Sinc.

• Blurring:

• LPF gains get multiplied while Downsampling and Upsampling.

• Information lost during Downsampling is irreversible. So in upsampling pixels

are filled with he help of existing information in Downsampled image.

• Priority: Nearest Neighbor, Bilinear, Lanczos, Bicubic.


Artifacts

Ringing Example

Aliasing ExampleBlurring Example

Fig 14: Examples of Ringing, Aliasing and Blurring


Content Aware Resampling – Seam Carving

• Seam: 8-Connected set of pixels that runs from top to bottom or Left to Right.• Principle: Low energy seam is not appealing to eyes.• Applications:

• Image Retargeting: Resizing image, Changing Aspect Ratio.• Object removal or insertion.

• Algorithm:• Find the Gradient Map of the input image I.

• In Gradient Map search a unique path (seam) from top to bottom or left to right such that Energy of the seam is minimum than all other possible seam.

• Remove the seam or Duplicate the seam from the image I and G which reduces/increases the width/height by 1 pixel.

• Iterate the above steps until desired size is achieved.



Fig 15: Seam Calculation using Gradient.



Fig 16: Comparison of Seam Carving with Scaling


Seam Carving – Failure Cases


Seam Carving – Failure Cases


Content Aware Resampling – New Edge Directed interpolation (NEDI)

• Principle: Inter pixel relations are retained while Downscaling. Hence Covariance in original Image and Downscaled image are nearly same.

• New Pixel Value = Weighted Sum of nearest 4 pixels. Weights are computed dynamically according to local image characteristics.

𝑌 2 𝑖+1, 2 𝑗+1=∑𝑘= 0

1

∑𝑙=0

1

𝛼2𝑘+𝑙𝑌 2 (𝑖+𝑘 ) ,2 ( 𝑗+𝑙 )

�⃗�=𝑅−1𝑟𝑅=

122𝐶𝑇𝐶 ,𝑎𝑛𝑑𝑟=

122𝐶𝑇 �⃗�

Where is the data vector containing the 2x2 pixels inside the local window and C is a 4x22 data matrix whose kth column vector is the four nearest neighbors of along the diagonal direction.


Content Aware Resampling – (NEDI)

Fig 16: Comparison of NEDI with Bicubic filter

Downscaled Image

Upscaled 4X using NEDI Upscaled 4X using Bicubic


Conclusion

• Trade off:

• Quality

• Speed of Operation

• Requirement

• Information lost during Downsampling cannot be recovered while upsampling.

• Future Work:

• Improvement of Content Aware Resizing methods.

• Adding Resolution.


References1. New Edge-Directed Interpolation, Xin Li and Michael T. Orchard. IEEE TRANSACTIONS ON

IMAGE PROCESSING, VOL. 10, NO. 10, OCTOBER 2012.2. Local and Nonlocal Regularization to Image Interpolation, Yi Zhan, Sheng Jie Li, and Meng Li,

Mathematical Problems in Engineering, Volume 2014 (2014), Article ID 230348.3. Adaptive multidirectional edge directed interpolation for selected edge regions. TENCON 2011 -

2011 IEEE Region 10 Conference.4. V.R. Algazi, G.E. Ford and R. Potharlanka, "Directional interpolation of images based on visual

properties and rank order filtering", Proceeding of ICASSP' 1991, pp.3005-3008.5. Seam Carving for Content-Aware Image Resizing. Shai Avidan and Ariel Shamir. Proceedings of

ACM SIGGRAPH, 417–424.6. J. Allebach and P.W. Wong, "Edge-directed interpolation", Proceeding of ICIP 1996, Page No

707-710.7. Keys, R., “Cubic Convolution Interpolation for Digital Image Processing”, IEEE Trans on ASSP, vol

ASSP-29, No. 6, Page No 1153-1160. Dec 1981.8. New Filters for Image Interpolation and Resizing, Amir Said, IEEE International Conference on

Image Processing, VOL. 8, 2007.9. Image Zooming Methods, Bax Smith.10. “Interpolation Theory”

http://sepwww.stanford.edu/public/docs/sep107/paper_html/node20.html

Technology

Resampling