Noisy Video Super-Resolution

Feng Liu, JinjunWang,ShenghuoZhu (MM’08)University of Wisconsin-Madison, NEC Laboratories America, Inc.

第一組： 資訊四 B95902105 黃彥達 資訊碩一 R98922046 蔡旻光 網媒碩二 R97944012 鄒志鴻

2

Outline

Introduction Goal File Format

Noise Reduced Image Proposed Approach Motion Estimation & Estimated Super-

Resolution Result Implementation Result Conclusion

3

Introduction

Low-quality videos often not only have limited resolution but also suffer from noise In fact, the requirements of de-noising & super-

resolution is quite similar

This paper present a unified framework which achieves simultaneous video de-noising and super-resolution algorithm by some measurements of visual quality

Goal

Refine low-quality videos from YouTube, and make the video better effects, which has better quality by human eyes.

Input is low-quality and noise-included (block effects or somewhat noise) videos

Noise-Reduced Image

mv-SAD Gaussian-space

Gaussian-time

| p(I,j) – p(i’, j’) | > threshold

Gaussian Space

Frame t

Pixel(I,j)

Standard deviation

Set Mean = 0

Motion Vector

Frame tPixel ( i , j , t)

Frame t+1

Pixel ( i + mv_i , j + mv_j , t+1)

(mv_i , mv_j)

Gaussian Time

Fram

e t

- 2Fram

e t

- 1Fram

e t

Space Gaussian

Time Gaussian

Pixel(I,j)

Fram

e

t+1Fram

e

t+2Fram

e t

Noise-Reduced ImageBefore After

10

Proposed Approach – 1 / 4 Consider the visual quality with respect to

the following 3 aspects: Fidelity Preserving▪ To achieve similar high-resolution result

Detail Preserving▪ Enhanced details (edge)

Spatial-Temporal Smoothness▪ Remove undesirable high-frequency contents (e.g. jitter)

11

Proposed Approach – 2 / 4 Fidelity Preserving

Conventional metrics:▪ Measure fidelity by the difference between Ih & Il would

be problematic & waste useful time-space information in video

Proposed metrics:▪ Estimate an approximation of super-resolution results

from space-time neighboring pixels▪ The fidelity measurement:

see next page for details

noised

12

Proposed Approach – 3 / 4 Detail Preserving

Enhanced details (edge)

Contrast preserving▪ Human visual system is more sensitive to contrast

than pixel values▪ Gradient fields of Ih & should be close

,where Wk is one or zero if the patchk with/o edges (canny detector)

13

Proposed Approach – 4 / 4 (Spatial-Temporal) Smoothness

Smooth results are often favored by the human system Encourage to minimize:

A 2-D Laplace filter may be

Spatial-temporal Laplacian

OR

14

An Optimization Problem

Proposed Measurements

A quadratic minimization problem to solve (AX = b):

Contrast

Similarity

Detail Information(edge)

Spatial-Temporal Smoothness

15

Implementation – 1 / 2

InputlowhI~

hI~

hG~

hI~

)1,()1,(~~

tIwtIw hmv

hmv

= X

I

6 -1 … -1-1 6 -1 … -1 -1 6 -1 … -1

Laplacian

Gradient-1 0 1 … 1 -1 0 1 … 1 -1 0 1 … 1

EdgeMinimize

Motion Estimation

+

Result (X)

Fidelity

Bilateral filter

mvw

fidv

fidg

dt

sm

16

Implementation – 2 / 2

Adjustments for the weight terms The measurement term is more emphasized if

the weight is larger

By iteratively experiments for our test data, we took

However, we found that for different videos, the best weight sets may be also different

3.0,1.0,1,1 smdtfidgfidv

17

Result

352 x 288 Result

18

Conclusion

The proposed framework formulates noisy video super-resolution as an optimization problem, aiming to maximize the visual quality of the result

The measurements of fidelity-preserving, detail-preserving and smoothness are considered to maximize the visual quality results

19

Thank you!!