My FInal Report!

May 25, 2014 Assoc. Prof. Dr. Maria TROCAN Associate Professor Institut Supérieur d'Electronique de Paris Signal & Image Processing Dept.

Dear Assoc. Prof. Dr. Maria TROCAN,

I wish to submit a training final report entitled “Image Super Resolution Techniques” for your review. This report summarizes our research during April - June 2014, in order to fulfill my knowledge and experience in the field of digital image processing.

Thank you in advance for your kind consideration of the enclosed report.

Sincerely,

Nutasn Ratchano

Enclosure

i

Summer Training Final Report

Image Super Resolution Techniques

Submitted to

Dr. Maria TROCAN Associate Professor

Institut Supérieur d'Electronique de Paris Signal & Image Processing Dept.

16 May 2014

by

Nutasn Ratchano

Electronics and Communication Program

Sirindhorn International Institute of Technology

Thammasat University

1

Abstract

Super Resolution has been one of the main topics of research in Image Processing

in the last twenty years. We should begin with the use of the term super resolution(SR). Super Resolution algorithms reconstruct a high resolution(HR) image from a set of low resolution (LR) images of a scene.

Super Resolution techniques are being applied to a variety of fields, such as obtaining improved still images from video sequence, astronomy and medical imaging.

Increasing Resolution of an image can be done by increase sensor quality but it increases production cost. Here is where Super Resolution algorithms can be applied, with less cost and still be able to obtain high resolution image is the main purpose of SR.

Methods of Super resolution algorithms will be reviewed in this report. Each kind of algorithm has its own advantages. SR performances depend on the application. We will also propose Optical-Flow based Super Resolution in the end of this report.

2

Table of Contents List of Figures……………………………………………………………………………iv

List of Tables ..................................................................................................................... 3

Introduction ....................................................................................................................... 4 1.1 Introduction ......................................................................................................................... 4 1.2 Classification of Super Resolution Algorithms .................................................................. 6 1.3 Problem Description ............................................................................................................. 7 1.3.1 Common Image Acquisition System ................................................................................. 8 1.3.2 Observation Model ............................................................................................................ 9

Motion Estimation .......................................................................................................... 13 2.1 Preliminaries ..................................................................................................................... 14

Reconstruction ................................................................................................................. 22 3.1 Optical-flow based super resolution reconstruction ............................................................. 22

Result ................................................................................................................................ 25

Bibliography: ................................................................................................................... 31

MATLAB code .................................................................................................................. 1

3

List of Figures Figure 1 : Comparison of visual quality for bilinear interpolation and super resolution 1 Figure 2: LR images and the HR image after applying SR algorithm 2 Figure 3: The basic idea for super resolution 2 Figure 4: Classifications of the Super Resolution algorithms 3 Figure 5: Basic premise of super resolution 4 Figure 6: Common Image Acquisition System 5 Figure 7: Common Image Acquisition System 5 Figure 8: Observation Model relating LR images to HR images 6 Figure 9: Common Super Resolution stages 6 Figure 10: Overview of common super resolution algorithms 10 Figure 11: Procedure to obtain a LR image sequence from a HR image sequence 13 Figure 12: Block-matching frame division in blocks of size m x n. 14 Figure 13: Coarse-to-fine optical flow estimation 16 Figure 14: Super resolution with mis-registration 18 Figure 15: Combination of LR images to create the HR image. 19 Figure 16: Deblurring block. 20

List of Tables Table 1: PSNR results of super resolution methods……………………………………..26

4

Chapter 1

Introduction

1.1 Introduction The term super resolution(SR) is used to describe the process of obtaining an high

resolution image or a sequence of HR images from a set of low resolution observation. In most electronic imaging applications, images with high resolution are desired

and often required. HR means that pixel density within an image is high, and therefore an HR image can be offer more details that may be critical in various applications. For instance, HR medical images are very helpful for a doctor to make a correct diagnosis. It may be easy to distinguish an object from similar ones using HR satellite images, and the performance of pattern recognition in computer vision can be improved if an HR image is provided.

The most direct solution to increase spatial resolution is to reduce the pixel size (i.e., increase the number of pixel per unit area) by sensor manufacturing techniques. As the pixel size decreases, however, the amount of light available also decreases. It generates shot noise that degrades the image quality. Therefore, there exists the limitation of the pixel size reduction. Another approach for enhancing the spatial resolution is to increase the chip size but it has many disadvantages. Increasing image sensors quality by manufacturing is one way to solve this problem but it is not the solution for us, either because of cost or hardware physical limits, super resolution algorithms are an inexpensive alternative.

Figure 1: Comparison of visual quality for bilinear interpolation(left) and one of super resolution algorithms

5

Super resolution is a computationally intensive process. For example some video applications may require super resolution to be done fast or data must be processed as they are received. Even it is a trade-off but speed should not be gained at the cost of robustness. SR algorithms must be robust to sources of image degradations. Including unknown sensor noise, unknown camera characteristics from frame-to-frame, etc. Moreover, the algorithms must be driven only by the sensor data as no information about noise or camera characteristics may be given.

The algorithm proposed by us will be evaluated by both the qualitative result and quantitative result in the end of the report as well.

Figure 2: LR images and the HR image after applying SR algorithm

Figure 3: The basic idea for super resolution reconstruction from multiple LR frames. Subpixel motion provides the complementary information among the LR frames that makes SR reconstruction possible.

6

1.2 Classification of Super Resolution Algorithms Super resolution was first proposed by Tsai and Huang [2], who suggested a frequency domain approach to SR. This first approach assumes a translational movement among the LR images and is based on Fourier shifting properties, aliasing and the assumption that the HR image is band-limited. In early works, frequency-based methods were the chosen approaches despite being restricted in the image observation model and the real scenarios, which they can handle. Now most super resolution algorithms are working on spatial-based, for its flexibility to model all kinds of image degradations.

Figure 4: Classifications of the Super Resolution algorithms

A. Single-image and multi-frame super-resolution

Super-resolution (SR) aims to produce a high-resolution (HR) image using one or several observed low-resolution (LR) images by upsampling, deblurring and denoising. For multi-frame SR, there can involve registration and fusion processes. It is interesting to point out that some SR algorithms do not involve the denoising process, or some interpolation algorithms are also referred to as super- resolution algorithms. Basically, the super- resolution methods can be classified into single-image super- resolution (only one LR image is observed) and multi-frame super-resolution. The multi-frame super- resolution for a video sequence can moreover use a recursive estimation approach of video frames, which is called the dynamic super-resolution. Dynamic super-resolution makes use of the previously reconstructed HR frame to estimate the current HR frame.

B. Reconstruction-based and learning-based super- resolution

There are two major categories of SR algorithms: they are the reconstruction-based and learning-based algorithms. Reconstruction-based algorithms use the data fidelity function, usually with the prior knowledge to regularize the ill-posed solution. Gradients are the main features to be used as the prior knowledge. Some more general prior knowledge using nonlocal self-similarity was also proposed. Learning-based algorithms moreover utilize the information from training images. Generally, when a LR image is observed, a patch within that LR image is extracted and searched within the

7

trained dictionary for estimating a suitable HR patch that reconstructs the HR image. Some recent investigations make use of the sparse representation for estimating the HR pair and training the dictionary. By combining the use of prior knowledge and dictionary training, some unified frameworks for single-image SR were proposed to further regularize the estimated solution

1.3 Problem Description How can we obtain an HR image from multiple LR images? The basic idea for increasing the spatial resolution in SR techniques is the availability of LR images captured from the same scene. Typically in super resolution the LR images represent different “perspectives” at the same scene. Therefore, LR images are aliased and shifted with subpixel precision so multiple LR images provide the non-redundant information but if LR images are shifted by integer unit, each image contains the same information and thus there is no additional information that can be used to construct an HR image so super resolution is not possible.

“If there exists subpixel motions between these low resolution frames thus LR images have different subpixel shifts (looks) from each other. In this case, the new information contained in each LR image then SR is possible.

Figure 5: Basic premise of super resolution

8

1.3.1 Common Image Acquisition System When we record a digital image with camera, there is a natural loss of spatial resolution due to the optical distortions (out of focus, diffraction limit, etc.) motion blur caused by limited shutter speed, noise occurs, and insufficient sensor density. The recorded image, therefore suffers from these distortions.

When it comes to set of images the image acquisition system can be illustrated as follows:

Figure 6-7: Common Image Acquisition System Super resolution is used to reconstruct HR images from undersampled LR images, however, it also can be used to restore HR image from a noisy,blurred images and aliased LR images but they are not the same , SR can restore image and increase size of the image as well.

9

1.3.2 Observation Model We first need to establish models that describe the acquisition of the LR images.

Figure 8: Observation Model relating LR images to HR images

We can represent the observation model as

yk =DBkMk+nk for k=1,…,p

when is kth observed LR image yk

D is a subsampling matrix

Bk is a blur matrix

M k is a warp matrix

n k represents a lexicographically ordered noise vector

Most of the SR image reconstruction methods proposed in the literature consist of the three stages illustrated in the following figure.

Registration or motion estimation, interpolation, and restoration. These steps can be implemented separately or simultaneously according to the reconstruction methods.

Figure 9: Common Super Resolution stages

10

1.4 Optical-flow based super resolution

Considering computational time and for more various types of transformation super resolution with optical flow based can be a good solution here. Super resolution optical flow based takes motion estimation information for warping an image in this step.

Our scheme will be, first, interpolate every scene in the input sequence using chosen interpolation method[1,2,3] (interpolation method that gives maximum PSNR results) to twice the input scenes’ resolution. Second, using optical flow algorithm (in this report we use Ce Liu [23]) warping the previous and the next scenes to the current scene and combine them with fusion. As a result, we obtain an initial approximation to the SR image with empty gaps. Then, third, to fill the gaps, we use an inpainting technique (in this report we regularized with Total Variation) to complete the resulting SR image. Fourth, after we get the SR of the current scene we will warp from the current scene using this SR image to previous scene and next scene in order to have improved neighbor scenes. Finally we redo the second and third steps until the result is converged.

11

The following figure illustrates our scheme.

previous scene low-resolution image current scene low-resolution image next scene low-resolution image

(1) interpolate interpolate interpolate

(2) warp to current scene warp to current scene

warped from previous warped from next

(3) fusion

(4) total variation

(5) warp to previous scene warp to next scene

Improved previous scene Improved next scene

Warp above improved scenes to Reconstructed image and iterate

step 3,4 until the result is converged.

Reconstructed image

12

1.5 Keywords

Spatial resolution: a digital image is made up of small picture elements called pixels. Spatial resolution refers to the pixel density in an image and measures in pixel per unit area.

Aliasing: Artifacts arising from sampling and consequent loss of information cause by sampling with too high sampling rate.

Prior information or Prior probability: Statistics the probability assigned to a parameter or to an event in advance of any empirical evidence, often subjectively or on the assumption of the principle of indifference.

Penalty method and Penalty function: A penalty method solving a constrained optimization problem by a series of unconstrained problems whose solutions ideally converge to the solution of the original constrained problem. The unconstrained problem are formed by adding a term, called a “penalty function”, to the objective function.

13

Chapter 2

Motion Estimation

Motion estimation is an important component of Image Super resolution also in the field of video compression as shown in figure 10,without information from motion estimation we cannot proceed to the reconstruction step.

Motion estimation is the process to identify how the information has moved within a sequence of images. In the other word Super resolution from multiple images or frames is possible only in the presence of motion. Frame-to-frame motions provide different view of the scene and motion estimation is used to detect the sub-pixel shifts displacement between such frame and its adjacent ones. These sub-pixel shifts are produced by the camera or by the motion of objects in the scene. Motion estimation or Image registration for imaging super resolution is widely considered the most influential factor when considering sources of poor performance in super resolution. Motion estimation has to be accurate for achieving good super resolution result. Different techniques have been developed to solve the estimation problem. Motion estimation mainly compute shifts (displacements of pixels) or rotation(angle rotated of pixels) or both between two frames.

Figure 10: Overview of common super resolution algorithms

Motion estimation plays a big role in super resolution algorithm. In the end of this chapter, we will show the effect of mis-registeration on result of HR image.

14

2.1 Preliminaries 2.1.1 Motion type

-‐ Translation : [!!!!] = [!!

!! + !

!]

-‐ Similarity : [!!!!]= s [ !"# (!)

!!"# (!) !"# (!)!"# !

] [!!!!+ !

!]

-‐ Affine : [!!!!] = [!"!

!"! + !"!

!"! + !

!]

-‐ Homography : [!!!!] = !

! [!"!!"! + !"!

!"! + !

!], z = 𝑔𝑥! + ℎ𝑦! + i

2.1.2 Intensity Conservation Assumption The total intensity is conserved from frame to frame. This assumption holds under ideal conditions, i.e. no large motions, consistent lighting conditions and no occlusions. We can express intensity conservation as follows:

I1(x,y) = I2(x+u,y+v)+r+g r ~ N(0,𝜎2), g ~ U(-1,1)

Noise r,outlier g (occlusion,lighting change)

2.2 Motion Estimation Methods In this part we will focus on two motion estimation techniques for being commonly used. These are block-matching method and Optical flow method. Block-matching techniques are slower but more accurate. Optical flow methods, on the other hand, are less accurate but faster and more suitable for obtaining dense motion vectors where the motion is small. Let’s denote by a pixel in the LR image. x and y represent the spatial coordinates in the image plane coordinate system. MxN is the size of the LR image. m x n is the size of the block. B = 𝒎

𝒏 x 𝑵

𝒏 is the total number of blocks in the image.

bm = 𝑴

𝒎 and bn = 𝑵

𝒏 are the number of blocks in the horizontal and vertical axis.

𝒈𝒍(𝒃𝒙, 𝒃𝒚) is a block in the LR image. 𝒈𝒍 is the notation for an entire image. 𝒈𝒌 denotes the image we want to improve its resolution.

15

𝒇𝒍(x,y) denotes HR image pixel. P is the magnification factor. PMxPN is the size of on HR image. 𝒇𝒍(bx,by) is a HR image block PmxPn is the size of each block 𝑭𝒌 is the HR image we seek to estimate 𝒅𝒍,𝒌 𝒙,𝒍𝒓 and 𝒅𝒍,𝒌

𝒚,𝒍𝒓 are the horizontal and vertical components of displacement of a pixel(x,y) from the k-th LR image into the l-th image. (𝒅𝒍,𝒌

𝒙,𝒉𝒓, 𝒅𝒍,𝒌 𝒚,𝒉𝒓) are the motion vectors for HR images

then,

To obtain a LR image 𝒈𝒍, the corresponding HR image𝒇𝒍, we will decimate the HR image by taking one pixel each P in rows and columns. This process is depicted in Figure11. Mathematically we have

and our goal is to estimate fk from(gk-2,…,gk+2)

2.2.1 Block-Based Motion Estimation Algorithms Block-matching Motion Estimation by least square estimate.

Block-matching algorithms is one of the most popular approach for motion estimation between images due to its low computational cost involved. They simply divide one frame into blocks of size m x n. These algorithms search for each block in certain area of another frame in the sequence. They are then assuming that every pixel in a block has the same movement. - Choosing a large block size implies less motion vectors to calculate, but the probabilities for each block to contain objects with different movements are higher. - Choosing a small block size will increases the computation, but it is also more sensitive to the noise present in the image. Matching Criteria Mean of Squared Error (MSE)

The intensity value of the pixel at coordinate(n1,n2) in the frame k-th is given by

s(n1,n2,k) of block size N x N. (i,j) is a displacement.

16

MSE is computed for each displacement position (i,j)within a specified search range in the reference image and the displacement that gives the minimum value of MSE is the motion vector and given by

Figure 11: Procedure to obtain a LR image sequence from a HR image sequence by down sampling Sum of Absolute Differences (SAD)

We may use SAD in order to reduce the computational cost because it use only addition and subtraction

we are considering low resolution data g for the motion estimation. Low values of this measure show high correlation between the current block and the displaced block according to a particular motion vector(𝒅𝒍,𝒌

𝒙,𝒉𝒓, 𝒅𝒍,𝒌 𝒚,𝒉𝒓).The block that turns out to be the

most similar to the current block(bx,by)in the reference frame, is given by the candidate vector with the minimum SAD:

for all motion vectors in the search area A.

17

Searching for the best block in every position of the search window is called full search block-matching. Faster algorithms exist that decrease the computational load, but they usually degrade the quality of the estimations. For this report we use Full search motion estimation. Here are the commonly used steps: Step1:Calculate the best motion vector at pixel precision. Step2:Search for the best half-pixel precision motion vector at eight half-pixel positions around the point of minimum given in step1. These new positions contain interpolated values.

Figure 12: Block-matching frame division in blocks of size m x n. The reference block that is the most similar to the current block is found within the search defined by the

parameter p. Step3: Compare the current block with eight new interpolated blocks around the best half-pixel position. These blocks contain values that are interpolated from pixel precision image data and half-pixel precision interpolated data. 2.2.2 Optical Flow-Based Motion Estimation Algorithm

Optical flow is used to compute the motion of the pixels of an image sequence. It provides a dense pixel correspondence i.e. determine where the pixel of an image at time t are in the image at time t+1.This method is good when motion is small.

Since the Lucas-Kanade algorithm was proposed in 1981 for image alignment it has been widely used in computer vision for multiple tasks. One of them has been to compute the motion field between a pair of images within an image sequence or video. This technique calculates motion vector estimates(𝒅𝒍,𝒌

𝒙,𝒍𝒓, 𝒅𝒍,𝒌 𝒚,𝒍𝒓) between LR images l-th

and k-th,which satisfy the optical flow equation with the minimum pixel to pixel variation in the velocity field.There are multiple variations in the literature of this method, depending on the purpose, we will use the following algorithm. The motion vector for each pixel(x,y) is defined by 𝒅𝒍,𝒌 𝒍𝒓 (𝒙,𝒚)= (𝒅𝒍,𝒌

𝒙,𝒍𝒓, 𝒅𝒍,𝒌 𝒚,𝒍𝒓) We

will center a window of size m x n on each pixel, the error to be minimized in the optical flow equation will be as follows

18

Using a second-order Taylor expansion for the second term of Equation above we have

……………(**) and

To obtain more accurate motion vectors, two processes has been added: Motion vectors refinement The refinement is based on an iterative estimation of the motion vectors. Once the initial motion vector are calculated the image is compensated via this motion field and equation (**) is again applied for each pixel. Hierarchical approximation The hierarchical improvement is achieved using a pyramid of three levels to calculate the motion vectors in a coarse to fine approximation. Coarse-to-fine refinement -Lucas-Kanade is a greedy algorithm that converges to local minimum. -Initialization is crucial:if initialized with zero,then the underlying motion must be small. -If underlying transform is significant, then coarse-to-fine is a must.

19

Figure 13:Coarse-to-fine optical flow estimation

The algorithm, including these improvements, can be summarized in the

following steps: Step1:Downsample the images by a factor of 2 and 4 to build the hierarchical pyramid. Step2:Compute the motion field for the lowest level in the pyramid by minimizing the optical flow equation for each pixel as shown in equation(**). Step3:Iterate over the motion vectors to refine. Step4:For each higher level in the pyramid Step4.1:Propagate the motion vectors to the higher level. Step4.2:Iterate to refine. 2.2.3 Frequency Domain Motion Estimation Algorithms[3],[4] The frequency domain approach makes explicit use of the aliasing that exists in each LR image to reconstruct an HR image. Tsai and Huang first derived a system equation that describes the relationship between LR images and a desired HR image by using the relative motion between LR images. The frequency domain approach is based on the following three principles: i) the shifting property of the Fourier transform ii) the aliasing relationship between the continuous Fourier transform(CFT) of an original HR image and the discrete Fourier transform(DFT) of observed LR images. iii) and the assumption that an original HR image is bandlimited.

20

For example, let us assume that there are two 1-D LR signals sampled below the Nyquist sampling rate. From the above three principles, the aliased LR signals can be decomposed into the unaliased HR signal as shown inFigure 14.

Figure 14: Aliasing relationship between LR image and HR image.

The basic premise of these techniques is the idea that if a sequence of images,

thought of as a 3-D function [in two-dimensional(2-D) space and time]. Contains a uniformly translating pattern or object, then the energy of the 3-D fast Fourier transform(FFT) of this function will be mostly concentrated along a plane through the origin whose orientation is related to the velocity of the moving pattern. Hence, by computing the 3-D FFT and then finding one or more planes with strong energy concentration, one can estimate the desired velocity. An important advantage of this technique is that because of the linear superposition of the Fourier transform, the existence of multiple superimposed motion will be manifested in the FFT magnitude simply as energy concentrations along more than one plane through the origin.

21

2.3 Effects of Motion Estimation on Performance of Super Resolution.

As mentioned in the beginning of this chapter that motion estimation quality has impact on result of super resolution. We will show the result of super resolution with registration error.

Figure 14: Super resolution with mis-registration

We can conclude that the motion estimation technique must be as accurate as possible. For the past literatures a little attention has been paid to the performance of motion estimation techniques when used in SR problems. Schultz et al. [6] compared several sub-pixel motion estimation techniques into the Bayesian multiframe enhancement algorithm, eight parameter projection motion model,block-matching and Horn-Schunck optical flow estimation. They concluded that each technique has its own advantages and disadvantages depending on the kind of motion and the image characteristics.

22

Chapter 3

Reconstruction

3.1 Optical-flow based super resolution reconstruction 3.1.1 Image Warping

Since we know the information between scenes (registration of input image sequence) by motion estimation .We can use it to combine input sequence images after interpolation by warping them to match the input reference image. Image warping is the process of digitally manipulating an image such that any shapes portrayed, which have been significantly distorted in the image. Warping may be used for correcting image distortion for motion compensation.

Performing this warp involves interpolating the input images, a task that is normally performed using one of the standard algorithms, such as nearest-neighbor,bilinear, or cubic-spline interpolation.

Figure 15: (left) input low resolution images, and (right) combination of LR images to create the HR image.

23

3.1.2 Image Fusion. The warped images( images after interpolation and warping) can be regarded as multiple estimates of an underlying high resolution image. From these warped images, we need to combine them. The simplest approach is to average the estimates, taking either the mean, the median , or a robust mean in this report we use mean to average and fuse the warped images. 3.1.3 Image Inpainting using Total Variation Regularization. Image inpainting refers to the process of filling-in missing data or damged part in a designated region of the visual input. Regularization is used to measurement information with some general prior information about the desirable SR solution. It takes an important role in inpainting like in super resolution as well. The Total Variation(TV) norm is a gradient penalty function. The TV criterion penaliz

3.1.4 Image restoration

Figure 16:Deblurring block. Image restoration consists of deblurring and denoising.

This process of super resolution algorithms can be done by apply deblurring and denoising. Deblurring With Camera motion causes the blur to occurred and considered to be a serious problem in Super Resolution. This problem occurs when low ambient light conditions prevent an imaging system from using sufficiently short exposure times. Power Spread Function(PSF) is a filter that models the image blur effect. There are two types of blur in all digital imaging systems 1)the optical blur : specifies the fine-resolving capability of the optics. 2)the sensor integration blur : caused by the accumulation of incoming light over a finite-size sensor element.

24

Deblurring allows the recover of an image with a good resolution. Hence, the estimate of the PSF is essential to the application of these algorithms.

Several super resolution algorithms have included an optional post-processing step to deblur the super resolution image. In this report we use Wiener Filter for deblurring.

Denoising

Denoising is the process that removes the image noise,which is a random variation of brightness in image.Denoising process plays a role in super resolution as well as deblurring.In this report median filter is used for denoising.

25

Chapter 4

Result Qualitative result Image Sequence I: Plant

low resolution images size 128*128pixels HR images size 256*256 pixels

original image after interpolate by bilinear method

our method our method after wiener filter

26

Image Sequence II: Baby


original image


27

Image Sequence III: Bowling


original image


28

Image Sequence IV: Flower Pots


original image


29

Quantitative result Image set

Scene PSNR in dB with method Bicubic Nearest Bilinear NEDI Our

method Our method after wiener filter

Plant 1 27.9231 26.5548 28.1876 29.1386 29.3183 29.2522 2 27.8827 26.5044 28.1494 29.1281 29.4855 29.3650 3 27.8674 28.4846 28.1301 29.0807 29.3487 29.3026 4 27.8775 26.5078 28.1386 29.0734 29.3883 29.2945 5 27.8967 26.5099 28.1604 29.1597 29.3767 29.3270

Baby 1 33.8193 32.4879 34.0689 34.9001 35.0704 35.2489 2 33.9751 32.6006 34.2318 35.0397 35.2206 35.3284 3 33.9529 32.5996 34.1998 35.0523 35.3768 35.4869 4 34.1184 32.7626 34.3760 35.1955 35.3543 35.5016 5 34.2442 32.8791 35.4867 35.3660 35.3851 35.5319

Bowling 1 33.1791 31.5532 33.3628 34.9857 32.1387 32.7856 2 33.1458 31.5250 33.3173 34.9343 31.9131 32.5933 3 33.0885 31.4674 33.2478 34.9311 31.7601 32.4243 4 33.1957 31.5903 33.3347 35.1195 31.9877 32.6261 5 33.1906 31.5863 33.3255 35.0714 32.0564 32.7074

Pots 1 33.6402 31.8266 33.7436 35.5478 32.6208 33.8836 2 33.9926 32.1868 34.0338 35.7892 32.6648 33.9363 3 33.6477 31.8356 33.7206 35.6193 32.5010 33.8138 4 33.5858 31.7728 33.6784 35.4704 32.9696 34.2235 5 33.5723 31.7598 33.6545 34.8822 32.8238 34.1682

Lamps 1 34.0099 32.3928 34.1879 35.9281 32.8042 34.2530 2 34.0681 32.4370 34.2582 36.0965 32.8904 34.3201 3 34.1466 32.5403 34.2767 36.0286 32.8309 34.3136 4 33.9895 32.3590 34.1925 36.2382 32.8455 34.2982 5 34.2015 32.5790 34.3640 36.4198 32.7324 34.3120

Table 1: PSNR results of super resolution methods.

30

Chapter 5

Conclusion& Future works. Super Resolution algorithms are made to enhance image quality by increasing resolution and also recovery image from blur and noise. The algorithms’ performances are limited under availability of input low-resolution images (i.e. better performance with more input LR images) but still can be considered as okay because we reduced production cost and still be able to get better image.

Optical-Flow based Super resolution based on works of Lucas-Kanade and Horn-schunck. The algorithms have flexibility to non-planar motion(i.e. Human faces) and use less computational time than other approach. Our Optical-flow based super resolution have a well performance, as you can see in chapter 4 the experiment result shown by PSNR result is higher than other interpolation methods for some set of image and slightly has lower PSNR in some set of image as well.

Our super resolution method was applied to gray scale image only in the future we will try to improve the algorithms to be able to handle color image as well.

31

Bibliography: [1] S.Baker and T.Kanade, “Super Resolution-Optical Flow, 1999 [2] Aggelos K.Katsaggelos, Rafael Molina, Javier Mateos, “Super Resolution of Image and Video”, Department of Electrical Engineering and Computer Science,Northwestern University, Evanston, Illinois, USA, 2007 [3] P. Milanfar, “Projection-based, frequency-domain estimation of super-imposed translational motions,” J.Opt. Soc. Amer. A, vol.133, pp.2151-2161, Nov. 1996 [4] P. Vandewalle, S. Susstrunk, M.Vetterli, “A Frequency Domain Approach to Registration of Aliased Images with Application to Super-Resolution” Ecole Polytechnique Federale de Lausanne(EPFL),Lausanne,Switzerland. [5] D. Rivero Alfonso, “Image Reconstruction and Restoration for Optical Flow-based Super Resolution”,Master Thesis, Universitat Politecnica de Catalunya-ETSETB Barcelona, 2012 [6] D. Barreto, L.D.Alvarez, J.Abad,” Motion Estimation Techniques in Super-Resolution Image Reconstruction. A Performace Evaluation”, Department of Computer Science and Artificial Intelligence, University of Granada,Spain,2005

[7] Ce Liu, “Motion Estimation I&II”, Microsoft Research New England, 2005

[8] Xin Li, Michael T. Orchard, “New Edge-Directed Interpolation”, IEEE Transactions on Image Processing, Vol.10,No.10,October 2001

[9]Nhat Xuan Nguyen, “Numerical Algorithms For Image Superresolution”, Scientific Computing And Computational Mathematics, Stanford University, July 2000

[10] S.C.Park, M.L.Park, M.G.Kang, “Super-resolution image reconstruction : a technical overview”, IEEE Signal Procesing Magazine, vol.20, no.3,pp.21-36,2003.

[11]Wan-Chi Siu and Kwok-Wai Hung, “Review of image interpolation and Super-resolution”The Hong Kong Polytechnic University, Hong Kong, China.

[12]H. Hirschmüller and D. Scharstein. Evaluation of cost functions for stereo matching. In IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 2007), Minneapolis, MN, June 2007.

1

Appendix

MATLAB code %Assigning global values [nRow, nCol] = size(curr_orig); blksize = 32; trans_mode = 'BWHT'; subrate =0.25; N= nRow*nCol; M = round(subrate*N); rand_vect = randperm(N); % Reordering vector rand_vect = sort(rand_vect)'; select_vect= rand_vect(1:2:end); %current one %modified here MAT=vec2mat(select_vect,nRow/2); select_vect=MAT(1:2:end); select_vect=select_vect(:); select_vect=sort(select_vect); A = @(z) blk_f1d_MT(z, select_vect, rand_vect, trans_mode, blksize); A_T = @(z) blk_t1d_MT(z, N, select_vect, rand_vect,trans_mode, blksize); %downsampling HR images sequence into LR images sequence [prev_LR ,curr_LR, next_LR] = downsamplingHRs(prev_orig, curr_orig, next_orig); %interpolate them with several methods %1st-Classical method (bicubic,bilinear,nearest) [prev_bicubic_HR,prev_bilinear_HR,prev_nearest_HR] = classicalinterp(prev_LR); [curr_bicubic_HR,curr_bilinear_HR,curr_nearest_HR] = classicalinterp(curr_LR); [next_bicubic_HR,next_bilinear_HR,next_nearest_HR] = classicalinterp(next_LR); %2nd-NEDI prev_NEDI_HR=nedi(prev_LR); curr_NEDI_HR=nedi(curr_LR); next_NEDI_HR=nedi(next_LR); %Evaluate the results of interpolation by PSNR %Previous scene prev_PSNR_bicubic=PSNR1(prev_bicubic_HR,prev_orig); prev_PSNR_bilinear=PSNR1(prev_bilinear_HR,prev_orig); prev_PSNR_nearest=PSNR1(prev_nearest_HR,prev_orig);

2

prev_PSNR_nedi=PSNR1(prev_NEDI_HR,prev_orig); %find maximum PSNR prev_PSNR=[prev_PSNR_bicubic,prev_PSNR_bilinear,prev_PSNR_nearest]; [max_prevPSNR i]=max(prev_PSNR); if i == 1 display(max_prevPSNR ,'previous scene maximum by bicubic interpolation in dB'); elseif i == 2 display(max_prevPSNR,'previous scene maximum by bilinear interpolation in dB'); elseif i == 3 display(max_prevPSNR,'previous scene maximum by nearest interpolation in dB'); % elseif i == 4 % display(max_prevPSNR,'previous scene maximum by New Edge-Directed interpolation in dB'); end %choose output from the method that gives maximum PSNR if i == 1 interpHR_prev=prev_bicubic_HR; elseif i == 2 interpHR_prev=prev_bilinear_HR; elseif i == 3 interpHR_prev=prev_nearest_HR; % elseif i == 4 % interpHR_prev=prev_NEDI_HR; end %Current scene curr_PSNR_bicubic=PSNR1(curr_bicubic_HR,curr_orig); curr_PSNR_bilinear=PSNR1(curr_bilinear_HR,curr_orig); curr_PSNR_nearest=PSNR1(curr_nearest_HR,curr_orig); curr_PSNR_nedi=PSNR1(curr_NEDI_HR,curr_orig); %find maximum PSNR curr_PSNR=[curr_PSNR_bicubic,curr_PSNR_bilinear,curr_PSNR_nearest]; [max_currPSNR j]=max(curr_PSNR); if j == 1 display(max_currPSNR ,'current scene maximum by bicubic interpolation in dB'); elseif j == 2 display(max_currPSNR,'current scene maximum by bilinear interpolation in dB'); elseif j == 3 display(max_currPSNR,'current scene maximum by nearest interpolation in dB'); % elseif j == 4 % display(max_currPSNR,'current scene maximum by New Edge-Directed interpolation in dB'); end %choose output from the method that gives maximum PSNR if j == 1 interpHR_curr=curr_bicubic_HR; elseif j == 2 interpHR_curr=curr_bilinear_HR; elseif j == 3 interpHR_curr=curr_nearest_HR;

3

% elseif j == 4 % interpHR_curr=curr_NEDI_HR; end %next scene next_PSNR_bicubic=PSNR1(next_bicubic_HR,next_orig); next_PSNR_bilinear=PSNR1(next_bilinear_HR,next_orig); next_PSNR_nearest=PSNR1(next_nearest_HR,next_orig); next_PSNR_nedi=PSNR1(next_NEDI_HR,next_orig); %find maximum PSNR next_PSNR=[next_PSNR_bicubic,next_PSNR_bilinear,next_PSNR_nearest]; [max_nextPSNR k]=max(next_PSNR); if k == 1 display(max_nextPSNR ,'next scene maximum by bicubic interpolation in dB'); elseif k == 2 display(max_nextPSNR,'next scene maximum by bilinear interpolation in dB'); elseif k == 3 display(max_nextPSNR,'next scene maximum by nearest interpolation in dB'); % elseif k == 4 % display(max_nextPSNR,'next scene maximum by New Edge-Directed interpolation in dB'); end %choose output from the method that gives maximum PSNR if k == 1 interpHR_next=next_bicubic_HR; elseif k == 2 interpHR_next=next_bilinear_HR; elseif k == 3 interpHR_next=next_nearest_HR; % elseif k == 4 % interpHR_next=next_NEDI_HR; end %Apply Optical Flow %assigning values of para alpha = 0.092; ratio = 0.75; minWidth = 20; nOuterFPIterations = 7; nInnerFPIterations = 1; nSORIterations = 30; [nRow, nCol] = size(curr_orig); para = [alpha,ratio,minWidth,nOuterFPIterations,nInnerFPIterations,nSORIterations]; %using Coarse2finetwoframe warp 2 images to make 2 new current images base %on motion between the previous scene and the current scene, motion between

4

%the next scene and the current scene. for k = 1 : 1 [vxl,vyl,warpedbyprev] = Coarse2FineTwoFrames(interpHR_curr,interpHR_prev,para); [vxl,vyl,warpedbynext] = Coarse2FineTwoFrames(interpHR_curr,interpHR_next,para); warped_image = (warpedbyprev + warpedbynext)/2; y = A(curr_orig(:)); Xpred = warped_image(:); y_pred = A(Xpred); r = y - y_pred; r_rec = tval3_residual(r,A,A_T, nRow, nCol); img_rec = r_rec + warped_image; reconstructedImg = curr_orig; reconstructedImg(2:2:end,:) = img_rec(2:2:end,:); beforeimprove= PSNR1(curr_orig, reconstructedImg) % interpHR_curr=reconstructedImg; %choose wiener filter block that gives maximum PSNR afterwiener2=wiener2(reconstructedImg,[2 2]); PSNRAfterwiener2= PSNR1(curr_orig,afterwiener2); afterwiener3=wiener2(reconstructedImg,[3 3]); PSNRAfterwiener3= PSNR1(curr_orig,afterwiener3); afterwiener=[PSNRAfterwiener2,PSNRAfterwiener3]; [max_afterwiener ,t]=max(afterwiener); if t == 1 afterwiener=afterwiener2; elseif t == 2 afterwiener=afterwiener3; end PSNRAfterwiener= PSNR1(curr_orig,afterwiener) figure;imshow(curr_orig),figure;imshow(reconstructedImg),figure;imshow(afterwiener); end

Documents

My FInal Report!