The Proposed Algorithm We suggest training a dictionary using the LR image itself and restore each decimated patch by a sparse

composition over the dictionary using a weighted version of the Simultaneous OMP.

The restored image is obtained by averaging the HR patches, followed by a simple projection of the known pixels in 𝐱 on the outcome.

Notations:

W𝑖,𝑗 sets a high weight for known pixels and a low one for the

unknown ones, multiplies by exp − 𝑥𝑖 − 𝑥𝑗 1/𝑐𝑤 .

𝛼𝑖𝑟 𝐀𝑖

𝑠𝑝

0,∞ counts the non-zero elements rows in the matrix 𝛼𝑖

𝑟 𝐀𝑖𝑠𝑝

.

𝛼𝑖𝑟 is the representation of the non-weighted version of the reference patch (stabilizer).

𝐀𝑖𝑠𝑝

is the representation of the weighted versions of the reference patch and its K – Nearest Neighbors.

𝐑𝑗 is an operator that extracts the jth-patch from the image.

The proposed two-stage algorithm:

1. First stage: Joint sparse-coding using the K-nearest “strong” patches and reconstructing the image using the “strongest” patches.

2. Second stage: Use all the patches (“strong” and “weak”), both in the sparse-coding and the reconstruction steps.

A Basic Observation The more known pixels within a patch, the better the restoration.

The number of known pixels depends on its location (“strong” and “weak” patches).

We suggest “increasing” the number of known pixels based on the self- similarity assumption (e.g. the bright patches are the K-Nearest Neighbors of each dark patch).

Motivation and Goals

Adaptive sparse representation modeling is a promising image prior, which has been shown to be powerful in filling-in missing pixels in an image.

Processing groups of related patches together (based on the self-similarity assumption) exploits their correspondence and leading often times to improved results.

{yaniv,matanpr,elad}@{tx,cs,cs}.technion.ac.il

The Interpolation Problem

Given a Low-Resolution (LR) image 𝐲 = 𝐔L𝐱 , where 𝐱 is the High-Resolution (HR) image and 𝐔L decimates the image by a factor of L along the horizontal and vertical dimensions, our goal is to recover 𝐱 from 𝐲.

Single Image Interpolation via Adaptive Non-Local

Sparsity-Based Modeling

The Core Idea A common patch-based image restoration scheme:

Problems…

Each patch is interpolated independently.

Sparse-coding tends to err due to small number of existing pixels.

Yaniv Romano The Electrical Engineering Department

Matan Protter The Computer Science Department

Michael Elad The Computer Science Department

Technion – Israel Institute of Technology

This research was supported by the European Research Council under EU’s 7th Framework Program, ERC Grant agreement no. 320649, and by the Intel Collaborative Research Institute for Computational Intelligence.

Dictionary Update

Initial Dictionary

Interpolate each patch

Zero-Filled Image

(Not so impressive) HR Image

- Known pixel

- Unknown pixel

LR image

Weighted OMP

Weighted KSVD

LR image

kth iteration interp. image

𝑟

ℎ

×W𝑖,𝑗 𝛼𝑖𝑟

𝑖=1𝑁 , 𝐀𝑖

𝑠𝑝

𝑖=1

𝑁≔ argmin 𝛼𝑖

𝑟 𝐀𝑖𝑠𝑝

0,∞

𝑁

𝑖=1

s. t ∀𝑖 𝐃𝛼𝑖𝑟 − 𝐑𝑖𝐱

𝑒𝑠𝑡22 + 𝐃 𝐀𝑖

𝑠𝑝

𝑗− 𝐑𝑗𝐱

𝑒𝑠𝑡

W𝑖,𝑗

2≤ T𝑖

𝑗∈𝑆𝑖

Joint weighted sparse-coding (weighted SOMP)

𝐃 , 𝛼𝑖 𝑖=1𝑁 , 𝐀𝑖 𝑖=1

𝑁 ≔

argmin 𝐃 𝛼𝑖 − 𝐑𝑖𝐱 𝑒𝑠𝑡

2

2+

𝑁

𝑖=1

𝐃 𝐀𝑖 𝑗 − 𝐑𝑗𝐱 𝑒𝑠𝑡

W𝑖,𝑗

2

𝑗∈𝑆𝑖

𝑁

𝑖=1

s. t ∀𝑖 𝑆𝑢𝑝𝑝 𝛼𝑖 𝐀𝑖 = 𝑆𝑢𝑝𝑝 𝛼𝑖𝑟 𝐀𝑖

𝑠𝑝

Dictionary Update (weighted KSVD)

𝐱 ≔

argmin 𝐃 𝐀𝑖𝑠𝑝

𝑗− 𝐑𝑗𝐱

W𝑖,𝑗

2

𝑗∈𝑆𝑖

𝑁

𝑖=1

Image Reconstruction

s. t. 𝐲 = 𝐔L𝐱

𝐃 ← 𝐃

𝐱 𝑒𝑠𝑡 ← 𝐱

Interpolate the missing

pixels ? S𝑖

Visual Results Interpolation by a factor of 2 (75% missing pixels)

Visual Results Interpolation by a factor of 3 (≈89% missing pixels)

…

𝑐

ℎ

L𝑐

L𝑟

0≠

0≠

0≠

0≠

Results

Peak Signal to Noise Ratio (PSNR) [dB] = 20 log10 255 MSE , Higher is better.

Method Cubic SAI SME PLE NARM Ours Cubic SAI SME PLE NARM Ours

Average PSNR over 18 well-

known images 28.98 29.51 29.62 29.62 29.98 30.09 25.52 25.83 25.95 26.08 26.21 26.44

State-of-the-art Performance

SAI SME

NARM Proposed

Original

PLE

Original SAI SME PLE NARM Proposed

Original SAI SME PLE NARM Proposed

References KSVD: M. Elad and M. Aharon, “Image denoising via sparse and redundant representations over learned dictionaries”, IEEE TIP, 2006.

SAI: X. Zhang and X. Wu, “Image interpolation by adaptive 2-D autoregressive modeling and soft-decision estimation,” IEEE TIP, 2008.

SME: S. Mallat and G. Yu, “Super-resolution with sparse mixing estimators,” IEEE TIP, 2010.

PLE: G. Yu, G. Sapiro, and S. Mallat, “Solving inverse problems with piecewise linear estimators: from Gaussian mixture models to structured sparsity,” IEEE TIP, 2012.

NARM: W. Dong, L. Zhang, R. Lukac, and G. Shi, “Sparse representation based image interpolation with nonlocal autoregressive modeling,” IEEE TIP, 2013.

Grouping

Original

SAI

SME

PLE

NARM

Proposed