Parallel and Distributed Audio (Image/Video) Concealment Using Nonlocal Sparse Representation

Parallel and Distributed Audio (Image/Video) Concealment Using

Nonlocal Sparse Representation

Xin Li

LDCSEE, WVU

Multimedia: a combination of content forms

Neural Basis of Media Content

“Missing Data” Problem

… … … …

x

kx

1x

q

p

p

packet loss

Sequential Parallel

Skype MSN

Error Concealment in Image Communication

videoimage

Sequential Parallel

Why Parallel?

• Avoid choosing scanning order and its associated error propagation problem

• Modeling multimedia signal at a patch (e.g., speech frame, image block) level instead of data (e.g., speech sample, image pixel) level

• Parallelism has to work with distributed processing together (redundancy is exploited to fight against uncertainty)

Prior Model in the Patch Space

MRPx

xyz

zy

xy

zManifold constraint of sensorysignals: dimensionality of localsubspace is <<patch dimension p

Sparsity-based Prior Model

k

tttw

10 BB

Nonlinear Dimensionality ReductionBy Locally Linear Embedding (LLE) Roweis and Saul, Science’2000

0WD

},...,,1{ 1 kwwdiag W],...,,[ 10 kBBBD

Sparsifying transform

B0 B3 B4B1B2

Nonlocal Sparse Representation

0FD

Approximated solution(FFT/DCT)

0WD

Optimal sparsifyingtransform (KLT)

B0 BkB1 Pack into(n+1)-D Array D (n+1)D-FFT

Thresholding

…

(n+1)D-IFFTPack into

(n+1)-D Array D

B0 BkB1…^ ^ ^

n-D

n-D

Why is it Nonlocal?

B4

B2

B3

B0

B1

RPP

Local neighborsin range (photometricsimilarity)

nonlocal patchesin domain (geometricproximity)

Algorithm Flowchart

Projectiononto

observationconstraint

Projectionontoprior

constraint

Data clustering

in Rp

PDACalgorithm

Better estimate of local neighborhood in the patch space

Inner loop

Outer loop

Better estimate of missing data in the time domain

Experimental Results (I)

sequential-I (parametric)

sequential-II (nonparametric)

parallel (this work)

Experimental Results (II)

outer loopproceeds

Extension into Images

27.06dB 29.14dB 30.16dB

original damagedexemplar

basedGuleryuz’s

schemeThiswork

18.06dB 17.57dB

35.72dB 17.93dB 37.37dB

22.36dB

26.75dB 24.63dB 31.84dB

original damagedexemplar

basedGuleryuz’s

schemeThiswork

26.52dB 25.84dB 33.08dB

28.78dB 28.09dB 34.26dB

21.21dB 14.66dB

17.83dB 17.66dB

26.58dB

20.24dB

Parallel and Distributed Processing (PDP)

• Dominant form of Connectionism (models mental or behavioral phenomena as the emergent processes of interconnected networks of simple units.)

• More widely known as neural networks since 1980s

Network Model Interpretation

Y

X

B

T

X1

B1X2

B2

B3

B4

I

Belief propagation in factor graph

observation

scene

patch

hidden