Upload
melanie-nicholson
View
214
Download
0
Tags:
Embed Size (px)
Citation preview
Hiroyuki Takeda, Hae Jong Seo, Peyman MilanfarEE Department
University of California, Santa Cruz
Jan 11, 2008
Statistical Image Quality Measures
UCSC MDSP Lab
Overview
BackgroundBackground
CCA-based Similarity Measure (Full-reference)CCA-based Similarity Measure (Full-reference)
Slide 1
ConclusionConclusion
SVD-based Quality Measure (No-reference)SVD-based Quality Measure (No-reference)
UCSC MDSP Lab
Develop quantitative measures that Develop quantitative measures that automatically predict the perceived image automatically predict the perceived image qualityquality
Objective Quality AssessmentObjective Quality Assessment
Full-reference Full-reference
No-reference No-reference
Reduced-reference Reduced-reference
Applications Applications
Image acquisition, compression, communication,Image acquisition, compression, communication,
displaying, printing, restoration displaying, printing, restoration
Slide 2
UCSC MDSP Lab
Overview
BackgroundBackground
CCA-based Similarity Measure (Full-reference)CCA-based Similarity Measure (Full-reference)
ConclusionConclusion
SVD-based Quality Measure (No-reference)SVD-based Quality Measure (No-reference)
UCSC MDSP Lab
Full-Reference Image Quality MeasureFull-Reference Image Quality Measure
Structural Similarity Measure [1]Structural Similarity Measure [1]
Focus on Focus on perceived changes in structural informationperceived changes in structural information variationvariation unlike error based approach ( i.e. MSE or unlike error based approach ( i.e. MSE or PSNR )PSNR )
MSE : 210MSE : 210
Original imageOriginal image
Contrast stretchedContrast stretched
JPEG compressedJPEG compressed Blurred Blurred
[1] [1] Zhou Wang et al, “ et al, “Image Quality Assessment: From Error Visibility to Structural SimilarityImage Quality Assessment: From Error Visibility to Structural Similarity ”, ”, IEEE TIP ‘IEEE TIP ‘ 04 04
Slide 3
Salt-pepperSalt-pepper
Mean shiftedMean shifted
UCSC MDSP Lab
Structural Similarity MeasureStructural Similarity Measure
Three components : Luminance , Contrast , StructureThree components : Luminance , Contrast , Structure
Small constant Small constant
Slide 4
Image patches being Image patches being compared compared
UCSC MDSP Lab
Drawback of SSIMDrawback of SSIM
Sensitive to spatial translation, rotation, and scale Sensitive to spatial translation, rotation, and scale changeschanges
due to due to simple correlation coefficientsimple correlation coefficient
A powerful statistical toolA powerful statistical tool
: Canonical Correlation Analysis (Hotelling, : Canonical Correlation Analysis (Hotelling, 1936)1936)
Solution Solution
OriginalOriginal RotatioRotationn
Zoom OutZoom Out TranslatioTranslationn
SSIM: 0.549SSIM: 0.549 SSIM: 0.551SSIM: 0.551SSIM: 0.505SSIM: 0.505
Slide 5
UCSC MDSP Lab
: : canonical correlationcanonical correlation
New Statistical Image Quality MeasureNew Statistical Image Quality Measure
Canonical Correlation Analysis (CCA)Canonical Correlation Analysis (CCA): : Find out a pair of direction vectors whichFind out a pair of direction vectors which
maximally correlatemaximally correlate the two datasets the two datasets
: : Useful propertyUseful property Affine–invariance Affine–invariance
Slide 6
UCSC MDSP Lab
CCA
CCA
CCA
New Statistical Image Quality MeasureNew Statistical Image Quality Measure
Canonical Correlation Structural Similarity Canonical Correlation Structural Similarity MeasureMeasure
original image
100 200 300 400 500
50
100
150
200
250
300
350
400
450
500
noisy, sigma= 35
100 200 300 400 500
50
100
150
200
250
300
350
400
450
500Gx GyP Gx GyP
P P
GyGx Gx Gy
: : Local Search Window at i Local Search Window at i thth position position
Slide 7
P : Pixel intensity
Gx,Gy : Gradients
AA BB
UCSC MDSP Lab
New Statistical Image Quality MeasureNew Statistical Image Quality Measure
Mathematical SolutionMathematical Solution
1) Calculate Covariance Matrix1) Calculate Covariance Matrix
2) Solve coupled eigen-value problems2) Solve coupled eigen-value problems
3) Define CCSIM as largest canonical 3) Define CCSIM as largest canonical correlation correlation
Slide 8
UCSC MDSP Lab
original image
Examples (1)Examples (1)
Original Original ImageImage
Zoom OutZoom Out
Slide 9
11 22
UCSC MDSP Lab
Distribution of CC1(compressed): Pixel + Gradient Value -->Mean(0.73212) Block Size:5
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1 SSIM: MSSIM = 0.3419WinSize = 5
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
original image
Examples (2)Examples (2)
CCSIMCCSIMSSISSIMM
11
0.340.34 0.730.73
22
Zoom OutZoom Out
Dis
trib
ution o
f C
C1(C
om
pre
ssed): P
ixel -->M
ean(0
.3098) B
lock S
ize:5
50
100
150
200
50
100
150
200
00.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
2211Original ImageOriginal Image
Slide 10
UCSC MDSP Lab
original image
Examples (2)Examples (2)
Original Original ImageImage
TranslationTranslation
11 33
Slide 11
UCSC MDSP Lab
Distribution of CC1(compressed): Pixel + Gradient Value -->Mean(0.75026) Block Size:5
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1 SSIM: MSSIM = 0.38452WinSize = 5
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
original image
Examples (2)Examples (2)
CCSIMCCSIMSSISSIMM
11
0.380.38 0.750.75
33
Original ImageOriginal Image
TranslationTranslation
Dis
trib
ution o
f C
C1(C
om
pre
ssed): P
ixel -->M
ean(0
.3098) B
lock S
ize:5
50
100
150
200
50
100
150
200
00.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
3311
Slide 12
UCSC MDSP Lab
original image
Examples (3)Examples (3)
Original Original ImageImage
RotationRotation
11 44
Slide 13
UCSC MDSP Lab
Distribution of CC1(compressed): Pixel + Gradient Value -->Mean(0.77315) Block Size:5
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1 SSIM: MSSIM = 0.41067WinSize = 5
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
original image
CCSIMCCSIMSSISSIMM
11
0.410.41 0.770.77
44
Original ImageOriginal Image
RotationRotation
Dis
trib
ution o
f C
C1(C
om
pre
ssed): P
ixel -->M
ean(0
.3098) B
lock S
ize:5
50
100
150
200
50
100
150
200
00.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
4411
Slide 14
Examples (3)Examples (3)
UCSC MDSP Lab
JPEG Compression ExampleJPEG Compression Example
Slide 15
Clean imageClean image(QF=100)(QF=100) JPEG(QF=10)JPEG(QF=10)JPEG(QF=50)JPEG(QF=50)
11 22 33
0.899 bits/pixel0.899 bits/pixel 0.352 bits/pixel0.352 bits/pixel8 bits/pixel8 bits/pixel
UCSC MDSP Lab
22
11
SSIM: MSSIM = 0.90153WinSize = 5
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
0.900.90
Distribution of CC1(compressed): Pixel + Gradient Value -->Mean(0.8533) Block Size:5
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
0.850.85
SSISSIMM
Clean ImageClean Image
JPEG (QF =50)JPEG (QF =50)
JPEG (QF =10)JPEG (QF =10)
Slide 16
JPEG Compression ExampleJPEG Compression Example
11
22
33
Distribution of CC1(Compressed): Pixel -->Mean(0.3098) Block Size:5
50 100 150 200
50
100
150
200
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
CCSIMCCSIM
SSIM: MSSIM = 0.786WinSize = 5
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
Distribution of CC1(compressed): Pixel + Gradient Value -->Mean(0.79459) Block Size:5
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
33
11
SSISSIMM
0.790.79
0.790.79
CCSIMCCSIM
UCSC MDSP Lab
Clean Image VS Compressed ImagesClean Image VS Compressed Images
Slide 17
QualityQuality
JPEG quality JPEG quality factorfactor
10 20 30 40 50 60 70 80 90 1000.75
0.8
0.85
0.9
0.95
1
Pixel + Gradient : Window Size = 5SSIM : Window Size = 5
SSISSIMM
CCSIMCCSIM
UCSC MDSP Lab
Denoising ExampleDenoising Example
Clean ImageClean Image Denoised by Denoised by SKR[2]SKR[2]
WGN(sigma=15WGN(sigma=15))
Slide 18
[2] Takeda[2] Takeda et al., “ Kernel Regression for image processing and reconstruction ”, et al., “ Kernel Regression for image processing and reconstruction ”, IEEE TIP ‘IEEE TIP ‘ 07 07
denoised by SKR, sigma= 15noisy, sigma= 15 original image
11 22 33
UCSC MDSP Lab
SSIM: MSSIM = 0.47514WinSize = 5
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
Distribution of CC1(Noisy): Pixel -->Mean(0.4755) Block Size:5
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
Denoising ExampleDenoising Example
CCSIMCCSIM
SSISSIMM
Slide 19
Clean ImageClean Image
WGN( sigma =15 )WGN( sigma =15 )
Denoised by SKRDenoised by SKR
SSIM: MSSIM = 0.88668WinSize = 5
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
Distribution of CC1(Denoised): Pixel + Gradient Value -->Mean(0.88536) Block Size:5
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
denoised by SKR, sigma= 15
noisy, sigma= 15
original image
Distribution of CC1(Compressed): Pixel -->Mean(0.3098) Block Size:5
50 100 150 200
50
100
150
200
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
CCSIMCCSIM
SSISSIMM
11
22
33
22
11
33
110.470.47
0.470.47
0.890.89
0.890.89
UCSC MDSP Lab
Clean VS (Noisy & Denoised images)Clean VS (Noisy & Denoised images)
WGN: Noise levelWGN: Noise level
QualityQuality
Slide 20
QualityQuality
WGN: Noise levelWGN: Noise level
5 10 15 20 25 300.5
0.6
0.7
0.8
0.9
Pixel+Gradient: Window Size = 5SSIM : Window Size = 5
5 10 15 20 25 300.2
0.4
0.6
0.8
1
Pixel: Window Size = 5SSIM : Window Size = 5
Clean VS NoisyClean VS Noisy Clean VS DenoisedClean VS Denoised
SSISSIMM
CCSIMCCSIM
SSISSIMM
CCSIMCCSIM
UCSC MDSP Lab
MotionEstimation
Resolution enhancement from video frames captured by a commercial webcam(3COM Model No.3719)
Steering Kernel Regression
Super-resolutionSuper-resolution
Slide 21
UCSC MDSP Lab
original image superresolved by SKR
Super-resolution ExampleSuper-resolution Example
Clean ImageClean Image
(512 x 512)(512 x 512)Super-resolved Super-resolved by SKRby SKR
Low resolution Low resolution
SequenceSequence
(64x64 32 frames)(64x64 32 frames)
Slide 22
11 22 33
UCSC MDSP Lab
Distribution of CC1(Denoised): Pixel + Gradient Value -->Mean(0.91575) Block Size:3
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1 SSIM: MSSIM = 0.86996WinSize = 5
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
Super-resolution ExampleSuper-resolution Example
SSISSIMM
Slide 23
Clean ImageClean Image
Super-resolved by Super-resolved by SKRSKR
Low resolution Low resolution Sequence( 32 frames)Sequence( 32 frames)
Dis
trib
utio
n of
CC
1(C
ompr
esse
d): P
ixel
-->
Mea
n(0.
3098
) B
lock
Siz
e:5
5010
015
020
0
50 100
150
200
00.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
CCSIMCCSIM
331111
22
33
0.870.87 0.910.91
original image
superresolved by SKR
UCSC MDSP Lab
Overview
BackgroundBackground
CCA-based Similarity Measure (Full-reference)CCA-based Similarity Measure (Full-reference)
ConclusionConclusion
SVD-based Quality Measure (No-reference)SVD-based Quality Measure (No-reference)
UCSC MDSP Lab
No-Reference SVD-Based MeasureNo-Reference SVD-Based Measure
SVD N x NSingular value decomposition of local gradient matrix:Singular value decomposition of local gradient matrix:
Local orientation dominanceLocal orientation dominance
It becomes close to 1 when there is one dominant It becomes close to 1 when there is one dominant orientation orientation
in a local area.in a local area.It takes on small values in flat or highly textured (or pure It takes on small values in flat or highly textured (or pure noise) noise)
area. area. So, this quantity tells us about the “edginess” of the So, this quantity tells us about the “edginess” of the region beingregion being
examined.examined.
UCSC MDSP Lab
Properties of Local Orientation Dominance(1)Properties of Local Orientation Dominance(1)
Slide 25
Density function for i.i.d. white Gaussian noiseDensity function for i.i.d. white Gaussian noise
N: the window size
[1] A. Edelman. Eigenvalues and condition numbers of random matrices, SIAM Journal on Matrix Analysisand Applications 9 (1988), 543-560.
Note : the PDF is independent from the noise variance, but depends on the window size.
[2] X. Feng and P. Milanfar. Multiscale principal component Analysis for Image Local Orientation Estimation, Proceedingof 36th Asilomar Conference on Signals, Systems, andComputers, Pacific Grove, CA, Nov. 2002
N=3
N=5
N=7
N=9
N=11
UCSC MDSP Lab
Properties of Local Orientation Dominance(2)Properties of Local Orientation Dominance(2)
Slide 26
The mean values for a variety of test images with The mean values for a variety of test images with added white Gaussian noise.added white Gaussian noise.
N = 11
The mean values for pure noise are always constant.
Rememberthe number
0.06
UCSC MDSP Lab
The Performance AnalysisThe Performance Analysis
Suppose we have a noisy image and a denoised version Suppose we have a noisy image and a denoised version
using some filter:using some filter:
The residual image is essentially just noise. The residual image is essentially just noise.
of the residual image must be close to the value of the residual image must be close to the value expectedexpected
for pure noise.for pure noise.
Slide 27
If the filter cleans up the given image effectively, If the filter cleans up the given image effectively,
: a given noisy image
: the estimated (denoised) image
: the residual image
UCSC MDSP Lab
Example (1)Example (1)
Image denoising by bilateral filterImage denoising by bilateral filter
Slide 28
Denoising experimentDenoising experiment
Bilateral filter has two parameters:Bilateral filter has two parameters:
Spatial smoothing parameterSpatial smoothing parameter , and , and radiometric smoothing parameterradiometric smoothing parameter
The original image A noisy image, Added white Gaussian noise,SNR=20dB, PSNR=29.25dB, RMSE = 8.67
C. Tomasi and R. Manduchi, “Bilateral filtering for gray and color images”, Proceedings of the 1998 IEEEInternational Conference of Computer Vision, Bombay, India, pp. 836-846, January 1998.
UCSC MDSP Lab
The Performance Analysis of Bilateral FilterThe Performance Analysis of Bilateral Filter
Slide 29
N = 11
The plot of as a function of the smoothing parameters: The plot of as a function of the smoothing parameters:
UCSC MDSP Lab
Denoising ResultDenoising Result
Slide 30
Bilateral filterPSNR = 42.87dB,RMSE = 1.833
The noisy imageResidual
UCSC MDSP Lab
The Performance Analysis of Bilateral FilterThe Performance Analysis of Bilateral Filter
Slide 31
The plot of as a function of the smoothing parameters: The plot of as a function of the smoothing parameters:
N = 11
UCSC MDSP Lab
Denoising ResultDenoising Result
Slide 32
Bilateral filterPSNR = 39.57dBRMSE = 2.68
The noisy imageResidual
The filter also removes image
contents.
UCSC MDSP Lab
What If We Pick the Parameters by the Best RMSE?What If We Pick the Parameters by the Best RMSE?
Slide 33
The plot of RMSE as a function of the smoothing The plot of RMSE as a function of the smoothing parameters: parameters:
UCSC MDSP LabSlide 34
Denoising Result Denoising Result
Bilateral filter, PSNR = 42.87dBRMSE = 1.832
The noisy imageResidual
UCSC MDSP LabSlide 35
Example (2)Example (2)
Iterative Steering Kernel RegressionIterative Steering Kernel Regression
The original image The noisy image,Added white Gaussian noise,SNR=5.6dB, PSNR = 20.22dB RMSE = 24.87
Iteratively cleaning up noisy imagesIteratively cleaning up noisy images
Using the local orientation dominance, we find the optimal Using the local orientation dominance, we find the optimal number of iterations.number of iterations.
UCSC MDSP LabSlide 36
Denoising Result (1)Denoising Result (1)
The plot of as a function of the smoothing parameters:The plot of as a function of the smoothing parameters:
UCSC MDSP LabSlide 37
Denoising Result Denoising Result
ISKR, IT = 15,PSNR = 31.33 dB
RMSE = 6.92
The noisy imageResidual
UCSC MDSP LabSlide 38
If the Ground Truth is Available,If the Ground Truth is Available,
The plot of RMSE as a function of the smoothing The plot of RMSE as a function of the smoothing parameters:parameters:
RMSERMSE
UCSC MDSP LabSlide 39
Denoising Result Denoising Result
ISKR, IT = 12, PSNR = 31.69 dB
RMSE = 6.64
The noisy imageResidual
UCSC MDSP Lab
Overview
BackgroundBackground
CCA-based Similarity Measure (Full-reference)CCA-based Similarity Measure (Full-reference)
ConclusionConclusion
SVD-based Quality Measure (No-reference)SVD-based Quality Measure (No-reference)
UCSC MDSP Lab
ConclusionConclusion
Two new statistical quality measuresTwo new statistical quality measuresCCSIM(CCA-based)CCSIM(CCA-based) : full-reference : full-reference
Slide 40
SVD-based measure: SVD-based measure: no-referenceno-reference
CCSIM is a general version of SSIMCCSIM is a general version of SSIM
The proposed methods can be easily The proposed methods can be easily extended to video using 3-d local window.extended to video using 3-d local window.
We showed examples of JPEG compression, denoising , and We showed examples of JPEG compression, denoising , and supersuper
Resolution with comparison to SSIMResolution with comparison to SSIMSVD-based measure is applicable for any SVD-based measure is applicable for any denoising filter.denoising filter.
We illustrated application to global parameter We illustrated application to global parameter optimization.optimization.Locally adaptive parameter optimization is also possible.Locally adaptive parameter optimization is also possible.
UCSC MDSP Lab
AuthorsAuthors
[1] Hiroyuki Takeda : [1] Hiroyuki Takeda : [email protected]
www.ucsc.edu/~htakedawww.ucsc.edu/~htakeda
[2] Hae Jong Seo : [2] Hae Jong Seo : [email protected]
www.ucsc.edu /~rokafwww.ucsc.edu /~rokaf
[3] Peyman Milanfar : [3] Peyman Milanfar : [email protected]
www.ucsc.edu/~milanfarwww.ucsc.edu/~milanfar
UCSC MDSP Lab
Thank you !Thank you !
UCSC MDSP Lab
Super-resolution ExampleSuper-resolution Example
Clean ImageClean ImageSuper-resolved Super-resolved by SKRby SKR
Down-sampled(2)Down-sampled(2)
+WGN(sigma=15)+WGN(sigma=15)
Extra 1
noisy, sigma= 15 original image
11 22 33
UCSC MDSP Lab
SSIM: MSSIM = 0.70632WinSize = 5
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
Super-resolution ExampleSuper-resolution Example
SSISSIMM
Extra 2
Clean ImageClean Image
Super-resolved by Super-resolved by SKRSKR
Distribution of CC1(Noisy): Pixel + Gradient Value -->Mean(0.85761) Block Size:5
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
original image
Down-sampled(2)Down-sampled(2)+WGN (sigma=15)+WGN (sigma=15)
noisy, sigma= 15
Dis
trib
utio
n of
CC
1(C
ompr
esse
d): P
ixel
-->
Mea
n(0.
3098
) B
lock
Siz
e:5
5010
015
020
0
50 100
150
200
00.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
CCSIMCCSIM
331111
22
33
0.710.71 0.850.85