Image deblocking using local segmentation

By Mirsad Makalic

Supervisor: Dr. Peter Tischer

Presentation Outline

An introduction to JPEG Lossy image compression Discrete Cosine Transform (DCT) and quantization

Local segmentation and prior research Measuring image quality Deblocking Filter Super-Resolution Filter Conclusion

Lossy image compression

JPEG is the most common lossy image compression format Easy to implement Good quality with high compression ratios

Block based transform approach Divide image into 8x8 blocks Perform discrete cosine transform (DCT) on each 8x8

block Quantize the DCT coefficients

Discrete Cosine Transform

Transform 8x8 block of pixels into a set of weighted basis functions (DCT coefficients)

DC coefficient

AC coefficients

DCT - Subbands

An image consisting of just one coefficient from each 8x8 block is called the sub-band image

An image which is divided into 8x8 blocks has 64 possible sub-band images

An 8x8 image split into 2x2 blocks has 4 subbands

Quantization

Exploit visual redundancy Quantization is a many-to-one mapping

Divide each DCT coefficient by a value and round to the nearest integer

The decoder makes a guess from a range of values (pick midpoint by default)

5 6 7 8 9 10 11 12 13 14

= 1Quantizer = 10

Quantization

Coarse quantization introduces artifacts into reconstructed image DCT coefficients are

reconstructed inaccurately

Most visually distracting artifact is blockiness

JPEG compressed image at quality 10 PSNR: 30.41 dB

Deblocking Techniques

Three approaches in the literature:Filter the reconstructed pixel valuesAttempt to reconstruct DCT coefficients more

accuratelyA hybrid approach

Local Segmentation

Most deblocking filters introduce excessive blurringDestroys the structure of the imageEdges lose their sharpness

Local segmentation takes into account the structure of the image

Local Segmentation

Divide a mask of pixels into N segments and filter each segment independently

Average of whole mask = 43.22, average of yellow segment = 20.8

Two questions:How many segments do we use?How do we segment a mask of pixels?

Prior Research

Lukasz Kizewski, BSE (hons) 2004 DC subband approach

Filter using a mask of DC subbands How do we segment a mask of pixels?

Segment the pixel mask using thresholding How many segments do we use?

Do-No-Harm heuristic

Prior Research

Do-No-Harm heuristicTry a 1-segment model (average of the whole

mask) If filtered value is implausible reject and try a 2-

segment model A plausible value is one which falls inside the

quantization range: midpoint +/- ½ QuantumTry a 2-segment model

If still implausible then don’t filter

Prior Research

Results

(a) Unfiltered image (a) Filtered image

Room for improvement

No objective measure used to test the effectiveness of the filterDifficult to make comparisonsDifficult to rate changes in filter

Works only on DC subbandAC subbands contain edge and texture

information A very simple local segmentation method

Measuring image quality

Peak-signal-to-noise-ratioMost commonly used metricDoes not necessarily reflect the subjective

visual quality Generalized Block-Edge Impairment

Metric (GBIM) – H.R. Wu, M. Yuen

Measuring image quality - GBIM

Measures the quality of DCT encoded images Assume that what happens inside a block is the same

as what happens across blocks Take absolute mean difference of pixels inside a block

(vertical/horizontal) Take absolute mean difference of pixels across blocks

(vertical/horizontal) Compare them, if the two differ greatly than it is a sign of

blockiness

100 101 130 131

101 101 132 131

102 100 131 130

102 101 132 129

2x2 block example with vertical blockiness

A new deblocking filter

Filter all coefficientsTreat each 8x8 block as a 64 element vector

where each value in the vector is one of the DCT coefficients

Local segmentation no longer as simple (need to segment masks of vectors)K-Means or K-Nearest Neighbours

DNH needs to work on vectors

A new deblocking filter

Basic structure of filter is same: Create an NxN mask where each item is a 64 element

DCT coefficient vector Segment mask using K-Means or K-Nearest

Neighbours segmentation Check if segmentation produces valid result using

DNH, if not, try different segment If no segmentation produces valid result, leave alone

K-Means

Start with one segment (average of mask) If segmentation is invalid, increase number

of segments by one until a maximum number of segments is reached

Try largest change first

K-Nearest Neighbours

Set the number of nearest neighbours to find as the number of items in the entire mask

Keep decreasing by segment size by one until a valid segment is found

Vector DNH

Center DNH Compare the filtered vector against only the center

vector in the mask Strict Segment DNH

Compare the filtered vector against all the vectors in the segment the center vector is in

Lesser Strict Segment DNH Same as strict segment DNH with some error

tolerance

Results

Best found parameters for the filter: 3x3 vector mask (covers 24x24 pixels) K-Nearest Neighbour segmentation Lesser strict segment DNH with 1% error tolerance

Tested: 5x5 mask, K-Means, Center DNH, Strict DNH etc. Many variations of the filter parameters

Results

(a) JPEG compressed image (b) Filtered image

An improvement of 0.07 dB in PSNR and a reduction of 0.62 in GBIM

Results

(a) JPEG compressed image (b) Filtered image

An improvement of 0.07 dB in PSNR and a reduction of 0.55 in GBIM

Other uses for local segmentation

Super-resolutionCombine multiple slightly different images to

form one higher quality imageCan we extract more information out of a

single image? Neighbouring pixels are similar and share

information Use local segmentation

Super-resolution filter

Very similar to the deblocking filter Instead of using a 64 element vector for each

DCT coefficient, use one element vector containing each pixel value in the mask

How to test if it works?Convert 8 bit image to 4 bits and attempt to

reconstruct back an 8 bit image

Results

Best found parameters for filter:3x3 maskK-Means segmentation with up to three

segmentsCenter DNHMore than one iteration of the filter can further

improve PSNR

Results

An improvement of 1.38 dB in PSNR

(a) An image rounded to 4 bits per pixel

(a) Filtered image (8 bits per pixel)

Results

(a) An image rounded to 4 bits per pixel

(a) Filtered image (8 bits per pixel)

An improvement of 1.32 dB in PSNR

Conclusion

Filtering AC subbands is difficult because most have been quantized to zero or have very large quantization ranges

Most improvements in image quality are from the stricter DNH and better local segmentation techniques

The increase in computational complexity may not be worth the increase in image quality

Conclusion

Super-resolution filter shows a lot of promiseLarge increase in PSNR and image quality

A different DNH heuristic may work better with the super-resolution filter

Future Research

Allow more variation in pixels for the strict DNHAssume local mask is linear and not constant

Try different segmentation techniquesRegion growing

Further investigate iterative filtering

The End

Questions?