Image and Multidimensional Signal Processing

Image and Multidimensional Signal Processing Colorado School of Mines

Colorado School of Mines

Image and Multidimensional Signal Processing

Professor William Hoff

Dept of Electrical Engineering &Computer Science

http://inside.mines.edu/~whoff/


Image Compression - Lossy


Lossy Compression

• Reconstructed image is different from original

• Hopefully differences are unnoticeable, or minor

• We will look at: – Block transform coding methods, using the discrete cosine transform

(such as the JPEG standard)

– Predictive coding

3


Block Transform Coding

• Divides the image into subimages, or blocks

• Apply a transform (e.g., Fourier) to each block

• Quantize and encode the coefficients

4

Compression


Transform Coding

• General forward transform of image g, size nxn

• Inverse transform

• Example: Fourier transform

1 1

0 0

( , ) ( , ) ( , , , )n n

x y

T u v g x y r x y u v

1 1

0 0

( , ) ( , ) ( , , , )n n

u v

g x y T u v s x y u v

2 ( )/ 2 ( )/

2

1,j ux vy n j ux vy nr e s e

n

r,s are the forward and inverse transformation kernels (also called basis functions)

5


Example: Walsh-Hadamard Transform (WHT)

• Kernels:

• where – nxn is the size of the kernel, and n = 2^m – bk is the kth bit – Summation is done in modulo 2 arithmetic – The p’s are

p0(u) = bm-1(u) p1(u) = bm-1(u) + bm-2(u) p2(u) = bm-2(u) + bm-3(u) : pm-1(u) = b1(u) + b0(u)

1

0

( ) ( ) ( ) ( )

( , , , ) ( , , , )

11

m

i i i i

i

b x p u b y p v

r x y u v s x y u v

n

6


Example WHT Basis Functions

7

+1 -1


Discrete Cosine Transform (DCT)

• Kernels:

• DCT used in JPEG (wavelets are used in JPEG2000)

1,...,2,1for2

0for1

)(

where

2

)12(cos

2

)12(cos)()(

),,,(),,,(

NuN

uNu

N

vy

N

uxvu

vuyxhvuyxg

8

% Show DCT kernels

N = 32;

x=0:N-1;

y=0:N-1;

u = 1;

v = 4;

au = sqrt(2/N);

av = sqrt(2/N);

if u==1

au = sqrt(1/N);

end

if v==1

av = sqrt(1/N);

end

gx = au*cos((2*x+1)*u*pi/(2*N));

gy = av*cos((2*y+1)*v*pi/(2*N));

figure, plot(x,gx);

figure, plot(y,gy);

g = gx'*gy;

figure, surf(g);


DCT 4x4 Basis Functions

9


Approximation Errors

Apply a transform to each 8x8 subimage block

Keep highest 50% of coefficients in each block

Then reconstruct image (by taking the inverse transform) using the remaining coefficients

Reconstructed Error Image

10

RMS error: 2.32 RMS error: 1.78 RMS error: 1.13


Effect of Subimage Size

Image: lena

Truncate smallest 75% coefficients in each subimage

Figure 8.26

11


12


Quantizing Transform Coefficients

• Methods: – Threshold coding: Within each 8x8 subimage, keep the top N% of the

coefficients; or those with magnitude greater than a threshold

• Matlab exercise with “blkproc”

– Zonal coding: Keep coefficients with maximum variance across all subimages

• Matlab’s “dctdemo”

• Then quantize to a fixed or variable number of bits

13


Threshold coding – Matlab example

• dct2

– The function dct2 performs 2D discrete cosine transform on a matrix

B = dct2(A)

– The function idct2 performs the reverse transformation A2 = idct2(B)

• blkproc

– Use “blkproc” to apply dct2 to each 8x8 block J = blkproc(I,[8 8],@dct2);

– You could also apply your own function (eg., threshold) to each block J = blkproc(I,[8 8],@mythresh);

14

Image and Multidimensional Signal Processing Colorado School of Mines 15

• DCT is performed on each 8x8 subimage (block) >> I = imread('cameraman.tif');

>> J = blkproc(I,[8 8],@dct2);

DCT coefficients


A note on functions in Matlab

• You can write a function and call it from your program

• Syntax: function B = myfunc(A)

% This function computes something from A and

returns B

:

B = …

• Store this in a file called myfunc.m

• Put in current working directory

16


Truncating coefficients

• Write a function called “mytrunc” that truncates the smallest 75% coefficients in an image

• The function should – Take the absolute value of each pixel in the image

– Sort the values in ascending order

– Find the value that is 75% down the list

– Threshold the image using that value

17

function B = mytrunc(A)

% Truncate the lowest 75% of the magnitudes within A

Aabs = abs(A);

vals = sort(Aabs(:)); % Sort the values from low to high

:

B = A .* (Aabs > thresh);


18

clear all

close all

I = double(imread('lena.tif'));

wsize = 8;

J = blkproc(I,[wsize wsize],@dct2);

imshow(J,[]), title('J');

% Truncate 75% of the values within each block

Jtrunc = blkproc(J,[wsize wsize],@mytrunc);

figure, imshow(Jtrunc,[]), title('Jtrunc');

pct = sum(sum(Jtrunc == 0))/(size(I,1)*size(I,2));

fprintf('Percentage of zero coeffs: %f\n',100*pct);

K=blkproc(Jtrunc,[wsize wsize],@idct2);

figure, imshow(K,[]), title('K');

R = I - K;

disp('RMS error:');

sqrt(mean2(R .^ 2))

• Try on different block sizes

• Do you get this result?


Quantizing Transform Coefficients

• Methods: – Threshold coding: Within each 8x8 subimage, keep the top N% of the

coefficients; or those with magnitude greater than a threshold

• Matlab exercise with “blkproc”

– Zonal coding: Keep coefficients with maximum variance across all subimages

• Matlab’s “dctdemo”

• Then quantize to a fixed or variable number of bits

19


Zonal coding – Matlab “dctdemo”

• Apply DCT to each 8x8 block

• Discard coefficients with the smallest variance

20

DCT coefficients

Original Saturn Image

Reconstructed Image Error Image


Ordering sequence

• Convert a nxn matrix to a one-dimensional vector

Typical threshold mask (we keep the coefficients in the shaded positions)

Ordering sequence, to convert 8x8 array to a 64x1 vector

Resulting vector will have long runs of 0s

Different mask for each subimage

21


Quantizing Coefficient Magnitudes

• Once we decide which coefficients to keep, we now quantize the remaining non-zero coefficients

• We divide each coefficient by a number (depending on its location) and round to integer – Smaller magnitudes can be represented by fewer bits

– Division by a large number will tend to give a zero result

22

Z(u,v)


Variable quantization

• You can achieve more or less compression by scaling the normalization matrix Z (i.e., dividing by larger values)

• Resulting compression:

(a) 12:1 (b) 19:1 (c) 30:1 (d) 49:1 (e) 85:1 (f) 182:1

23


JPEG Algorithm

• Divide into 8x8 subimages

• Discrete cosine transform on each

• Quantize the coefficients

– Uses threshold coding

– Order coefficients in zig-zag pattern

• Encode the 1D sequence using run-length encoding and Huffman encoding

24

Ordering of coefficients

Threshold quantization array


Input 8x8 subimage

Subtract 128 from each value

25


Do forward DCT

Quantize and truncate values

26

Example: round(-415/16) = -26


JPEG Algorithm (continued)

• Re-order in zig-zag pattern

• Use variable length code words to encode non-zero values

• Use run length encoding to encode # zeros

• Results – bit count = 92

– Compression: 512/92 = 5.6:1

27


Details of coding the coefficients

• We use a pre-computed Huffman code (Appendices A.4-A.5)

• It assumes that values are clustered around zero

• The code word consists of a “base” code (which represents the most significant bits), followed by a coding of the least significant bits

• The base code is determined by the magnitude of the coefficient

First find what range the coefficient value lies in, and the corresponding category K

Then look up the base code for that category

28


Predictive Coding

• Takes advantage of interpixel redundancy

• Predict next pixel from previous pixel, encode only the difference from the actual and the predicted

29


30

A simple predictor: fpred(x,y) = f(x,y-1)


Lossy Predictive Coding

• Error values are quantized

• Predictions by encoder and decoder must be same to prevent error buildup

31


Optimal quantization of Error Values

• Lloyd-Max interval quantizer: staircase function with L values

32


Choosing Intervals

• Assume a Laplacian pdf

• The optimal 2-bit quantizer is

e

e eep

2

2

1)(

Quantized output

Error (e) 0 1.102 -1.102

-0.395 0.395 1.81 -1.81

33


Lloyd-Max Quantization

Quantized output

Error 0 1.102 -1.102

-0.395 0.395 1.81 -1.81

2 bit (4 level):

34


Compression of Image Sequences

• To compress a video we take advantage of the redundancy between successive frames

• See NASA Shuttle Movie

– 1829 color frames (~1 minute)

– Compression using Quicktime (H.264)

• Reduction from 5 GB to 45 MB (100:1)

• Predict the value of each pixel, transmit the residual error • Simplest prediction method: Prediction is the value of the pixel in the

previous image (forward prediction) • Periodically insert I-frames (“independent” frames)

– These are compressed as single images (like DCT) – Needed for initialization – Or to handle cases where there are too many changes between successive

images

• Can also base the prediction on the next frame (backward prediction)

35

nasa_shuttle-r_m720p.mov


Motion Compensation

• Predict motion of small blocks (e.g, 16x16)

Encoder has to estimate motion of each macroblock … usually finds dx,dy to minimize “mean absolute distortion”, which is the average of the absolute values of the differences

36


Example

std dev = 12.7 std dev = 5.6

37


Example – Subpixel Motion Estimation

std dev = 12.7 std dev = 4.4

std dev = 4 std dev = 3.8

Need to interpolate values

38


Video Compression Standards

P: prediction (forward) B: backward prediction

39

Documents

Image and Multidimensional Signal Processing