Quantization - Stanford University · 2012-01-26 · Bernd Girod: EE398A Image and Video Compression Quantization no. 4 Iterative Lloyd-Max quantizer design 1. Guess initial set of

Bernd Girod: EE398A Image and Video Compression Quantization no. 1

Quantization

Q q

Q -1

Sometimes, this

convention is used:

M represen-

tative levels

Q

Input-output characteristic of a scalar quantizer

M-1 decision thresholds

input signal x

Output

t q+1 t q+2 t q

x x̂

x̂2

ˆqx

1ˆ

qx

ˆqx

x

q x̂


Example of a quantized waveform

Original and Quantized Signal

Quantization Error


Lloyd-Max scalar quantizer

Problem : For a signal x with given PDF find a quantizer with

M representative levels such that

( )Xf x

Solution : Lloyd-Max quantizer

[Lloyd,1957] [Max,1960]

M-1 decision thresholds exactly

half-way between representative

levels.

M representative levels in the

centroid of the PDF between two

successive decision thresholds.

Necessary (but not sufficient)

conditions

1

1

1

1ˆ ˆ 1, 2, , -1

2

( )

ˆ 0,1, , -1

( )

q

q

q

q

q q q

t

X

t

q t

X

t

t x x q M

x f x dx

x q M

f x dx

2

ˆ min.d MSE E X X


Iterative Lloyd-Max quantizer design

1. Guess initial set of representative levels

2. Calculate decision thresholds

3. Calculate new representative levels

4. Repeat 2. and 3. until no further distortion reduction

ˆ 0,1,2, , -1qx q M

1

1

( )

ˆ 0,1, , -1

( )

q

q

q

q

t

X

t

q t

X

t

x f x dx

x q M

f x dx

1

1ˆ ˆ 1,2, , -1

2q q qt x x q M


Example of use of the Lloyd algorithm (I)

X zero-mean, unit-variance Gaussian r.v.

Design scalar quantizer with 4 quantization indices with

minimum expected distortion D*

Optimum quantizer, obtained with the Lloyd algorithm Decision thresholds -0.98, 0, 0.98

Representative levels –1.51, -0.45, 0.45, 1.51

D*=0.12=9.30 dB

Boundary

Reconstruction


Example of use of the Lloyd algorithm (II)

Convergence

Initial quantizer A:

decision thresholds –3, 0 3

Initial quantizer B:

decision thresholds –½, 0, ½

After 6 iterations, in both cases, (D-D*)/D*<1%

5 10 15-6

-4

-2

0

2

4

6

Qu

an

tizati

on

Fu

ncti

on

Iteration Number

0 5 10 150

0.2

0.4

D

0 5 10 150

5

10

SN

RO

UT

[dB

]

Iteration Number

SNROUT

final

= 9.2978

5 10 15-6

-4

-2

0

2

4

6

Qu

an

tizati

on

Fu

ncti

on

Iteration Number

0 5 10 150.1

0.15

0.2

D

0 5 10 156

8

10

SN

RO

UT

[dB

]

Iteration Number

SNROUT

final

= 9.298


Example of use of the Lloyd algorithm (III)

X zero-mean, unit-variance Laplacian r.v.

Design scalar quantizer with 4 quantization indices with

minimum expected distortion D*

Optimum quantizer, obtained with the Lloyd algorithm Decision thresholds -1.13, 0, 1.13

Representative levels -1.83, -0.42, 0.42, 1.83

D*=0.18=7.54 dB

Threshold

Representative


Example of use of the Lloyd algorithm (IV)

Convergence

Initial quantizer A,

decision thresholds –3, 0 3

Initial quantizer B,

decision thresholds –½, 0, ½

After 6 iterations, in both cases, (D-D*)/D*<1%

0 5 10 150

0.2

0.4

D

0 5 10 154

6

8

SN

R [

dB

]

Iteration Number

SNRfinal

= 7.5415

2 4 6 8 10 12 14 16-10

-5

0

5

10

Qu

an

tizati

on

Fu

ncti

on

Iteration Number2 4 6 8 10 12 14 16

-10

-5

0

5

10

Qu

an

tizati

on

Fu

ncti

on

Iteration Number

0 5 10 150

0.2

0.4

D

0 5 10 154

6

8

SN

R [

dB

]

Iteration Number

SNRfinal

= 7.5415


Lloyd algorithm with training data


2. Assign each sample xi in training set T to closest representative

3. Calculate new representative levels

4. Repeat 2. and 3. until no further distortion reduction

ˆ ; 0,1,2, , -1qx q M

x

1ˆ 0,1, , -1

q

q

Bq

x x q MB

ˆqx

B

q x T :Q x q q 0,1,2,, M -1


Lloyd-Max quantizer properties

Zero-mean quantization error

Quantization error and reconstruction decorrelated

Variance subtraction property

ˆ 0E X X

ˆ ˆ 0E X X X

2

2 2

ˆˆ

XXE X X


High rate approximation

Approximate solution of the "Max quantization problem,"

assuming high rate and smooth PDF [Panter, Dite, 1951]

Approximation for the quantization error variance:

3

1( )

( )X

x x constf x

Distance between two

successive quantizer

representative levels

Probability density

function of x

3

23

2

1ˆ ( )12

X

x

d E X X f x dxM

Number of representative levels


High rate approximation (cont.)

High-rate distortion-rate function for scalar Lloyd-Max quantizer

Some example values for

2 2 2

3

2 2 3

2

1with ( )

12

R

X

X X

x

d R

f x dx

2

92

uniform 1

Laplacian 4.5

3Gaussian 2.721

2


High rate approximation (cont.)

Partial distortion theorem: each interval makes an (approximately)

equal contribution to overall mean-squared error

2

1 1

2

1 1

ˆPr

ˆPr for all ,

i i i i

j j j j

t X t E X X t X t

t X t E X X t X t i j

[Panter, Dite, 1951], [Fejes Toth, 1959], [Gersho, 1979]


Entropy-constrained scalar quantizer

Lloyd-Max quantizer optimum for fixed-rate encoding, how can we

do better for variable-length encoding of quantizer index?

Problem : For a signal x with given pdf find a quantizer with

rate

such that

Solution: Lagrangian cost function

2

ˆ min.d MSE E X X

( )Xf x

1

2

0

ˆ logM

q q

q

R H X p p

2

ˆ ˆ min.J d R E X X H X


Iterative entropy-constrained scalar quantizer design


and corresponding probabilities pq

2. Calculate M-1 decision thresholds

3. Calculate M new representative levels and probabilities pq

4. Repeat 2. and 3. until no further reduction in Lagrangian cost

ˆ ; 0,1,2, , -1qx q M

1

1

( )

ˆ 0,1, , -1

( )

q

q

q

q

t

X

t

q t

X

t

xf x dx

x q M

f x dx

-1 2 1 2

-1

ˆ ˆ log log= 1,2, , -1

2 ˆ ˆ2

q q q q

q

q q

x x p pt q M

x x


Lloyd algorithm for entropy-constrained

quantizer design based on training set


and corresponding probabilities pq

2. Assign each sample xi in training set T to representative minimizing Lagrangian cost

3. Calculate new representative levels and probabilities pq

4. Repeat 2. and 3. until no further reduction in overall Lagrangian cost

: 0,1,2, , -1qB x Q x q q M T

ˆ ; 0,1,2, , -1qx q M

x

1ˆ 0,1, , -1

q

q

Bq

x x q MB

ˆqx

2

2ˆ log

ix i q qJ q x x p

2

2logi i

i i

x i i q xx x

J J x Q x p


Example of the EC Lloyd algorithm (I)

X zero-mean, unit-variance Gaussian r.v.

Design entropy-constrained scalar quantizer with rate R≈2 bits, and

minimum distortion D*

Optimum quantizer, obtained with the entropy-constrained Lloyd

algorithm 11 intervals (in [-6,6]), almost uniform

D*=0.09=10.53 dB, R=2.0035 bits (compare to fixed-length example)

Threshold

Representative level


Example of the EC Lloyd algorithm (II)

Initial quantizer B, only 4 intervals

(<11) in [-6,6], with the same length

Same Lagrangian multiplier λ used in all experiments

Initial quantizer A, 15 intervals

(>11) in [-6,6], with the same length

5 10 15 20-6

-4

-2

0

2

4

6

Qu

an

tizati

on

Fu

ncti

on

Iteration Number

0 5 10 15 20

6

8

10

12

SN

R [

dB

]

SNRfinal

= 8.9452

0 5 10 15 201

1.5

2

2.5

R [

bit

/sym

bo

l]

Rfinal

= 1.7723

Iteration Number

0 5 10 15 20 25 30 35 40

6

8

10

12

SN

R [

dB

]

SNRfinal

= 10.5363

0 5 10 15 20 25 30 35 401

1.5

2

2.5

R [

bit

/sym

bo

l]

Rfinal

= 2.0044

Iteration Number

5 10 15 20 25 30 35 40-6

-4

-2

0

2

4

6

Qu

an

tizati

on

Fu

ncti

on

Iteration Number


Example of the EC Lloyd algorithm (III)

X zero-mean, unit-variance Laplacian r.v.

Design entropy-constrained scalar quantizer with rate R≈2 bits

and minimum distortion D*

Optimum quantizer, obtained with the entropy-constrained Lloyd

algorithm 21 intervals (in [-10,10]), almost uniform

D*= 0.07=11.38 dB, R= 2.0023 bits (compare to fixed-length example)

Decision threshold

Representative level


Example of the EC Lloyd algorithm (IV)

Initial quantizer B, only 4 intervals

(<21) in [-10,10], with the same length

Same Lagrangian multiplier λ used in all experiments

Initial quantizer A, 25 intervals

(>21 & odd) in [-10,10], with the

same length

Convergence in cost faster than convergence of decision thresholds

5 10 15 20-10

-5

0

5

10

Qu

an

tizati

on

Fu

ncti

on

Iteration Number

0 5 10 15 20

5

10

15

SN

R [

dB

]SNR

final = 7.4111

0 5 10 15 201

2

3

R [

bit

/sym

bo

l]

Rfinal

= 1.6186

Iteration Number

0 5 10 15 20 25 30 35 40

5

10

15

SN

R [

dB

]

SNRfinal

= 11.407

0 5 10 15 20 25 30 35 401

2

3

R [

bit

/sym

bo

l]

Rfinal

= 2.0063

Iteration Number

5 10 15 20 25 30 35 40-10

-5

0

5

10

Qu

an

tizati

on

Fu

ncti

on

Iteration Number


High-rate results for EC scalar quantizers

For MSE distortion and high rates, uniform quantizers (followed by

entropy coding) are optimum [Gish, Pierce, 1968]

Distortion and entropy for smooth PDF and fine quantizer interval

Distortion-rate function

is factor or 1.53 dB from Shannon Lower Bound

222

2

12d d

2

ˆ logH X h X

2 212 2

12

h X Rd R

6

e

2 212 2

2

h X RD Re


Comparison of high-rate performance of

scalar quantizers

High-rate distortion-rate function

Scaling factor

2 2 22 R

Xd R

2

2

Shannon LowBd Lloyd-Max Entropy-coded

6Uniform 0.703 1 1

e

9Laplacian 0.865 4.5 1.232

2 6

3Gaussian 1 2.721 1.423

2 6

e e

e


Deadzone uniform quantizer

Quantizer

input

Quantizer

output ̂x

x


Embedded quantizers

Motivation: “scalability” – decoding of compressed

bitstreams at different rates (with different qualities)

Nested quantization intervals

In general, only one quantizer can be optimum

(exception: uniform quantizers)

Q0

Q1

Q2

x


Example: Lloyd-Max quantizers for Gaussian PDF

2-bit and 3-bit optimal

quantizers not embeddable

Performance loss for

embedded quantizers

0 1

0

0

0

1

1

0

1

1

0

0

0

0

0

1

0

1

0

0

1

1

1

0

0

1

0

1

1

1

0

1

1

1

-.98 .98

-1.75 -1.05 -.50 .50 1.05 1.75


Information theoretic analysis

“Successive refinement” – Embedded coding at multiple

rates w/o loss relative to R-D function

“Successive refinement” with distortions and

can be achieved iff there exists a conditional distribution

Markov chain condition

1D 2 1D D

1 1

2 2

ˆ( , )

ˆ( , )

E d X X D

E d X X D

1 1

2 2

ˆ( ; ) ( )

ˆ( ; ) ( )

I X X R D

I X X R D

1 2 2 1 2ˆ ˆ ˆ1 2 2 1 2ˆ ˆ,

ˆ ˆ ˆ ˆ ˆ( , , ) ( , ) ( , )X X X X X X X

f x x x f x x f x x

2 1ˆ ˆX X X

[Equitz,Cover, 1991]


Embedded deadzone uniform quantizers

Q0

Q1

Q2

x

8

4

2

x=0

4

2

Supported in JPEG-2000 with general for

quantization of wavelet coefficients.


Vector quantization


LBG algorithm

Lloyd algorithm generalized for VQ [Linde, Buzo, Gray, 1980]

Assumption: fixed code word length

Code book unstructured: full search

Best representative vectors

for given partitioning of training set

Best partitioning of training set

for given representative

vectors


Design of entropy-coded vector quantizers

Extended LBG algorithm for entropy-coded VQ [Chou, Lookabaugh, Gray, 1989]

Lagrangian cost function: solve unconstrained problem rather than

constrained problem

Unstructured code book: full search for

2

ˆ ˆ min.J d R E X X H X

The most general coder structure:

Any source coder can be interpreted as VQ with VLC!

2

2ˆ log

ix i q qJ q x x p


Lattice vector quantization

•

•

• • •• • • •

• • • • • • • •

• • • • • ••

• • • • • • •• •

• • • • • • • •

•• • • • • •

• • • • • • •

•

•

•

Amplitude 1

Am

plit

ude

2

cell

pdf

representative vector


8D VQ of a Gauss-Markov source

r 0.95

SN

R [

dB

]

18

12

6

0

0 0.5 1.0

Rate [bit/sample]

scalar

E8-lattice

LBG variable CWL

LBG fixed CWL


8D VQ of memoryless Laplacian source S

NR

[dB

]

Rate [bit/sample]

scalar

E8-lattice LBG variable CWL

LBG fixed CWL

9

6

3

0.5 1.0

0

0


Reading

Taubman, Marcellin, Sections 3.2, 3.4

J. Max, “Quantizing for Minimum Distortion,” IEEE Trans.

Information Theory, vol. 6, no. 1, pp. 7-12, March 1960.

S. P. Lloyd, “Least Squares Quantization in PCM,” IEEE

Trans. Information Theory, vol. 28, no. 2, pp. 129-137,

March 1982.

P. A. Chou, T. Lookabaugh, R. M. Gray, “Entropy-

constrained vector quantization,” IEEE Trans. Signal

Processing, vol. 37, no. 1, pp. 31-42, January 1989.

Documents

Quantization - Stanford University · 2012-01-26 · Bernd Girod: EE398A Image and Video Compression Quantization no. 4 Iterative Lloyd-Max quantizer design 1. Guess initial set of