A Method for Signalling Block-AdaptiveQuantization in Baseline Sequential JPEG

Ramakrishna Kakarala and Ravikiran BagadiSchool of Computer Engineering

Nanyang Technological UniversitySingapore 639798

Email: [ramakrishna,ravi0024]@ntu.edu.sg

Abstract—The traditional JPEG compression standard, unlikeJPEG2000, does not prescribe a means for spatially-variablequantization of transform coefficients. This prevents useful fea-tures such as region-of-interest (ROI) coding, where, for example,the subject in a photograph may be lightly compressed whilethe background is heavily compressed to hide irrelevant details.In this paper, we show how the quantization may be adaptedin each block and, more importantly, how the adaptation maybe signalled to the decoder in a memory-efficient manner. Themethod takes advantage of previously unused slots in the Huffmantables employed for entropy coding. The adaptation allows, forexample, ROI coding of subject and background. We demonstratehow the encoder may be modified to produce an output thatis standards-compliant, and how the decoder may be modifiedto correctly recover the adaptively quantized image. We showthat the dequantization of an adaptively-quantized image with astandard, non-adaptive decoder, amounts to filtering in the DCTdomain. Examples are shown to illustrate adaptive quantization,signalling, and decoding.

I. INTRODUCTION

The JPEG standard [1], completed in 1992, relies on thewell-known method of separating an image into luminance(denoted Y ) and chrominance (denoted Cb, Cr) color planes,partitioning the planes into 8 × 8 blocks, each of which iscompressed using quantization of discrete cosine transform(DCT) coefficients followed by entropy coding of run-lengths.The main contributor to compression is the quantization (Q)matrix, which is applied to each set of DCT coefficients.The Q matrix may be changed from image to image, anddifferent Q matrices may be used for the Y and Cb, Cr colorplanes. However, in the JPEG standard, the Q matrix mustremain the same across a color plane, i.e., spatially-variablequantization is not allowed. This prevents the encoder from, forexample, varying the quantization within an image to achieve adesired output file size, or quantizing subject and backgroundseparately to permit region-of-interest (ROI) coding. Thoseuseful features are built into JPEG2000, but are not availablein JPEG as prescribed by the standard. However JPEG2000 isnot yet widely adopted. Hence, if we implement the featurein baseline JPEG, then it can have significant impact, eventhough JPEG is an old standard.

This paper describes a method of varying the quantizationmatrix from block to block, and, more importantly, shows howthe adaptive quantization may be signalled to the decoder in a

manner that is compliant with the JPEG standard1. Adaptivequantization only works if the decoder varies the Q matrixacross the image in exactly the same way as the encoder.To allow maximum freedom in adaptation, the encoder mustsignal to the decoder on each block which Q matrix was used.Furthmore, the signalling must be compliant syntactically withthe JPEG standard if any “standard” decoder (as opposed toone modified for the signalling method) is able to processthe adaptively-quantized image. Our method for signallingtakes advantage of previously-unused slots for end-of-block(EOB) symbols in the Huffman table to signal up to 14different levels of quantization across a color plane. The syntaxused is compliant with the JPEG standard in that a standarddecoder treats all 14 EOB symbols as the same. However, amodified decoder that is aware of the adaptive signalling isable to recognize the EOB symbol and modify the decodingQ matrix appropriately to recover the image. The modificationis relatively simple, as we show below. Moreover, since theEOB symbols by definition come as the end of a 8× 8 block,there is no need for either the encoder or the decoder to keepmore than one block in memory at a time. This is a crucialconsideration for hardware implementation of encoders anddecoders, as memory consumes a significant proportion of chiparea in embedded cameras such as used in cellphones.

Adaptive quantization within JPEG has been a subject ofinterest since the standard was published. Standards-compliantmethods for signalling the adaptation have been described.Since the JPEG standard allows any number of color planesto be stored in the output file, Pennebaker [2] describes amethod that use an extra “color” plane whose sole purpose isto contain a spatial map of how the quantization matrices arevaried. However, this requires that the decoder read more thanone color plane at a time to vary the Q matrix appropriately.Moreover, it is not clear that a standard decoder, especiallythose relying on the JFIF file format that is widely used to storeJPEG images, will not confuse the extra color plane with thosedesigned for display. Another approach [3] searches the up to64 coefficients in a 8 × 8 block to find the coefficient whosequantized value, in fractional portion, is closest to 0.5: for ex-ample, a fractional value of 11.499 may be rounded to either 11or 12 with approximately the same error. Rounding up or down

1The method has been granted a patent[4].

978–1–4244–4547–9/09/$26.00 c© 2009 IEEE TENCON 20091

produces a sum of quantized coefficients that is either even orodd, and the parity of the sum therefore becomes a methodof signalling one of two possible Q matrices. However, thescheme requires a search to be carried out prior to quantization,and moreover does not extend to more than two Q matrices.Other related works use rate-distortion based thresholding ofDCT coefficients to achieve a target file size [5][11]. While thiscould, in principle, be made spatially adaptive, the methods donot appear to be suitable for embedded applications.

In this paper we show how previously unused slots in thebaseline JPEG Huffman table may be used to signal Q matrixadaptation. Those slots allow up to 14 different EOB codes tobe used on each block, and therefore provide for the encoderto signal that many different Q matrices to the decoder. Weshow how the Huffman codes may be constructed to efficientlyencode the adaptively-quantized image. We demonstrate howthe syntax is compatible with the baseline sequential JPEGstandard, so that any standard decoder may decode the adap-tively quantized image, although it reconstructs the imagewith frequency-domain filtering. Examples of ROI coding andadaptive quantization are demonstrated below. Details whichwe discuss in this paper and which are not discussed inthe related patent[4] are as follows: the modification of theHuffman tree in a memory efficient manner; the effect ofdequantizing with a different Q matrix at the decoder sideinstead of the Q matrix which is used at encoder side; andhow we can use the adaptive quantization with zero additionalcomputation over what is required for standard JPEG decoding.

II. BACKGROUND

Although the principles behind JPEG are well known, thedetails of the standard are not as commonly known. We reviewthe relevant portions of the JPEG standard as described in ref.[6] here.

JPEG image compression is available in two formats, base-line sequential, and progressive. The baseline format storesthe full resolution of the image in a single sequential top-to-bottom scan, and it is the one that has been universallyadopted. The progressive format stores several scans of theimage, each with increased resolution over the previous scan,in order to provide the viewer with incremental update in alow-bandwidth channel. Progressive JPEG is not commonlyused. What is important for this paper is that JPEG syntax, inparticular the organization of the Huffman table, is designedto accomodate both formats. This dual-purpose nature of theHuffman table allows us to exploit one feature of progressiveformat to permit signalling of adaptive quantization in baselinesequential format.

The operation of baseline sequential JPEG compression ofa color image is as follows.

1) The image is converted from RGB to YCbCr, usuallyfollowed by subsampling of the Cb, Cr planes.

2) Each of the Y , Cb, Cr planes is divided into blocks ofsize 8× 8.

3) The DCT of each block is calculated. Let x(i, j) denotethe (i, j)-the pixel in block, and X(u, v) the (u, v)-

TABLE IHUFFMAN TABLE LAYOUT FOR AC COEFFICIENTS IN BASELINE

SEQUENTIAL MODE. ENTRIES MARKED ”X” CONTAIN CODES FOR OTHERVALUES.

R \ S 0 1 2 · · · 9 100 EOB x x x x x1 Not used x x x x x... Not used x x x x x14 Not used x x x x x15 ZRL x x x x x

th frequency in the DCT. In particular, X(1, 1) is the“DC” coefficient, and all others are “AC”. The DCTcoefficients are quantized using a 8 × 8 matrix Q asfollows:

XQ(u, v) = round{X(u, v)Q(u, v)

}, u, v = 1, . . . , 8.

(1)The Q matrix may be different for Y , Cb, Cr planes,but as mentioned earlier, it does not vary spatially.

4) The quantized block XQ is scanned in “zig-zag” order.The quantized DC coefficient is stored after subtractionof the DC value of the previously coded block. The scanof AC coefficients is stored in (R,S) format, where Ris the number of consecutive zeros followed by the non-negative value S indicating the size of the next non-zerovalue. If V is the value of the quantized coefficient, then

S = blog2 |V |c+ 1. (2)

Here b·c is the floor function. Two special symbols arealso allowed: a ZRL symbol representing a run of 16consecutive zeros; and a EOB symbol indicating that allsubsequent coefficients in the scan are zero.

5) Each scan is entropy-encoded using Huffman coding,where the Huffman table for AC coefficients is organizedas a 16×11 structure shown in Table I to accomodate 16different R values and 11 possible values of size S. TheHuffman code for each (R,S) pair in a scan is emitted,followed by S additional bits to indicate the actual valueof the coefficient.

Note that the entry in Table I for R = 0, S = 0 is usedfor the EOB symbol, while R = 15, S = 0 indicates 15consecutive zeros followed by a size 0, which is the conditionfor the ZRL symbol. The entries in Table I for S = 0 andR = 1, . . ., 14 are not used in baseline sequential format.They are reserved for the progressive format. We propose totake advantage of those unused entries for signalling adaptivequantization.

The JPEG standard defines the decoder action on theHuffman table in Table I. The flowchart in ref. [6, pg 463]shows that an (R,S) pair is extracted from each codeword,followed by the magnitude of the coefficient. If S = 0, thena check is made if R = 15; if so, 16 consecutive zerosare appended to the zig-zag scan reconstruction. If not, thenthe symbol is treated an EOB. In other words, all entries

2

in Table I with S = 0 and R < 15 have the same effect:they are considered to the same EOB symbol in a standards-compliant decoder. Therefore, we may replace any of the 14unused entries in Table I with additional codes for alternativeconditions without disrupting a standards compliant decoder.

III. ANALYSIS OF QUANTIZATION

The Q matrix used in sequential JPEG is fixed acrossthe color plane. This limitation precludes certain possibilitiessuch as ROI coding as described in the introduction. We mayremove the limitation by signalling to the decoder with EOBcodes which Q was used. But it is important to understandwhat happens if the decoder is a standard one, and thereforedequantizes an input block with a different matrix than it wasquantized with.

Suppose that the DCT of an 8 × 8 block, X(u, v), 1 ≤u, v ≤ 8, is quantized at the encoder using matrix Q as ineq.(1). At the decoder, the quantized matrix XQ is used toreconstruct the DCT matrix X using the equation

X̂(u, v) = Q(u, v)XQ(u, v), u, v = 1, . . . , 8. (3)

Obviously, X̂ will not the same as X due to rounding. If welet P = αQ, where α is a scale factor, then the effect ofdequantizing XQ with P instead of Q is as follows:

X̃(u, v) = P (u, v)XQ(u, v)

= αQ(u, v) round{X(u, v)Q(u, v)

}= αX(u, v) + αEq(u, v) (4)

where Eq = Q round{XQ/Q} − X is the quantization errorintroduced by rounding. Note that

|Eq(u, v)| < Q(u, v)/2. (5)

The quantization error is often modelled as a uniformly dis-tributed random variable within its range [8, pg 684].

We may consider eq. (4) to show that using P rather than Qscales the X matrix by α, and also scales the quantization noisesource by the same factor. Due to linearity of the DCT, theeffect of the scaling on reconstruction of the DCT coefficientsis to scale the reconstructed pixel values by the same amount.To see this, note that the forward and inverse DCTs may beviewed as matrix operations [9, pg 129]. With D denoting the8 × 8 orthogonal DCT matrix, x the 8 × 8 matrix of pixelvalues, and X the corresponding matrix of DCT coefficients,we have

X = DxD; x = DtXDt. (6)

Therefore, if we let x̃ denote the inverse transform of X̃ , weobtain that

x̃ = DtX̃Dt = αx+ αeq, (7)

where eq = DtEqDt is the inverse transform of the quantiza-

tion error.In the general case, we may choose P = H ◦ Q, where ◦

denotes the elementwise product and H a “filtering” matrix.Then, we obtain that

X̃(u, v) = H(u, v)X(u, v) +H(u, v)Eq(u, v). (8)

Consequently, we see that dequantization with H ◦Q insteadof Q is equivalent to filtering by H in the DCT domain, ifwe ignore the quantization noise term. We may therefore useadaptive quantization as a means to perform spatially adaptivefiltering of the image, with zero additional computation to whatis required for standard JPEG decoding.

A. Interaction with DC Prediction

A potential complication with adaptive quantization is thedependence of adjacent blocks. In baseline sequential JPEG,the DC coefficient of each block is coded as the differencefrom the corresponding value in the previously consideredblock in the scan. The prediction is reset by “restart” markers[6, pg 109], which may optionally be placed in the bitstreamby the encoder. If adaptive quantization is used, predictionwill accumulate error when the DC coefficient is dequantizeddifferently than it was quantized. The DC coefficient is uniquein this respect: all other coefficients in a block are codedindependently of adjacent blocks and therefore varying the Qmatrix has no effect. We therefore constrain the quantizationchange matrix H discussed above so that H(1, 1) = 1, whichensures that X̃(1, 1) = X(1, 1) + EQ(1, 1). Alternatively, wemay emit restart markers to reset the prediction on each blockwhere the Q matrix changes. However, this adds to the outputfile size, which, while small, may be avoided by using theconstraint.

B. Reduction of artifacts

It is well-known that JPEG compression results in at leasttwo types of artifacts: 1) visible boundaries between 8 × 8blocks; 2) ringing (Gibbs phenomenon) around edges. Al-though deblocking and de-ringing filters may be applied at theoutput to suppress those artifacts, it is obviously desireable toavoid the creation of the artifact. Adaptive quantization maybe applied to this purpose. For example, if a given Q matrixproduces both types of artifacts, we may apply a “smaller”matrix Q1 for smooth blocks, and a “larger” matrix Q2 foredge blocks, where Q2(u, v) > Q1(u, v). The identificationof smooth vs edge blocks may be accomplished by notingwhether the last non-zero coefficient occurs early or late in azig-zag scan relative to a predetermined threshold. We chosethe threshold value of 26 for the experiments described below.

IV. HUFFMAN TABLE MODIFICATION

If we use the unoccupied slots in the Huffman table in Ta-ble I for S = 0, R = 1, . . ., 14 to signal adaptive quantization,then we must modify the entire Huffman codebook to allow forunique decoding. One way to proceed is simply to substitute aHuffman table for progresssive coding in place of the baselinetable. The progressive Huffman table uses all of the slots,including the 14 slots for 1 ≤ R ≤ 14 for additional codesEOB1, . . ., EOB14. This allows up to 15 different Q matricesto be signalled. The organization of a typical progressive tableis shown in Table II.

Tables can also be designed to support fewer EOB codes,which in turn reduces the average code length. In the examples

3

TABLE IIHUFFMAN TABLE LAYOUT FOR PROGRESSIVE ENCODING

R \ S 0 1 2 · · · 9 100 EOB0 x x x x x1 EOB1 x x x x x...

... x x x x x14 EOB14 x x x x x15 ZRL x x x x x

(a) Standard Huffman tree for baseline sequential JPEG.See text for description of the symbols.

(b) Modified tree to allow two EOB codes.

Fig. 1. Huffman trees for standard and adaptive coding

described below, we use two EOB codes to signal the samenumber of Q matrices. Their placement in the Huffman tableis illustrated in the tree structure of Figure 1(a)-1(b). In thosefigures, the square boxes represent the symbols as describedin [6, Table K.5], while the circle represents the remainder ofthe code tree.

V. EXPERIMENTAL RESULTS

We used the EOB signalling method to apply adaptivequantization to the well-known image “Lena”. We used twolevels of quantization on the luminance (Y ) component ofthe image. While it is certainly possible to apply the same

(a) Interior ROI coding using adaptivequantization when reconstructed withstandard decoder

(b) Interior ROI coding using adaptivequantization reconstructed with adaptivedecoder signalled by EOB

Fig. 2. Example of ROI coding of the boundary by adaptive quantizationsignalled by EOB matrices

type of adaptation to the chrominance planes Cb, Cr, forillustrative purposes it suffices to use Y . Some experimentsare conducted with the public-domain software available fromthe Independent JPEG group (IJG), and others are carried outin MATLAB.

Our first experiment demonstrates ROI coding using adap-tive quantization. In this experiment, the two matrices appliedto the Y plane are as follows: the first matrix, denoted Q1,is the default matrix shown in ref.[6, pg 503], which resultsin a roughly 20 : 1 compression ratio; the second, denotedQ2, is 4 ×Q1 for all entries except that Q2(1, 1) = Q1(1, 1)to ensure that DC prediction is not affected. We applied Q1

to a selected region in the interior of the image, and Q2 tothe remaining portion in the exterior. The resulting imagewere decoded with a standard decoder which assumes theQ matrix is Q1 for the entire image, and with an adaptivedecoder which receives the signalling from the EOB codes andswitches between Q1 and Q2 appropriately. Figure 2(a) showsthe result from a standard decoder and Fig. 2(b) shows theresult from an adaptive decoder. We see that in Fig. 2(a), theexterior loses detail because of the filtering in the DCT domainperformed by H(u, v) = Q1(u, v)/Q2(u, v); while Fig. 2(b)shows an improved result due to a matching of quantization.The remaining blockiness of image 2(b) is due to the strongquantization of Q2, which corresponds roughly to a “quality

4

(a) Adaptively quantized exterior ROIimage reconstructed with standard de-coder

(b) Adaptively quantized exterior ROIimage reconstructed with adaptive de-coder signalled by EOB

Fig. 3. Example of ROI coding of the interior by adaptive quantizationsignalled by EOB matrices

factor” of 12.5 in the IJG code. This pattern of inside-outsideROI coding may be used to provide privacy about backgroundregions in standard decoders, without permanently losing thedetails of those regions for later use. For example, in onescenario, the standard decoder in an internet browser shows theROI coded image with a blurry background, while the sameimage may be provided to an adaptive decoder (perhaps aftera password is entered) to enable the background details tobecome apparent.

The second experiment is the reverse of the previous one,where the ROI is the exterior: here the exterior of the imageis quantized with Q1, and the interior with Q2 = 3 × Q1

except Q2(1, 1) = Q1(1, 1). Figures 3(a) and 3(b) show theresults. This pattern of inside-outside ROI coding may be usedto provide privacy about subjects without permanently losingthe details such as faces for later use.

The third experiment demonstrates signalling of adaptationby block type. Here again, two Q matrices are used, with Q2 =4×Q1 and Q2(1, 1) = Q1(1, 1). Smooth blocks are classifiedas those where the EOB condition occurs early in the zig-zag scan when quantized with Q1, and the remaining blocksare considered edge blocks. Edge blocks are quantized withQ2. This form of adaptive quantization does not entail makingtwo sequential passes on each block; it is possible that the 64coefficients may be quantized in parallel with both Q1 and

(a) Lena image using Q1 for all blocks

(b) Edge/smooth adaptively quantizedimage reconstructed with adaptive de-coder signalled by EOB

(c) Lena image using Q2 for all blocks

Fig. 4. Example of edge/smooth coding by adaptive quantization signalledby EOB matrices

Q2, and the decision as which quantized output to use may bebased on the EOB location in the Q1 set. Efficient hardwareimplementations are reasonably simple to devise. Figures 4(a)-4(c) show the results.

The PSNR of the image with Q1 for all blocks in Fig. 4(a)is 30.4 dB. When the edge/smooth adaptive quantization isused, the PSNR of the result shown Fig. 4(b) drops to 29.2dB. In comparison, if Q2 is used for all blocks, then the resultshown in Fig. 4(c) has PSNR of 27.8 dB. Obviously, adaptivequantization will not produce an image with quality that isas good as simply choosing the lesser of the quantizationmatrices, but the PSNR results show the penalty is slightcompared to the benefit of reducing the file size. A moreextensive study is currently under way to examine the rate-

5

distortion curve for adaptively quantized matrices.

VI. CONCLUSION

In this paper, we describe a relatively simple, standards-compliant, method for signalling adaptive quantization: usethe unoccupied slots for EOB1, . . ., EOB14 in the standardbaseline Huffman table. The signalling method is efficient inthat the encoder and decoder need not keep more than oneblock in memory at a time. A standard decoder will treatall EOB codes as the same, and if adaptive quantizationis performed, will perform frequency-domain filtering upondequantization. This is useful for ROI coding. We may alsoperform adaptive quantization by block type, which has thebenefit of improving image quality selectively in regions wherethe reconstruction artifact is most apparent.

REFERENCES

[1] G. K. Wallace, “The JPEG still picture compression standard,” IEEEtransactions on consumer electronics, Vol. 38, No. 1, Feb. 1992, pp xviii-xxxiv.

[2] W. B. Pennebaker, “Adaptive quantization within the JPEG sequentialmode,” U. S. Patent 5,157,488.

[3] A. M. Silverstein and S. A. Klein, “Precomputing and encoding com-pressed image enhancement instructions,” U. S. Patent 5,822,458.

[4] R. Kakarala and J. S. Gibson, “Signaling adaptive-quantization matricesin JPEG using end-of-block codes”, U. S. Patent 7,092,578.

[5] T. A. Simpson and Y. Hu, “Adaptive image data compression”,U. S. Patent 6,724,817.

[6] W. B. Pennebaker and J. L. Mitchell, JPEG: Still Image Data CompressionStandard, New York: Springer-Verlag, 1992.

[7] M. Stirner and G. Seelman,“Improved redundancy reduction for JPEGfiles”, Proc. of the Picture Coding Symposium (PCS 2007), Lisbon,Portugal: November 2007.

[8] S. K. Mitra, Digital signal processing: A computer-based approach, 3rdEd, New York : McGraw-Hill, 2006.

[9] R. A. Gonzalez and R. E. Woods, Digital image processing, 1st Ed,Reading, MA: Addison-Wesley, 2002.

[10] J. Zhao and Y. Shimazu and K. Ohta and R. Hayasaka and Y. Matsushita,“A JPEG codec adaptive to region importance”. In Proceedings ofthe Fourth ACM international Conference on Multimedia Boston, MA:November 18 - 22, 1996, pp 209-218.

[11] K. Ramchandran and M. Vetterli, “Rate-distortion optimal fast thresh-olding with complete JPEG/MPEG decoder compatibility”,IEEE Trans-actions on image processing, Vol. 3, No. 5, Sept. 1994, pp 700-704.

[12] G. Lakhani and V. Ayyagari, “Improved Huffman code tables for JPEG’sencoder”, IEEE Transactions on circuits and systems for video technology,Vol. 5, No. 6, Dec. 1995, pp 562-564.

[13] G. Lakhani, “Modified JPEG Huffman coding”, IEEE Transactions onimage processing, Vol. 12, No. 2, pp 159- 169.

6