Advanced Topics Multimedia Video (5LSH0), …vca.ele.tue.nl/sites/default/files/5LSH0 mod 01B JPEG2000...1 1 PdW-AP-EB-FS-FG-SZ-EB / 2017 Fac. EE SPS-VCA Adv. Topics MMedia Vid. Cod

1

1

PdW-AP-EB-FS-FG-SZ-EB / 2017

Fac. EE SPS-VCA

Adv. Topics MMedia Vid. Cod. /

5LSH0 / Module 01B JPEG2000

Advanced Topics Multimedia Video

(5LSH0), Module 01 B

Introduction to JPEG2000: the next

generation still image coding system

Peter H.N. de With

( [email protected] )

2


Fac. EE SPS-VCA



Module 1B: JPEG2000 – Part 1

Standardization issues,

Requirements, Comparisons

3


Fac. EE SPS-VCA



JPEG: Summary – (1)

JPEG (Joint Photographic Experts Group)

“Digital Compression and Coding of Continuous-tone Still

Images”

• Joint ISO and ITU-T

• Published in 4 Parts:

– ISO/IEC 10918-1 | ITU-T T.81 : Requirements and guidelines

– ISO/IEC 10918-2 | ITU-T T.83 : Compliance testing

– ISO/IEC 10918-3 | ITU-T T.84: Extensions

– ISO/IEC 10918-4 | ITU-T T.86: Registration of JPEG Parameters,

Profiles, Tags, Color Spaces, APPn Markers

Compression Types, and Registration Authorities (REGAUT)

4


Fac. EE SPS-VCA



JPEG: Summary – (2)

JPEG derived industry standards

• JFIF (JPEG File Interchange Format, <xxxxxx.jpg>)

• JTIP (JPEG Tiled, Pyramid Format)

• TIFF (Tagged Image File Format)

• SPIFF (Still Picture Interchange File Format, JPEG Part 3)

• FlashPix

– Developed by Hewlett-Packard, Kodak, Microsoft, Live Picture (1996)

– Transferred to Digital Imaging Group (DIG), an industry consortium

5


Fac. EE SPS-VCA



Motivation new still image st’dard – (1)

Current standards fail to produce the best quality/performance, hence

• Should complement JPEG, not replace!

• Low bit-rate compression: For example below 0.25 bpp

• Lossless and lossy compression: No current standard exists providing superior lossy and lossless compression in a single codestream.

• Computer generated imagery: JPEG was optimized for natural imagery; does not perform well on computer generated imagery.

• Large images: JPEG does not go beyond 64Kx64K without tiling

6


Fac. EE SPS-VCA



• Random codestream access and processing

• Compound documents: Currently, JPEG is seldom used in the compression of compound documents because of its poor performance when applied to bi-level (text) imagery

• Transmission in noisy environments: The current JPEG standard has provision for restart intervals, but image quality suffers dramatically when bit errors are encountered.

• Open Architecture: Desirable to allow open architecture to optimise the system for different image types and applications. Also, JPEG2000 offers a single architecture, whereas JPEG has 44 modes of which most are not used.

• Progressive transmission by pixel accuracy and resolution

Motivation new still image st’dard – (2)

2

7


Fac. EE SPS-VCA



• Internet

• Mobile

• Printing

• Scanning

• Digital Photography

• Remote Sensing

• Facsimile

• Medical

• Digital Libraries

• E-Commerce

JPEG2000 Markets and Applications

First real application:

Digital Cinema!

8


Fac. EE SPS-VCA



JPEG2000 Requirements∗ New still image coding standard for

– Bi-level images, like faxes and masks

– Gray-level images, like photography and newspapers

– Color images, like consumer & professional photography

– Multi-component images, like in professional printing

– Hyper-component images, like in art and prof. applications

∗ Characteristics/Applications– natural, medical, scientific, remote sensing, professional

∗ Imaging models– Client-server models, Real-time transmission

– Image library retrieval, Imaging limited resources (embedded, mobile)

9


Fac. EE SPS-VCA



JPEG2000 Features in Part I

∗ High compression efficiency (superior to JPEG)

∗ Lossless colour transformations (continuous tone supp.)

∗ Lossy and lossless coding in one algorithm

∗ Embedded lossy to lossless coding (continuous vs. bi-level)

∗ Progressive by resolution, quality, position, …

∗ Static and dynamic Region-of-Interest coding/decoding

∗ Error resilience (enable random access, robustness)

∗ Perceptual quality coding (facilitate regions of interest)

∗ Multiple component image coding

∗ Tiling (supports real-time coding, seq. build-up, limited space)

∗ Light file format (optional) (pursue open architecture, formats)

10


Fac. EE SPS-VCA



J2K Application requirements – (1)

∗ Image type

– Image width and height: 1 – (2**32 – 1)

– Component depth: 1 – 32 bits

– Number of components: 1 – 255 (or more)

– Dissimilar comp. depths: each component has a different depth

– Dissimilar comp. spans: each component has its own coverage

JPEG2000

11


Fac. EE SPS-VCA



J2K Application requirements – (2)∗ Application profiles

– Internet: image 32x32 up to 4Kx4K, with 3 (Y Cr Cb, RGB, …)

comp’s, or 4 comp’s with 1-8 bit alpha channel

– Printing: compound images with typically 4800x6600 pixels (600 ppi,

8”x11”) with 1, 3 or 4 components

– Scanning: compound images, typically 10Kx10K up to 20Kx20K or

more, with 1, 3, or 4 components and 16-bit/comp.

– Digital Photography: natural, sizes up to 4K x 4K, with 1, or 3

components, and between 8-16 bits/comp.

– Remote sensing: infra-red, electro-optical, multi-spectral, SAR

images, horiz. res. up to 24K pixels, 1-500 components, 8-20 b/comp.

– Mobile: compound, 32x32 till 4Kx4K pixels, with 1 or 3 components

and 1-8 bits/component

JPEG2000

12


Fac. EE SPS-VCA



J2K Application requirements – (3)∗ Uncompressed: image is stored in bit stream without compression

∗ Lossless compression: reconstructed image is identical, bit for bit, to the

original, with a compression at least as good as JPEG-LS

∗ Visually lossless compression: reconstructed image differs numerically

from original, the differences are not perceptible

∗ Visually lossy compression: differences are perceptible

∗ Progressive Spatial Resolution: ability to extract lower resolution images

from a stream without redundant coding

∗ Progressive Quality Resolution: ability to extract lower bit-rate images

without redundant coding

∗ Complexity scalability: depends on applications, different levels complexity

∗ Strip processing: ability to compress and decompress images in a single

sequential pass

JPEG2000

3

13


Fac. EE SPS-VCA



Some examples

JPEG2000 versus

JPEG baseline

14


Fac. EE SPS-VCA



JPEG at 0.125 bpp

15


Fac. EE SPS-VCA



JPEG2000 at 0.125 bpp16


Fac. EE SPS-VCA



JPEG at 0.25 bpp

17


Fac. EE SPS-VCA



JPEG2000 at 0.25 bpp18


Fac. EE SPS-VCA



JPEG at 0.5 bpp

4

19


Fac. EE SPS-VCA



JPEG2000 at 0.5 bpp20


Fac. EE SPS-VCA



JPEG compound image 1.0 bpp

21


Fac. EE SPS-VCA



JPEG2000 compound im’g 1.0 bpp22


Fac. EE SPS-VCA



JPEG vs. JPEG2000: Major Differences

∗ New functionalities– Region-Of-Interest (ROI) Coding

– Better error resilience

– More flexible progressive coding (both in accuracy & resol.)

– Object-oriented functions (coding, information embedding)

∗ Lossy to lossless in one system

∗ Better compression at low bit-rates

∗ Better at compound images and graphics (palletized)

23


Fac. EE SPS-VCA



JPEG2000 and other standards

24

26

28

30

32

34

36

38

40

42

44

46

0 0.5 1 1.5 2

bpp

PSNR (dB)

J2K R J2K NR JPEG VTC

24


Fac. EE SPS-VCA



Some lossless compress. results

0.00

1.00

2.00

3.00

4.00

5.00

6.00

7.00

8.00

bike café cmpnd1 chart aerial2 target us average

co

mp

ress

ion

ra

tio

JPEG2000 JPEG-LS L-JPEG BZIP2

5

25


Fac. EE SPS-VCA



Comparison of various algorithms

on functionality

JPEG 2000 JPEG-LS JPEG MPEG-4 VTC

lossless compression performance +++ ++++ + -

lossy compression performance +++++ + +++ ++++

progressive bitstreams ++++ - + ++

Region of Interest (ROI) coding +++ - - +

arbitrary shaped objects - - - ++

random access ++ - - -

low complexity ++ +++++ +++++ +

error resilience +++ + + +++

non-iterative rate control +++ - - +

genericity +++ +++ ++ ++

26


Fac. EE SPS-VCA



More comparisons between

JPEG2000 & other standards

∗ « JPEG 2000 still image coding versus other standards », D. Santa-Cruz,

T. Ebrahimi, J. Askelöf, M. Larsson and Ch. Christopoulos, in Proc. of

SPIE, Vol. 4115

∗ « A study of JPEG 2000 still image coding versus other standards », D.

Santa-Cruz, T. Ebrahimi, in Proc. of the X European Signal Processing

Conference (EUSIPCO), Tampere, Finland, September 5-8, 2000

∗ « An analytical study of JPEG 2000 functionalities », D. Santa-Cruz, T.

Ebrahimi, in Proc. of the IEEE International Conference on Image

Processing (ICIP), Vancouver, Canada, September 10-13, 2000

27


Fac. EE SPS-VCA



JPEG2000

Part 2: Algorithm description

C

28


Fac. EE SPS-VCA



JPEG2000 / Basic encoding scheme

Wavelet

transform

Codeblock

partition

Entropy

codingRate

allocation

Coefficient

Quantization

Codeblock

partition

Entropy

codingCoefficient

Quantization

Codeblock

partition

Entropy

codingCoefficient

Quantization

Joint RDO

analysis

Analysis

coeff.dataPre & post

coding analysis!

Multiple

coeff. bands

29


Fac. EE SPS-VCA



Embedded Block Coding with

Optimized Truncation (EBCOT)

∗ Each subband is partitioned into a set of blocks

∗ All blocks within a subband have the same size

– (possible exception for the blocks at the image boundaries)

∗ Blocks are encoded independently

∗ Post-processing operation determines the extent to which each

block’s bitstream should be truncated

∗ Final bitstream is composed of a collection of “layers”

30


Fac. EE SPS-VCA



Motivation block coding in bands?

∗ Exploit local variations in the statistics of the image

from block to block

∗ Provide support for applications requiring random

access to the image

∗ Reduce memory consumption in hardware

implementations

– Both for compression or decompression engine

∗ Enable parallel implementation

∗ Typical blocks are 64x64 coefficients, later 4x256 to

reduce memory

6

31


Fac. EE SPS-VCA



EBCOT coding operations

∗ T2: Layered bitstream

formation

∗ T1: Generation of

embedded block bit-

streams

T1Low-level embedded block

coding engine

T2Layer formation and block

summary informationcoding engine

Full-featured bit stream

Emb. block bit streams and

summary information

Blocks subband samples

32


Fac. EE SPS-VCA



EBCOT /

Layered bitstream formation – (1)

∗ Each bitstream is organized as a succession of layers

∗ Each layer contains additional contributions from each

block (some contributions might be empty)

∗ Block truncation points associated with each layer are

optimal in the rate distortion sense

∗ Rate distortion optimization can be performed but it does

not need to be standardized

33


Fac. EE SPS-VCA



EBCOT / Layered formation – (2)

empty

empty empty

empty

empty

empty

empty

empty

layer 5

layer 4

layer 3

layer 2

layer 1

block 1 bit-stream

block 2 bit-stream

block 3 bit-stream

block 4 bit-stream

block 5 bit-stream

block 6 bit-stream

block 7 bit-stream

34


Fac. EE SPS-VCA



JPEG2000 / Coding types – (1)∗ Zero Coding

∗ Run-Length Coding (RLC)

– Used in conjunction with ZC, in order to reduce the average binary symbols count that must be encoded with arithmetic coding

∗ Sign Coding (SC)

– Used at most once for each sample in the block immediately, a previously insignificant symbol is found to be significant during a ZC or RLC operation

∗ Magnitude refinement (MR)– Used to encode an already significant sample

35


Fac. EE SPS-VCA



JPEG2000 / Coding types – (2)

∗ If the sample is not yet significant, a combination of Zero Coding and RLC primitives is used to encode whether or not a symbol is significant in the current bit-plane

∗ If significant, the Sign Coding (SC) primitive must also be invoked to send the sign

∗ If the sample is already significant, the Magnitude Refinement (MR) primitive is used to encode the new bit position

36


Fac. EE SPS-VCA



JPEG2000 / Zero Coding (ZC)

∗ Use 1 of 9 different context states to code the value of the symbol, depending on the significance state variables of:

∗ Immediate horizontal neighbors (h)

∗ Immediate vertical neighbors (v)

∗ Immediate diagonal neighbors (d)

∗ Non-immediate neighbors (f)

v

v

hh

d

7

37


Fac. EE SPS-VCA



JPEG2000 / Wavelet Transform

∗ Two filters supported– W9x7 (Floating point)for lossy coding– W5x3 (Integer) forlossless coding

∗ Only dyadic decomposition supported

– Earlier experiments also with packet and spatial decompositions

Dyadic decomposition

38


Fac. EE SPS-VCA



JPEG2000 / Filters supported

∗ Floating-point wavelets– Daubechies (9 x 7), also (10 x 18)

– (9 x 7) proposed in Part 1 of the standard, coefficients in working p.

∗ Integer– (13 x 7), CRF (13 x 7), (5 x 3), (2 x 10), etc. …

∗ Default integer for lossy coding– CRF (13 x 7)

∗ Default integer for lossless coding– (5 x 3)

∗ User defined filters

39


Fac. EE SPS-VCA



JPEG2000 / QuantizationScalar dead-zone quantization or Thresh. Coeff. Quant.

∗ Explicit

– Define a specific quantization step for each subband

– Smaller quantization steps for lower resolution subbands

∗ Implicit

– Quantization steps derived from LL subband quantization steps

– Smaller quantization steps for lower resolution subbands

∗ Reversible (Quantization and Coding)

– No quantization but pure bit-plane coding of transform coefficients

• Possibility of visual weighting− Fixed visual weighting

− Visual progressive coding (VIP)

40


Fac. EE SPS-VCA



JPEG2000 Further capabilities – (1)

∗ Advanced Rate Control– Post-compression rate-distortion optimization

– Requires second pass algorithm

– Target one bit rate or an arbitrary set of specified bit rates

∗ Scalability– Resolution scalability

– SNR scalability: select degree of SNR or Progr. Visual Weighting

∗ Random access into bit stream

∗ Arbitrary resolutions (up to 16 levels, up to 1x1 decom)

∗ Multi-component imaging (up to 256 components)

41


Fac. EE SPS-VCA



J2K Multiresolution decomposition – (1)

Original

Image

42


Fac. EE SPS-VCA



LH1

HL1

HH1

LL1LL1

Multiresolution decomposition – (2)

8

43


Fac. EE SPS-VCA




LL2

LH1

HL1

HH1

LH2 HH2

HL2LL2

44


Fac. EE SPS-VCA



LH1 HH1

LH2 HH2

HL2

HL3

HH3LH3

LL3

HL1


45


Fac. EE SPS-VCA



Example of dyadic decomposition into subbands

Multiresolution decomposition – (5)46


Fac. EE SPS-VCA



JPEG2000

Part 3: Scalability and

Progressive coding

47


Fac. EE SPS-VCA



– Different modes are realized depending on the way

information is written into the codestream

Code

stream

JPEG2000 / Scalability forms48


Fac. EE SPS-VCA



Scalability / Progressive Resolution – (1)

9

49


Fac. EE SPS-VCA



Scalability / Progressive Resolution – (2)50


Fac. EE SPS-VCA



Scalability / Progressive Resolution – (3)

51


Fac. EE SPS-VCA



Scalability / Progressive Resolution – (4)52


Fac. EE SPS-VCA



Scalability / Progressive Accuracy – (1)

53


Fac. EE SPS-VCA



Scalability / Progressive Accuracy – (2)54


Fac. EE SPS-VCA



Scalability / Progressive Accuracy – (3)

10

55


Fac. EE SPS-VCA



Example: Progressive by resolution

∗ Image: Woman

∗ Resolution levels: 5

∗ Decoded sizes: 1/16

1/8

1/4

1/2

1

56


Fac. EE SPS-VCA



57


Fac. EE SPS-VCA



58


Fac. EE SPS-VCA



59


Fac. EE SPS-VCA



60


Fac. EE SPS-VCA



11

61


Fac. EE SPS-VCA



Example: Progressive by quality

∗ Image: Woman

∗ Bitrates: 0.125 bpp

0.25 bpp

0.5 bpp

1.0 bpp

2.0 bpp

62


Fac. EE SPS-VCA



0.125 bpp

63


Fac. EE SPS-VCA



0.25 bpp

64


Fac. EE SPS-VCA



0.5 bpp

65


Fac. EE SPS-VCA



1.0 bpp

66


Fac. EE SPS-VCA



2.0 bpp

12

67


Fac. EE SPS-VCA



JPEG2000

Part 4: Specific features:

Region Of Interest (ROI) coding

and Conclusions

68


Fac. EE SPS-VCA



JPEG2000 / Region Of Interest coding

∗ Allows certain parts of an image to be coded

or decoded in better quality

∗ Static: • ROI is decided and coded once for all at the encoder side

∗ Dynamic:• ROI can be decided and decoded on-the-fly from a same

bitstream

69


Fac. EE SPS-VCA



JPEG2000 ROI: Some visual results

No ROI

69:1 overall compression ratio

Rectangular ROI

70


Fac. EE SPS-VCA



Regions Of Interest – (1)

71


Fac. EE SPS-VCA



Regions Of Interest – (2)72


Fac. EE SPS-VCA



Regions Of Interest – (3)

13

73


Fac. EE SPS-VCA



J2K / ROI coding: mask computation

Filter length

over the

boundaries!

Mask adapted

to filter

characteristics

Because of wavelet filtering, there are issues….

74


Fac. EE SPS-VCA



JPEG2000 / Region Of Interest coding

∗ Basic idea:• Calculate wavelet transform of whole image/time

• calculate ROI mask == set of coefficients that are

needed for up to lossless ROI coding

• Encoding is progressive by accuracy and

resolution

• NOTE: ROI mask need NOT be transmitted to decoder

(instead, location and shape of ROI is required)

75


Fac. EE SPS-VCA



JPEG2000 / Creation of ROI mask

∗ The ROI masks are acquired

by looking at the inverse

transform

∗ For each pixel (X) that is in

the ROI, the low- and high-

frequency coefficients (L:s

and H:s) that are needed to

reconstruct the pixel, are

included in the ROI mask

n-1 n n+1

Low High

n-1 n n+1

X:s

2n 2n+1

Inverse transform of the 5-3 filter

76


Fac. EE SPS-VCA



JPEG2000 / Modes for ROI Coding

∗ Block-based mode– Comes for free but not the optimal solution for static ROI

– Useful for dynamic ROI

∗ Arbitrary shape scaling based mode– Scale ROI mask coefficients up (decoder down)

– During encoding, the ROI mask coefficients are found significant at early stages of the coding

– ROI always coded in better PQ than background

∗ MaxShift method

77


Fac. EE SPS-VCA



JPEG2000 / ROI Scaling-based method

Coefficient values

ROI Coefficients

Highest BG coeff value is found

78


Fac. EE SPS-VCA



JPEG2000 / ROI MaxShift method

ROI Coefficients

Coefficient values

After shifting, all the ROI coefficients are larger than the largest BG coefficient

If min. coefficient in ROI

is larger than BG, then

MaxShift method!

14

79


Fac. EE SPS-VCA



JPEG2000 / Example ROI coding

∗ Image: Woman

∗ ROI: rectangular

∗ Scaling value: 6

∗ Progressive type: SNR

∗ Bitrate: 4 bpp

80


Fac. EE SPS-VCA



0.125 bpp

81


Fac. EE SPS-VCA



0.25 bpp

82


Fac. EE SPS-VCA



0.5 bpp

83


Fac. EE SPS-VCA



1.0 bpp

84


Fac. EE SPS-VCA



2.0 bpp

15

85


Fac. EE SPS-VCA



4.0 bpp

86


Fac. EE SPS-VCA



ROI Maxshift mode /

What is the gain?

– Support for arbitrary shaped ROI’s with minimal

complexity

– No shape information to send

– No shape encoder and decoder is needed

– No ROI mask at decoder side is needed

– Decoder as simple as non-ROI capable decoder

– Can decide in which subband the ROI will begin– therefore it can give similar results to the general scaling method

87


Fac. EE SPS-VCA



ROI coding: what do we pay? – (1)

Lossless image coding with ROI

Gold: Rectangular ROI

0,99

0,995

1

1,005

1,01

1,015

1,02

No ROI

S=2

S=4

Maxshift

88


Fac. EE SPS-VCA



ROI coding: what do we pay? – (2)

Lossless image coding with ROITarget - approx. 25% circular ROI - Relative sizes

0,94

0,96

0,98

1

1,02

1,04

1,06

1,08

1,1

No ROI

S=2

S=4

Max shift

89


Fac. EE SPS-VCA



JPEG2000 / Conclusions

∗ Advanced Still Image Coding System

– Better performance than JPEG and forms of scalability

∗ Offers many functionalities, yet more complex than JPEG

– RDO quantization, progressive resolution & accuracy, ROI, etc.

∗ No IPR associated to Part I of the standard (free licensing)

∗ Intended key standard for still image coding this decade

– Killer application upcoming: Digital Cinema

90


Fac. EE SPS-VCA



JPEG2000 / More background info

∗ JJ2000– http://jj2000.epfl.ch

∗ JPEG Web site:– http://www.jpeg.org

∗ EUROSTILL– http://ltswww.epfl.ch/~eurostill

∗ SPEAR– http://spear.jpeg.org/

Documents

Advanced Topics Multimedia Video (5LSH0), …vca.ele.tue.nl/sites/default/files/5LSH0 mod 01B JPEG2000...1 1 PdW-AP-EB-FS-FG-SZ-EB / 2017 Fac. EE SPS-VCA Adv. Topics MMedia Vid. Cod