Multimedia Standards Multimedia Standards MPEGMPEG

Instructor: Dr. Ritu

Presented by Jijun(Justin) Huang

0360520 Section 2

Winter 2003

OUTLINE• IntroductionIntroduction• MPEG OverviewMPEG Overview• MPEG Standards MPEG Standards

– MPEG-1MPEG-1– MPEG-2MPEG-2– MPEG-4MPEG-4– MPEG-7MPEG-7– MPEG-21MPEG-21

• Influence of Other Standards to Influence of Other Standards to MPEGMPEG

• Implementation AreasImplementation Areas• ConclusionConclusion

INTRODUCTION• MPEGMPEG is a working group of theis a working group of the ISO/IEC.ISO/IEC.

– MPEG (Moving Pictures Expert GroupMPEG (Moving Pictures Expert Group))– ISO (International Organization for Standardization)ISO (International Organization for Standardization)– IEC (International Electronics Commission)IEC (International Electronics Commission)

• The aim of MPEG is:The aim of MPEG is:– compression, compression, – decompression, decompression, – processing, processing, – and coded representation of video, audio, and their and coded representation of video, audio, and their

combination. combination.

MPEG Video Overview• Idea is to remove Idea is to remove

– Spatial redundancy:Spatial redundancy:• Discrete Cosine Transform (DCT) is used:Discrete Cosine Transform (DCT) is used:

– Transforms a video frame from spatial domain into a Transforms a video frame from spatial domain into a frequency domain.frequency domain.

– Temporal redundancyTemporal redundancy• Motion-compensation is used:Motion-compensation is used:

– Encodes a video frame based on other video frames Encodes a video frame based on other video frames which are temporally close to it.which are temporally close to it.

MPEG Video Format

• A video stream is a sequence of video frames A video stream is a sequence of video frames

– digitized in a standard RGB format, 24 bits per pixel;digitized in a standard RGB format, 24 bits per pixel;

• 8 bits for each Red, Green, and Blue.8 bits for each Red, Green, and Blue.

– RGB format must be converted to YUV format:RGB format must be converted to YUV format:

• Represented in 24 bits per pixel: Represented in 24 bits per pixel:

– 8 bits for each Y, U, and V information.8 bits for each Y, U, and V information.

MPEG Video Format

• In MPEG-1 images represented in YUV color In MPEG-1 images represented in YUV color

space space (YCbCr)(YCbCr)

– Frame is divided into macro blocks of 16x16 pixelsFrame is divided into macro blocks of 16x16 pixels

• 4 luminance blocks and 4 luminance blocks and

• 2 chrominance blocks2 chrominance blocks..

– Every block has a dimension of 8x8 valuesEvery block has a dimension of 8x8 values

• all luminance inf. (Y) are kept as it is, all luminance inf. (Y) are kept as it is,

• chrominance inf. is divided into 2 parts chrominance inf. is divided into 2 parts (Cb and Cr)(Cb and Cr)

MPEG Video Format

• This subsampling is a lossy step and This subsampling is a lossy step and • Does not affect the qualityDoes not affect the quality

• The 24 bits RGB format --> 12 bits YUV inf.,The 24 bits RGB format --> 12 bits YUV inf.,• which gives 2:1 compressionwhich gives 2:1 compression

• MPEG-1 allows video compression ratio:MPEG-1 allows video compression ratio:• 50:1 to 100:150:1 to 100:1

MPEG Frame Encoding• A MPEG “film” is a sequence of :A MPEG “film” is a sequence of :

– I- frames: is a still image,I- frames: is a still image,

– P-frames: are forward predicted from last I or P frameP-frames: are forward predicted from last I or P frame

– B-frames: are both forward & backward predicted from the B-frames: are both forward & backward predicted from the

last/next I or P framelast/next I or P frame

• Frame is divided into macro blocks of 16x16 pixelsFrame is divided into macro blocks of 16x16 pixels

– Depending on the kind of macro block:Depending on the kind of macro block:

• blocks contain pixel information or blocks contain pixel information or

• prediction error information.prediction error information.

– The information is compressed using the DCT.The information is compressed using the DCT.

MPEG Frame Encoding• Macro blocks are units for motion-compensated compr.Macro blocks are units for motion-compensated compr.

– Blocks are used for DCT compression.Blocks are used for DCT compression.

– I- frames: encoded as a single image,I- frames: encoded as a single image,

• Each 8x8 block transformed into a freq. domainEach 8x8 block transformed into a freq. domain

• Data is encoded in a zig-zag orderingData is encoded in a zig-zag ordering

– P-frames: encoded relative to the past reference frame; P-frames: encoded relative to the past reference frame;

• Encoded either as an I-macroblock or P-macroblock.Encoded either as an I-macroblock or P-macroblock.

– B-frames: encoded relative to the past/future or both B-frames: encoded relative to the past/future or both

reference frame.reference frame.

• Encoded similar to P-framesEncoded similar to P-frames

MPEG Audio

• To compress audio MPEG tries to remove:To compress audio MPEG tries to remove:

– The irrelevant parts, andThe irrelevant parts, and

– Redundant parts.Redundant parts.

• Parts of the sound that we don’t hear:Parts of the sound that we don’t hear:• Can be thrown away.Can be thrown away.

– MPEG Audio uses psycho-acoustic principles.MPEG Audio uses psycho-acoustic principles.

MPEG STANDARDS

• MPEG-1:(release in 1993, ISO 11172)MPEG-1:(release in 1993, ISO 11172)

– Standard for Standard for

• storage and storage and

• retrievalretrieval

of video and audio on digital storage media.of video and audio on digital storage media.

– Provides video compression rate:Provides video compression rate:

• Around 1.5 Mbit/s.Around 1.5 Mbit/s.

– Provides audio compression rate:Provides audio compression rate:

• Around 32Kbit/s to 384Kbit/s.Around 32Kbit/s to 384Kbit/s.

Description of MPEG-1 System

• Synchronized video and audio with timing information• 90kHz system clock

MPEG STANDARDS (continued)

• MPEG-2:(1994, ISO 13818)MPEG-2:(1994, ISO 13818)

• Standard for digital television.Standard for digital television.

• Tires for a higher resolution.Tires for a higher resolution.– A color resolution in:A color resolution in:

• MPEG-1: 4:2:0 YCbCrMPEG-1: 4:2:0 YCbCr

• MPEG-2: 4:2:2 YCbCrMPEG-2: 4:2:2 YCbCr

• Support video transmission about 2-80 Mbits/s.Support video transmission about 2-80 Mbits/s.

Description of MPEG-2 System

– PES Packetised Elementary Streams– PS: combines 1 or more PES with a common time base into a stream– TS: combines 1 or more PES with 1 or more independent time base

MPEG STANDARDS (continued)

• MPEG-4:(1998, ISO 14496; version 2, late 2001)MPEG-4:(1998, ISO 14496; version 2, late 2001)

– Include the ability to represent as a single object, andInclude the ability to represent as a single object, and– Audio, video, synthetic audio, graphics, image, etc.Audio, video, synthetic audio, graphics, image, etc.

– multiplex and synch. to form complex appearance.multiplex and synch. to form complex appearance.

– aims to provide a set of technologies to:aims to provide a set of technologies to:

• Satisfy the needs of authorsSatisfy the needs of authors

• Service providersService providers

• End usersEnd users

By avoiding incompatible formats and players.By avoiding incompatible formats and players.

– MPEG-4 video encodes frames into YUVA format.MPEG-4 video encodes frames into YUVA format.

MPEG STANDARDS (continued)

• MPEG-7:(release late 2001, ISO 15938)MPEG-7:(release late 2001, ISO 15938)– common interface for describing multimedia materials.common interface for describing multimedia materials.

– MPEG-7 aims to standardize:MPEG-7 aims to standardize:• A set of Description Schemes and Descriptors to describe dataA set of Description Schemes and Descriptors to describe data

• A language to specify Description Schemes, andA language to specify Description Schemes, and

• A scheme for coding the descriptionA scheme for coding the description

– MPEG-7 goalsMPEG-7 goals• Describe multimedia contentDescribe multimedia content

• Manage data flexibly, andManage data flexibly, and

• Globalize data resourcesGlobalize data resources

MPEG STANDARDS(continued)

• MPEG-21MPEG-21

– Describes “big picture” across wide range of networksDescribes “big picture” across wide range of networks

– Creates framework for multimedia delivery and consumptionCreates framework for multimedia delivery and consumption

– Transparent use of multimedia resources across networksTransparent use of multimedia resources across networks

– Provides access to information and services from almost Provides access to information and services from almost

anywhere at anytime with terminals and networksanywhere at anytime with terminals and networks

MPEG-21 Parts• Vision, Technologies and StrategyVision, Technologies and Strategy

– Technical roadmap, identify key steps in value chainTechnical roadmap, identify key steps in value chain

• Digital Item Declaration (DID)Digital Item Declaration (DID)

– Schema defining the structure of a digital item (resources + descriptors)Schema defining the structure of a digital item (resources + descriptors)

• Digital Item Identification & Description (DII & D)Digital Item Identification & Description (DII & D)

– Provides a tools to identify a Digital ItemProvides a tools to identify a Digital Item

• Intellectual Property Management & ProtectionIntellectual Property Management & Protection

• Rights Data Directory ( RDD) Rights Data Directory ( RDD) deal withdeal with

• Rights Expression Language ( REL) Rights Expression Language ( REL) digital rightsdigital rights

• Digital Item Adaptation ( DIA)Digital Item Adaptation ( DIA)

– SGML (SGML (Standard Generalized Markup LanguageStandard Generalized Markup Language ): provides a good ): provides a good

example for MPEG’s DDL (Description Definition Language)example for MPEG’s DDL (Description Definition Language)

StructuringStructuring

Describing contentDescribing content

– XML: XML Schema Language is used for the basis of it.XML: XML Schema Language is used for the basis of it.

– HyTime: parts of the standard are considered;HyTime: parts of the standard are considered;

Too genericToo generic

Difficult to understand and handleDifficult to understand and handle

Influence of Other Standards

DDL and XML <Character-info><Character-info> <Actor>Bruce Willis</Actor><Actor>Bruce Willis</Actor> <Character name>Bo Weinberg</Character name><Character name>Bo Weinberg</Character name> </Character-info></Character-info>

An example:An example:• A web site about boxingA web site about boxing• Huge databaseHuge database• Advanced queries on keywords describing certain Advanced queries on keywords describing certain

events in the video filesevents in the video files

Boxing DDL example

<Mpeg7><Mpeg7> .. <Video><Video> <TemporalDecomposition><TemporalDecomposition> <VideoSegment><VideoSegment> <TextAnnotation type = “Scene description” ><TextAnnotation type = “Scene description” > <FreeTextAnnotation>Tyson knocks some poor sap <FreeTextAnnotation>Tyson knocks some poor sap

out</FreeTextAnnotation>out</FreeTextAnnotation> </TextAnnotation></TextAnnotation> <MediaTime><MediaTime> <MediaTimePoint>T00:01:00:0F24 </MediaTimePoint>...<MediaTimePoint>T00:01:00:0F24 </MediaTimePoint>...

Tyson knocks some poor sap outTyson knocks some poor sap out

Implementation Areas• Social and economic issues :Social and economic issues :

– EducationEducation

– SurveillanceSurveillance

– ShoppingShopping

– Film, video, and radio archivesFilm, video, and radio archives

– Geographical Information Geographical Information

Systems (GIS)Systems (GIS)

– Face recognition technologyFace recognition technology

CONCLUSION• MPEG-1: Audio & video coding for digital storageMPEG-1: Audio & video coding for digital storage

• MPEG-2: For Digital TV ( DVD, Satellite)MPEG-2: For Digital TV ( DVD, Satellite)

• MPEG-4: For coding of audio & video objectsMPEG-4: For coding of audio & video objects

• MPEG-7: Identifies and manages audio-video content; multimedia indexing, MPEG-7: Identifies and manages audio-video content; multimedia indexing,

standardizes a set of high level schemas (XML-Schema)standardizes a set of high level schemas (XML-Schema)

• MPEG-21: Multimedia Framework...MPEG-21: Multimedia Framework...

• The complexity of streams are increasing ….The complexity of streams are increasing ….

• They are making the work enjoyable with computer.They are making the work enjoyable with computer.

REFERENCESREFERENCES

[1] S. Battista, F. Casalino, C. LandeLande, “MPEG-4: A Multimedia Standard for the Third Millennium, Part 2”, IEEE Multimedia, Vol.7, No.1, 2000,pp. 76-84

[2] S. Battista, F. Casalino, C. Lande, “MPEG-4: A Multimedia Standard for the S. Battista, F. Casalino, C. Lande, “MPEG-4: A Multimedia Standard for the Third Millennium, Part 1”, IEEE Multimedia, Vol.6, No.4, 1999,pp. 74-83Third Millennium, Part 1”, IEEE Multimedia, Vol.6, No.4, 1999,pp. 74-83

[3] F. Nack, A.T. Lindsay, “Everything You Wanted to Know About MPEG-7: Part F. Nack, A.T. Lindsay, “Everything You Wanted to Know About MPEG-7: Part 1”,IEEE Multimedia, Vol.6, No.3,1999, pp. 65-761”,IEEE Multimedia, Vol.6, No.3,1999, pp. 65-76

[4] F. Nack, A.T. Lindsay, “Everything You Wanted to Know About MPEG-7: Part F. Nack, A.T. Lindsay, “Everything You Wanted to Know About MPEG-7: Part 2”,IEEE Multimedia, Vol.6, No.4,1999, pp. 64-732”,IEEE Multimedia, Vol.6, No.4,1999, pp. 64-73

[5] S. Gringeri, R. Egorov, K. Shuaib, A. Lewis, B. Basch, “Robust Compression S. Gringeri, R. Egorov, K. Shuaib, A. Lewis, B. Basch, “Robust Compression and Transmission of MPEG-4 Video”, 7and Transmission of MPEG-4 Video”, 7thth ACM International Conference on ACM International Conference on Multimedia, Part I, 1999Multimedia, Part I, 1999

[6]   http://mpeg.telecomitalialab.com/standards/mpeg-1/mpeg-1.htm http://mpeg.telecomitalialab.com/standards/mpeg-1/mpeg-1.htm

    http://www.crs4.it/~luigi/MPEG/mpeg2.htmhttp://www.crs4.it/~luigi/MPEG/mpeg2.htm[7]

Thank you!Thank you!