1

CS 558 : CS 558 : Multimedia Systems Multimedia Systems

Lecturer: Don Adjeroh

Office: Rm937, ESB

e-mail: don@csee.wvu.edu

Course Schedule: Tuesday 9:30 - 10:55am (Rm G449, ESB)

Thursday 9:30 - 10:55pm (Rm G449, ESB)

2

Important Dates

Assignment-1 Week of Feb. 11, 2008 (Due 2 weeks after)

Assignment-2 Week of Mar 3, 2008 (Due 2 weeks after)

Assignment-3 Week of Mar 17, 2008 (Due 2 weeks after)

Student Projects Week of Mar 3, 2008 (due Apr. 28, 2008)

Student Seminar From Apr 21, 2008

Assessment

Weekly Quiz 20%

3 Assignments 30%

Student Projects 20%

Student Seminar 20%

Class Participation 5% extra on Assignments

3

References:

Lu G., Communication and Computing for Distributed Multimedia Systems, Artech House, Norwood, Massachusetts, 1996.

Ze Nian Li , Mark S. Drew, Fundamentals of Multimedia, Prentice Hall, 2004

P. A. Chou and M. van der Schaar, (eds.), Multimedia over IP and Wireless Networks: Compression, Networking, and Systems, Elsevier, 2007.

4. Gonzalez R. C. and Woods R. E., Digital Image Processing. Addison-Wesley Publishing Co., Reading, Massachusetts, 2001.

Others Raghavan, S. V. and Tripathi, S. K., Networked Multimedia Systems:

Concepts, Architecture and Design, Prentice Hall, 1997 Papers and other reference materials to be provided on need basis

4

Course OutlineCourse Outline

Introduction

Multimedia Communications and Transport

Multimedia Data Compression

Multimedia Information Systems

Information Networks

5

Elements of MM Computing SystemsElements of MM Computing Systems

MM data acquisition/generation MM storage MM information processing MM communication MM presentation MM QoS specification and performance measurement.

6

Data Acquisition/generationData Acquisition/generation

Multimedia data can be computer generated - e.g. graphic objects or animation sequences, or images.

Most MM data are however externally acquired. Externally acquired MM data requires two basic units: A transducer – a physical sensing device that transforms the

physical data such as sound or light into electrical signals. A digitizer – a device that converts the electrical signal into a

digital form, example the scanner.

Some devices perform the two basic functions at the same time, example digital cameras.

7

Analogue to Digital ConversionAnalogue to Digital Conversion

Stages in ADC : sampling, quantization, and coding.

Sampling – converts the continuous-time signal into a discrete-time signal.

Quantization - converts the discrete-time continuous values into discrete-amplitude values. A sample value that falls into a given quantization level is assigned that quantization level.

Coding - represents the quantized amplitude values in the form of digital codes. Facilitates the transmission and storage of the digital signal.

8

9

MM Data StorageMM Data StorageText: (One page)

Plain text at 60 lines/page, with 80 characters 48KB

Formatted text: (2 bytes/character) 96KB

Images: (e.g. a 1024x1024 image at 8-bits/colour level)

Grey-scale image requires about 1MB of storage.

Colour image will require 3MB of storage.

Video:

A 5-minute video clip at 30fps will require:

5*60*30 *3MB=27000 MB or 27GB of storage

Audio - Storage depends on: the sampling rate used, and the number of bits used to represent each sample value

Generally, Bit rate =sampling rate * no of bits per sample

10

Example:

For CD audio, the frequency range is 20 – 20,000Hz, or a bandwidth of about 20kHz. For accurate reconstruction, the minimum sampling rate is thus 40kHz. The industry standard is 44.1kHz. Using 16 bit quantization, and for the 2 audio (i.e. stereo) channels, we will have 44100x2x16 or 1410kbps. Thus, to store a 5 minute CD audio, we will need 1410*5*60 kb (or > 50MB) of storage.

Four types of storage used for multimedia data: Short-term storage: main memory On-line storage: magnetic disks, optical disks, CDs, MOs Near-line storage: juke-boxes, Off-line (archival) storage: magnetic tapes, WORM optical

disks

11

MM Information ProcessingMultimedia information usually involves

media-specific processing and analysis, and considerations across different media.

E.g: MM indexing, search and retrieval, MM data compression, MM information summarization, etc.

MM Communication limited network bandwidth large data sizes

A typical voice-phone line can transmit at rates up to 9600bps. To transmit the 8-bit 1024x1024 grey-scale image over this line will require about 14 minutes.

Approaches: compression, network & transport protocols, network scheduling and multimedia shaping algorithms

12

MM PresentationHow is the multimedia data finally presented to the user?

Key issues : Human perception of MM information Play-out or media consumption rate Interactivity – what level of control does the user have over the

presentation, Synchronization – both inter- and intra-media Consideration of spatio-temporal constraints and Media integration.

Performance and QoSPerformance of a MM system is often checked with respect to the

acceptable QoS specified by the user.

The overall performance is typically affected by the performance of the different functional elements.

13

Nature of MM DataNature of MM Data

MM Data type - form in which the MM data is presented

Example MM data types Text – plain text, formatted text Images – colour, black & white photos, maps, paintings Graphic objects – drawings, sketches, illustrations, 2D or

3D objects Animation sequences – moving images or graphic objects,

usually independently generated Video – sequence of images showing a record of real-life

event, usually produced by a video recorder Audio – speech and non-speech audio, music, etc Composites – formed from a combination of two or more

media types.

14

Classification of MM Data Static Vs Continuous media

Static media – are time independent. Meaning remains the same independent of presentation time.

Continuous media – are time-dependent. Their contents and meaning depend on the time and rate at which they are presented.

Stored Vs live, -for continuous media.

Multimedia system usually refers to systems a that is capable of handling at least one continuous-media data type, and one static media type.

Multimedia document - a document that contains at least one type of each media class.

15

General characteristics of MM dataGeneral characteristics of MM data

Huge volumes of dataMM data typically involves huge volumes of data.

Compression and storage techniques to the rescue! Temporal constraints –

Continuous MM data have temporal dependencies. These must be played-out at a given standard rate and in a certain order .

Implications for storage, processing, communication and presentation of these data types.

Inter/Intra media dependence Intra-media dependence - dependence among different

components of data belonging to the same media type. Inter-media dependence involves multimedia data across

different media types.

16

Semantic structure

Users have the final say on information content. MM data items may be described differently by different users. It is often difficult to precisely describe the information content of a multimedia data. Querying and searching MM data is thus performed in terms of similarity or approximate matches.

Error tolerance

Some level of error is often tolerated in considering the presented MM data. MM system designers usually make use of the limitations of human perception in providing acceptable levels of service to the user

Criterion: the presented results should not degrade below the user’s stated acceptable level, as indicated in the QoS specification.

17

Characteristics of MM SystemsCharacteristics of MM Systems Integration - aims at minimizing the number of

different systems that need to be connected.

Interactivity - how much control the user has over the presentation.

Degree of interactivity or user customization: time to start order of presentation (linear or non-linear) form of presentation speed of presentation automation of the presentation

Pro: could provide statistical info about user interaction

Con: Added level of complexity.

18

Computer control - MM systems require computer control of the presentation of information to the users.

Digital Representation - Some of the advantages: Uniformity in representation and information handling Error-free transformations and reliable transmission Ease of encryption Ease of regeneration or replication.

Two problems with digitization: Digitization is generally a not completely reversible. some

errors may be introduces. Digitized data often results in large data sizes, which in turn

requires more storage. Performance

QoS Traditional performance measures not adequate !

19

Classification of MM systems/applicationsClassification of MM systems/applications

Stored Vs Live (usually applicable to continuous media)

Interactive Vs non-interactiveWhat level of control does the user have over the presentation

of the MM data? Networked Vs non-networked (local)

Local – involves only a stand-alone system. Examples applications are authoring, training, etc.

Networked – distributed, usually involves clients and one or more servers communicating over a network.

20

Some application areasSome application areas

Entertainment - VoD, games, tour, museums, Education- tele-teaching, authoring tools, tutoring Medicine- telemedicine Business - video conferencing, video phone,

training, presentation Information Kiosks (IoD) Digital libraries Digital TV broadcasting Personal communication - video phone Others.

21

Other related issues MM, Hypermedia, and Hypertext

MM and the Internet

MM and Virtual Reality

Course Structure General approach taken in this

course - see diagram

DistributedMM

MMCompression

MM InfoSystems