Chapter 4

Hardware & Software

That

Enables Multimedia

Hardware That Enables

Multimedia

Hardware needs to be considered when you are

specifying, designing, creating, and

maintaining a multimedia presentation

Why?

Display Adapters

Resolution: pixel by pixel size that defines the size of the display area limited by hardware

E.g., my laptop: max resolution 1600 x 1200 pixels

Video memory

Used to hold RGB (or CYM) pixel intensity information

Motion video

Frame rate indicates how many times per second video memory must be updated

Amount of memory updated depends on

Size of video image

Color resolution

Frame rate

Image and Text Capture: OCR,

Digital Cameras, Scanners

Real image, digital image, still image, motion image

OCR: Optical Character Recognition

A method to capture printed text, as characters

Depends on

OCR software

e.g., range of fonts recognized

Quality of page being scanned

Likely will require proof reading

Still image capture: digital camera, scanner

Issues: Optics quality, precision of photocells (e.g., CCDs), dpi: dots per inch resolution

Speakers/Sound

Sound formats

.WAV, .MP3

MIDI (musical instrument digital interface)

Microphone

Microphone

Generates analog signal for sound

Can digitize with an A/D converter (sound input board)

Video

Video Cameras, frame grabbers

Analog video:

Generates analog video signal (e.g., NTSC, PAL)

Need a frame grabber (video input card)

Converts an analog video signal to a digital image frame

Digital video:

Directly Digitizes camera image at some frame rate

E.g., digital camcorder with firewire interface

Graphics Card and Graphical Processing

Unit (GPU)

Positional Information

Mouse, joystick, digitizing tablets, track pads, touch screens

Positional information is analog

E.g., rotations of the wheels on a mouse device

E.g., coordinate information on a digitizing tablet; stylus pressure

This information is digitized

Change information is often important

Future developments

Eye tracking, eye gaze

In combination with mouse or other devices

Mass Storage

DVD, CD-R, CD-RW, Mini disc (MD)

Direct access devices

Capacity

CD-ROM: about 720MB

MD: 74 minutes audio @ 44.1 kHz

DVD: from 4.4 GB to 16GB (8 hrs video)

Speed: Write speed, read speed, rewrite speed

Write once (CD-R), write multiple times (CD-RW; MD)

Laser disks: Analog format only?

60 minutes of 30 fps video

Virtual Reality (VR)

Goal: “to provide the user with a computer-controlled environment that is as close to physical reality as possible”

VR system equipped with sophisticated output devices

And specialized input devices

Allow the user to react in a similar to “natural” manner

Getting the computer system to adapt (in part) to the human

As opposed to the human adapting to the computer system

Emphasis similar to that in interactive multimedia

A great deal of output from the computer to the user

Less input from the user to the computer

Networks:

Modems & Network Interfaces

Modems Connection to phone, cable, ADSL, or ISDN

Network Computers connected via some physical

networking (e.g., LAN, Internet) hardware

Need network interface hardware (e.g., ethernet)

Required to interface to a network

Network bandwidth Modems: Lower bandwidth

Cable modem, ADSL, LAN connection: Higher bandwidth

Character encoding ASCII, EBCDIC, Unicode (16 bit), ISO/IEC

standards

Software That Enables Multimedia

Operating systems

Development software

Delivery software

Operating System Software

Device drivers

Multi-tasking, multi-threading

Memory management

Disk space management

Processor (CPU) management & scheduling

Window management

Servers (e.g., printing, email, web, telnet, ssh)

User level software

Includes programs to access and deal with lower levels of operating system (e.g., format a disk, partition a disk, change the priority of a task or process)

end user level applications to get work done

E.g., Macromedia Director, Power Point, Web browser

Device Drivers

E.g., Display adapter driver, Firewire card driver, Network card driver, Video Camera Driver, Microphone driver

Devices have specific hardware

Drivers are specific to the hardware

Enable the operating system to use that specific hardware

Device drivers depend

on operating system

on the specific device

Device drivers enable programs

To be less dependent on hardware; more generic

Codec’s

Compression and Decompression software for video and audio

“drivers” for the particular format used for the files

Depend on format, not on hardware

Within a particular general format (e.g., AVI or Quicktime)

E.g., with AVI, a Cinepak or Indeo codec can be used

Sorensen is available with Quicktime

Many codec’s can be used

Codec must be on development system and on the destination system

Networking - 1

Networking functions are incorporated

into operating systems

Remote disk mounting

Network device (e.g., printer) access

Lower levels of network data transmission

(e.g., physical, data link layers)

Multimedia software often depends on

networks

Dynamic loading of codec & other components

Access to web-based presentations

Networking - 2

Communication protocols

Physical layer protocols (ISO layer 1)

E.g., Ethernet, token ring, ATM

Higher protocol layers

TCP/IP (layers 2 & 3)

SMTP – simple mail transfer protocol (layer 6)

HTTP – hypertext transfer protocol (layer 6)

Performance objectives

Accuracy, transmission speed, error correction, user-perceived delays, overall throughput

Client/Server Architecture - 1

Server

A process that provides access to particular hardware or software resources

Process: A program that runs independently on a computer

Often used in the context of networks

Used within software design generally, as well

“Server” may designate a software process

Or may designate a computer (hardware)

Depends on the kind of service being provided

Client/Server Architecture - 2

E.g., database server

Computer will have database software

May also have additional hardware resources (e.g., more disk space)

E.g., a software license server (“metering”)

Enables some maximum number of people to be using a software resource at a given time.

Server receives requests

Remote requests from a client process on another computer

Local requests from a client process on same computer

Requests are sometimes for data sometimes for programs

Operating System &

Multimedia Requirements

Development of multimedia

Running multimedia

Some operating systems may be more suited for running multimedia

Issues include:

Memory (RAM) required

Operating system availability

Application availability

Speed

Networking

Multimedia Development

Software

Software needed to develop multimedia

And software needed to deliver

E.g., Shockwave playback in a browser has different requirements than running Macromedia Director

Development

Preproduction, production, postproduction

Types of development software

2D image capture, scanning & processing; Digital video capture & processing

Animation; 3D modeling, rendering; Sound capture & processing; Text; Web development; Authoring

Vector vs. Bitmap Graphics - 1

Representation of objects in images E.g., a rectangle with a black outline and a gray

interior

Bitmap (raster) graphics Specify value for each pixel

Application data file (e.g., Photoshop) stores collection of pixels making up rectangle

E.g., 300 x 600 = 180,000 pixels of rectangle

Requires 176KB to 527KB of data (uncompressed)

Vector graphics Use rectangle drawing tool & color-fill

Application program generates a data file that represents rectangle as a command and parameters

E.g.,

Rectangle, 0, 0, 300, 600, black, gray (32)

Requires about 100 bytes of information

Vector vs. Bitmap Graphics - 2

Space Tradeoff

Bitmap graphics (uncompressed) take more space

Vector graphics take less space

Speed Tradeoff

Bitmap graphics (uncompressed) are drawn faster

Vector graphics need to be decoded and then the bitmap (pixels) generated

Will be slower than bitmap representation

An issue on slower computers or if a very large number of objects need to be displayed

Expressiveness

Some objects can’t be easily drawn in vector fashion

Contemporary drawing programs may provide both vector and bitmap graphics

Video - 1

Video is captured in discrete units called frames Frame: Similar to a still picture

Temporal and spatial compression Spatial: within a frame

Temporal: across frames

Key-frame concept in temporal compression Some compression methods use this form of coding

Store information on one frame at particular intervals

This is a “key frame”

Then store changes from this point

Uses frame differencing

Makes use of common background often shared between multiple frames

Video - 2

Time-dependent medium

Illusion of motion depends on frame rate

Don’t perceive motion below about 15 fps

Also: animation and sound

Time-dependent media place high computational demands on a computer

E.g., with 1024 x 768 image resolution, and 24 bit color depth @ 30 fps, what is the (apparent) data transfer rate in KB/second?

Video - 3

1024 * 768 = 786, 432 pixels/frame

786, 432 pixels/frame* 3 bytes/pixel =

2,359,296 bytes/frame

2,359,296 bytes/frame * 30 frame/sec =

70,778,880 bytes/sec

(70,778,880 bytes/sec) / (1024 bytes/KB) =

69,120 KB/sec

(Apparent: because this is without any data compression)

Animation

Similar to video

Each frame created via software

not from a camera source

History

traditionally done by hand, frame by frame

Model-based animation

Online (real-time) versus offline

Model-based: software representation of world

Online: Frames are generated on-the-fly, in response to dynamic changes of the model

Offline: Frames generated in response to model changes

Related Techniques

Biological animation

Animal gait (مشية الحيوان), bipedal motion (حركة القدمين)

Morphing (تتحول) Gradual changes in one image to transform

it to a second image

Applying changes gradually

E.g., taking a few seconds @ 30 fps

Can be used for animation purposes

VRML

Virtual reality modeling language

"an open standard for 3D multimedia and shared virtual worlds on the Internet."

MPEG, Quicktime, AVI

Standard formats for video and audio

information

Includes techniques/standards for

compression, decompression

MPEG

MPEG Audio

ISO/IEC standardization

Open standard

Specification is available to everyone (for a fee)

No single company “owns” the standard

Quicktime, AVI

Quicktime

Primarily Macintosh/Apple

http://www.quicktime.com

AVI

Primarily Windows/Microsoft

AVI: “Audio Video Interleave”

E.g., see http://www.jmcgowan.com/avi.html

MPEG Standards

MPEG-1:

Applications: MP3, Video CD

MPEG-2

MP3, Digital Television set top boxes, Digital Versatile Discs (DVD).

MPEG-4

Proposed fixed & mobile web standard

MPEG-7: Multimedia Content Description

MPEG-21: Multimedia Framework

MPEG: Video

MPEG-1

Roughly VHS quality at 1,15 Mbit/s

MPEG-2

“MPEG-2 Video was at least good, and in many cases even

better than standards or specifications developed for high

bitrate or studio applications”

http://mpeg.telecomitalialab.com/standards/

mpeg-2/mpeg-2.htm

MPEG: Audio

MPEG standards have layers

Layer: Operating modes with increasing complexity and performance

MPEG-1 provides

Mono and stereo coding at 32, 44.1, and 48 kHz sampling rate

Layer I, II, III: varying bit rates from 32 to 448 kbit/s

Optimized for 128 kbits/s stero

MPEG-2 BC provides

Backwards compatible multichannel extension to MPEG-1

Up to 5 channels plus a 'low frequent enhancement' channel

Bit rate range is extended up to about 1 Mbit/s;

an extension of MPEG-1 towards lower sampling rates 16, 22.05, and 24 kHz for bitrates from 32 to 256 kbit/s (Layer I) and from 8 to 160 kbit/s (Layer II & Layer III).

http://mpeg.telecomitalialab.com/faq/audio.htm#5

MPEG-2 & MPEG-4: Audio

MPEG-2 Advanced Audio Coding (AAC)

High-quality audio coding standard for 1 to 48 channels

Sampling rates of 8 to 96 kHz

Multichannel, multilingual, and multiprogram capabilities

Bit rates from 8 kbit/s to more than 160 kbit/s/channel

Multiple encode/decode cycles

Three profiles (layers) of AAC provide varying levels of complexity and scalability.

MPEG-4 will provide

coding and composition of natural and synthetic audio objects at a very wide range of bit rates.

Emphasis on new functionalities

MPEG-4

Goal: Higher levels of interaction

1. represent units of natural, visual or audiovisual content, called "media objects". These media objects can be of natural or synthetic origin; this means they could be recorded with a camera or microphone, or generated with a computer;

2. describe the composition of these objects to create compound media objects that form audiovisual scenes;

3. multiplex and synchronize the data associated with media objects, so that they can be transported over network channels providing a QoS appropriate for the nature of the specific media objects; and

4. interact with the audiovisual scene generated at the receiver’s end.

http://mpeg.telecomitalialab.com/standards/mpeg-4/mpeg-4.htm

MP3

MPEG-1/2 Audio Layer-3

MPEG-1 Audio Layer-3

MPEG-2 Audio Layer-3 (successor)

MP3 and AAC Explained

http://citeseer.nj.nec.com/421409.html

MP3 MPEG-3!

Sound Representation

On a sample basis E.g., about 44 kHz for CD quality audio

Each sample is a number

gives the amplitude of the analog sound wave at that point

On a feature basis E.g., frequency, duration, musical note

Also possible: for speech, phonemes

Phoneme is a unit of sound in a particular language

Phonemes may sound differently dependent on context

E.g. /p/ (personal, paternal, computer, rapid), /b/, and /s/

Difference is analogous to the difference between bitmapped (raster) & vector graphics

Tradeoffs– Similar to those between bitmap & vector graphics

Example 1

With a sampling rate of 44.1 kHz, and 16 bitsound resolution, how many KB does it take to represent 3 minutes of sound? (Without compression)

Example 1

44,100 samples/sec * 2 bytes/sample =88, 200 bytes/sec

88, 200 bytes/sec * 60 sec/minute =

5,292,000 bytes/min

5, 292, 000 bytes/min * 3 min =

15,876,000 bytes

15,876,000 bytes/(1024 bytes/KB) =

15, 503.90625 KB

(Or about 15 MB)

Example 2

If we are representing speech phonemes, and we want to represent individual words, how much data does it take to encode the English word “Hello”?

Assume that English has 39 phonemes

Assume that we have one phoneme per character

Problem is to represent sequences of phonemes

Each phoneme can take on 39 possible values

Need at least 6 bits of information to represent 64 different values , So, need at least 6 bits per phoneme

Word “Hello” has 5 characters

Assuming also 5 phonemes 5 * 6 = 30 bits

30 bits/(8 bits/byte) = 3.75 bytes Need 4 bytes

Sound Filtering

A sound can be filtered to remove a certain set of frequencies

Some types of filtering

Low pass: Allows only low frequencies through

High pass: Allows only high through

Band pass: Allows only a band range of frequencies through

Other types of filtering or processing

Trimming, splicing, volume

Web browsers

Process various file types

HTML

Media formats: images, sounds, movies

Some types of data are processed directly

Built into the browser

Other types

Require additional programs or “plugin-ins”

E.g., a plugin to display PDF or Adobe Portable Document format files or a plugin to display Shockwave files

Some plugins are called “players” or “viewers”

E.g., a viewer to display MS Word documents

Multimedia platform

requirements:

Multimedia platform should

Support

the development of a wide variety of multimedia applications

applications in a distributed system

Be able to

establish and control streams of continuous media

to choose the quality of service for a data stream

to express relations between several media streams

Multimedia Delivery Software

Two general means of delivery Compiled or Interpreted

Compiled Machine code generated for a computer CPU

(e.g., Intel-based) and operating system (O/S)

Faster; limited to computer type & O/S; stand-alone

E.g., Macromedia Director “Projector”

Interpreted Instructions created that can be interpreted by

a program; E.g., a plugin

Slower; more portable; need interpreter (e.g., player) software

E.g., Macromedia Shockwave

In both cases, media files are embedded and/or retained as separate files