Challenges in Wireless Multimedia

CSE Department Seminar Series

September 26, 2003

Borko Furht

OUTLINE

Scalable Video Source Coding Channel Coding and Error Control Power-Aware Coding and

Transmission Techniques Networking Issues

Rate control Multimedia Security Application: Virtual Workplace

Mobile Internet Access

Source: Ericsson

0

200

400

600

800

1000

1200

2000

2001

2002

2003

2004

2005

2006

Fixed

Mobile

Internet

Subscribers

(millions)

Year

Wireless Multimedia Architecture

Video/Audio PlaybackVideo/Audio Recording

2.5G Data Packet Mobile

Video/Audio Playback

Video/Audio PlaybackVideo/Audio Recording

Video/Audio Playback

2G Data Packet Mobile

Desktop PC

Streaming Server

Pocket PC

T1-T3 Pipe

Wireless Packet Data Connection

Asynchronous Serial Data Access

Bandwidth Problem

“Bandwidth is like money and sex -only too much seems to be enough.”

Arnold Penzias, former chief scientist of Bell Labs

Generations of WAN Air Interfaces Based on Access Technologies

1G: FDMA (Frequency Division Multiple Access)

1980s - each caller has a dedicated frequency channel (3 callers use 3 channels)

2G: TDMA (Time Division Multiple Access) and GSM (Groupe Speciale Mobile)

1990s - callers timeshare a frequency channel (9 callers use 3 channels)

3G: CDMA (Code Division Multiple Access) and WCDMA (Wide Code Division Multiple Access) 1990s - callers use a shorter bandwidth 2000s - “spread spectrum”. Each code is spread, randomly broken

down and mixed (14 callers use the full bandwidth of 1 channel)

Data Services

2,000

0

32

64

9.6

128

144

384

1G 2G 3G

VoiceVoice

Text MessagingText Messaging

Video StreamingVideo Streaming

Still Still ImagingImaging

Audio StreamingAudio Streaming

Da

ta T

ran

sm

iss

ion

Sp

ee

d -

k b

ps

ElectronicNewspaper

RemoteMedical Service(Medical image)

Video Conference(High quality)

Telephone (Voice)

Voice Mail

E-MailFax

ElectronicPublishing

Karaoke

Video Conference(Lower quality)

JPEG Still Photos

Mobile Radio

Video Surveillance,Video Mail, Travel

Image

Audio

Voice-driven Web PagesStreaming Audio

DataWeather, Traffic, News,Sports, Stock updates

Mobile TV

E-Commerce

Video on Demand:Sports, News Weather

M-Commerce Applications

Transaction Management Digital Content Delivery Telemetry Services Searching for Killer Applications!

Transaction Management

On-line shopping tailored to mobile phones and PDAs on-line catalogs shopping carts back office functions

Initiate and pay for purchases and services

Micro-transactions - subway fees, digital cash

Digital Content Delivery

Information browsing weather transit schedules sport scores ticket availability market prices

Downloading entertainment products Transferring software, high-resolution

images, and full-motion video Innovative video applications

Telemetry Services

Wide range of new applications Transmission of receipt of status,

sensing, and measurement information Communication with various devices

from homes, offices, or in the field Activation of remote recording devices

or service systems

AT&T Wireless Welcome to mlife

Find breaking news,

flight information,entertainment..

Get the latest weather forecasts

Get the business andinvestments news

Get the latest weather forecasts

Get the business andinvestments news

Future of Wireless Technology

Mobile networks have already begun the migration to IP-based networks IP as the routing protocol

4G, New spectrum, and Emerging wireless air interfaces (very high bandwidth 10 Mbps+) It may entirely be IP-based and packet-switched

Increasing usage of wireless spectrum On average, the number of channels has doubled

every 30 months since 1985 (Cooper’s law)

Wireless Multimedia Challenges

• Adaptive Decoding - Optimizing rich digital media for mobile information devices with limited processing power, limited battery life and varying display sizes

• Error Resilience - Delivering rich digital media over wireless networks that have high error rates and low and varying transmission speeds

• Network Access - Delivering rich digital media without adversely affecting the delivery of voice and data services

• Negotiable QoS for IP multimedia sessions as well as for individual media components

Components of a Wireless Video System

VideoEncoder

VideoDecoder Depacketizer

Packetizer

Demodulator

Modulator

ChannelDecoder

ChannelEncoder

WirelessChannel

InputVideo

OutputVideo

Transport + Network Layer

Tradeoff: Throughput, Reliability, Delay

Source and Channel CodingTrade-off

Classic goal of source coding Achieve the lowest possible distortion for a given

target bit rate Classic goal of channel coding

Deliver reliable information at a rate that is as close as possible to the channel capacity

Shannon’s separation principle: It is possible to independently consider source and

channel coding without loss in performance The separation principle applies only to point-to-point

communications and it is not valid for multiuser or broadcast scenarios

Pragmatic Approach

Keep the source coder and channel coder separate, but optimize their parameters jointly Key problem in this optimization is the bit allocation

between the source and channel coder Joint source-channel coding schemes

In the infancy today Exploit the redundancy in the source signal for

channel decoding (Source-controlled channel decoding)

Designing the source codec for a given channel characteristic (Channel-optimized source coding)

Characteristics of a Wireless Video System

The capacity of wireless channel is limited by the available bandwidth of the radio spectrum and various types of noise and interference

The wireless channel is the weakest link of multimedia networks – mobility causes fading and error bursts

Resulting transmission errors require error control techniques (such as FEC - forward error control and ARQ – automatic repeat request)

The Case for Scalable Video Coding

In emerging wireless applications, multimedia data will be streamed: over various access networks (GPRS, UMTS,

WLANs, etc.) to a variety of devices (PCs, TVs, PDAs, cellular

phones, etc.) The transmission of multimedia data need to

cope with unpredictable bandwidth variations: due to heterogeneous access technologies of

receivers (3G, 802.11a, etc.) or due to dynamic changes of network conditions

(interference, etc.)

Scalable Video Coding Techniques

Scalable video coding methods can adapt in real time to the bandwidth variations over heterogeneous networks and to the terminal capabilities while using the same pre-encoded system.

Scalable video coding uses multiple bit streams – layered video coding

For example, in a two-layer coding, the codec generates two bit streams: Base layer – the most vital video information Enhancement layer – the residual information to enhance the

quality of the base layer image This form of two-layer coding is known as SNR scalability

Scalability Techniques

Data partitioning SNR scalability Spatial scalability Temporal scalability Hybrid scalability

Data Partitioning Data partitioning is used when two channels

are available for transmission (it is not true scalable coding)

Divides the bitstream of a single layer into two parts, or layers.

Single layerencoder

DataPartitioner

Multiplexer

Video inBase-layerbitstream

Enhancement-layer bitstream

Outputbitstream

Block DiagramTwo-Layer SNR Scalable Coder

Base layerEncoder(MPEG 1)

Multiplexer

Base layerDecoder(MPEG 1)

Enhancement layerEncoder(MPEG 2)

Video in

+

-

Base layerbitstream

Enhancementlayer bitstream

Outputbitstream

8

8

8

8

8

8

8

8

8

8

8

8

8

8

8

8

Frames 0 1 2 3 4 5 6 7

8X8X8videocube

Adaptive Video Coder Based on 3D-DCT

Original video cube 8x8x8 3D Discrete Cosine Transform

F u v w C u C v C w f x y z

x u y v z w

zyx

( , , ) , ,

cos cos cos

0

7

0

7

0

7

2 1

16

2 1

16

2 1

16

Motion Analysis for Various Blocks

Partition image into NxN inspection areas

Examine each area for motion content based on Normalized

Pixel Difference (NPD) between frames 1 and 8

Three motion types defined: No Motion

Low Motion, and

High Motion

3D-DCT block size adapts based on determined motion content

Example of a Video Hallway Clip

8 Frames of luminance

(Y) component Inspection area size =>

16x16 Inspection areas used

to determine NPD thresholds

Video Example, 4:2:0Original Cr=120

Cr=190 Cr=408

Architecture of 3D-DCT Adaptive Encoder

Forward3D-DCT

MotionAnalyzer

Quantization Tables

Huffman Tables

Quantizer HuffmanEncoder

Inputvideo

Compressedvideo

Videocube

Videocube

Selectionof cube size

Level ofmotion

Selectionof Q tables

..

Figure 2. The architecture of the adaptive 3D-DCT encoder.

Example of a Scalable CodingAdaptive 3D-DCT Coder

Original

Adaptive 3D-DCT Coder

Layer 1: Cr=164 (in vehicles, 144 Kbps)

Adaptive 3D-DCT Coder

Adding Enhancement Layer 2: Cr=96(For pedestrians, 384 Kbps)

Adaptive 3D-DCT Coder

Adding Enhancement Layer 3: Cr=54(for indoor use, 2 Mbps)

Channel Coding and Error ControlEffects of Transmission Errors

Error-free frame

Example 2: Corrupted group number causing a GOB misplacement

Example 1: The extra insertion bit causing the loss of the first GOB

Example 3: Corruption of the group quantizer parameter that resulted in employing the wrong quantizer in decoder

Channel Coding and Error Control

Trade-off between throughput, reliability, and delay

Forward Error Correction (FEC) Automatic Repeat Request (ARQ) Error Resilience Techniques for Low Bit

Rate Video Techniques that reduce the amount of

introduced errors for a given error event (Resynchronization)

Techniques that limit interframe error propagation

Recovery From Packet Loss FEC scheme

• “Piggyback lower quality stream” • Send lower resolutionaudio stream as theredundant information• For example, nominal stream PCM at 64 kbpsand redundant streamGSM at 13 kbps.• Sender creates packet by taking the nth chunk from nominal stream and appending to it the (n-1)st chunk from redundant stream.

• Whenever there is non-consecutive loss, thereceiver can conceal the loss. • Only two packets need to be received before playback• Can also append (n-1)st and (n-2)nd low-bit ratechunk

Joint Source Coding and Transmission Power Management

Goal: to limit the amount of distortion in the received video sequence, while minimizing transmission energy

Combines: Error resilience and concealment techniques at

the source coding level, and Transmission power management at the

physical layer Optimization problem: Minimizing the energy

required to transmit video under distortion and delay constraints

Joint Source Coding andTransmission Power Management

VideoEncoder

WirelessChannel

Controller

DemodulatorChannelDecoder

VideoDecoder

ChannelEncoder

Modulator

DecoderConcealment Strategy

Channel StateInformation

Video in

Video out

Goal: to limit the amount of distortion in the received video sequence, while minimizing transmission energy

Control powerControl codingparameters

Transmission Energy

Total energy to transmit all the packets in a frame:

The algorithm calculates the power needed to achieve the desired probability of loss

kK

k

kK

k

k PR

BEEtot

11

Controlling the Bit Rate

Most video codecs use variable-length coding techniques

Most existing mobile radio systems transmit at a fixed bit rate

Goal: Constant signaling rate leading to a different constant bit rate for each modulation scheme

Rate Control Techniques - determine the sending rate of video traffic based on the estimated bandwidth in the network Source-Based Rate Control Receiver-Based Rate Control Hybrid-Based Rate Control

Rate Shaping Techniques

Techniques that adapt the rate of pre-compressed video stream to a target rate constraint

Rate shaper is an interface (or filter) between the compression layer and the network transport layer

CompressionLayer

RateShaper

NetworkTransport

Layer

Video in

Variable rate

Constant bit rate

Rate Shapers

Codec filters Frame-dropping filters (dropping B,P, or I

frames) Layer-dropping filters (in scalable video

coding schemes) Frequency filters (discard DCT coefficients of

the highest frequency) Requantization filters (reqauntizes the DCT

coefficients with a larger quantizers, resulting in rate reduction)

Multimedia Content Security

Access control in applications such as video-on-demand and videoconferencing, so only selected users can access the data

Established encryption algorithms (DES or AES) are very complicated and involve large number of computations.

Software implementations of these schemes are not fast enough to process the large amount of multimedia data

Hardware implementations require additional costs to both data generation and receivers

General ArchitectureSelective Encryption System

Colorspaceconvertor

FDCT Quantization Entropyencoder

I frame

RGB --> YUV

Compressed data10001110000...

Colorspaceconvertor

FDCT

Entropyencoder

RGB --> YUV

P/B frame

+

Compressed data00111100101...

+

-

Errorterms

MotionestimatorReference

frames

Example of Video EncryptionMPEG Encoder

Secret KeySelective encryption algorithm

That operates on sign bits of DC coefficients

Secret KeyRandomly change the sign

bits of motion vectors

Secret KeyPermutation of

the Huffman codeword list

Example: Encrypting Frames of a MPEG-4 Video Sequence

Original frame Encrypted VLC only Encrypted FLC only Encrypted VLC and FLC

Virtues of the Virtual Workplace

Universal access to information, applications, services, processes, and

people, from any device, over any network connection - wired, wireless,

or Web

Virtual Workplace Video Clip

Wireless Internet and Web Wireless appliances Security Redundant systems Wireless applications:

videoconferencing

The Portable Office

Take the office with you, wherever you go

Secure Authentication

High Security Authentication, including Bio-Authentication

IntegratedMessaging and Communication

Integrated messaging (eg. voice, chat), voice to text, with intelligent alerting

Information Portability

Access information over any connection – wired or wireless, regardless of form factor

Business Collaboration

Collaborative capabilities allow on-line information sharing and communication

Business Continuity

Resilient to network interruptions

Further Readings

Hanzo, Cherriman, and Streit, “Wireless Video Communications,” IEEE Press, 2001.

IEEE Trans. On Circuits and Systems for Video Technology, Special Issue on Wireless Video, June 2002.

Sun and Reibman, “Compressed Video over Networks,” Marcel Dekker, 2001

Wang, Ostermann, and Zhang, “Video Processing and Communications, Prentice Hall, 2002.

Furht and Ilyas, “Wireless Internet Handbook,” CRC Press, 2003.