Ubiquitous Multimedia Computing

1

Ubiquitous Multimedia Computing and Communication:

Challenges and Future Trends

C.-C. Jay Kuo

University of Southern California

2

Content

Introduction Review of embedded system design

– Choice of RISC, VLIW and ASIC– Software or hardware?– Low power design

Several new design issues– Design of multi-format codec– Joint compression/encryption algorithm– Joint R-D-C optimization

Conclusion

3

Mobile Multimedia

Features– Portability

• Pervasive computing• Ubiquitous computing

– Communication capability

• 2G/2.5G/3G• WLAN, Bluetooth, UWB• Overlay networks

– Multimedia capability• A/V capturing• A/V display

– Marketing• Consumer electronics

oriented• Rather than PC-oriented

Embedded processors

http://www.nokia.com/cda1/0,1080,825,00.html

4

Main Challenges

00111010001111010110100

wireless channels

power management

security and rights issue

Poor Channel Conditions:Low SNRMulti-path FadingDoppler Fading

5

Embedded Computing Systems

Two-module solution– Communication module

• RF-Modem• Baseband processing-channel decoding, despreading,

etc.

– Multimedia module• Codec, DRM, etc.

Embedded Media Processor ArchitectureCPU

DSP

Image Co-Processor

On-Chip Memory

CPUImage Co-Processor

On-Chip Memory

6

Standards of Mobile Communications

Resource: http://plus.ric.co.jp/wireless/wl003_01_0410.html

Eff

ecti

ve D

ista

nce

Data Rate

7

Trends of Mobile Communications (2)

Orthogonal Frequency Division Modulation Wireless Systems

– Wi-Fi (IEEE 802.11a/g/n)

– WiMax (IEEE 802.16)

– DVB-H Broadcasting Systems

http://www.wi-fi.com/

8

Trends of Mobile Multimedia Applications

Two-way communication– Video telephony– 3G provides the solution

Web access– Short video clips for on-demand service

• Download or streaming

– Video gaming– Something better than 3G is desirable

Live video broadcasting– DVB-H or DMB

• Vision: Bring live TV programs to your cellular phone

9

Content




Conclusion

10

Tradeoff between Different Architectures

Dedicated coprocessorsASIC

Specific processorsSIMD, VLIW

General processorsCISC or RISC

Power consumption, chip area

Fle

xibi

lity

11

CISC versus RISC

Main difference– CISC: multi-clock complex instructions

• Minimizing the no. of instructions per program, while sacrificing the no. of cycles per instruction

• Example: Intel Pentium

– RISC: one-clock reduced instructions• Go to the opposite direction of CISC• Example: ARM

Advantages of RISC design– Lower power (suitable for mobile applications)– Lower chip area (for lower cost)

12

Single Instruction Multiple Data (SIMD)

Why is SIMD?- Multimedia data’s low-precision

- 8-bit pixels for image/video application- 16-bit samples for audio application Challenges: representation, storage and processing

- Multimedia algorithm’s inherit data parallelism- Add, subtract, and simple forms of multiplication and

division are common operations

First developed by UIUC – Used as imaging processing engine (CM series)

in early days

Popular engine: Intel MMX, TI iMX

13

Examples of Media Processors

Texas Instruments (TI)– DSC-25, DM-270, DM-320– OMAP for cellular phone– C64xx series

Intel– XScale Processor

Trimedia: TM1300– Speech/Image/Video– Somehow, not well received

Equator media processor

14

Integrated Solution to Video Coding

RISC (ARM processor)– Irregularity operations – Sequential computation (no inherent parallelism)– e.g. streaming parsing & entropy decoding

SIMD (Image co-processor)– Semi-regular operations– e.g. Intra prediction, quantization, motion

prediction/compensation

ASIC (dedicated coprocessors)– Regular yet computational intensive operations– To save SOC area and power – e.g. DCT, loop filtering, half-pel interpolation etc.

15

Content




Conclusion

16

Paradigm shift from hardware to software– Faster algorithm adoption– Easier for add-on functionalities

– Reuse of platform independent codes

Increased use of programmable processor core in system-on-chip (SoC)– Easier use of high level languages such

as C, C++

Advantages of Software Solution

17

Relative importance of embedded software development tools

Compiler for Embedded Processors

Need more sophisticated compiler to generate codes to meet stringent embedded application requirements

Use of C code versus assembly language by design team at

Northern Telecom area

18

Content




Conclusion

19

Why Multi-Codec Design?

Joint S/W & H/W Decoder

S/W ComponentsEntropy decoding

Selected mode decoding

H/W ComponentsInverse transform

Inverse quantizationMotion search

De-blocking filter

MPEG-2Stream

H.264/MPEG-4 AVC Stream

VC-1 Stream

OutputVideo/Audio

20

Flexible Design (1): Processor Consideration

Bitstream parsing– Sequential processing– Mutimedia standards define different bitstream

formats

Solution– Programmable architecture

• Pros: flexible, easy to be upgraded to compliance multiple standards

• Cons: not cost-effective on power and area

– Dedicated architecture• Pros: Save power, chip area• Cons: fixed, not easy to be modified

21

Flexible Design (2): Bus Consideration

Bus architecture– Integrating components on multimedia SoC

Multi-format codec is usually a hybrid SoC• RISC, SIMD, ASIC, on-chip memory

– Interfacing with off-chip memory

How to integrate efficiently?– Common bus architecture

• Flexible, accommodate to modules from all parties• Example: AMBA (used in ARM)

– Dedicated bus architecture• High utilization of bus bandwidth, cost-effective

22

Flexible Design (3) ---- A Case Study

“Real-time Audio/Video Decoders for Digital Multimedia Broadcasting”Victor H. S. Ha, Samsung, IWSOC’04

Common busAMBA

High bandwidthDedicated bus

23

Content




Conclusion

24

Confidentiality for Multimedia Data

What separates multimedia data from traditional alpha numeric data?– Large in file size– May require real-time processing (especially for

continuous media)– Portable and mobile applications

25

Approaches to Multimedia Encryption

Signal scrambling– Historical approach– Not compatible with modern multimedia

compression– Fast speed but low security

Total encryption with cryptographic ciphers– Trivial solution– High security but slow speed

Selective encryption– Most popular approach today– Limited in its range of application

Integrating encryption into entropy coding– Complementary to selective encryption– Very fast computation speed

Others

26

Selective Encryption

Select the most important coefficients and then encrypt them with traditional ciphers such as DES

Advantages– Lower complexity– High security level provided by traditional

cryptology– Less error correction coding redundancy– Compatible with existing software and hardware

modules

MediaCompression

System

Coefficient

Selection

Cryptographic

Cipher

ErrorCorrection

Coding

DigitizedAudiovisual

data

Coefficients SelectedCoefficients

Non-selectedCoefficients

Transmission channel or storage media

27

Example: Selective Encryption for G.723.1 Speech Coder

ITU-T Recommendation G.723.1– A popular low bit rate speech codec

Based on the human voice generation model– Vocoder– Decoder synthesizes speech using the model

LSPDecoder

PitchDecoder

ExcitationDecoder

+Synthesis

FilterPitch

Postfilter

LSPInterpolator

FormantPostfilter

Gain Scaling

Unit

LSP codebook indices

Lag of pitch predictorsGain vectors

Fixed codebook gainsand others

Vocal CordExcitation signal

generation

Vocal TractLinear filter

28

Effect of Encrypting Different Coefficients on Speech Intelligibility

Decoder Coefficient Type Size

(bits per frame)

LSP Decoder LSP codebook indices 24

lag of pitch predictors 14

Differential adaptive codebook lag 4Pitch Decoder

Pitch gain vectors 30

fixed codebook gains 18

Pulse positions index 76 / 48

Pulse sign index 22 / 16Excitation Decoder

Grid index 4

Original Speech

29

Effective Selective Encryption Scheme

Encrypt the most significant bits of all important coefficients given below

The best restoration approach– replace encrypted bits with the average value of its

rangeCoefficient Type Coefficient Name

LSP codebook indices

LPC_B23, LPC_B22, LPC_B21, LPC_B20,



LPC_B7, LPC_B6

lag of pitch predictorsACL0_B6, ACL0_B5, ACL0_B4,

ACL1_B6, ACL1_B5, ACL1_B4

Pitch gain vectorsGAIN0_B11, GAIN0_B10, GAIN1_B11, GAIN1_B10,

GAIN2_B11, GAIN2_B10, GAIN3_B11, GAIN3_B10

fixed codebook gainsGAIN0_B4, GAIN0_B3, GAIN1_B4, GAIN1_B3,

GAIN2_B4, GAIN2_B3, GAIN3_B4, GAIN3_B3

VAD mode flag VADFLAG_B0

Encrypt 37 bits/frame

roughly 20% of total encryption

30

Randomized Huffman Table Encryption

0

0

0 0

11

1

1 1

A

0 1

B CD E F G

0

1

0 1

10

1

1 0

A

0 1

B CD E F G

0 0

BADCAEFG

Huffman code #0 Huffman code #1

00000000

10011010

100011001010110111101111

110011001110110110111111

isomorphic tree!

31

Multimedia Encryption with Randomized Entropy Coder

Select a good PRBG Select an r-bit random seed s (encryption key) Pseudo-random sequence output from PRBG(s)

becomes the key hoping sequence (KHS)

Entropy

Coder

PRBGs KHS = 011000110 …

Input symbol

1110110001…

1011110

32

Simulation Results

Pentium III CPU, 600MHz, 256MB RAM Video compression standard: H.264 Cipher: the Randomized Huffman Table (RHT)

encryption PRBG: emulated by 128-bit MD5 hash function Test video clip: “foreman”

CIF size: 352 x 288 YUV 4:2:0 format The first 10 frames encrypted using key

0x246CCA6B103C95

33

Encoding Speed

Frame No.

Bit-stream size (no encryption)

Bit-stream size(encryption)

Encoding time (no encryption)

Encoding time (encryption)

Time increase

1

2

3

4

5

6

7

8

9

10

47848 bits

12712

15232

14776

16744

15384

11864

15640

15928

18184

47848 bits

12712

15232

14776

16744

15384

11864

15640

15928

18184

414 ms

338

350

351

358

350

330

356

360

365

415 ms

340

351

354

361

355

334

362

364

369

0.24% 0.59%

0.28%

0.85%

0.85%

1.41%

1.21%

1.68%

1.11%

1.09%

34

Video Clip “Foreman”

35

Standard H.264 Decoding

36

Attacking with Random Seed0x17460FD05B9EDF

37

Content




Conclusion

38

Research Motivation

There is no trivial relationship between quality (R-D performance) and complexity– Does the variable block size motion estimation always

help?• From 16x16 to 2 modes 16x16 and 8x8 (MPEG-4)• From 16x16 to 7 modes 16x16, 8x16, 16x8, 8x8, 4x8, 8x4 and 4x4

– Does the subpel motion search always help?• The subpel interpolation is one of the most time-consuming jobs in

the decoder implementation

– Does the deblocking filter always help?• The deblocking filer is another time-consuming job in the decoder

implementation

– Does the long-term memory always help?• This is probably not to be used in mobile video

The answer:– It is content dependent

39

Decoding-Friendly Encoder Design

The encoder selects the decoding-friendly modes– Stick to the integer motion pel– Stick to the option of no deblocking filter– Stick to the 1-reference frame– Stick to blocks of larger sizesif the other “fancier” choices do not help much

How to cast these in a formal framework– Rate-Distortion-Complexity optimization

40

Content




Conclusion

41

Conclusion (1)

Mobile multimedia communication is the major trend– Convergence of IT, CE, telecom, gaming, etc.

Requirements from consumers– Low power– Low cost– Broadband access– Reliability (quality)– Mobility

Requirements from company executives– Short design cycle, fast turn-around time– Fast adaptation to new markets

42

Conclusion (2)

Requirements from content owners– Digital Rights Management

Technology barriers and R&D opportunities– More flexible architectures

• Example: multi-format video decoder design (MPEG-2,H.264 and VC-1)

– Lightweight encryption algorithms• Examples: selective speech encryption, randomized

Huffman entropy coder

– Decoder-friendly coding methods• Example: Joint R-D-C optimization• Complexity is introduced effectively and in a controlled

fashion

43

Conclusion (3)

Technology barriers and R&D opportunities (Cont’d)– Broadband wireless communication

• More spatial diversity (MIMO) to be exploited– MIMO-OFDM

• New standard activities– 4G & 802.11n

– Cross-layer design• No clear layer boundary as observed in wired

communication systems• Integrated QoS across physical, MAC, transport,

application layers