40
nus.soc.cs5248 Ooi Wei Tsang 1 Intermedia Synchronization Ooi Wei Tsang

Nus.soc.cs5248 Ooi Wei Tsang 1 Intermedia Synchronization Ooi Wei Tsang

Embed Size (px)

Citation preview

Page 1: Nus.soc.cs5248 Ooi Wei Tsang 1 Intermedia Synchronization Ooi Wei Tsang

nus.soc.cs5248Ooi Wei Tsang

1

Intermedia Synchronization

Ooi Wei Tsang

Page 2: Nus.soc.cs5248 Ooi Wei Tsang 1 Intermedia Synchronization Ooi Wei Tsang

2

Overview

• Why we need to sync?• How close must we sync?• How to sync?• How to measure sync?• Applications

Page 3: Nus.soc.cs5248 Ooi Wei Tsang 1 Intermedia Synchronization Ooi Wei Tsang

3

Synchronization

Intrastream Interstream

Local playback Networked

Page 4: Nus.soc.cs5248 Ooi Wei Tsang 1 Intermedia Synchronization Ooi Wei Tsang

4

Problem

• Ensure that audio/video captured at the same instance at the source, are displayed at the same instance at the receivers.

Page 5: Nus.soc.cs5248 Ooi Wei Tsang 1 Intermedia Synchronization Ooi Wei Tsang

nus.soc.cs5248Ooi Wei Tsang

5

Why we need Synchronization?

Page 6: Nus.soc.cs5248 Ooi Wei Tsang 1 Intermedia Synchronization Ooi Wei Tsang

6

Why not Pre-sync?

V A V A

Sender Receiver

Page 7: Nus.soc.cs5248 Ooi Wei Tsang 1 Intermedia Synchronization Ooi Wei Tsang

7

Seperating A/V Flows

Sender Receiver

V

A

V

ASender Receiver

• Can receive audio without video• Can prioritize audio over video

Page 8: Nus.soc.cs5248 Ooi Wei Tsang 1 Intermedia Synchronization Ooi Wei Tsang

8

May Come from Different Sources

• UNLV Access Grid

Sender Receiver

V

A

V

ASender Receiver

Page 9: Nus.soc.cs5248 Ooi Wei Tsang 1 Intermedia Synchronization Ooi Wei Tsang

9

or Displayed at Different Sources

Sender Receiver

V

ASender Receiver

A

V

Page 10: Nus.soc.cs5248 Ooi Wei Tsang 1 Intermedia Synchronization Ooi Wei Tsang

10

More Complications

• Different decoding time for video and audio

• Network delay and jitter• Different clocks

Page 11: Nus.soc.cs5248 Ooi Wei Tsang 1 Intermedia Synchronization Ooi Wei Tsang

11

Bottom Line

• Must synchronize at the receiver

Page 12: Nus.soc.cs5248 Ooi Wei Tsang 1 Intermedia Synchronization Ooi Wei Tsang

nus.soc.cs5248Ooi Wei Tsang

12

Synchronization Requirements

Page 13: Nus.soc.cs5248 Ooi Wei Tsang 1 Intermedia Synchronization Ooi Wei Tsang

13

Reference

• Human Perception of Jitter and Media Synchronization

by Ralf Steinmetz IEEE JSAC 14(1) 1996

Page 14: Nus.soc.cs5248 Ooi Wei Tsang 1 Intermedia Synchronization Ooi Wei Tsang

14

Results “Badness”

A/V offset-80ms +80ms

Page 15: Nus.soc.cs5248 Ooi Wei Tsang 1 Intermedia Synchronization Ooi Wei Tsang

15

What matters

• Video ahead of audio vs. audio ahead of video

• Size of head• Occupancy

Page 16: Nus.soc.cs5248 Ooi Wei Tsang 1 Intermedia Synchronization Ooi Wei Tsang

16

What does not matter

• Language• Speaking or Singing• Hobby• Speed

Page 17: Nus.soc.cs5248 Ooi Wei Tsang 1 Intermedia Synchronization Ooi Wei Tsang

17

Conclusion

• -80ms to +80ms acceptable• -160ms to +240ms noticable

but not annoying

• - audio > video• + video > audio

Page 18: Nus.soc.cs5248 Ooi Wei Tsang 1 Intermedia Synchronization Ooi Wei Tsang

nus.soc.cs5248Ooi Wei Tsang

18

How to Synchronize?

Page 19: Nus.soc.cs5248 Ooi Wei Tsang 1 Intermedia Synchronization Ooi Wei Tsang

19

Receiver needs

• Exact time a sample is generated

• Each packet needs a timestamp

Page 20: Nus.soc.cs5248 Ooi Wei Tsang 1 Intermedia Synchronization Ooi Wei Tsang

20

RTP Protocol

Media TimestampRTPHdr

Page 21: Nus.soc.cs5248 Ooi Wei Tsang 1 Intermedia Synchronization Ooi Wei Tsang

21

But..

• Media timestamp has a random offset

Page 22: Nus.soc.cs5248 Ooi Wei Tsang 1 Intermedia Synchronization Ooi Wei Tsang

22

RTCP Protocol

:

Media Timestamp

NTP Timestamp

RTCPHdr

:

Page 23: Nus.soc.cs5248 Ooi Wei Tsang 1 Intermedia Synchronization Ooi Wei Tsang

23

NTP

• Network Time Protocol• Synchronize clock up to few

milliseconds

Page 24: Nus.soc.cs5248 Ooi Wei Tsang 1 Intermedia Synchronization Ooi Wei Tsang

24

Sender Audio

Sender Video

Receiver Audio

Receiver Video

Audio RTCP

Video RTCP

Audio RTP

Video RTP

Page 25: Nus.soc.cs5248 Ooi Wei Tsang 1 Intermedia Synchronization Ooi Wei Tsang

25

Sender Audio

Sender Video

Receiver Audio

Receiver Video

Audio RTCP

Video RTCP

Audio RTP

Video RTP

Synchronization Delay

Page 26: Nus.soc.cs5248 Ooi Wei Tsang 1 Intermedia Synchronization Ooi Wei Tsang

26

Multiple Receivers

• Everyone must have the same synchronization delay

Page 27: Nus.soc.cs5248 Ooi Wei Tsang 1 Intermedia Synchronization Ooi Wei Tsang

27

How to set Sync Delay?• Receivers tell each other their

network delay• Synchronization Delay =

max {Network Delays} + Buffer

Page 28: Nus.soc.cs5248 Ooi Wei Tsang 1 Intermedia Synchronization Ooi Wei Tsang

28

Buffer Control

Page 29: Nus.soc.cs5248 Ooi Wei Tsang 1 Intermedia Synchronization Ooi Wei Tsang

29

No Delay No Jitter

Page 30: Nus.soc.cs5248 Ooi Wei Tsang 1 Intermedia Synchronization Ooi Wei Tsang

30

Delay, No Jitter

Page 31: Nus.soc.cs5248 Ooi Wei Tsang 1 Intermedia Synchronization Ooi Wei Tsang

31

Delay, Jitter

Page 32: Nus.soc.cs5248 Ooi Wei Tsang 1 Intermedia Synchronization Ooi Wei Tsang

32

If Buffer too Full/Empty• Send message “Play faster!” or

“Play slower!” to other receivers

Page 33: Nus.soc.cs5248 Ooi Wei Tsang 1 Intermedia Synchronization Ooi Wei Tsang

33

Must Synchronize!

• Send message “Play faster at time T!” or “Play slower at time T!” to other receivers

Page 34: Nus.soc.cs5248 Ooi Wei Tsang 1 Intermedia Synchronization Ooi Wei Tsang

nus.soc.cs5248Ooi Wei Tsang

34

How do we Measure Synchronization?

Page 35: Nus.soc.cs5248 Ooi Wei Tsang 1 Intermedia Synchronization Ooi Wei Tsang

35

Reference

• A Method and Apparatus for Measuring Media Synchronization

• by Brian Smhmidt etc.

• NOSSDAV 1995

Page 36: Nus.soc.cs5248 Ooi Wei Tsang 1 Intermedia Synchronization Ooi Wei Tsang

36

System under Test

Measurement System

Generator

Page 37: Nus.soc.cs5248 Ooi Wei Tsang 1 Intermedia Synchronization Ooi Wei Tsang

nus.soc.cs5248Ooi Wei Tsang

37

Beyond Video Conferencing

Page 38: Nus.soc.cs5248 Ooi Wei Tsang 1 Intermedia Synchronization Ooi Wei Tsang

38

HQ Video and Audio

• Multiple Channels• Multiple Displays

Page 39: Nus.soc.cs5248 Ooi Wei Tsang 1 Intermedia Synchronization Ooi Wei Tsang

39

Network Musical Performance

BOS

BOS

SF

BOS

SF

ACM Multimedia ‘94

Page 40: Nus.soc.cs5248 Ooi Wei Tsang 1 Intermedia Synchronization Ooi Wei Tsang

40

NASDAQ

• Synchronize display of stock quotes