A Semantic-based Middleware for Multimedia Collaborative Applications

A Semantic-based Middleware for Multimedia Collaborative Applications

Agustín J. GonzálezAdvisor: Dr. Hussein Abdel-

WahabDoctoral Dissertation Defense

Old Dominion University

February 2000

2

Outline

Introduction

MiddlewareObjectives

Extension of operating systems network services

Stream synchronization

Floor control framework

Protocol for dynamic image transmission

Experimental results

ConclusionsQuestions

3

Introduction • Large-scale Multimedia Applications

* Desktop computer performance increase

* Internet growth in bandwidth and # of hosts

• A challenging class of applications* Processing power & bandwidth

* Scalability

* Heterogeneity (Ethernet/modem, WinNT/Solaris, MPEG/H263)

* Timely data delivery

• Traditional services* Network layer: UDP & TCP (real time was not a concern)

* Operating systems: Abstractions are not adequate for multimedia.» Example: Real time is not well supported.

• Gap between multimedia requirements and system services

more

more

more Outline

4

Multimedia Resource Requirements

Bandwidth

Processing

Quality

Q1Q2

Q3

CPU performance

5

Source: Dr. David Patterson University of Virginia Distinguished Lecture Series, May 19,1998. http://www.cs.berkeley.edu/~pattrsn/talks/Stanford.pdf

Processor Performance Increase

3

Processor Performance Increase

µProc60%/yr.

1

10

100

10001980

1981

1983

1984

1985

1986

1987

1988

1989

1990

1991

1992

1993

1994

1995

1996

1997

1998

1999

2000

CPU

1982

Performance

Time

Effect on MM

6


Bandwidth

Processing

MultimediaRequirements

Quality

Q1Q2

Q3

Bandwidth

7


Bandwidth

Processing

MultimediaRequirements

Quality

Q1Q2

Q3

High processing + high bandwidth + Others

Introduction

8

Internet Growth

56 M

Introduction

9

Traditional requirementsTrad. Services

Trad. Application

Gap between system services and application requirements

Developers need to fill this gap by implementing common services for multimedia applications.

Real-timeScalabilityHeterogeneity

Multimedia applications

Middleware

outline

10

Objective “Our main objective is to investigate and propose

heterogeneous, scalable, reliable, flexible, and reusable solutions and enhancements to common needs in developing multimedia collaborative applications.”

Needs we addressed:

* Extension of network services

* Media synchronization

* Floor control

* Data sharing Outline

11

Extension of Network Services

• New services* Asynchronous data reception

* Quality of service monitoring

* Transmission traffic rate control

• New convenient facilities* Unified Multicast/Unicast API

* Efficient buffer management for Application Data Unit more

more

more

more

more

Outline

12

Asynchronous data reception

Network packet arrivals

packet

Thread watching

packet“application function”

Ext. Net. Srvs

Event-driven model

13

Quality of services monitoring

Traffic monitoring

packet packet“application function”

3,67 Bps789

2,15 Bps789

packet packet“application function”

Ext. Net. Srvs

ti-3 ti

si

Time

Packet Size

ti-1ti-2 t

Window k=3

ki

i

kijj

k tt

s

tSTTR

1)(

14

Transmission traffic rate control

Ext. Net. Srvs

ti-2 ti-1

si

Time

Packet Size

t`i ti

Send() call Actual Tx time

packet

i

2,15 Bps789

packet“application function”

Traffic Limit2,200 Bps

i-1

15

Unified Multicast/Unicast API

• Datagram transmission* A send to a machine or a multicast group does not make a difference.

• Datagram reception* if the given IP address is a multicast, join group.

* if address is not multicast, do not bind (I’m client).

port,addressUnicast

addressMulticast

Ext. Net. Srvs

16

Efficient buffer management for Application Data Unit

A B

AH BH

Tx Rx

BA

BHAH

* Goal: to prevent payload movements in memory

* Sender modules create an output buffer that can hold following “headers” and “tails” .

* Receiver module needs to allocate worst case buffer size.

Ext. Net. Srvs

17

Stream Synchronization• Problem: processing times and network delays are not

deterministic.

• The objective of synchronization is to faithfully reconstruct the temporal relationship between events (“pieces of data”).

• Main characteristics of our solution:* It depends on one-way messages only

» No need of feedback

* It only requires sender’s and receivers’ clock rates to be constant. » These clocks might be off.

» These clocks might even have different rates of change.

» No need of globally synchronized clocks

* It supports policies to handle late packets and delay adjustments.

Details

18

Stream Synchronization (details)

• Time model• Intra-stream synchronization• Inter-stream synchronization• Clock skew estimation and removal

more

more

more

more

Outline

19

Time ModelVirtual Observer

Network

Original Phenomenon

Capture

TimestampingEqualization

Buffer

Remote view

Playout

tiai

ciqi pi

Virtual Observer’smachine time

Receivertime

Absolute time

ci

i

pi

ti

TransmissionPath

Sync

20

ci

pi

ai

Receiver

Sender

ti-1 ti

ai-1 ai

cici-1

qi-1

qi

pi-1 pi

Perception

Timestamping

Arrival

Equalization

Playout

ci-1

pi-1

Intra-stream Synchronization (model)“Seen” by virtual observer

“Seen” by receiver

ctecp ii Synchronization condition:

Virtual delay

cte

Solution

Tradeoff:% late packets

Total delay

Adaptive compromise

21

Intra-stream Synchronization (solution)

Adjust “virtual delay” to achieve a given % of late packets

otherwise 0

arrival latefor 1*11 ii ll

NASA MBone 1% late packets

sync

more

Slow start1 min !

Estimator for % of late packets:

22

Fast start refinement

Less than 5 s ! sync

23

Inter-stream Synchronization

Global synchronization model v/s Differentiated synchronization model

Synchronizes streams coming from one virtual observer

Actual network delay Global Sync Model Differentiated

Synchronizes streams coming from anywhere with worst case delay

Solution

24

Inter-stream Synchronization (solution)

AudioSync

DataSyncDataSync

VideoSyncInter-stream

coordinatorVirtual delay

Max. virtual delay

sync

25

Clock skew estimation and removal

Goal: Remove differences in clock frequencies

Correction

The algorithm adjusts a straight line as new packets arrive

sync

Before After

26

Active Resource

Everywhere Resource

Inactive resourceNode (participant)

Localized Resource

“Audio” “Shared tool”

Lightweight Framework for Floor Control

• Problem: How to manage exclusive resources in large-scale multimedia applications?

• We recognize two cases:

ResourceUserUser Resource Communication Channel

1:1

:

nn

Solution

27

Floor Control (Solution)

* We propose two protocols for floor control, one per architecture.

* Features: lightweight, scalable, robust

(1) Request(2) Granted

(2) Taken

(1) Request

(3) Granted

Preemptive

Coordinator Floor holder Participant

(1) Request(2) Release

(4) Granted

(3) Taken or Released

TCP connectionHeartbeat

Delayedpreemptive

* The coordinator is stationary for localized resources.

* The coordinator migrates with floor for everywhere resources

Localized res.

28

Architecture for localized resources

Requester Control

Requester

RequesterListener

Coordinator

RequestReleased

HolderRefreshTakenGrantedRelease

GrantedRelease

RequestReleased

Withdrew

GrantedTakenRelease

NewHolder

getResourceInfo

Object implementing interface xObject related with floor architectureMain floor architecture objectsOptional Object

x

*

NewHolderListener*

ResourceInfo *

ResourceUserListener*

1-1 reliable remote invocation1-N unreliable remote invocationLocal invocation

Monitor/LogListener

*

log

Monitor/LogListener

*

log

HeartbeatHeartbeat

Policy

RequestNotiWithdrawalNotiHolderTimeoutSelectNextHolder

Everywhere res.

Outline

29

Architecture for everywhere resources

Requester Control

Requester

RequesterListener

RequestReleased

HolderRefreshTakenGrantedRelease

GrantedRelease

RequestWithdrew

Granted

getResourceInfo

ResourceListener*

1-1 Temporary connection for reliable remote invocation1-N unreliable remote invocationLocal invocation

Monitor/LogListener*

log

Monitor/LogListener*

log

Heartbeat

RequestNotiWithdrawalNotiHolderTimeoutSelectNextHolder

Coordinator

Policy

ActivateRequestWithdrewReleased

GrantedTakenRelease

Coordinator

Other Objects(Same Architecture as above)

To all Requesters

Before Granted

After Granted

Heartbeat

Outline

30

Protocol for Dynamic Image Transmission

• Problem: In addition to audio and video, multimedia sessions needs a component to convey the main idea of discussion.

• Traditional solutions: * Use video (size limitation & high bandwidth)

* Shared tools: XTV, co-browsers, VNC,.. (do not scale well)

• Our solution:

* Video-like protocol tuned to send dynamic images

Solution

31

Protocol for Dynamic Image Transmission

• Sender:* Temporal redundancy removal

» Sample image at regular period

» Divide image in tiles

» Process only changed tiles

* Spatial redundancy removal» compress and send changed tiles

• Receiver:» Receive data unit

» Decompress tile

» Update tile in image

Losses?

32

Overcoming losses

• Each tile is retransmitted after a random time

• This also accommodates late comers

Performance Study* How to select a tile compression technique? (JPEG, GIF, PNG?)

* Is there a “best” tile size? What does it depend on?

* How often to sample the image?

* How can two tiles be compared efficiently?

* Maximum data transmission rate? What does it depend on?

Outline

33

Implementation and Experimental Results

• Implementation:* Network support implemented

* Synchronization: implemented and used with real RTP data in off-line analysis

* Floor control: partially implemented for localized resources

* Image protocol implemented

• Putting everything together: Odust* A prototypical sharing tool built on top of the middleware.

It uses:* Network support, floor control, dynamic image protocol, other

application specific modules.Odust

34

MulticastNetwork

User: EduardoOS: WinNT

User: RodrigoOS: WinNT

User: AgustínOS: Solaris

User: CeciliaOS: Solaris

Odust Description

Architecture

35

MulticastNetwork





Odust Description: Cecilia’s view

Architecture

UNIX

36

MulticastNetwork





Odust Description: Rodrigo’s view

Architecture

WinNT

37

MulticastNetwork





Odust Description: Eduardo’s view

Architecture

WinNT

38

MulticastNetwork





Odust Description: Agustín’s view

Architecture

UNIX

39

Odust Architecture

Capture andDynamic Compound

Image ProtocolSender

Dynamic CompoundImage Protocol

Receiver and Display

EventInjector Event

Capture

TokenManager

TokenClient

Application A

WinNT

NativeLibrary

Application A Receiver

Sharing Tool Receiver

Application A Sender

Sharing Tool Sender

ApplicationB’s View

ApplicationA’s View

JDesktop

Java VM

Mx Dx

Temporary TCPMulticast Method Invocation

c

a

d

e

g

b

f

h

i

j

k

l

mn

Outline

40

Conclusion• We observed the convenience of a middleware

Traditional requirementsTrad. Services

Trad. Application

Real-timeScalabilityHeterogeneity

Multimedia applications

Middleware

• Future work* Add more components

* Continue implementation

* Try new ideas (see thesis)

• It offers:* Multimedia network services

* Synchronization

* Floor control

* Dynamic image transmission

Outline

Documents

A Semantic-based Middleware for Multimedia Collaborative Applications