Real-Time Collaborative Environments Colin Perkins

Real-Time Collaborative Environments

Colin Perkinshttp://csperkins.org/

Cop

yrig

ht ©

200

5 U

nive

rsit

y of

Gla

sgow

Talk Outline

• Real-time Collaborative Environments– Definition and Requirements

– Example: AccessGrid

– History and Current Status

– Underlying Technologies and Standards

– Capture and Analysis of Real-time Sessions

• Research Issues and Future Directions

…from the perspective of a developer of those applications, and of the relevant technical standards

Cop

yrig

ht ©

200

5 U

nive

rsit

y of

Gla

sgow

Original Picture ©1998 UCL

QuickTime™ and aTIFF (Uncompressed) decompressor

are needed to see this picture.

Picture from http://www.accessgrid.org/

Real-Time Collaborative Environments

• Support collaborative work by scientists, industry and others– Both one-to-one and group communication

– Widely distributed participants; heterogeneity

• Provide high-quality audio-visual media

• Provide sense of community and presence– Venues as a rendezvous point; familiar virtual

meeting places

• Integrate with other computing resources– Visualization, shared state & data repositories,

computational resources, applications

• Secure and trusted infrastructure

Cop

yrig

ht ©

200

5 U

nive

rsit

y of

Gla

sgow






The AccessGrid

• Example: A widely used real-time collaborative environment– Supports many-to-many interactive meetings

– Typically room based, although not required by the underlying technology

• Components:– Venue client

– Media tools

– User experience

• A framework for initiating real-timecollaborative work sessions– Default media tools provided, can be

replaced

Cop

yrig

ht ©

200

5 U

nive

rsit

y of

Gla

sgow

The AccessGrid: Venue Client

• Venue client provides the rendezvous point to join a session

• Maintains state, participant list, links to other venues, etc.






Cop

yrig

ht ©

200

5 U

nive

rsit

y of

Gla

sgow

The AccessGrid: Typical Media Tools



• Joining a venue starts multicast audio and video tools on the controller display– Audio level and activity indicator

– Video thumbnails and statistics

• A separate display shows the main video windows…



Cop

yrig

ht ©

200

5 U

nive

rsit

y of

Gla

sgow

The AccessGrid: User Experience




Tool development ongoing - theuser experience will change withtime…

Cop

yrig

ht ©

200

5 U

nive

rsit

y of

Gla

sgow

History and Standards Development

• Long development history• Strong open standards• Commercial interest

19851975 1995 2005

Mbone tools AccessGridvic

vat rat nte

wb sdr

Grid Computing

RTPNVP

ST ST-II SIP

H.323

NVP-II

H.320

3G phones

Video conferencing

IP Multicast

RFC 741

Cop

yrig

ht ©

200

5 U

nive

rsit

y of

Gla

sgow

Current Deployment Status



• Hundreds of nodes at sites worldwide• Large scale multicast video conferences• Standard, if primitive, media formats• Non-standard web service control infrastructure

QuickTime™ and aTIFF (Uncompressed) decompressorare needed to see this picture.

• Tens of millions of mobile phones• Voice telephony and multicast streaming• Standard, advanced, media formats• Standard SIP-based control infrastructure

• Dedicated video conferencing• Wi-Fi and wired SIP-based VoIP• PC-based video streaming• Mixture of standard and proprietary

Others…

• Largely successful deployment of open standard media coding and transport protocols– With notable exceptions, and

continuing patent issues…

• Largely successful deployment of open standard session setup and control protocols– AccessGrid, although open, is

the main exception to the use of standards-based protocols

• Independent and interoperable implementations available and widely used

Cop

yrig

ht ©

200

5 U

nive

rsit

y of

Gla

sgow

Underlying Technologies

• Complex protocol stack, many options

• Two phase operation: control then data

Mediacodecs

Light weightsessions

IP

TCP UDP/UDPlite

SIP SAP RTP

SDP, SDPng

Session and call control

RTSP

DCCP

Web/Grid Services

STUN

ICE/TURN

NAT Traversal

SIP+SDP

Web services

RTP + media codecs

Cop

yrig

ht ©

200

5 U

nive

rsit

y of

Gla

sgow

Control Protocols: Concepts

• Many control protocols exist– For telephony and integration with the PSTN

– For streaming video, television and radio

– For collaborative work in a Grid environment

• Some common features:– Initial rendezvous point and user location

– Media format negotiation

– Address and port negotiation

– Session metadata• Dates and times

• Additional information

• Participant scheduling

• Shared state

– Negotiation of security parameters

v=0o=jdoe 2890844526 2890842807 IN IP4 10.47.16.5s=SDP Seminari=A Seminar on the session description protocolu=http://www.example.com/seminars/[email protected] (Jane Doe)c=IN IP4 224.2.17.12/127t=2873397496 2873404696a=recvonlym=audio 49170 RTP/AVP 0m=video 51372 RTP/AVP 99a=rtpmap:99 h263-1998/90000

Cop

yrig

ht ©

200

5 U

nive

rsit

y of

Gla

sgow

Control Protocols: Implementations

• Session Initiation Protocol (SIP)– Commercial video conferencing products

– Commercial telephony and voice-over-IP

• Real-Time Streaming Protocol– Video streaming (e.g. QuickTime)

• Web services– AccessGrid

All evolutions of HTTP designedto support negotiation and control

of real-time media sessions

Complex, flexible

Large toolkits

Initial negotiation

Ongoing control

Cop

yrig

ht ©

200

5 U

nive

rsit

y of

Gla

sgow

Data Transfer Protocols

• Once negotiation completed, media data transferred

• Complex due to nature of IP networks– A “best effort” packet delivery service

– Performance not guaranteed• Loss, delay, reordering, corruption, duplication

– Higher level protocols must compensate• Application complexity, simple network

• An application must correct errors, recover media timing, etc., in addition to processing audio/visual data

• Cannot use TCP, since require accurate timing

• Main standard is RTP: Real-time Transport Protocol

IP

EthernetADSL

PPP

Optical FibreTwisted Pair

TCP UDP

HTTP

FTP

SMTP RTP

SIP RTSP

HTML MIME Media codecs

Application programs

Wireless

Cop

yrig

ht ©

200

5 U

nive

rsit

y of

Gla

sgow

• A standard protocol for real-time collaborative environments on IP networks

• Provides features for:– Timing recovery and synchronisation

– Loss detection and recovery

– Media format identification

– Participant identification andpresence

– Reception quality statistics

but must be implemented inan application!

• A toolkit for real-time data transfer

Payload data

Padding

Synchronization source (SSRC) identifier

Timestamp

Sequence NumberPTMP XV CC

Contributing source (CSRC) identifiers

RTP: Real-time Transport Protocol

Cop

yrig

ht ©

200

5 U

nive

rsit

y of

Gla

sgow

Capture and Analysis of Media Sessions

• Media capture and decoding is complex– One cannot simply dump packets to a file!

– Need to understand transport protocol• Reconstruct timing, correct errors, log metadata

– Need to understand the signalling protocol• Metadata needed to find and understand the media

• Data storage format:– Standard file formats exist, but often lacking in metadata and context

– Formats based on the ISO multimedia file format better?

• Relevant? “RTP Payload Format for ETSI ES 201 108 Distributed Speech Recognition Encoding”, RFC 3557

Cop

yrig

ht ©

200

5 U

nive

rsit

y of

Gla

sgow

Underlying Technologies

• Flexible and sophisticated control protocols– Initial rendezvous and negotiation

– Ongoing control during a collaborative work session

• Capable data transfer protocols; range of media formats

• “Some assembly required”– Protocols are expected to be tightly coupled to applications

– The design rejects much of the traditional protocol layering

– Collaborative work environments use a wide range of protocols; often in different ways to commercial products

Cop

yrig

ht ©

200

5 U

nive

rsit

y of

Gla

sgow

Research Issues and Future Directions

• Continual evolution of protocols– New media formats

– Extension of the control frameworks• Emergency calls (E.911)

• Presence and messaging

• NAT traversal, seamless operation over present-day Internet

• Integration of AccessGrid with standard control protocols– SIP-based architecture? Desirable to move away from “Grid” protocols, to

a more standard architecture, to leverage massive commercial development

• Evolution of IP multicast peer-to-peer overlays

Cop

yrig

ht ©

200

5 U

nive

rsit

y of

Gla

sgow

Summary

• Real-time Collaborative Environments– Definition and Requirements

– Example: AccessGrid

– History and Current Status

– Underlying Technologies and Standards

– Capture and Analysis of Real-time Sessions

• Research Issues and Future Directions

Any Questions?

Documents

Real-Time Collaborative Environments Colin Perkins