Jin Li, Principal Researcher(Collaborators: Cheng Huang, Keith Ross)Communication and Collaboration Systems

Microsoft Research

1

2

Introduction: Internet Video on the Rise

Internet Video is on the RiseVideo streams served increased 38.8% in 2006

to 24.92 billion (Source: AccuStream iMedia Research)

53 web-video startup in 2006 (US), $521M VC funding (Source: DowJones VentureOne)

Major studio goes into Web Video

$410M video ads revenue in 2006 and grow by 89% in 2007 (0.6% of $74B TV ad market, Internet ads $16.4B in 2006, expect to grow 19% in 2007) [source: Emarketer]

3

Internet Video is Growing: in Popularity & Quality

4

Apr. 2006 Dec. 2006 Up (%)

# of views (million) 41.1 64.7 57.4

# of users (million) 9.03 12.02 33.1

video quality evolution

popularity evolution

5

Internet Video Delivery: Data Center vs. CDN vs. P2P

Just Build More Powerful Data Center?

6

Data Center CapacityVHS quality video streaming: 500 kbps

(H.264)200,000 viewers = 100 Gbps

Data center capacity: Tera Grid (UIUC)30 petabyte of storage40 Gbps backbone: 80k video viewers

7

CDN Is Not The Answer

8

Akamai20,000 servers, 900 point of presence, 71 countries400Gbps bandwidthNetwork optimized for latency

Limelight25 point of presence, hundreds servers per

presence1,000Gbps bandwidth

Akamai+Limelight: 2.8 million viewers. Current TV audience

Olympics: 2.5 billion viewersEach viewer may have his/her own interest (different

sport event, athlete nationality, etc.)

Peer Assisted Delivery is the Way To Scale

9

Economical to runSaves server/CDN bandwidth, disk I/O, CPU,

memoryRobust

no single point of failure in networkSuper-scalable

system capacity increases with number of nodes

peer resource

bandwidth

CPU

memory

harddrive

P2P Benefit Consumers: Better Video Quality, More Selection

10

11

Case Study 1: On Demand Internet Video

Peer Assisted Delivery: Mode

File Sharing

broadcast On Demand Streaming(Interactive TV)

Live MessengerFolderShare

Groove

12

Peer Buffer Map: File Sharing

13

Peer 1

Peer 2

Peer 3

Peer 4

Peer 5

Peer 6

Peer Buffer Map: Broadcast

14

Peer 1

Peer 2

Peer 3

Peer 4

Peer 5

Peer 6

Peer Buffer Map: On Demand

15

Peer 1

Peer 2

Peer 3

Peer 4

Peer 5

Peer 6

MSN VoD Service

16

Traces from the on-demand service of MSN Video9-month period: Apr. – Dec. 2006520M streaming requests59,000 unique videos

Peer-assisted VoD Model

17

Guaranteed QoS: always available serverPerformance metric: server bandwidthPeers upload what / when they are

watchingconservative assumption

servers in data

centers/CDN

Peer Bandwidth

Download BW is measured by Windows Media Serverno accurate measurement

beyond 3.5MbpsUpload BW is inferred

Average upload: 500+ kbps

18

DSL2 Cable Ethernet

Modem

ISDN

DSL1

Bandwidth Allocation Policies

19

Assumptionspeers always start watching from beginningVoD: earlier peers upload to later peers

1st policy: no-prefetchingonly satisfy demand for smooth playback, do not

further build up the bufferused by commercial live streaming companies to offer

VoD

servers in data centers

1

23

4

arrival

• ask• ask• ask• ask server

321

Bandwidth allocation policies (2)

20

Prefetching – to utilize remaining upload capacity2nd policy: water-leveling3rd policy: greedy

Lower boundallow later peers upload to earlier ones

no arrival order constraints

servers in data centers

1

23

4

arrival

water-leveling:

greedy:

4

2

Observations on Policies(Simulated: Peer Poisson Arrival)

Prefetching is crucial“free” to increase

video bitrate“balanced mode” is

most difficultS ≈ D

Greedy policy works bestlowest server loadvery close to bound

more available upload

Server BW Reduction – Two Videos

select top two most popular videos~800,000 views during April, 2006

significant server bandwidth reduction using peer assistance

less server BW even increase quality 3 times (@3x bitrate)22

gold stream silver stream

Server BW reduction – two videos

select top two most popular videos~800,000 views during April, 2006

significant server bandwidth reduction using peer assistance

less server BW even increase quality 3 times (@3x bitrate)23

gold stream silver stream

P2P @3x

Server BW reduction – all videos

24

12,000+ videosserver bandwidth

reduction in all categories1.23Gbps 36.9Mbps (97%)1.23Gbps 770Mbps

@3X bitrate (38%)April 2006

25

Locality Aware P2P Delivery

P2P Traffic Today

26

1999 to present: fueled by Napster, KaZaA, eDonkey and BitTorrent

CacheLogic ResearchInternet Protocol Breakdown 1993 - 2006

Internet Traffic on the Rise

27

Internet traffic trend: grow at a compound monthly average of 7.4% in 2006

Internet traffic d

oubles per y

ear

Traffic at Amsterdam Internet Exchange (AMS-IX)

Locality to the Rescue

28

Internet HierarchyASISP POPHome/corporationBranch office of a corporation

Delivery content in a locality aware fashionBeyond ISP aware delivery

Internet : Grand View

29

Impact on ISPs

30

Tier 1 ISP

Tier 2 ISP

AS

sibling

peering

peering entityboundary

sibling entityboundary

transit

Tier 2 ISP

AS AS AS

Economics of ISP relationships sibling relationship

several ISPs belong to same org

peering relationship mutual beneficial free

agreement (to certain extent)

transit relationship one ISP pays another

Inside ISP

31

ISP POP (Point of Presence)

32

Inside Home/Branch Office

33

neighborhood

home corporation

Branchoffice

Identify Peers Locality

34

Information usedExternal IP addressInternal IP addressSubnet mask

Peer localityISP (AS)ISP POPHome/corporationCorporate branch office

Peers are considered closer if they are in a smaller common neighborhood

MAP External IP Address to AS

35Using BGP dump

Identify POP

36

POP neighborhoodIdentify one peer that is directly connected to

the Internet at some point of timeCollect its external IP address and the subnet

maskInfer the subnet neighborhood where other

peers belong, even if they are not directly connected to the Internet

Below POP

37

Home/corporation neighborhoodAll peers with the same external IP address

Corporation branch officeAll peers with the same external IP address,

and on the same internal subnet (based on subnet mask)

Locality Aware Topology Building

38

Preferentially link peers within the same ISP neighborhoodSay if we need to establish 20 connections

We assign 50% of links to be within branch office neighborhoodIf there are less peers than the allocated links,

we simply put the unused links back to the pool

We then assign 50% of unused links to be within home/office neighborhood

The next 50% of unused links are assigned within POP neighborhood

The next 50% of unused links are assigned within AS neighborhood

The rest of the links are used for cross-AS connections

Example

39

Scenario Neighborhood Branch office Home/corporation

ISP AS Outside AS

1 Total peers 20 80 0 900 9000

Connected peers 10 5 0 3 2

2 Total peers 0 10 100 1000 9000

Connected peers 0 10 5 3 2

3 Total peers 0 2 0 1000 9000

Connected peers 0 2 0 9 9

Locality Aware P2P Scheduling

40

Preferentially deliver content to peers within closer neighborhoodPropagate neighborhood availability

informationExchange with a outside peer preferentially

content that is not available in the neighborhood

Preliminary Result: ISP Friendly

Without ISP-friendlyMuch more cross

sibling than peering boundary

Significant crossing boundary traffic

41

Without ISP-friendly

Preliminary Result: ISP Friendly

Pure ISP-friendly1 video 5000+

separate distributionsstill surprising

reductions but unnecessarily conservative

ISP could help by sharing information

42

svr rate (Mbps)

no P2P

sibling partition

peering partition

silver 39.0 19.6 15.8

top 10 295.2 90.3 75.1

cut cross boundary traffic completely

43

Conclusions

Conclusion

44

Peer assisted delivery is the way to go for mass content delivery over the worldPeer assistance can significantly reduce

server bandwidth requirementDemonstrated in real world for file sharing &

broadcastShown in our work for on demand streaming

Locality aware P2P delivery is the way to scale