Network Breaking Point- Bruce peterson panel

Focus: reliable KPIs

Aggregation and filtering.

?? ??!

Internet, S

Client!

Server,ApplicationRealtime

1Measuring E2E Quality Dr.Stefan Rugel 15.2.2011 slide

Measuring E2E throughput as provided by the network

Checking efficiency of time sampling method:

20 M t P b l d t diff t it ithi T 2 N t k• 20 Measurement Probes placed at different sites within To2 Network

• 7.2 Mbit/s max throughput, good coverage placement

• Testcase: 2 Mbyte http downloads

Probe Measurements Network-based measurements



=> Perfect match with true throughput measured by the probe.

1600

Final Improvement

1200

1400

sec]

800

1000

hput [kbit / s

400

600

Through

0

200

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38

Measurement Sample

"Effective Throughput": considering the type of the idle times Probe Measurements

3Measuring E2E Quality Dr.Stefan Rugel 15.2..2011 slide

User Experience vs Network Monitoring

Service Flow Measures User Experience Monitoring

RAN Core IPUser Data

User Data

FlRAN Core IPFlows Flows

RANStats

CoreStats

RouterStats

NetworkMonitoring = + +

Quick Wins (vague specific)

F l i t S ifi D t il• For a complaint – can now see Specific Details• Which network elements involved (name of GGSN, SGSN, etc)• PDP Context Set-Up records (Accept / Reject)PDP Context Set-Up records (Accept / Reject)• Data Flow

Network Initial RNC GGSN SGSN Sig Name SGSN SigSGSN DatStart Time End Time APN Create/Update Closure DL Vol (kB) UL Vol (kB)g g p ( ) ( )

Hutchison 3RMHB13 BM0FXG01 HH0SGN07 217.171.1217.171.1424/09/2010 02:55 24/09/2010 03:55 234.20.three.co.uk Hourly Update 3.88 2.06




Hutchison 3RMHB13 BM0FXG01 HH0SGN07 217.171.1217.171.1423/09/2010 22:57 23/09/2010 22:57 234.20.three.co.uk Hourly Update 0 0


RMHB13 HH0FXG02 HH0SGN07 23/09/2010 22 57 23/09/2010 22 57 234 20 th k R j t d b GGSN 0 0Hutchison 3RMHB13 HH0FXG02 HH0SGN07 217.171.1217.171.1423/09/2010 22:57 23/09/2010 22:57 234.20.three.co.uk Rejected by GGSN 0 0

Hutchison 3RMHB13 HH0FXG01 HH0SGN07 217.171.1217.171.1423/09/2010 20:55 23/09/2010 21:01 234.20.three.co.uk Success Coverage Lost 0 0

Hutchison 3RMHB13 HH0FXG01 HH0SGN07 217.171.1217.171.1423/09/2010 19:55 23/09/2010 20:55 234.20.three.co.uk Hourly Update 25.4 1.15

Hutchison 3RMHB13 HH0FXG01 HH0SGN07 217.171.1217.171.1423/09/2010 19:04 23/09/2010 19:55 234.20.three.co.uk Hourly Update 97.07 26.01

Orange UK Roaming HH0FXG01 194.33.x.x 194.33.24194.33.24.23/09/2010 18:51 23/09/2010 18:51 234.20.three.co.uk Success Success 4.04 1.12


Hutchison 3RMBS06 HH0FXG02 BM0SGN05 217.171.1217.171.1423/09/2010 16:55 23/09/2010 17:49 234.20.three.co.uk Success Coverage Lost 0 0

Hutchison 3RMBS06 HH0FXG02 BM0SGN05 217.171.1217.171.1423/09/2010 16:49 23/09/2010 16:55 234.20.three.co.uk Hourly Update 0 0


Hutchison 3RMBS06 BM0FXG01 BM0SGN05 217.171.1217.171.1423/09/2010 16:17 23/09/2010 16:42 234.20.three.co.uk Success Success 144.02 48.14

Peak Speed Metric Calibration (591 Customers Surveyed 10 = Excellent, 1 = Poor)( y , )

gePe

rcen

tag

ed M

etric

Customer Score Customer ScoreSpee

• Good Correlation between Customers’ Scores and Speed Metric• Wide disagreement about what speed is good – perhaps due to g p g p p

different expectations and uses of Internet.• Vote Good – Actual speeds measured are also good• Vote Poor – Actual speeds measured may be good or bad.

Call Centre Use

• Per Session Per Customer Data flow Metrics

S d i tti

• Per-Session, Per-Customer Data-flow Metrics has resulted in a 8% reduction in Call Handling Times.

Speed Metric(last 30 days)

Speed is getting worse in last

fortnight

Which devices customer has used and data volumes

Shouldn’t You Be Scoring Your Customers’Customers Experience?

Per Session, Per User QoE Scoring

Experience Latency Packet Error EffectiveExperienceScore

Latency Packet Error Effective Throughput

5. Excellent4. Good3. Fair2. Poor1. Bad > 600 ms >6% < 400 Kbps

2220016142216790

2220016142247380

Impacted PDP Sessionsby Cell ID

2220016142216790

2220016142247380

2220016142211010

2220016142216860

2220016142204710

2220016142247400

2220016142216790

0 200 400 600 800 1000 1200 1400

2220016142216770

2220016142216790

Real-Time Measurements & Analytics Feed Business ProcessesBusiness Processes

End-to-End SScoring

+Analytics & Reporting

+g

Customer Experience

Resolution

Trouble Ticket

Resolution Isolation

Proactive Problem Isolation Effectiveness

Capacity Planning

Effectiveness

SolutionsValidate experience issues and quickly

Isolate problems with detailed session data

Aggregate scores of actual user experience

Verify planned capacity expansions based onissues and quickly

develop corrective actions

detailed session data, network element

reports

actual user experience and alarm about

problems

expansions based on user experience

scoring

BackupBackup

The Challenge:M i Q li f M bil IManaging Quality for Mobile Internet

iatio

niff

eren

tias

ing

Di

Incr

ea


Monitoring E2E-Quality: Network based Approach

Advantage: Full network coverage, real time availability.Problem: Away from Client.Challenge: Drawing reliable conclusion from observations.

?

Internet, S

Client:Throughput,Latency ServerLatency


E2E-Protocols for E2E KPIs

Packet Inspection:• E2E Protocols • Flow specific• Gn-If

Move from Network PM counter evaluation to E2E Protocol analysis.

A li ti

• Gn-If• Real Time

y

PDCP PDCPIPTCP

Application

GTP GTP

RLC

MAC-d

MAC-c/sh

RLC

MAC-d

MAC-c/sh IP / MPLS(ATM) IP

UDP

IP / MPLS(ATM) IP

UDP

L3 L3

PHY

MAC-hs

PHY

MAC-hs

PHY

L2 / L3

HS_DSCHFP

PHY PHY

L2

(ATM)

PHYL2

PHYL2

IP

PHYL2L2 / L3

HS_DSCHFP

(ATM) IP

PHYL2

PHYL2

UE Nb RNC SGSN GGSN Server

IU-PSIub GnGi /

InternetUu


UE Nb RNC SGSN GGSN Server

Protocols providing E2E Quality information

E il Web Vid St iService AEmail Web

Browsing Video Streaming Access, Topology2nd Step:

• Classification• Detailed KQI

• Internet Access• Location• Network Topology

POP3,SMTP,

Containers (.flv, .f4v, .mpg, .mp4, .avi, .swf ...)

CodecsHTTP GTP

DNS

• Network Topology• Device Type• Radio Access Type

RTMPIMAPRTCP RTP

DNS

UDP(<1% of traffc)

HTTP

TCP (> 99% of Traffic) UDP(<1% of traffc)

IPTCP (> 99% of Traffic)

1st Step:• Throughput• RTT (Round Trip Time)• Packet Retransmissions• Service Access

Measuring E2E Quality Dr.Stefan Rugel 17.11.2010 slide 15

Agenda

1 Introduction

g

1. Introduction2. TCP Throughput Measurements3. Conclusion

M t i & Q EMetrics & QoEUser perceived Quality (QoE)User perceived Quality (QoE)

QoE

M

appi

ng

Mapping

VoIP, Life Streaming

File Download

Web VideoM Web Browsing

BufferinTime to loadApplication

ifi ty

P k t L

gApplication

Met

rics

Time to load

k t lThroughput

specific

mpl

exit

Packet LossRound Trip Time

TCP

M

RTCP

packet loss jitter

utQoS C

o

Network, L2, IP, UDP

TCPRTCP

Which kind of Throughput?

• Network provided ThroughputNetwork provided Throughput.

• Realistic download scenario, no peak throughput.

Usecases:

• Strategic Marketing.g g

• Customer Care

• Capacity Engineering

• Traffic ManagementTraffic Management

• ……….



Basic Approach: Data VolumeConnection Time

Download Throughput =Connection Time

Problem: several downloads per connection, separated by idle times.

TCP Setup TCP TeardownC

Idle Idle Idle Idle

Connection Time

DownlinkUplink

Ti

TCP Control Packet

C

Time

TCP Packet with Payload19Measuring E2E Quality Dr.Stefan Rugel 17.11.2010 slide


Improved approach: sampling fixed time intervals (e.g. 0.5 sec)

Data VolumeDownload Throughput =Active Time

Download Throughput

Active Time

Idle Idle Idle Idle

DownlinkUplink

Ti

TCP Control Packet

C

Time



Checking efficiency of time sampling method:

20 M t P b l d t diff t it ithi T 2 N t k• 20 Measurement Probes placed at different sites within To2 Network

• 7.2 Mbit/s max throughput, good coverage placement

• Testcase: 2 Mbyte http downloads

• Correlation with network based time filtering taken from inside the

network

Probe MeasurementsNetwork Based:Time sampling



6000

Probe Throughput Measurement versus Active ThroughputMeasurement Results (Time Filter 0.5sec)

5000Over‐Estimation due to neglecting idle periodst [

kbps

]

4000

kbps]

Over Estimation due to neglecting idle periods

mt R

esul

t

3000

red Throughp

ut [k

Active Throughput

ProbeThroughptused

Msm

!!!2000

Measur ProbeThroughptu

wor

k ba

s !!!1000

Under‐Estimation due to sampling‐intervalmissmatch at Download Start and ‐End

Net

w

0

0 1000 2000 3000 4000 5000 6000

Throughput [kbps]Probe Measurement[kbps]O fOver Estimation for bad throughput!



Final Correction: considering the type of idle times.

• Netw. BufferingTCP Timeout Bad“ Idle Times

Idle Times: due to• TCP Timeout

Time between two

„Bad Idle Times

Good“ Idle Times

TCP Setup TCP TeardownC

separate Downloads „Good Idle Times

Idle Idle Idle Idle

Connection Time

DownlinkUplink

Ti

TCP Control Packet

C

Time



=> Perfect match with true throughput measured by the probe.

1600

Final Improvement

1200

1400

sec]

800

1000

hput [kbit / s

400

600

Through

0

200

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38

Measurement Sample

"Effective Throughput": considering the type of the idle times Probe Measurements


TCP: E2E property

Client ServerMSMTTCP handshake enables PING-like

round trip time measurements.

Client ServerMSMT

RTTAcknowledged data transfer shows

packet retransmissions and delivered

RTT

data volume.

Various methods for throughput Duration

measurement.Duration,Volume


P i t f tPoint of measurement

Measurement at the Clients Side.

V l t Q E

Network based measurement.

F ll t k/tiVery close to QoE

No full network/time coverage

Full network/time coverage

Real time capable

Expensive Only part of information available

RTT+ k t l > Th h t?RTT+packet loss => Throughput?2500

Life Network Measurement, static probes, 2Mbyte file download

2000

e [m

sec]

1500

d Trip Tim

e

< 1000kbps

< 1500kbps

Throughput

1000

erage Ro

und

< 2000kbps

< 2500kbps

>= 2500kbps

500

Ave

0

0 5 10 15 20 25 30 35

Packet Retransmission Rate [%]

A dAgenda

• Introduction

• Network based measurement on QoE in mobile data networksmobile data networks.– Why do we need it

– Approach

– Example: Web Video– Example: Web Video

– Conclusion

W b VidWeb VideoApproach:

Buffer simulation: estimate buffered time by correlating packet arrival‐ and playoutBuffer simulation: estimate buffered time by correlating packet arrival and playout time.

ViewingViewing Time Buffered

Time

1 t ti i i f ti1. get timing informationInternet Video Server

TCP Acknowledgements

IP Packets of VideostreamFirst Step:

Measure packet arrivalPoint of Measurement

Get:TCP-Sequence Number,Timing Info of last FrameCalculate:Playout Time

ptimes.Extract correspondingframe playout times (flv, Playout Time

Save:TCP-Sequence Number,Playout Time

Table

frame playout times (flv, mp4)

Seq-Nr. Playout Timen1 x1

.. ..nm xm

nm+1 xm+1

Get:TCP-Ack Number,Timestamp of ArrivalLookup:TCP-Ack NumberSave:Timestamp of Arrival

Ack Timey1

..ym

.. ..

Calculate QoE – KPI

KPI = f(xi yi)Codec

Protocols used( i,yi)

Video File Format

Codec

TCP

HTTP

30Video Client

TCP

2 B ff i l ti2. Buffer simulationBuffering Mode = “ON“Buffering Mode = ON

BufferedTime = 0Init TB

i=2Get x1, y1

Second Step: Simulate Buffered Time

BufferingMode =

“OFF“

BufferedTime =

Yes

No

BufferedTime = BufferedTimei =BufferedTimei-1+ (xi-xi-1) - (yi-yi-1)

BufferedTimei>

BufferedTimei≤

BufferedTimei =BufferedTimei-1+

(xi-xi-1)

KPIs per Video Flow: Total re-bufferred playtime fe

rin

Playo

> TB

BufferingMode =“OFF“

≤0

YesYes

NoBufferingMode =

“ON“

Total re bufferred playtime.Number of rebufferingevents.

Input Parameter:

Buf

fg

out

Last iEnd Yes

BufferedTime =0

Input Parameter: Buffering threshold TB.

Last iEnd

Next i

Yes

No

31

Test setupTest setup

Compare packet trace Analysis with client measurements

Network Based:• Initial Buffering TimeR b ff i Ti

Client based (manually):• Initial Buffering TimeR b ff i Ti • Rebuffering Time

• Number of Rebuffering Events• Rebuffering Time• Number of Rebuffering Events

32

Fi t R lt (1)First Results (1)

16

18

12

14

ec]

8

10

ered

Tim

e [se

4

6

Buff

0

2

0

0 20 40 60 80 100 120

Viewing Time [sec]Buffering Period:Good agreement with

33

Client Measurement

Fi t R lt (2)First Results (2)2.1Mbit/s required < 2.3Mbit/s achieved

34

C ll C t UCall Centre Use• Per Session Per Customer Data flow Metrics has resulted in a

S d i tti

• Per-Session, Per-Customer Data-flow Metrics has resulted in a 8% reduction in Call Handling Times.

Speed Metric(last 30 days)

Speed is getting worse in last

fortnight

35

Which devices customer has used and data volumes

E i i I ti tiEngineering Investigation Site Throughput Problem detectedSite Throughput Problem detected

Site flat-lined from 9am!

36

Not all user activity uses the available speedNot all user activity uses the available speed(Unloaded Cell with 7.2 HSPA dongle)

Download 3 5MBLarge Web Page Asset

Download 2MB Photo(unloaded website)

Download 3.5MB Application (popular website)

BBC iPlayer

5.8 Mbits/sec

Browsing Web Pages Flash Video

(Speed limited by External Site to ~1.6Mbits/s)

37

Technology

Network Breaking Point- Bruce peterson panel