The Statistics of Web Performance Analysis

• Philip Tellis

• .com• philip@lognormal.com

• @bluesmoon• geek paranoid speedfreak• http://bluesmoon.info/

Boston #WebPerf Meetup / 2012-08-14 The Statistics of Web Performance Analysis 1

I’m a Web Speedfreak

We measure real user website performance

This talk is about the Statistics we learned while building it

The Statistics of Web Performance Analysis

Philip Tellis / philip@lognormal.com

Boston #WebPerf Meetup / 2012-08-14

0Numbers

Accurately measure page performance∗

Be unintrusive

If you try to measure something accurately, you will changesomething related

– Heisenberg’s uncertainty principle

And one number to rule them all

What do we measure?

• Network Throughput• Network Latency• User perceived page load time

We measure real user data

Which is noisy

1Statistics - 1

Disclaimer

I am not a statistician

1-1Random Sampling

Population

All possible users of your system

Sample

Representative subset of the population

Bad sample

Sometimes it’s not

How to randomize?

http://xkcd.com/221/

How to randomize?

• Pick 10% of users at random and always test them

• For each user, decide at random if they should be tested

http://tech.bluesmoon.info/2010/01/statistics-of-performance-measurement.html

Select 10% of users - I

if($sessionid % 10 === 0) {// instrument code for measurement

• Once a user enters the measurement bucket, they staythere until they log out

• Fixed set of users, so tests may be more consistent• Error in the sample results in positive feedback

Select 10% of users - II

if(rand() < 0.1 * getrandmax()) {// instrument code for measurement

• For every request, a user has a 10% chance of beingtested

• Gets rid of positive feedback errors, but sample size !=10% of population

How big a sample is representative?

Select n such that∣∣∣1.96 σ√n

∣∣∣ ≤ 5%µ

1-2Margin of Error

Standard Deviation

• Standard deviation tells you the spread of the curve• The narrower the curve, the more confident you can be

MoE at 95% confidence

±1.96 σ√n

MoE & Sample size

There is an inverse square root correlation between sample sizeand margin of error

1-3Central Tendency

One number

• Mean (Arithmetic)• Good for symmetric curves• Affected by outliers

Mean(10, 11, 12, 11, 109) = 30

One number

• Median• Middle value measures central tendency well• Not trivial to pull out of a DB

Median(10, 11, 12, 11, 109) = 11

One number

• Mode• Not often used• Multi-modal distributions suggest problems

Mode(10, 11, 12, 11, 109) = 11

Other numbers

• A percentile point in the distribution: 95th, 98.5th or 99th

• Used to find out the worst user experience• Makes more sense if you filter data first

P95th(10, 11, 12, 11, 109) = 12

Other means

• Geometric mean• Good if your data is exponential in nature

(with the tail on the right)

GMean(10, 11, 12, 11, 109) = 16.68

Wait... how did I get that?

√ΠN

i=1xi — could lead to overflow

i=1 loge(xi )N

)— computationally simpler

√ΠN

i=1 loge(xi )N

√ΠN

i=1 loge(xi )N

√ΠN

i=1 loge(xi )N

Other means

And there is also the Harmonic mean, but forget about that

...though consequently

We have other margins of error• Geometric margin of error

• Uses geometric standard deviation• Median margin of error

• Uses ranges of actual values from data set

• Stick to the arithmetic MoE– simpler to calculate, simpler to read and not incorrect

...though consequently

We have other margins of error• Geometric margin of error

• Uses geometric standard deviation• Median margin of error

• Uses ranges of actual values from data set

• Stick to the arithmetic MoE– simpler to calculate, simpler to read and not incorrect

2Statistics - 2

2-1Distributions

Let’s look at some real charts

Sparse Distribution

Log-normal distribution

Bimodal distribution

What does all of this mean?

Distributions

• Sparse distribution suggests that you don’t have enoughdata points

• Log-normal distribution is typical• Bi-modal distribution suggests two (or more) distributions

combined

In practice, a bi-modal distribution is not uncommon

Hint: Does your site do a lot of back-end caching?

2-2Filtering

Outliers

• Out of range data points• Nothing you can fix here• There’s even a book about

Outliers

DNS problems can cause outliers

• 2 or 3 DNS servers for an ISP• 30 second timeout if first fails• ... 30 second increase in page load time• Maybe measure both and fix what you can• http://nms.lcs.mit.edu/papers/dns-ton2002.pdf

Band-pass filtering

• Strip everything outside a reasonable range• Bandwidth range: 4kbps - 4Gbps• Page load time: 50ms - 120s

• You may need to relook at the ranges all the time

IQR filtering

Here, we derive the range from the data

The Statistics of Web Performance Analysis

Technology

Error analysis statistics

Vital Statistics Analysis Results

Chapter 12 Statistics for engineering analysis · Statistics for Engineering Analysis Chapter 12 Statistics for engineering analysis ... a mathematical model for statistical analysis

Evaluating Student Statistics Performance Using SAS/STAT ... · Evaluating Student Statistics Performance ... same introductory statistics course within the past several years. The

Design and Performance Analysis of a DRAM-based Statistics Counter Array Architecture

PORT PERFORMANCE FREIGHT STATISTICS WORKING GROUP … September 23... · PORT PERFORMANCE FREIGHT STATISTICS WORKING GROUP ... BTS to establish a Port Performance Freight Statistics

Performance Analysis on Blue Gene/P Tulin Kaman Department of Applied Mathematics and Statistics Stony Brook University

STAR Year 2001 Data and Analysis RHIC performance STAR Trigger Recorded Event Statistics Offline Production…

Categorical statistics analysis

Statistics and Data Analysis - New York Universitypeople.stern.nyu.edu/wgreene/Statistics/Statistics-Problem1... · Assignment 1 1 Statistics and Data Analysis Professor William Greene

Statistics (Data Analysis)

NHS Performance Statistics

Statistics & Analysis

Statistics for Performance Evaluation

Planning Performance Statistics - 2019-20 - Annual - Infographic …€¦ · Annual Planning Performance Statistics, 2019-20 Key Findings 04 Planning Performance Statistics 2019/20

statistics analysis

Unido.org/statistics Composite measure of industrial performance for cross-country analysis Shyam Upadhyaya UNIDO The 59th World Statistics Congress Hong

Multivariate analysis on performance in statistics, self

Divorce Statistics Analysis

Statistics, Data Analysis, and Decision Modeling, 5e …Statistics, Data Analysis, and Decision Modeling, 5e (Evans) Chapter 2 Descriptive Statistics and Data Analysis 1) ________