Embed Size (px)
DESCRIPTION
From Velocity 2012 in Santa Clara, CA. Buddy Brewer, Philip Tellis, and Carlos Bueno talk about real user measurement collection, analysis, and insights.
Citation preview
RUM for Breakfast
Buddy Brewer, Carlos Bueno, Philip Tellis
Velocity 2012 / 2012-06-26
Velocity 2012 / 2012-06-26 RUM for Breakfast 2
https://github.com/lognormal/boomerang/
Velocity 2012 / 2012-06-26 RUM for Breakfast 5
Buddy built the navtiming plugin
Velocity 2012 / 2012-06-26 RUM for Breakfast 7
tl;dr
1 Measure a bunch of stuff in the browser2 Use high school stats that we vaguely remember3 Randomly invent insights
Velocity 2012 / 2012-06-26 RUM for Breakfast 8
Log-Normal Distribution
The logarithm of the x-axis follows a Normal distribution
Velocity 2012 / 2012-06-26 RUM for Breakfast 11
Log-Normal Distribution
Use the Geometric Mean for pure Log-Normal distributions
Velocity 2012 / 2012-06-26 RUM for Breakfast 12
Log-Normal Distribution
Performance data does not always follow a "pure" Log-Normal
distribution
Velocity 2012 / 2012-06-26 RUM for Breakfast 13
Look at the entire spread
. . .
Velocity 2012 / 2012-06-26 RUM for Breakfast 14
Look at the entire spread
which often approaches an infinite width
Velocity 2012 / 2012-06-26 RUM for Breakfast 14
• 0.8% of hits are fake/abusive• 0.2-0.5% of hits are from a stale cache• 0.1% of hits are absurd• Timestamps in the future (or past depending on how you
interpret it)• Bots ignore robots.txt across domains• "Interesting" caches/copies
Velocity 2012 / 2012-06-26 RUM for Breakfast 16
Even with beacons, you need to sanitize your input
Velocity 2012 / 2012-06-26 RUM for Breakfast 17
Band-pass filtering
• Strip everything outside a reasonable range• Bandwidth range: 4kbps - 4Gbps• Page load time: 0ms - 600s
• You may need to relook at the ranges all the time
Velocity 2012 / 2012-06-26 RUM for Breakfast 18
IQR filtering
Derive the range from the data
Velocity 2012 / 2012-06-26 RUM for Breakfast 19
MoE & Sample size
There is an inverse square root correlation between sample size
and margin of error
Velocity 2012 / 2012-06-26 RUM for Breakfast 22
How big a sample is representative?
Select n such that���1.96 σ√n
��� ≤ 5%µ
Velocity 2012 / 2012-06-26 RUM for Breakfast 23
This needs to be at your lowest drilldown level
Velocity 2012 / 2012-06-26 RUM for Breakfast 24
How does performance impact human behavior?
8 million pages
1.5 million visits
50 different dimensions
0%
17.50%
35.00%
52.50%
70.00%
0 2 4 6 8 10 12 14 16 18 20 22 24 26 28 30
very fast sessions had high bounce rates
0%
17.50%
35.00%
52.50%
70.00%
1 3 5 7 9 11 13 15 17 19 21 23 25 27 29
bounce rate vs. load time
0%
17.50%
35.00%
52.50%
70.00%
1 1.5 2 2.5 3 3.5 4 4.5 5 5.5 6 6.5 7 7.5 8 8.5 9 9.5 10 10.5 11 11.5 12 12.5
bounce rate vs. DOM interactive
0%
20.00%
40.00%
60.00%
80.00%
0.5 2 3.5 5 6.5 8 9.5 11 12.5 14 15.5 17 18.5 20 21.5 23 24.5 26 27.5 29
bounce rate vs. front end time
http://www.flickr.com/photos/21560098@N06/3796822070
is my web site performance toxic to my users?
http://www.flickr.com/photos/thecosmopolitan/6117530924
LD50 - when do half the users bounce?
Back end time
DOM Loading
DOM Interactive
Front end time
DOM Complete
Load event
1.7 sec
1.8 sec
2.75 sec
3.5 sec
4.75 sec
5.5 sec
Bounce rate >=50%
What is the LD50 for your site?
Other bounce rates? 40%? 30%?
Other variables? (critical content visible, etc)
Other behaviors? Conversions, revenue, pages per session, actions, when do people make tea?
Future directions
Questions?
Buddy Brewer @bbrewerPhilip Tellis @bluesmoon
Carlos Bueno @archivd
