View
301
Download
0
Category
Preview:
Citation preview
Measuring What Matters
© AKAMAI - EDGE 2016
Introductions
Cliff CrockerAkamai
Buddy BrewerSOASTA
© AKAMAI - EDGE 2016
Three eras of modern performance measurement
2001-2006 Basically one browser
2007-2010 Transitional period
2011-2016 WTF just happened
© AKAMAI - EDGE 2016
2001-2006: Internet Explorer 6
2001 - 2006
© AKAMAI - EDGE 2016
2007-2010: Transitional period
2007
IE7iPhoneFirefox 2
2008 2009 2010
© AKAMAI - EDGE 2016
2007-2010: Transitional period
2007
IE7iPhoneFirefox 2
2008
IE7iPhone 3GFirefox 3AndroidChrome 1
2009 2010
© AKAMAI - EDGE 2016
2007-2010: Transitional period
2007
IE7iPhoneFirefox 2
2008
IE7iPhone 3GFirefox 3AndroidChrome 1
2009
IE8iPhone 3GSFirefox 3Android 2.1Chrome 1
2010
© AKAMAI - EDGE 2016
2007-2010: Transitional period
2007
IE7iPhoneFirefox 2
2008
IE7iPhone 3GFirefox 3AndroidChrome 1
2009
IE8iPhone 3GSFirefox 3Android 2.1Chrome 1
2010
IE8iPhone 3GSFirefox 3Android 2.1Chrome 1iPadNav Timing
© AKAMAI - EDGE 2016
2011-2016: WTF just happened
?How many browsers in a typical Thanksgiving weekend?
© AKAMAI - EDGE 2016
2011-2016: WTF just happened
572How many browsers in a typical Thanksgiving weekend?
© AKAMAI - EDGE 2016
Meanwhile...
© AKAMAI - EDGE 2016
© AKAMAI - EDGE 2016
So what can I measure?
© AKAMAI - EDGE 2016
What is available today - Synthetic Tooling
ProsCPU metricsTest environments & other places without real usersAccess to everything including headers
ConsGaps in: Geos, browsers, pagesImpossible to precisely match user demo- & technographicsNo way to realistically measure browser cache impact
© AKAMAI - EDGE 2016
What is available today - RUM
ProsNear perfect coverage of browsers, geos, pagesAbility to understand impact to performanceRealistic measurement of browser cache impact
ConsGaps in: Geos, browsers, pagesImpossible to precisely match user demo- & technographicsNo access to HTTP header data
© AKAMAI - EDGE 2016
EngagementRevenueCoverageCaching
CPUPre-prod
Availability
Page detailsResourcesHeaders
SyntheticReal
© AKAMAI - EDGE 2016
© AKAMAI - EDGE 2016
https://www.w3.org/wiki/Web_Performance/Publications
W3C - Performance Working Group
© AKAMAI - EDGE 2016
Navigation Timing API
© AKAMAI - EDGE 2016
Resource Timing API
© AKAMAI - EDGE 2016
Coming soon to a browser near you! Measure the size of
assets in addition to performance.
Resource Timing (2.0) - Transfer Size
© AKAMAI - EDGE 2016
● Measure performance using high-precision timestamps
● Consists of “marks” and “measures”● PerformanceMark: Timestamp● PerformanceMeasure: Duration between two given
marks
Browser Support - User Timing API
© AKAMAI - EDGE 2016
Browser Support - Navigation Timing API
© AKAMAI - EDGE 2016
Browser Support - Resource Timing API
© AKAMAI - EDGE 2016
Browser Support - User Timing API
© AKAMAI - EDGE 2016
So what should I measure?
© AKAMAI - EDGE 2016
© AKAMAI - EDGE 2016
What matters depends on you
© AKAMAI - EDGE 2016
What matters to an ops engineer?
● Preference for binary metrics● Looks for indicators of system performance● Affinity to network and back-end metrics
MTTR
© AKAMAI - EDGE 2016
What matters to an ops engineer?
Network
Server
© AKAMAI - EDGE 2016
What matters to an ops engineer?
Network
Server
function getPerf() { var timing = window.performance.timing; return { dns: timing.domainLookupEnd - timing.domainLookupStart, connect: timing.connectEnd - timing.connectStart };}
ttfb: timing.responseStart - timing.connectEnd};
© AKAMAI - EDGE 2016
Help answers questions like...
● Do I have an issue with an upstream DNS resolver?● Am I having Tier1 connectivity issues?● How fast is my application server responding?
© AKAMAI - EDGE 2016
What matters to an ops engineer?
Don’t be misled by “super fast” responses● DNS Caching● Prefetching● Connection reuse● Pages served from browser cache
© AKAMAI - EDGE 2016
What matters to an ops engineer?
Object Level Timing
● Resource timing API● Binary metric for testing infrastructure● Measure performance of a third-party● CDN healthcheck
© AKAMAI - EDGE 2016
What matters to an ops engineer?
Object Timing
var rUrl = ‘https://edge.akamai.com/ec/us/multimedia/images/edge/akamai-edge-conference-speaker-buddy-brewer.jpg’;var me = performance.getEntriesByName(rUrl)[0];var timings = { loadTime: me.duration, dns: me.domainLookupEnd - me.domainLookupStart, tcp: me.connectEnd - me.connectStart, waiting: me.responseStart - me.requestStart, fetch: me.responseEnd - me.responseStart}
© AKAMAI - EDGE 2016
What matters to a front-end developer?
Optimized Load Times
● “Do no harm”● Looks for indicators of application performance● Looks to measure a “fully loaded” page● Affinity for front-end metrics
© AKAMAI - EDGE 2016
What matters to a front-end developer?
Client
© AKAMAI - EDGE 2016
What matters to a front-end developer?
© AKAMAI - EDGE 2016
What matters to a front-end developer?
function getPerf() { var timing = window.performance.timing; return { Basepage: timing.responseEnd - timing.responseStart, DLoading: timing.domLoading – timing.navigationStart, DInt: timing.domIneractive – timing.navigationStart, DContLoaded: timing.domContentLoadedEventEnd – timing.navigationStart, DContLoadTime: timing.domContentLoadedEventEnd – timing.domContentLoadedEventStart, DComplete: timing.domComplete - timing.navigationStart, PLoad: timing.loadEventStart - timing.navigationStart
}; }
Client
© AKAMAI - EDGE 2016
Be wary of onload...
~1s OnLoad ~3.5s visibly complete
© AKAMAI - EDGE 2016
What matters to a designer?
UX
● Performance by design (D4P)● Looks for indicators of human perception● How quickly the page renders● Affinity for custom metrics and render metrics
© AKAMAI - EDGE 2016
RUM first paint and start render
Chrome - “First Paint” Visual Start Render
© AKAMAI - EDGE 2016
User perception
Courtesy of the filament group: https://www.filamentgroup.com/lab/weight-wait.html
© AKAMAI - EDGE 2016
Custom Metrics
© AKAMAI - EDGE 2016
Custom Metrics - User Timing API
performance.mark(“startTask”);//Some stuff you want to measure happens hereperformance.mark(“stopTask”);
//Measure the duration between the two marksperformance.measure(“taskDuration”,“startTask”,“stopTask”);
//Measure the load of an image<img src=”image1.gif” onload=”performance.mark(‘image1’)”>
© AKAMAI - EDGE 2016
What matters to a business executive?
$$$$$$$● Business impact analysis● Simplified/lean metrics● Actionable data
© AKAMAI - EDGE 2016
What matters to a business executive?
© AKAMAI - EDGE 2016
What matters to a business executive?
© AKAMAI - EDGE 2016
What matters to a business executive?
© AKAMAI - EDGE 2016
Don’t create unintended silos
Biz
Ops
Dev
Design
© AKAMAI - EDGE 2016
Don’t create unintended silosThe success of my business is a reflection of...
The digital experience I provide, which depends on...
how quickly I can render content in the browser, which is impacted by...
the performance and stability of my infrastructure.
1
2
3
4
Biz
Design
Ops
Dev
© AKAMAI - EDGE 2016
“I just want to buy my Mom’s birthday present.”
“I just want to know if it will rain today.”
“I just want to share pictures of my vacation”
© AKAMAI - EDGE 2016
How do I collect it?
© AKAMAI - EDGE 2016
© AKAMAI - EDGE 2016
Real users are not normal
What do I do with it?
© AKAMAI - EDGE 2016
What do I do with it?
© AKAMAI - EDGE 2016
2.9s - p50 (median)
4.4s - Average
10.4s - p95
What do I do with it?
© AKAMAI - EDGE 2016
What should I throw out?
© AKAMAI - EDGE 2016
Load Time = 3500 years
© AKAMAI - EDGE 2016
Load Time = -(10years)
© AKAMAI - EDGE 2016
What should I keep?
© AKAMAI - EDGE 2016
© AKAMAI - EDGE 2016
© AKAMAI - EDGE 2016
Other opportunities for bots...
1. How does performance impact bot behavior?2. Are bots skewing the numbers I’m reporting?3. Am I providing an optimal experience for search bots?4. Does any of this impact SEO?
© AKAMAI - EDGE 2016
Summary
● We have amazing tools at our disposal - your users are wired for sound!
● Measuring what matters is situation independent● Important to look at all metrics from the same context● Use robust statistics when looking at your data
Recommended