Upload
mark-watson
View
12.356
Download
2
Tags:
Embed Size (px)
DESCRIPTION
#seconf
Citation preview
Performance Metrics In A Day
Shane Hender
Mark Watson
Agenda
Goal is to show how you can enhance your automation tests to gather performance metrics
Keep the short presentation as practical as possible with enough code snippets to give you a starting point
One more thing…
HTTP Archive (HAR)
What? UTF8 Text based JSON encoding to represent
performance data
Why? Many libraries to convert from JSON to objects More tools allowing import/export/view HAR Becoming the standard format
Snippet of HAR format
{ "log": { "version": "1.1", "creator": { "name": "Firebug", "version": "1.7.0" }, "browser": { "name": "Firefox", "version": "4.0" }, "pages": [ { "startedDateTime": "2011-03-31T16:56:50.455-07:00", "id": "page_62901", "title": "Google", "pageTimings": { "onContentLoad": 431, "onLoad": 3148 } } ],
"entries": [ { "pageref": "page_62901", "startedDateTime": "2011-03-31T16:56:50.455-07:00", "time": 250, "request": { "method": "GET", "url": "http://www.google.com/", "httpVersion": "HTTP/1.1", "cookies": [ { "name": "PREF", "value": "ID" },
HAR Viewers http://www.softwareishard.com/har/viewer/
Fiddler UI can import HAR files ShowSlow webapp can be used to archive and view
HAR files
Which Metrics? Overall page load time DOM loading/interactive/complete, browser 1st
render, … Per-item timings
DNS, SSL connect, time to first byte, total time to download bytes, …
Headers, status codes, and content For a more detailed picture of the reasons for the
page performance Can help with debugging performance, e.g. were
items compressed, or not being loaded dynamically, or certain items not being prioritized for above-the-fold rendering
Methods for gathering metrics Setting your own timings in test code Using the new ‘Navigation.Timings’ or Web
Timings standard built into browsers Using browser plugins that report back the
timings to you, e.g. WebPageTest Per HTTP request logs via Selenium’s built-in
proxy Routing the browser traffic through a local
proxy and gathering the statistics from there. Network traffic capture
Method 1: Own Timings
WebDriver driver = new ChromeDriver();Date start = new Date();driver.get("http://www.webmetrics.com");Date end = new Date();
Method 2: Web Timings Currently Chrome and IE9 supported, coming soon
for Firefox http://w3c-test.org/webperf/specs/NavigationTiming/
WebDriver driver = new ChromeDriver();// A "base url", used by selenium to resolve relative URLs String baseUrl = "http://www.webmetrics.com"; driver.get(baseUrl); JavascriptExecutor js = (JavascriptExecutor)driver; Long loadEventEnd = (Long) js.executeScript("return
window.performance.timing.loadEventEnd");Long navigationStart= (Long) js.executeScript("return window.performance.timing.navigationStart");Long answerToAllProblems = loadEventEnd - navigationStart;
Method 2: Web Timings Could also modify this by using your own
JavaScript loaded into the page to set timings
Also would work in combination with the already set Web Timings E.g. Set a Date().getTime() variable when some dynamically
loaded content is loaded into the DOM
Long perceivedLoadTime = (Long) js.executeScript("return elementInsertTime – window.performance.timing.navigationStart");
Unfortunately it doesn't give timings per item downloaded, e.g. images, css, js, ....
Method 3: Browser Plugins Typically can give compute time metrics
Javascript execution time CPU usage Render times
IE8 - WebPageTest CPU usage Video capture
Firefox - Firebug Net Panel + NetExport https://github.com/lightbody/browsermob-page-perf https://github.com/AutomatedTester/AutomatedTester.PagePerf.git
DynaTrace browser plugin (FireFox/IE on Windows) Good breakdown of stats (Javascript execution times, CSS times,
render, 'first impression' time). http://ajax.dynatrace.com/ajax/en/
Chrome – some export from the Developer Tools Network Panel?
FirefoxProfile p = new FirefoxProfile();
try { p.addExtension(new File("c:/firebug-1.6.0.xpi")); p.addExtension(new File("c:/netExport-0.8b9.xpi")); p.addExtension(new File("c:/firestarter-0.1.a5.xpi"));} catch (IOException e) { throw new RuntimeException(“Failed to load extensions:", e);}p.setPreference("extensions.firebug.netexport.autoExportActive", true);//p.setPreference("extensions.firebug.netexport.defaultLogDir", "c:/");p.setPreference("extensions.firebug.onByDefault", true);p.setPreference("extensions.firebug.defaultPanelName", "net");p.setPreference("extensions.firebug.net.enableSites", true);p.setPreference("extensions.firebug.previousPlacement", 1);driver = new FirefoxDriver(p);
Method 3: Example - Firebug + NetExport
Method 4: Example - Firebug + NetExport
Method 4: Use a Proxy
Use built in Selenium 1 proxy
Use a 3rd party proxy library Many available, few capture metrics in a
convenient way Two good ones:
BrowserMob Proxy Fiddler
Method 4: Selenium built-in Proxy
Use API Selenium.captureNetworkTraffic()
Selenium selenium = new DefaultSelenium("localhost", 4444, "*firefox", "http://www.webmetrics.com"); selenium.start("captureNetworkTraffic=true");selenium.open("http://www.webmetrics.com"); String json = selenium.captureNetworkTraffic("json"); selenium.stop();
Method 4: Selenium built-in Proxy Not in HTTP Archive (HAR) format, but does
report: HTTP status code, URL and HTTP method Request & response headers Overall request->response timing Bytes downloaded
Works with Chrome and Firefox Doesn’t work for WebDriver tests.
Still work with Selenium 2 RC, but only if send jobs via Selenese API.
Not much control over the proxy
Method 4: Advantages of using a Proxy Testing benefits beyond performance
monitoring Blacklisting/whitelisting URLs
Comparing page load times of site with/without external content
Not hitting analytics/advertising content Simulating external content being offline
URL rewrites Redirect production URLs back to staging/test
environment Pretend to be production as far as the browser is
concerned (e.g. for cookies) Make sure ad/metrics requests are being made
Method 4: Advantages of using a Proxy Header changes
Set the user agent manually to test different browser behavior
Auto authentication
Wait until all content is downloaded HTTP idle for X seconds
Limit bandwidth
Method 4: BrowserMob Proxy
Open source cross platform proxy written in Java. HTTP Archive support Friendly API
Source code available: https://github.com/lightbody/browsermob-prox
y
Method 4: BrowserMob Proxy
Export HAR Sample
import org.browsermob.proxy.ProxyServer;
ProxyServer proxy = new ProxyServer(9090);proxy.start();
Proxy mobProxy = new Proxy().setHttpProxy("localhost:9090");
DesiredCapabilities caps = new DesiredCapabilities();caps.setCapability("proxy", mobProxy);
FirefoxDriver driver = new FirefoxDriver(caps);
Method 4: BrowserMob Proxy
Run test, denoting pages in HAR
proxy.newHar("Yahoo");driver.get("http://yahoo.com");proxy.endPage();
proxy.newPage("CNN");driver.get("http://cnn.com");proxy.endPage();
Method 4: BrowserMob Proxy
Write out HTTP Archive file
proxy.getHar().writeTo(new File("test.har"));
Method 4: BrowserMob Proxy
Redirecting URLs
proxy.blacklistRequests(regex, responseCode)proxy.whitelistRequests(regex, responseCode)
Blacklist/Whitelist URLs
proxy.rewriteUrl(regex, replace)
Limit Bandwidth
proxy.setDownstreamKbps(kbps)proxy.setUpstreamKbps(kbps)
Method 4: BrowserMob Proxy
When to use Cross platform Java Can set the browser proxy
Method 4: Proxy - FiddlerCore Fiddler is an application for viewing HTTP
traffic on Windows. Works with Chrome, FireFox, Internet Explorer. Fiddler Application is built on FiddlerCore
.NET library Allows extensive programmatic control over
the proxy Captures HTTP timings Allows request/responses to be intercepted and modified
Configures itself as default Windows proxy (supports proxy chaining)
Can decrypt SSL traffic
To start the proxy and register it as the default system wide proxy.
Each HTTP transaction can then be recorded as follows:
Method 4: Proxy - FiddlerCore
// Initialize the proxyFiddler.FiddlerApplication.Startup(
8877, FiddlerCoreStartupFlags.Default);
var items = new List<Fiddler.Session>(); Fiddler.FiddlerApplication.AfterSessionComplete += delegate(Fiddler.Session oS){
items.Add(oS);};
Method 4: Proxy - FiddlerCore Run Selenium test as normal
As each item is downloaded it will be added to the ’items’ var in previous slide.
string baseUrl = "http://www.webmetrics.com";var webDriver =
new OpenQA.Selenium.Firefox.FirefoxDriver();var selenium =
new Selenium.WebDriverBackedSelenium(webDriver, baseUrl);
selenium.Start();selenium.Open(baseUrl);selenium.WaitForPageToLoad("30000");selenium.Stop();
Method 4: Proxy (Fiddler -> HAR) Using the HAR Exporter to convert the
sessions to HAR Add FiddlerCore-BasicFormats.dll to the project
references and load the assembly: https://www.fiddler2.com/dl/FiddlerCore-BasicFormats
.zip
String exePath = Assembly.GetExecutingAssembly().Location;String path = Path.Combine( Path.GetDirectoryName(exePath), @"FiddlerCore-BasicFormats.dll");
FiddlerApplication.oTranscoders.ImportTranscoders(path);
Method 4: Proxy (Fiddler -> HAR) Finally, export the HAR to a file:
var oExportOptions = new Dictionary<string, object>();string filename = @"output.har”;oExportOptions.Add("Filename", filename);
Fiddler.FiddlerApplication.DoExport( "HTTPArchive v1.2", sessions, oExportOptions, null);
Method 4: Proxy Bonus Features Modifying HTTP requests
Fiddler.FiddlerApplication.BeforeRequest +=delegate(Fiddler.Session oS)
{// Ignore requests made to the main site.Regex matchUrl = new Regex("webmetrics\\.com”);if (matchUrl.IsMatch(oS.fullUrl)) {
oS.utilCreateResponseAndBypassServer();oS.responseCode = 200;
}};
Fiddler.FiddlerApplication.BeforeResponse
In the same way responses from the server can be modified
Method 4: Proxy – FiddlerCore
When to use Windows Only Cross browser .NET
Method 5: Wire HTTP capture Captured network traffic can be converted to HAR
On Unix/OSX use tcpdumptcpdump -i en0 -n -s 0 -w traffic.pcap
On Windows use WinPCapwinpcap -i 0 -n -s 0 -w traffic.pcap
Then use pcap2har to do the conversion:main.py traffic.pcap http-requests.har
Pcap2har https://github.com/andrewf/pcap2har
Method 5: Wire HTTP capture Captures all HTTP traffic Timings based on actual network traffic No interference from proxy
No extra network delays/context switches talking to the proxy
Browsers sometimes behave differently when talking to a proxy
Summary Discussed 5 methods for recording
performance metrics Use timers around selenium commands Use WebTimings JavaScript API Firebug & NetExport Proxy
Use the built-in Selenium proxy Use a 3rd party proxy
Sniff network traffic
Links BrowserMob proxy
https://github.com/lightbody/browsermob-proxy
Fiddler Cookbook http://www.fiddler2.com/fiddler/dev/ScriptSamples.
asp
Examples from this talk https://github.com/watsonmw/selenium-pageload
metrics