Perform Realistic Web Performance Testing ToPerformance Testing To Achieve Peak User ExperiencesExperiencesMike GualtieriSenior Analyst
May 13, 2009
yForrester Research
Realistic web ea st c ebperformance testing is
critical to achieving c t ca to ac e gflawless user experiences.p
Agenda
• Why your customers won’t wait
Wh t i li ti f t ti• What is realistic performance testing
• Recommendations
4 Entire contents © 2008 Forrester Research, Inc. All rights reserved.
Agenda
• Why your customers won’t waitWh t i li ti f t ti• What is realistic performance testing
• Recommendations
5 Entire contents © 2008 Forrester Research, Inc. All rights reserved.
Performance means:• Raw page load speed Gomez Media Home
Page BenchmarkApril 10 ’09 – May 10 ’09
0:00 - 24:00 EST
6 Entire contents © 2008 Forrester Research, Inc. All rights reserved.
Gomez Media ArticlePerformance means:
Gomez Media Article Search Transaction
BenchmarkApril 10 ’09 – May 10 ’09
0:00 24:00 EST
• Customer workflow speed1
0:00 - 24:00 EST
2
33
7 Entire contents © 2008 Forrester Research, Inc. All rights reserved.
Realistic Web Performance Testing Is A F t B t P tiForrester Best Practice• Best Practices: Attaining And Maintaining Blazing Fast Web Sites
• Best Practices: Blazing Fast Web Site Infrastructure
• Case Study: How Orbitz.com Takes You Places Faster
• Case Study: How Msnbc.com Is First In News
8 Entire contents © 2008 Forrester Research, Inc. All rights reserved.
Blazing FastWeb Sites = CustomersWeb Sites Customers
Traditional web performance testing d ’t t itdoesn’t cut it• Traditional testing is expensive
• Cost of licenses, hardware and training
• Time consuming to implement
• Idiosyncrasies are often difficult to master
• Traditional testing is not realistic• Scope is limited to internal clean-room testing
• Not focused on end-user point of viewp
• Misses too many pieces in the web application delivery chain such as CDN, 3rd-party components,
11 Entire contents © 2008 Forrester Research, Inc. All rights reserved.
geography, etc…
Most Developers Are Building Web Apps.“What types of new custom applications are your firm’s developers building”
(multiple responses accepted)
12 Entire contents © 2008 Forrester Research, Inc. All rights reserved.
2008 Base: 977 platform software decision-makers at North American and European enterprises and SMBs
New trends make optimal user i i t texperiences even more important
• The Millennial generation expects more– 18 to 27 year olds– The tech-savvy generation
• Web site redesigns– RIA
• More Web sites are composed of 3rd-party componentsVi t li ti f i f t t i l di l d• Virtualization of infrastructure including cloud
• Mobile
13 Entire contents © 2008 Forrester Research, Inc. All rights reserved.
Agenda
• Why your customers won’t wait?
Wh t i li ti f t ti• What is realistic performance testing• Recommendations
14 Entire contents © 2008 Forrester Research, Inc. All rights reserved.
“Realistic testing” means testing from your customer’s point of viewcustomer s point of view
• Where they do it– LA, Edmonton, London, Bangkok, Des Moines,
Miami, Cannes, Dubrovnik, Osaka, Pusan
Wh h d• What they do– Key tasks, transactions, and business processes
• How they do it– Browsers galore, mobile devices
• When they do it (and how frequently)– Peak times, spikes
15 Entire contents © 2008 Forrester Research, Inc. All rights reserved.
p
The user’s distance from the content ttmatters
16 Entire contents © 2008 Forrester Research, Inc. All rights reserved.
Browsers render differently
17 Entire contents © 2008 Forrester Research, Inc. All rights reserved.
Source: Actual Experience XF Real-User Monitoring Data; Gomez, Inc.
What features should a realistic web performance testing platform have? It should:performance testing platform have? It should:
• Test from the end user’s point of view– Different locations
– Different browsers
– Different loads
– Different tasks
• Be easy for developers, testers and business people to use.
• Identify and resolve problems along the entire web application delivery chain.
18 Entire contents © 2008 Forrester Research, Inc. All rights reserved.
Agenda
• What do customers want?
Wh t i li ti f t ti• What is realistic performance testing
• Recommendations
19 Entire contents © 2008 Forrester Research, Inc. All rights reserved.
Recommendations (1 of 2)• Design a test plan based on how real users interact with your web application
• Use testing tools that allow you to test from your user’s point of view:
Wh Wh t H Wh– Where, What, How, When
• Test the entire Web application
• Democratize testing: – Build a test team that includes anyone who has a
stake in the outcome
20 Entire contents © 2008 Forrester Research, Inc. All rights reserved.
Recommendations (2 of 2)• Testing early to nip problems in the bud
• Test often to maintain a consistently supreme user experience
• Follow Blazing Fast Web Performance Best Practices to keep your Web site humming
21 Entire contents © 2008 Forrester Research, Inc. All rights reserved.
In this fiercely competitive environment, realistic web performance testing is the
right answer.
Thank you
Mike Gualtieri
1 617 613 6145+1 617.613.6145
www.forrester.com
23 Entire contents © 2008 Forrester Research, Inc. All rights reserved.
Gomez Profile
▪ First SaaS Web Application Experience Management Suite▪ Headquarters pe e ce a age e t Su te
▪ Web Performance and Load Testing
▪ Cross-Browser Testing
▪ Lexington, MA
▪ Global Offices Cross Browser Testing
▪ Web Performance Monitoring
▪ 80 000+ internet experience
▪ UK
▪ Germany ▪ 80,000+ internet experience measurement points in 162 countries
Ge a y
▪ Switzerland
▪ China▪ Over 2,500 Customers Worldwide
▪ 12 out of 20 most visited US Websites tili G
24
utilize Gomez
▪ 300 Employees
Today’s Testing Challenges
Never Enough Time Small QA Teams
25Testing the End-User PerspectiveCatching All the Problems
When To Load Test
Launching New Apps/Features Deploying New Infrastructure
26Penetrating New MarketsMajor Marketing Campaigns
Web Application Delivery In A Web 2.0 World
▪ Web applications ppincreasingly assembled at browser level▪ 3rd party
performance from ads, videos, shopping cart vary widely cart vary widely under load
▪ Content Delivery Network configuration and effectiveness impact
d i
27
end-user experience
Load 2.0 Testing Solutions
Overcome Limitations of 1.0 Testing Approaches
▪ Find and fix problems across the entire web application delivery chain▪ Inside and outside the firewall▪ Inside and outside the firewall
▪ Flexible and easy to useFlexible and easy to use▪ On-demand – no installation required▪ Highly scalable to accommodate
peak loadspeak loads▪ Supports Agile development
28
▪ Test from the end-user perspective
Ensure Web Experiences Scale Under Load With Reality Load XF
Deploy web applications and ▪ Deploy web applications and infrastructure with confidence
▪ Combine high volume web application load testing with broad geographic experience testing
▪ Ensure on-demand testing gflexibility
▪ Detect failures in user experience not caught by traditional load not caught by traditional load testing
▪ Testing across the entire web li ti d li h i
29
application delivery chain
Critical To Test Across Entire Web Application Delivery Chain
Key Load Testing Metrics• Success rate of end-user workflow• Response time of end-user workflowResponse time of end user workflow• Quality of end-user workflow delivery
30
Load Testing 2.0Performance Testing For A Web 2.0 World
▪ Self-service testing gives you control to test as neededcontrol to test as needed
▪ Test internal & external web application components to ensure transactions perform ensure transactions perform under load
▪ Test cloud-based applications & servicesservices
▪ SaaS based Load Testing 2.0 solutions enable on-demand, scalable testing without
C bi hi h l g
associated hardware investment & maintenance costs
▪ Test the full web application
Combine high volume application load testing with broad geographic
31
ppdelivery chain all the way to your end users’ desktops
experience testing
Critical To Test Across Entire Web Application Delivery Chain
22 33
11
Ensuring Performance Of Login Process
Company• Online presence for a popular television show
11
p p p
Situation• Following episodes of the television show the web site
sees high traffic spikessees high traffic spikes
• Goal was to achieve 1500 logins per minute
• Load tested database optimizations aimed to improve performance in anticipation of another large traffic spikeperformance in anticipation of another large traffic spike
33
Application Bottleneck Causes Immediate Response Time Issue
• As users were added the• As users were added, the response time of step 3 (the login) climbed immediately
• The test bottlenecked at 160 logins per minute (Goal 1500)
• But quickly dropped off as users received server errors
• New login query was not optimized and was bottlenecking the database
’ CPU
34
servers’ CPUs
Ensuring Performance Of E-Commerce Transaction
22
Company• Online Retailer
Situation• Typical e-Commerce type application
S l 3 d P ti i l d i• Several 3rd Parties now involved in serving up key content
• Goal was to validate performance of entire application
Response Times Rise Due To Third-Party Object Error
The load increases throughout the test
The transaction rate increases and then falls off as response times climb
Errors are seen, all on a 3rd party object
3 d t h d i ffi i t f ll d d li ti• 3rd party hardware was insufficient for overall demands on application• Based on SLAs 3rd party had to improve performance to get paid
Ensuring Performance In Target Markets
33
Company• Regional Online News Source
Situation• Began testing for the 2008 election season• Goal was to validate performance of overall
application, and specifically for users in 2 key geographies
• Testing started with the homepage
No Performance Issues Detected From Data-Center
There was only Increase and hold load and not exceed response times of 4 seconds and Success Rate of 99%
Page response
1 page error and 11 errors total out of 60000+ transactions
Rate of 99%
Page response times stayed under 4 seconds, outside of one brief blip
transactions
p
By traditional test standards the test passed
Performance Issues Detected From Real User Desktops
Key geographies for this customer are New York and Pennsylvania
Last Mile data showing substantial number of measurements greater than 4 seconds
Performance Issues Detected For Target Market End Users
• Last Mile Users exceeding the 4-second goal – 29%
• Overall availability was 96.3%
• Availability for Last Mile UsersAvailability for Last Mile Users exceeding 4 seconds – 87%
• Last Mile Users reporting 10+ gsecond response times – 8%
B th t d d th t t f il dBy these standards the test failed
• Cutting 1 5 to 2 sec off the response time for the page would move• Cutting 1.5 to 2 sec. off the response time for the page would move another 16% of the Last Mile users into the 4 sec. response time range
Gomez Offers Full Lifecycle SolutionsFor Web Experience Management
“Gomez has made spectacular progress in the last 18 months…to
41
p p gbecome the leader in web experience management.”
Jean-Pierre Garbani, Vice President, Principal Analyst, Forrester ResearchTech Horizons: Evaluating Gomez’s Web Experience Management Services, August 2008
Find Out How Gomez Can Help You
Some of Our CustomersEnsuring great experiences in a Web 2.0 world requires a new approach to web application load testing Reality Load XFapplication load testing - Reality Load XF
Reality Load XF• Self-service or full turnkey solution with Gomez
P f i l S i ff iProfessional Services offerings • Tests outside-in, with drill down to web application
components• SaaS with no investment or maintenance costsSaaS with no investment or maintenance costs
and rapid payback• Full desktop browser testing across globally
distributed geographies
Learn More:Reality Load XF Solution Briefhttp://www.gomez.com/pdfs/SB_RealityLoad.pdf
42Speak To A Gomez Solutions ExpertToll Free: 877.372.6732