Upload
soasta
View
219
Download
0
Embed Size (px)
Citation preview
©2 016 S OA S TA , A l l r i gh t s r es e rv ed .
Tom Chavez @tomchavezMike Ostenberg @mikeostenberg
Load testing for peak season:What to test and how to test
©2 016 S OA S TA , A l l r i gh t s r es e rv ed .
• How fast is your site now?• Measure everything• Too much data? Science!• Prioritize what to fix• Optimize your site• What to test?• Test early and often – shift left and right• How to test?
©2 016 S OA S TA , A l l r i gh t s r es e rv ed . 3
Measure, Optimize, Test – Repeat
How fast is your site now?
©2 016 S OA S TA , A l l r i gh t s r es e rv ed . 5
How fast is your site now?
• Google says 1000ms!
• No single right answer
• Not all pages are equalo Some pages should be faster than others
o The slowest pages might not be the ones to fix first
©2 016 S OA S TA , A l l r i gh t s r es e rv ed . 6
How is your site now?
• mCommerce is globalo US is not the benchmark
• 39% of mobile users
are unhappy with their online experienceso Top reasons: slow pages and site freezes/crashes
• 57% have experienced problems when using mobile
• 3 out of 4 consider mobile slower than desktop
• 46% will not return to a site that performs poorly
• 27% of smartphone users exceed data cap every month
©2 016 S OA S TA , A l l r i gh t s r es e rv ed . 7
When is your Peak Season?
• Peak is year-roundo Super Bowl Sunday – 110 million viewers
o Bi-weekly or monthly payroll processing
o Valentine’s Day, President’s Day, Mother’s Day, Father’s Day
o Quarterly processing and filing
o US Tax filing
o Spring, Summer, Fall & Winter season sales
o July 4th, Christmas in July
o Back to School We are here
o Olympics
o Halloween
o Election Day
o Singles Day – world’s biggest shopping day, $9B in 12 hours
o Black Friday, Cyber Monday and Holidays
Measure Everything
©2 016 S OA S TA , A l l r i gh t s r es e rv ed . 9
Measure Real, Synthetic, and Virtual Users
• Real Users (RUM) provide deep data about your visitorso No other way to know everything about your users on your site
• Synthetic Users tell you how parts of your site are doingo When no real users are visiting, synthetic users are still coming
• Virtual Users tell you how your site will do at scaleo Can’t get a million friends to visit your site at the same time?
• Collect all the data and save it in the cloud!o Storage is cheap in the cloud, plus easier to analyze there
• Add App Performance Management (APM) for a complete picture
The Three Musketeers of Performance
©2 016 S OA S TA , A l l r i gh t s r es e rv ed . 10
Other Things to Measure
• Session pathso How are your users going through your site
• Mobile App Performanceo How are your mobile apps performing?
o What are users doing in your mobile apps?
• Revenueo How much money are you making from each user and where?
• How fast are your competitors’ pages?o You can’t insert code on their sites but you can measure
Your company may be measuring these already
Too much data?Science!
©2 016 S OA S TA , A l l r i gh t s r es e rv ed . 12
Analyze
• Benchmark and baseline your siteo How fast is your site now? How about the competition?
• Correlate performance and outcomeo Is poor performance affecting the business?
• Find root causes for performance issueso Use RUM and APM data
• Fix pages that impact mosto Make a prioritized punch list of the pages to fix
Use the data you’ve collected to gain insights
©2 016 S OA S TA , A l l r i gh t s r es e rv ed . 13
Baseline Your Performance
How is your site doing? How does it compare to the competition?
©2 016 S OA S TA , A l l r i gh t s r es e rv ed . 14
Find Root Causes for Poor Performance
Not all problems are from your app or back end systems
©2 016 S OA S TA , A l l r i gh t s r es e rv ed . 15
Page Visits vs. Performance
©2 016 S OA S TA , A l l r i gh t s r es e rv ed . 16
Page Visits vs. Performance by Device
©2 016 S OA S TA , A l l r i gh t s r es e rv ed . 17
Prioritize What to Fix – Greatest Impact
The poorest performing page might not be the most important page
Fix and Optimize
©2 016 S OA S TA , A l l r i gh t s r es e rv ed . 19
Page Bloat
The average web page is more than 2MB in size
©2 016 S OA S TA , A l l r i gh t s r es e rv ed . 20
Page Bloat
Where are all those kilobytes going
©2 016 S OA S TA , A l l r i gh t s r es e rv ed . 21
Doesn’t Bandwidth Solve the Problem?
©2 016 S OA S TA , A l l r i gh t s r es e rv ed . 22
Comparing Network Performance
©2 016 S OA S TA , A l l r i gh t s r es e rv ed . 23
Comparing Network Performance
©2 016 S OA S TA , A l l r i gh t s r es e rv ed . 24
Doesn’t Bandwidth Solve the Problem?
©2 016 S OA S TA , A l l r i gh t s r es e rv ed . 25
Doesn’t Bandwidth Solve the Problem?
25x Faster Bandwidth != 25x Faster Load Time
©2 016 S OA S TA , A l l r i gh t s r es e rv ed . 26
Prepare for Peak Events
Optimize content for peak load events
• Simplify website contento Keep only the most important content on the page
• Reduce infrastructure requestso Block unnecessary requests, out-of-region visitors
• Leverage historical comparisonso Review past traffic levels to predict future levels
What Should I Test?
©2 016 S OA S TA , A l l r i gh t s r es e rv ed . 28
Creating a Test Plan – Session Analysis
Which paths are the most important to users?
©2 016 S OA S TA , A l l r i gh t s r es e rv ed . 29
Creating a Test Plan – Session Analysis
Which paths are the most important to users?
©2 016 S OA S TA , A l l r i gh t s r es e rv ed . 30
Creating a Test Plan – Peak Load
What was the busiest Day, Hour, Minute, Second?
©2 016 S OA S TA , A l l r i gh t s r es e rv ed . 31
Creating a Test Plan – Page Flows
What were most popular page flows with think times?
©2 016 S OA S TA , A l l r i gh t s r es e rv ed . 32
Creating a Test Plan -
Which paths should I test, what think times, at what load, from where?
©2 016 S OA S TA , A l l r i gh t s r es e rv ed . 33
Network Emulation
• Testing to match realistic situationso Users are not all from one location on a fast network
o Packets get delayed, lost, re-ordered
• Apps perform differently as network conditions worseno Reduced throughput
o Timeouts occur, connections are lost
o Transactions take longer, longer sessions
Adding real network modeling to load and performance testing
Test Earlyand Often
©2 016 S OA S TA , A l l r i gh t s r es e rv ed . 35
Start Testing Yesterday
• How long will it take?o 300 test cases x 2 browsers = 600 cases
o 600 cases x 30-minutes/case = 300 hours
o 300 hours / 6 hours/day = 50 days
o 50 days / 5 people = 10 days
• And for mobile:o 300 text cases x 26 devices = 7800 cases
o 7800 cases x 30 minutes/case = 3900 hours
o 3900 hours / 6 hours/day = 650 days
o 650- days/5 people = 130 days of test – 26 weeks!
And even earlier for Mobile
©2 016 S OA S TA , A l l r i gh t s r es e rv ed . 36
Don’t Wait Til the Last Minute to Test!
©2 016 S OA S TA , A l l r i gh t s r es e rv ed . 37
Test Right and Test Left, even into Production
• Performance Testing in developmento Set performance SLAs for new code and test in development,
same as unit and functional tests which code must pass
• Performance Testing in stagingo Higher loads in near-production systems
• Performance Testing in Productiono Full scale load tests to expected load and beyond
o Find weak and stress points and prioritize fixes into project
Performance Testing is Continuous
©2 016 S OA S TA , A l l r i gh t s r es e rv ed . 38
Test Continuously
Test from Development to Integration to Staging to Production
©2 016 S OA S TA , A l l r i gh t s r es e rv ed . 39
Test in Production
Some issues will only be found at scale load on production systems
Don’t forget your third party providers: CDNs, external scripts
©2 016 S OA S TA , A l l r i gh t s r es e rv ed . 40
Measure Mobile App Performance Under Load
All mobile apps perform well when there is no load
©2 016 S OA S TA , A l l r i gh t s r es e rv ed . 41
Automate Your Testing
Use Continuous Integration tools to automate testing
Demo
©2 016 S OA S TA , A l l r i gh t s r es e rv ed . 43
Now That the Test is Over
• Analyze results for future areas to addresso Weak or stress areas that aren’t likely this time but …
• Identify third parties that were affected by loado Coordinate with suppliers to address issues
o Align schedules or at least be aware of their changes
Even if the test passed, you’re not done
©2 016 S OA S TA , A l l r i gh t s r es e rv ed . 44
Issues Will Happen
• “I am 100% confident that everything we tested will work
just fine.” – Nordstrom Performance Leadero When loads are different than what you modeled, may have
problems
o E.g. At Nordstrom, during a recent anniversary sale, mobile sign-
ins were 9x greater than anticipated
o Used RUM to get to the root cause of the issue
Realize that Issues Will Happen Where You Can’t Predict Them
Q & AResourcesNext Steps
©2 016 S OA S TA , A l l r i gh t s r es e rv ed . 46
Questions, Resources & Next Steps
• Holiday Readinesso soasta.com/holiday-readiness
• The Performance Beacono soasta.com/blogo soasta.com/blog/2015-popular-posts-web-performance
• Upcoming Webinarso soasta.com/webinars: RUM-Based Testing
o Recordings of past webinars
• CloudTest on-Demando soasta.com/cloudtest: SOASTA will build & run tests for you
Resources
©2 016 S OA S TA , A l l r i gh t s r es e rv ed . 47
“The leader in performance analytics.”
www.soasta.com
SOASTA
©2 016 S OA S TA , A l l r i gh t s r es e rv ed . 48
CloudTest
• Load testing from development to productiono Continuous testing at any scale, on-prem or in the cloud
• Real-time analytics with customizable dashboardso Safely test in production with full visibility
• Run tests live or automaticallyo Full hands-on control or hands-off automation
• Full end-to-end viewo Front-end performance to back-end systems
Fast, scalable, affordable load testing for performance optimization