Upload
stephen-thair
View
1.407
Download
1
Tags:
Embed Size (px)
DESCRIPTION
Presentation given at the 2009 TestExpo in London www.testexpo.co.uk
Citation preview
Commercial in Confidence - © Copyright 2008 NCC Group plc - all rights reserved
SERITI CONSULTING
Load Testing Case Study
Some hints and tips on load-testing your website...
1
Commercial in Confidence - © Copyright 2008 NCC Group plc - all rights reserved
Introductions
Before you start
Designing the test
On the Day…
Analysing the results and taking action
The vendor bit…
Questions?
2
Agenda
Commercial in Confidence - © Copyright 2008 NCC Group plc - all rights reserved
SERITI CONSULTING
Before you start
Working out what to test and how much to test it
5
Commercial in Confidence - © Copyright 2008 NCC Group plc - all rights reserved
New website still under development?
Existing website about to head into a business peak period?
New marketing campaign about to launch?
The Met Office is predicting snow...
6
Why do you want to load test?
Commercial in Confidence - © Copyright 2008 NCC Group plc - all rights reserved
“Load test” vs. “Performance Test”
Performance vs Load Test?
This affects:How you will do itWhat you will do during the testWho needs to be involvedWhat information you record!
7
Commercial in Confidence - © Copyright 2008 NCC Group plc - all rights reserved
Load vs. Performance?
Load Test = "Black Box"I only care if the whole system can handle the
loadIf it can't I probably don't have a lot of clues as
to why not...
Performance Test = "White Box"Looking inside the App and Infrastructure
architecture to identify the key elementsDesigning tests to stress each elementMonitoring those key elements during the test
8
Commercial in Confidence - © Copyright 2008 NCC Group plc - all rights reserved
Test In-house or use 3rd party?
Project triangle decisionTimeMoneyScope
Resources required...Testing software e.g. LoadRunner, Selenium, Site ConfidencePeople to script itHardware & network to generate the loadAccessibility of test environments (or use Prod)Reporting interface to get the right stats & graphsPeople to interpret the results...
9
Commercial in Confidence - © Copyright 2008 NCC Group plc - all rights reserved
SERITI CONSULTING
Designing the test
Working out what to test and how much to test it
10
Commercial in Confidence - © Copyright 2008 NCC Group plc - all rights reserved
Decide on your critical user journeys
Home page
Search pages
Details pages
Upload/download
Checkout/e-commerce
SEO Crawl – Google can hammer sites!
12
Commercial in Confidence - © Copyright 2008 NCC Group plc - all rights reserved
Set your targets
How many transactions/sec?
How many concurrent users?
What is the mix of the user journeys?Don't underestimate the impact of concurrence...
What's your "safety margin"We used 40%
13
Commercial in Confidence - © Copyright 2008 NCC Group plc - all rights reserved
TIP - Web Analytics is your BFF
Leverage your site metrics!Google Analytics, WebTrends or Omniture etc
Use the information to identify your peak loads across different user journeys
It is very common for people to test their sites at very unrealistic rates... Or focus on the wrong parts…• Don’t forget deep linking!
14
Commercial in Confidence - © Copyright 2008 NCC Group plc - all rights reserved
TIP – Management Speak..
The Management tends to ask “how many users will the site support”…
How many users do what, how often?Users on playing online poker are different to users reading
“The Register”…
I find it best to talk about “page impressions” since that is a metrics they are probably familiar with by now…
15
Commercial in Confidence - © Copyright 2008 NCC Group plc - all rights reserved
Concurrent users & “think time”
Take the number of page impression per hour
Divide by 3600
That gives you the page impressions ("transactions") per second target
Then take the number of "concurrent users" and divide by the "think time" (dwell time) to give you transactions a second1200 users with a 60s think time = 20 transactions /sec
600 users with 30s think time = 20 transactions /sec
300 users with 15s think time = 20 transactions /sec
100 users with a 5s think time = 20 transactions /sec
16
Commercial in Confidence - © Copyright 2008 NCC Group plc - all rights reserved
Scripts Profiles Scenarios
Design the test…
Scripts - the actual user journey (“clicks”)
Profiles - how many, how fast, how long etc
Scenarios - combining profiles to match the traffic patterns on your site…
The “mix” of user journeys at any one time
17
Commercial in Confidence - © Copyright 2008 NCC Group plc - all rights reserved
What does a Profile look like?
19
Commercial in Confidence - © Copyright 2008 NCC Group plc - all rights reserved
3 Quick Tips…
Baseline your network firstDo a “GET” for a 100K JPEG and see how many you get before you
saturate your network
Do “single profile” scenarios firstWork from simple to complexTest each element before you try combinations
Think about “keyhole” tests to isolate specific application functionsRather than simulate the “click by click” of the user just “POST” (form
submit) directly to the siteThis enables you to focus on the app performance rather than the UI
elements (you need to do both though!)
21
Commercial in Confidence - © Copyright 2008 NCC Group plc - all rights reserved
One more tip…
DON’T FORGET YOUR BATCH PROCESSESHow does the system perform during…
• Backups• Re-indexes• ETL activities• Any other batch process that stresses the system…
Many a site dies when these processes run
And if you are truly 24x7 online – when is your window?
22
Commercial in Confidence - © Copyright 2008 NCC Group plc - all rights reserved
On the day – running the test!
Who does what, where & when?
23
Commercial in Confidence - © Copyright 2008 NCC Group plc - all rights reserved
Plan out the timetable!
Make the most of your testing window…
Plan carefully but BE REALISTIC!
Allow a lot of time for review between each scenario run...
You WILL change your plans on the day...Be flexible!
Get the right people – Operations, QA, Network, Database etc
24
Commercial in Confidence - © Copyright 2008 NCC Group plc - all rights reserved
On the day...
DON’T - Suddenly remember that you forgot to tell your hosting provider...And watch their IDS system think it was a DDOS attack and drop all
your packets...
DO – keep an eye on the time (and ensure that time is sync’s across the systems)You will want to correlate events!
DO – use a meaningful name for each run and add commentsSo you know how many users are added, over what ramp up period,
and run for how long at a glance
DO – think about cleaning up after you! If you have been running against Prod you might have created a lot of
dummy transactions (and your web logs will be HUGE!)
25
Commercial in Confidence - © Copyright 2008 NCC Group plc - all rights reserved
SERITI CONSULTING
Analysing the results and taking action
Data -> Information -> Knowledge -> Action
26
Commercial in Confidence - © Copyright 2008 NCC Group plc - all rights reserved
Did you hit your targets?
Where there any odd peaks/troughs?
Was “mixed scenario” performance different?
Was performance consistent across test runs?Did your releases make things better (or worse)?
What was the Max/Mean/Min valuesAnd what was the variance?
Lots of questions…
27
Commercial in Confidence - © Copyright 2008 NCC Group plc - all rights reserved
Before and After a Network upgrade…
28
Commercial in Confidence - © Copyright 2008 NCC Group plc - all rights reserved
Lower & longer is better
29
Commercial in Confidence - © Copyright 2008 NCC Group plc - all rights reserved
Faster response, with more users…
30
Commercial in Confidence - © Copyright 2008 NCC Group plc - all rights reserved
Use the performance stat (if you gathered any) to isolate the weak linksUpgrade/re-factor & re-test
Use the (bad) stats to get more budget to fix the problem…
Use the (good) stats to get more a bonus because you did such a good job…
Taking action…
31
Commercial in Confidence - © Copyright 2008 NCC Group plc - all rights reserved
SERITI CONSULTING
The vendor bit…
Over to James…
32
Commercial in Confidence - © Copyright 2008 NCC Group plc - all rights reserved
Site Confidence Clients…
34
Commercial in Confidence - © Copyright 2008 NCC Group plc - all rights reserved
Talk to Site Confidence on their stand at the TestExpo after the presentation or if you would like a copy of the presentation
Email James – [email protected]
Visit their website - www.siteconfidence.com
Contact Details – Site Confidence
35
Commercial in Confidence - © Copyright 2008 NCC Group plc - all rights reserved
If you have any questions around web hosting (IIS, Cloud)Performance tools (Appmetrics, Avicode)Operations & ITIL (service-now.com)Configuration Management (Configuresoft ECM)Near-shore development, functional test (Endava)
Email me – [email protected]
Connect with me on LinkedInhttp://www.linkedin.com/in/stephenthair
Contact Details – Seriti Consulting
36
Commercial in Confidence - © Copyright 2008 NCC Group plc - all rights reserved
Any Questions?
37