Optimisation Case Study

January 2011

Ged Waring & Perry Dyball

CONFIDENTIAL DRAFT

Headline Results of Project

• Reduced HTTP Requests by 30%

• Reduced Page Load time by between 50% and 70%

• Reduced Page Size by between 22% and 33%

• Reduced Hard Bandwidth requirements by 43%

• Reduced DB CPU requirements by 75%

• Increased Concurrent Users ceiling by 300%

CONFIDENTIAL DRAFT

Presentation Outline

• Origins of Problem

• Why Optimise?

• Project Constraints and Approach

• Measurement Techniques & Tools

• Project Steps

• Wrap Up / Challenges

CONFIDENTIAL DRAFT

Origins of Problem

• Frequent releases with poor measurement andtesting

• Speed to market versus slower engineering process

• Problem compounds & worsens over time

• No single culprit – all tiers of the platform

• Did the problem harm our business?

• In the short-to-medium term – almost certainly not.

• In the medium-to-longer term – yes.

CONFIDENTIAL DRAFT

Why Optimise?

• User Experience / Conversion / Retention

• Increased capacity to serve customers

• Google/SEO

• Return on investment

• Hard bandwidth ceiling and DB Server capacity

• Handle large traffic spikes

CONFIDENTIAL DRAFT

Constraints

• No additional hardware spend

• Continual product / feature development

CONFIDENTIAL DRAFT

Project Approach

• Path of least resistance

• Iterative – Change / Measure / Report / Repeat

• Phase 1 – Fairly Easy

• Infrastructure Configuration / Leverage Existing Resources

• Client Side Optimization

• CDN

• Phase 2 - Harder

• Database Optimisation

• Middle Tier Refactoring / Caching

CONFIDENTIAL DRAFT

Project Approach

Business buy-in

Make a singlechange

Measure itAdapt your plan

Analyseresults

Baseline performance

CONFIDENTIAL DRAFT

Project Approach

• Release Discreet Changes• Script & run tests

• Record metrics

• Collate results

• Report & analyse

• Repeat

• Minimise Other Environment Variables / Changes

• Time of day

• Hardware / Network changes

• Skewed traffic (Ad. campaigns / Extraordinary spikes)

CONFIDENTIAL DRAFT

Tools we used

• HttpWatch Professional (Object Library)• WebPageTest.org• Y-Slow / Firebug / Fiddler• Site Confidence Monitoring Portal

• IDERA (DB Monitoring)

• SQL Profiling (Server Side)• SQL Reporting Services

• F5 Load Balancing Consultants – Quadrant Networks

Note : In our case the new Site Confidence Performance Analyser toolhas replaced HTTP Watch / WebPageTest / Y-Slow

CONFIDENTIAL DRAFT

Phase 1 : Client Side / Page Optimisation

• Areas yielding highest benefit• Compression (check your configuration)

• Compression moved from IIS to F5 Load Balancer

• Object caching at F5 Load Balancer

• Reduction in HTTP Requests / Spriting / File Consolidation

• Image size consistency

• Removal of third party killers (Images/JS)

• Parallelism (CDN)

CONFIDENTIAL DRAFT

Client Side / Page Optimization Results

Home Page Load Time (ms) (2mbps download speed) Empty Cache Primed Cache

Baseline 4,435 3,279

P10 F5 Optimisation 4,145 3,123

P20 10 March (Ptix, Concat) 4,608 2,781

P30 24 March (Spriting and Verisign) 1,804 1,177

P40 24 March (Parallelism) 1,839 1,084

R50 21st April Release 1,393 824

Home Page Load Times (ms)(source: HttpWatch)

0500

100015002000250030003500400045005000

P01 Baseline P10 F5Optimisation

P20 10March (Ptix,

Concat)

P30 24March

(Spriting andVerisign)

P40 24March

(Parallelism)

R50 21stApril Release

Optimisation Phase

Load

tim

e (m

s)

Empty

Primed

CONFIDENTIAL DRAFT

Client Side / Page Optimization Results

Http Requests by Mime * css flash html image javascript redirectGrandTotal

01 Baseline 1 4 1 1 37 13 3 60

02 Post F5 Optimisation 1 4 1 1 37 13 3 60

03 Page Optimisation 10 March (Ptix, Concat) 1 4 1 1 37 12 3 59

04 Page Optimisation 25 March (Spriting and Verisign) 1 4 1 1 23 12 3 45

04 Parallelism 25th March - 4 1 1 21 11 38

05 Release 21st April 4 1 22 10 37

Season PageHttpRequests by Phase/Mime

0

10

20

30

40

50

60

70

01 Baseline 02 Post F5Optimisation

03 PageOptimisation 10

March (Ptix,Concat)

04 PageOptimisation 25

March (Spriting andVerisign)

04 Parallelism 25thMarch

05 Release 21stApril

redirect

javascript

image

html

flash

css

*

CONFIDENTIAL DRAFT

Client Side / Page Optimisation Results

CONFIDENTIAL DRAFT

Client Side / Page Optimisation Results

Page load performanceboth improved and

became more consistent

CONFIDENTIAL DRAFT

Phase 1 : Results

• Page load performance• Page load times reduced by between 38% and 60%

• Bandwidth requirements reduced by 18%

• How long did it take?

• 4 months elapsed.

• Work completed alongside normal product development

CONFIDENTIAL DRAFT

CDN Evaluation Overview

Load time drop Monday to Wednesday when test speed increased from 512kbps to 2mbps

CONFIDENTIAL DRAFT

Phase 2 : What did we do?

• Database• Analyse web page db interaction

• Repeated tracing of all db calls

• Reports on worst performing aspects of the db

• Computing DB CPU per user session

• F5 Load Balancing

• Upgrade internal network layer configuration to 1Gbps

• Upgrade F5 O/S to V10.2

• F5 traffic management rules / log

• CDN

• Non image assets – JS / CSS serve from CDN

CONFIDENTIAL DRAFT

Phase 2 – Why did we do it?

• Identify and remove redundant db calls from web code

• Identify data that was a candidate for caching

• Identify which site features were resource hungry

• Save more bandwidth

• Control traffic at the F5 inreal time

Overflow traffic redirectedand managed in a cloudbased queuing system

CONFIDENTIAL DRAFT

Effect on DB Resource Usage

CONFIDENTIAL DRAFT

Phase 2 Achievements

• Benefits• Capacity increased from 2,100 to 6,300 concurrent users

• 8.3m sessions in October 2010 on 8 web servers

• Queuing capability for another 4,000 concurrent users

• DB resource use reduced by 75%

• Reduce bandwidth demands by further 25%

• Reduce internal network traffic

• Financial Benefit• Single biggest return this year

• How long did it take?• 3 months elapsed

• Work completed alongside normal product development

CONFIDENTIAL DRAFT

Normal Traffic Pattern

CONFIDENTIAL DRAFT

Major On-Sale Traffic Pattern

Traffic spike sustained for long period.Gradual inflow of traffic from queuingsystem until queue expired

CONFIDENTIAL DRAFT

TV Advertising Traffic Pattern

Note 3rd / 4th ad break spikes when wedidn’t run ads

CONFIDENTIAL DRAFT

Wrap Up

• It’s hard work

• You need to do it repeatedly

• You need to be consistent

• It will generate large amounts of data

• Use automation wherever possible

• Expect unanticipated bottlenecks / pinch points

• Bake optimisation into the release process

• It will yield material benefits