CONFIDENTIAL and not for reproduction without prior written consent. © of the Sporting Exchange Limited.

London Web Meetup – Dec 2011Continuous Integration – A Performance Engineers Tale

CONFIDENTIAL and not for reproduction without prior written consent. © of the Sporting Exchange Limited.

2

“WE WILL DO CONTINUOUS INTEGRATION TESTING”1128th commandment - The Management

CONFIDENTIAL and not for reproduction without prior written consent. © of the Sporting Exchange Limited.

INITIAL THOUGHTS

•Automate all of our Performance Tests

•Add tests to the Build process

3

CI

CONFIDENTIAL and not for reproduction without prior written consent. © of the Sporting Exchange Limited.

WHY IT WONT WORK

•Performance testing needs a functional application

•The length of time needed to gather significant amount of data points

•Performance testing requires analysis of results to find meaningful conclusions

4

CONFIDENTIAL and not for reproduction without prior written consent. © of the Sporting Exchange Limited.

MORE INFORMATION

5

we'll receive immediate

feedback

continuous integration (CI) implements continuous processes of applying quality

control

For a build to be self-testing the failure of a

test should cause the build to fail.

CONFIDENTIAL and not for reproduction without prior written consent. © of the Sporting Exchange Limited.

GOALS FOR PERFORMANCE

•Measure Performance as part of every build

•Compare between builds (regression)•Alert if performance deviates from normal

•Fail the build if performance does not meet SLA’s

•Automatic process

6

CONFIDENTIAL and not for reproduction without prior written consent. © of the Sporting Exchange Limited.

7

Build

Delivery Team

CONFIDENTIAL and not for reproduction without prior written consent. © of the Sporting Exchange Limited.

8

Build

Environment

Delivery Team

Deploy

Ready for

Test?

CONFIDENTIAL and not for reproduction without prior written consent. © of the Sporting Exchange Limited.

9

Build

Environment

Performance Measurement

s

Perf Trends

Delivery Team

Deploy

Ready for

Test?

CONFIDENTIAL and not for reproduction without prior written consent. © of the Sporting Exchange Limited.

10

Build

Environment

Performance Measurement

s

Perf Trends

Delivery Team

Alert

Deploy

Fail Fail to meet

SLA’s?

Deviation from

normal?

Ready for

Test?

CONFIDENTIAL and not for reproduction without prior written consent. © of the Sporting Exchange Limited.

11

Build

Environment

ContinuousLoad

Injection

Production logs

Performance Measurement

s

Perf Trends

Delivery Team

Usage Profile

Alert

Deploy

Fail Fail to meet

SLA’s?

Deviation from

normal?

Ready for

Test?

CONFIDENTIAL and not for reproduction without prior written consent. © of the Sporting Exchange Limited.

12

Build

Environment

ContinuousLoad

Injection

Production logs

End User Response

Perf Trends

Delivery Team

Usage Profile

Alert

Deploy

Service/API response

Measurements

System monitors

Fail Fail to meet

SLA’s?

Deviation from

normal?

Ready for

Test?

CONFIDENTIAL and not for reproduction without prior written consent. © of the Sporting Exchange Limited.

13

Build

Environment

ContinuousLoad

Injection

Production logs

End User Response

Perf Trends

Perf Details

Delivery Team

Usage Profile

For Each Data Point

Alert

Deploy

Service/API response

Measurements

System monitors

Fail Fail to meet

SLA’s?

Deviation from

normal?

Ready for

Test?

CONFIDENTIAL and not for reproduction without prior written consent. © of the Sporting Exchange Limited.

14

“THE POINT OF TESTING IS TO FLUSH OUT, UNDER CONTROLLED CONDITIONS, ANY PROBLEM THAT THE SYSTEM WILL HAVE IN PRODUCTION. A SIGNIFICANT PART OF THIS IS THE ENVIRONMENT WITHIN WHICH THE PRODUCTION SYSTEM WILL RUN. IF YOU TEST IN A DIFFERENT ENVIRONMENT, EVERY DIFFERENCE RESULTS IN A RISK THAT WHAT HAPPENS UNDER TEST WON’T HAPPEN IN PRODUCTION”

MARTIN FOWLER

CONFIDENTIAL and not for reproduction without prior written consent. © of the Sporting Exchange Limited.

15

Build

Production Environment

End User Response

Perf Trends

Perf Details

Delivery Team

For Each Data Point

Alert

Deploy

Service/API response

Measurements

System monitors

Fail Fail to meet

SLA’s?

Deviation from

normal?

Ready for

Test?

CONFIDENTIAL and not for reproduction without prior written consent. © of the Sporting Exchange Limited.

16

Build

Environment

ContinuousLoad

Injection

Production logs

End User Response

Perf Trends

Perf Details

Delivery Team

Usage Profile

For Each Data Point

Alert

Deploy

Service/API response

Measurements

System monitors

Fail Fail to meet

SLA’s?

Deviation from

normal?

Ready for

Test?

CONFIDENTIAL and not for reproduction without prior written consent. © of the Sporting Exchange Limited.

17

Build

Environment

ContinuousLoad

Injection

Production logs

End User Response

Perf Trends

Perf Details

Delivery Team

Usage Profile

For Each Data Point

Alert

Deploy

Service/API response

Measurements

System monitors

Fail Fail to meet

SLA’s?

Deviation from

normal?

Ready for

Test?

LoadRunner

CONFIDENTIAL and not for reproduction without prior written consent. © of the Sporting Exchange Limited.

18

Build

Environment

ContinuousLoad

Injection

Production logs

End User Response

Perf Trends

Perf Details

Delivery Team

Usage Profile

For Each Data Point

Alert

Deploy

Service/API response

Measurements

System monitors

Fail Fail to meet

SLA’s?

Deviation from

normal?

Ready for

Test?

Keynote

CONFIDENTIAL and not for reproduction without prior written consent. © of the Sporting Exchange Limited.

19

Build

Environment

ContinuousLoad

Injection

Production logs

End User Response

Perf Trends

Perf Details

Delivery Team

Usage Profile

For Each Data Point

Alert

Deploy

Service/API response

Measurements

System monitors

Fail Fail to meet

SLA’s?

Deviation from

normal?

Ready for

Test?

Dynatrace Ajax Premium

CONFIDENTIAL and not for reproduction without prior written consent. © of the Sporting Exchange Limited.

ADVANTAGES

•Allows standard performance regression testing to be fully automated, freeing up time for more complex performance work

•Separating load injection from performance measurements enables performance to be included in build cycle

•Provides immediate feedback to delivery teams20

CONFIDENTIAL and not for reproduction without prior written consent. © of the Sporting Exchange Limited.

NOT IN SCOPE

•Capacity Testing•Soak Testing•Failure Testing•Stress Testing

21

Still Plenty of Traditional work to do

CONFIDENTIAL and not for reproduction without prior written consent. © of the Sporting Exchange Limited.

THINGS I LEARNT

•Keep an open mind to new stuff

22

CONFIDENTIAL and not for reproduction without prior written consent. © of the Sporting Exchange Limited.

THINGS I LEARNT

•Keep an open mind to new stuff•Give it more than 2 minutes of thought

23

CONFIDENTIAL and not for reproduction without prior written consent. © of the Sporting Exchange Limited.

THINGS I LEARNT

•Keep an open mind to new stuff•Give it more than 2 minutes of thought•Get past the reasons it won’t work

24

CONFIDENTIAL and not for reproduction without prior written consent. © of the Sporting Exchange Limited.

THINGS I LEARNT

•Keep an open mind to new stuff•Give it more than 2 minutes of thought•Get past the reasons it won’t work•Use Google

25

CONFIDENTIAL and not for reproduction without prior written consent. © of the Sporting Exchange Limited.

THINGS I LEARNT

•Keep an open mind to new stuff•Give it more than 2 minutes of thought•Get past the reasons it won’t work•Use Google•Design for the ideal, compromise later

26

CONFIDENTIAL and not for reproduction without prior written consent. © of the Sporting Exchange Limited.

THANK YOU