Realizing Continuous Performance Management

Realizing Continuous Performance ManagementSteven HainesQuest SoftwareJava Domain ExpertFebruary 14, 2008

Your logo here

Agenda

State of the Market Cost of Mistakes Test-Driven Development Continuous Integration Continuous Performance Management CPM in Practice Business Value Next Steps… Q&A

Industry Experts Say…

Forrester Research reports nearly 85 percent of companies with revenue of more than $1 billion reported incidents of significant application performance problems. Survey respondents identified the architecture and deployment as the primary causes of these problems.

-Forrester Research

Medium-sized businesses* are losing an average of 1% of their annual revenue, or $867K, to downtime. Application outages and degradations are the biggest sources of downtime, costing these companies $213K annually.

-Infonetics Research

*101 to 1000 employees

Impact of Poorly Performing Applications

Business-to-consumer

− Direct loss of customer revenue

− Loss of confidence

Business-to-business

− Loss of reputation

− Potential loss of business relationships

Intranet

− Loss of productivity

− Longer time-to-market = loss of competitive advantage

Cost of Mistakes

Performance problems found later in production cost almost 10x more to fix… Test Early, Test Often

Why is the Current State of Testing Failing?

Test suites are written to out-of-date artifacts

Test suites are not comprehensive enough

Test suites are not automated

− Not performed regularly

− No regression testing

Test-Driven Development

Test cases must be written prior to writing code

Test cases written by developers

Each code component must include a test suite

Test Suite

TestCode

TestCode

TestCode

Test-Driven Development Continuous Integration

Source codeRepository

Continuous Performance Management

Picture to show profiling

of a test: JProbe or PAS

type of screen

Profile Analyze React

Test-Driven Development

Test-Driven Development Lifecycle

Add a Test

Test Suite

Prove Failure

Add a Test

Test Suite



Prove Failure

Add a Test

Implement New Functionality

Test Suite


Prove Failure

Add a Test

Prove Success


Test Suite

Prove Failure

Re-factorthe Code

Add a Test

Prove Success


Test Suite


Shorter Development Cycles

Limited Debugging

Faster Triaging

Clearly Solved Business Problems

Confidence in Deployment

Business Value:

Continuous Integration

The Challenge

− With complicated applications, integration can take longer than development

The Solution

− Integrate after each code check-in

The Tool

− Continuous Integration Server

Test Suite

TestCode

TestCode

TestCode



CIServer

Build

Test

Publish


Picture to show profiling

of a test: JProbe or PAS

type of screen

Profile Analyze React

Test-Driven Development + Continuous Integration

Continuous Integration Lifecycle

Monitors SourceCode Repository

ContinuousIntegration

Server


Detects New Check-Ins



Server




Compiles Entire Application


Server




Runs Exhaustive Test Suite

• Commit Build• Secondary Builds



Server



Publishes Build Results


Runs Exhaustive Test Suite

• Commit Build• Secondary Builds



Server

Shorter integration times

Integration issues resolved as they occur

Higher quality end product

Business Value:

What is Continuous Performance Management?

Challenge in Performance Testing

− Time consuming and error prone because manual

− Problems discovered at the end of development effort often affect assumptions made at the beginning of a development effort

Requires re-factoring the architecture Solution

− Performance Management in a Continuous Integration Environment

Proactive approach Execute performance tests on every

developer commit− On every code check-in

Continuous Performance Management (CPM)

• Create additional Secondary Builds that implement performance tests

1. Performance Unit Tests

2. Performance Integration Tests• Business case

• Use case scenario

3. Performance Stress/Load Tests

SecondaryBuild 2

SecondaryBuild 3

SecondaryBuild 1

CPM

Test Suite

TestCode

TestCode

TestCode



CIServer

Build

Test

Publish


Profile Analyze Repair

Test-Driven Development + Continuous Integration + CPM

Performance Unit Tests• Memory

– Lingering Object References (memory leaks)

– Object cycling• Performance

– Identify slow running code• Coverage

– Quantify the percentage of classes, methods, lines of code and conditions exercised by unit tests

Performance Integration Tests Performance analysis of a single user

− Identify request response times that exceed service-level agreements

− Triage to offending methods and SQL statements

− Follow requests across tiers (and across JVMs) Identify application problems prior to load

− Find problems in isolation

− Easier to pinpoint root cause

Performance Stress/Load Tests Performance analysis under load (multiple users)

− Identify request response times that exceed service-level agreements

− Triage to offending methods and SQL statements

− Follow requests across tiers (and across JVMs)

− Validate environmental configuration

− Identify resource contentions

Identify scalability problems prior to deployment

− Identify environmental configuration issues

− Resolve scalability and reliability problems

Continuous Performance Management Toolset Continuous Integration Server –

CruiseControl

Build Environment – Apache Ant

Test Bed – JUnit

Load Generator – Apache JMeter

Performance Testing:

− Unit tests: Quest JProbe

− Integration/Load tests: Quest PerformaSure

http://jakarta.apache.org/jmeter/

CPM – Too Arduous a Task?

It is a common misconception that CPM is too risky and time consuming of an undertaking

We’ve gone through painstaking efforts to make CPM as easy as possible to implement

Only requires a handful of build script additions

CPM In Practice – Performance Unit Tests Additions to Ant build script to profile all your JUnit tests

Easy to do – can paste on a single slide



<property name="jprobe.enterprisetools.home“ location="c:\jprobe-ent" />



<path id="classpath.preprocessor">

<fileset dir="${jprobe.enterprisetools.home}" includes="lib/*.jar"/>

</path>



<java classname="com.javasrc.anttools.JProbePreprocessor">

<sysproperty key="src.dirs" value="${src};${src.test}" />

<sysproperty key="jprobe.home" value="C:\Program Files\JProbe 7.0" />

<sysproperty key="jprobe.build.dest" value="${jprobe.enterprisetools.home}\jprobe.xml" />

<classpath refid="classpath.preprocessor"/>

</java>



<import file="${jprobe.enterprisetools.home}\jprobe.xml" />

CPM In Practice – Unit Test Report

CPM In Practice – Unit Test Report

Performance Unit Testing – Quest JProbe

Three analysis enablers:

− Memory

− Performance

− Coverage

CPM In Practice – Integration / Load Tests Additions to Ant build script to launch JMeter Easy to do – can paste on a single slide

<taskdef name="jmeter“ classname="org.programmerplanet.ant.taskdefs.jmeter.JMeterTask"/>



<jmeter jmeterhome="C:\lib\jakarta-jmeter-2.2"

testplan="${basedir}/mytests/MyTest.jmx"

resultlog="${basedir}/mytests/MyTestResults.jtl">

<property name="jmeter.save.saveservice.output_format" value="xml" />

</jmeter>



<xslt in="${basedir}/mytests/MyTestResults.jtl"

out="${basedir}/mytests/MyTestResults.html"

style="${basedir}/mytests/jmeter-results-report.xsl" />

CPM In Practice – Integration / Load Tests Additions to Ant build script to record a performance session<property name="pas.home" location="j:\PerformaSure5.0" />

<target name="pas" description= "Start a PerformaSure recording" >



<exec executable="cmd">

<arg value="/c" />

<arg value="${pas.home}/scripts/nexusctl.cmd" />

<arg value="start-recording" />

<arg value="-user" />

<arg value="user" />

<arg value="-l" />

<arg value="30m" />

<arg value="-t" />

<arg value="10s" />

<arg value="-f" />

</exec>

</target>

Performance Integration/Load Testing – Quest PerformaSure

Quest Management Server

Tag-and-Follow™

Performance Integration/Load Testing – Quest PerformaSure

Business Value Benefits

− Reduce R&D and QA time in performance testing

− Better performing applications

Results

− Faster time-to-market = more time to add features that give you a competitive advantage over your competition

− Higher quality applications = higher customer satisfaction

Next Steps…

Download one of my whitepapers from www.quest.com under Application Management, solutions for Java:

− Extending the Continuous Integration Environment with Performance Testing (Part 1)

− Achieving Continuous Performance Management (Part II)

− Best Practices in Java Environment Performance Testing

− A Formal Performance Tuning Methodology: Wait-Based Tuning

Sign up for my upcoming “Java Performance Winner’s Circle” webcast series on performance testing, tuning, and management at www.quest.com under Application Management, solutions for Java (starts in March 2008)

Watch my Continuous Integration OnSoftware Video Podcast on www.informit.com (or iTunes)

Read my forthcoming digital shortcut book, Agile Java Development with Test-Driven Development and Continuous Integration, on www.informit.com

Read my latest book, Pro Java EE 5 Performance Management and Optimization, at your favorite book store

http://www.quest.com/

http://www.quest.com/

http://www.informit.com/

http://www.informit.com/

Next Steps cont’d…

Quest Software Product Information:

JProbe – www.quest.com/jprobe

PerformaSure – www.quest.com/performasure Other Resources:

JProbe Live Product Demo – www.quest.com/events

PerformaSure Product Demand – www.quest.com/performasure Contact Quest Software Sales:

www.quest.com/company/contact-us.aspx

http://www.quest.com/jprobe

http://www.quest.com/performasure

http://www.quest.com/events

http://www.quest.com/performasure

http://www.quest.com/company/contact-us.aspx

State of the Market Test-Driven Development Continuous Integration Continuous Performance Management CPM in Practice Business Value Next Steps…

Thank You -- Q&A

Agenda

Documents

Realizing Continuous Performance Management