Enabling Agile Testing Through Continuous Integration

Sean Stolberg, Pacific Northwest National Laboratory

Where We’re Going Today

Introduction - Being thrown into the deep end of Agile Testing

Our Plan – Deciding to Implement Agile Practices

The Problem

The Solution - Continuous Integration (CI)

What is CI?

Implementing CI

Assessing the Team’s CI Practices

Assessing Our New Agile Development Practices and Testing Capabilities

Retrospective

Being Thrown Into the Deep End

Me

Eight years as traditional software tester/QA engineer.

November 2007 – First job as a tester on an Agile team.

First two sprints

Humbling

All the “moves” I knew didn’t work too well.

Kept coming up short on time, test coverage, and accumulation of technical debt.

Re-evaluation

“What I’m doing is not working, so what will?”

Practices We Decided to Implement

1. Define and execute “just-enough” acceptance tests. [1]

2. Automate as close to 100% of the acceptance tests as possible. [2]

3. Automate acceptance tests using a “subcutaneous” test approach with a xUnit test framework. [2]

[1] “eXtreme Rules of the Road: How a tester can steer an eXtreme Programming project toward success”, Lisa Crispin, STQE Jul/Aug 2001

[2] “Testing Extreme Programming”, Lisa Crispin and Tip House, 2003, Addison Wesley

Practices We Decided to Implement (continued)

5. Run all acceptance tests in the regression test suite with the build, daily (at a minimum). [4]

6. Develop unit tests for all new code during a sprint. [5]

7. Run all unit tests with every build. [4]

8. Run multiple builds per day. [4]

[4] “Continuous Integration”, Martin Fowler, http://www.martinfowler.com/articles/continuousIntegration.html

[5] “Code the Unit Test First”, http://www.extremeprogramming.org/rules/testfirst.html

The Problem

We didn’t have an automation framework.

How could we:

Implement our new development practices?

Run all acceptance tests in the regression test suite with the build?

Run all unit tests with every build?

Run multiple builds per day?

The Solution

Continuous Integration

Commonly used by teams already successfully implementing Agile Testing.

Provides framework to implement several of the practices.

We needed to find out more.

What is Continuous Integration?

Describes a set of software engineering practices that

speed up the delivery of softwareby

decreasing integration times

Emerged in the XP community.

Martin Fowler and Kent Beck – helped make it popular.

What CI Might Look Like

1. Automated source repository change detection Change is detected resulting to a chain of events being put

into motion

2. Get latest source

3. Compile (or run through other check; interpreted code)

4. Run unit tests

5. Deploy to test environment

6. Run acceptance tests

7. Send results of testing

8. Publish build if tests pass

Speeding Up the Feedback Loop

Continuous Integration

Compile

Unit Testing

Deploy

Acceptance Testing

Publish build

Notify

Report

Tools Used

• First step in build

automation

• Configuration

• Dual Quad Core Xeon

1.6ghz

• 8GB Ram

• 4 x 146GB 15k RPM

SCSI drives in RAID-5.

• Unified build environment

and administration

• Get build machines out

from under the desks and

into the server room

Virtual Server Host

Virtual Build Server

IF a source code change is detect

• Refresh source code

• Build (ABS using VS2008)

• Clean build

• Build All

• Build Installer

IF no build errors were found

• Prep the test VM for product

install and testing (VBScript

helper script called via STAF)

• Copy install files to test VM

(STAF)

• Install app-under-test on test

VM (STAF)

• Discover and deploy all Unit

and Acceptance tests to test

VM (custom C# helper

app/STAF)

• Execute tests on test VM

(custom C# helper app/STAF)

• Copy test result .xml file(s)

from test VM to build machine

(STAF)

• Process the test result .xml

files (custom C# helper app)

Communication

Send mail with

• test results (PASS or FAIL

with details of failures)

• Link to location of build

• Build logs

• Change logs

Assessing our CI Practices

Maintain a Single Source Repository

Before CI: Seapine Surround SCM source repository.

After CI: No change.

Automate the Build

Before CI: Some partial automation; still very manual. Heterogeneous build environments.

After CI: Yes, with Automated Build Studio (ABS).

Make Your Build Self-Testing

Before CI: Not self testing.

After CI: NUnit framework unit and acceptance tests.

Assessing our CI Practices (continued)

Everyone Commits Every Day

Before CI: Unknown, varied probably.

After CI: We can only hope.

Every Commit Should Build the Mainline on an Integration Machine

Before CI: No, it was not happening.

After CI: Yes, ABS's Continuous Integration Tasks helped us do this.

Keep the Build Fast

Before CI: Multi-pass builds, unordered dependencies.

After CI: 10 to 15 min; refactoring needed sooner than later.

Assessing our CI Practices (continued)

Test in a Clone of the Production Environment

Before CI: Yes, but not automated

After CI: Using clean virtual machine test clients to install and test

Make it Easy for Anyone to Get the Latest Executable

Before CI: Not all projects using the common build repository. Some private file share locations for production code

After CI: All products building to common location now. Build mail contains link to new build location

Assessing our CI Practices (continued)

Everyone can see what's happening

Before CI: Limited to ad hoc e-mails, no web, no reporting, different project worked differently.

After CI: Use ABS's web interface to see the progress of builds, and email build and test status.

Automate Deployment

Before CI: Not being done.

After CI: Yes, automatically deploy the build, then test it.

Assessing Our Agile Practices

Define and execute “just-enough” acceptance tests

Made acceptance test definition a required task during sprint planning.

Developers and customer liked visibility into testing.

Automate as close to 100% of the acceptance tests as possible

Now tried to automate as close to 100% acceptance tests as possible.

Slowed our accumulation of technical test debt.

Assessing Our Agile Practices (continued)

Automate acceptance tests using a “subcutaneous” test approach with an xUnit test framework

NUnit test framework used to automate our acceptance/functional testing using the API of the application.

Acceptance tests versioned code in the repository and possible to run in a daily build.

Run all acceptance tests in the regression test suite with the build, daily (at a minimum)

NUnit acceptance tests now ran with a daily build.

Practice enabled using our new continuous integration system.

Assessing Our Agile Practices (continued)

Develop unit tests for all new code during a sprint

Developers started putting more emphasis on developing unit tests during the same sprint.

Practice encouraged due to ease with which unit tests could now be run with the build.

Run all unit tests with every build

NUnit tests now ran with every build.

Run multiple builds per day

Builds started automatically when changes to the source repository were checked in.

Retrospective

Personal “biggest discoveries”

Testing in sync with development.

Embracing fully automated acceptance tests

Exciting and intimidating.

Hard to convince developers to go through the automation “hump-of-pain”

But it’s worth it.

Devoting personal time toward investigation and proof-of-concept paid off.

Can’t imagine working on a team not using Agile Testing or development

Get “hooked” on injecting testing and quality early into the cycle.

Adds immediate value.

Recommended Reading

Crispin, Lisa, Tip House. Testing Extreme Programming. Addison-Wesley Professional, 2002

Gerard Meszaros, xUnit Test Patterns: Refactoring Test Code.Addison-Wesley Signature Series, 2007

Gregory, Janet, Lisa Crispin. Agile Testing: A Practical Guide for Testers and Agile Teams. Addison-Wesley Signature Series, 2008

References

[1] “eXtreme Rules of the Road: How a tester can steer an eXtremeProgramming project toward success”, Lisa Crispin, STQE Jul/Aug 2001

[2] “Testing Extreme Programming”, Lisa Crispin and Tip House, 2003, Addison Wesley

[3] “Continuous integration”, http://en.wikipedia.org/wiki/Continuous_Integration

[4] “Continuous Integration”, Martin Fowler, http://www.martinfowler.com/articles/continuousIntegration.html

[5] “Code the Unit Test First”, http://www.extremeprogramming.org/rules/testfirst.html

Questions???