CS 5150 1

CS 5150 Software Engineering

Lecture 23

Testing Large Systems

Guest Lecturer: Stephen Purpura

CS 5150 2

Testing Begins on Day 1

A professional project team begins with:

(a) An idea (usually mused over for a while between friends)

(b) A program/product manager

(c) A development manager

(d) A test manager

Today, we’re going to focus on how the Test Manager thinks

CS 5150 3

Good Test Managers

• Before the project starts, they build the first draft of the Test Exit Criteria which answers:

“How do we know when we’re done?”

• The Test Manager:

– Drives the testing framework and approach

– Is a great communicator

– Understands the value of defect-fix-build-retest cycles that work like clockwork. THINK BOTTLENECKS.

CS 5150 4

Example of a large scale system test: a browser

At one point, the second largest automated software testing infrastructure in commercial software development was used to test Internet Explorer.

The largest tested Windows.

Why so many computers?

CS 5150 5

Different Testing Methodologies

• In large projects there are many stakeholders.

• Usually, they believe in different schools or methodologies.

• You need to balance this from the start.

• An example of a methodology: Agilehttp://pettichord.com/http://www.io.com/~wazmo/papers/four_schools.pdf

• Five different schools (from pdf above):

– Analytic, Standard, Quality, Context-Driven, Agile

CS 5150 6

A Test Pass

A Test Pass: all of the actions involved in assessing whether a software system meets its requirements.

This definition implies:

• All of the requirements are expressed as “tests” (usually with instrumented indicators)

• The team documents all of the steps required to run the tests, so that the test pass can be replicated.

• The team runs the test pass multiple times. Else why write it all down?

CS 5150 7

Automated Testing Methods

Two different types of

(a) Scripted testing

• Programs that manipulate internal or external interfaces

(b) “Screen Scrapers”

• Webload (http://www.webload.org) can automate tests by capturing mouse movements, keyboard events, etc.

Both methods can be “fed” into scripted load testing solutions like WebLoad.

CS 5150 8

The Test Harness

• Test Harness: the program that controls the automated testing scripts.

– Usually first built for “smoke testing”

– Expands to replicate the maximum required load (simultaneous transactions, MTBF, more scenarios)

• Smoke Testing: the minimal set of tests one runs to make certain that daily check-ins to the source control system didn’t break critical functionality in the daily build.

– I integrate the smoke test into the “make” command

CS 5150 9

Scenarios Make Great Test Cases

• Success and failure scenarios and use cases are the foundation of great testing systems.

• Over the course of product development, you usually uncover additional scenarios by watching people break the system.

• How does this happen …

CS 5150 10

Eating your own Dog Food

• “Eating your own dogfood” refers to a situation when a team decides that it will use the software it is developing to conduct its daily business and operations.

• If people use the system every day as part of their work, they find problems with it. These problems frequently become new scenarios for the test infrastructure.

• With operating systems and browsers, this is easy. With IT systems (like human resource systems), this is less possible. But get creative.

CS 5150 11

Additional Sources of Test Scenarios

• Ergonomics• Usability labs• In-home testing• Customer Support• Consulting• Sales Engineers• “Mail a PC” to family members program• Drive to Fry’s and buy one of everything• Executive dogfood• Executive Mission Statements• Other

CS 5150 12

Dealing with Mixed User Interface Feedback

• Market segmentation (try dividing the feedback by demographic groups to look for patterns)

• Example 1: Men over age 65; Income < Poverty Line

• Example 2: Ages 20 – 24; college educated;

– “Interface is not cool”

• Ask questions like:

– Why is this group buying the product?

– Which demos like the product and why?

– Is their issue related to the core “buying story”?

CS 5150 13

Beta Testing

• Beta Testing: early release of a product for the purpose of accumulating feedback that can be integrated into the product.

• Validates requirements

• Uncovers buying issues

• Generates testing scenarios

• Users have a lot of diversity in both their uses and their environments. Exploit the diversity to get better data about the coverage of your test pass.

CS 5150 14

Permutations and Combinations

• How many versions of Microsoft Windows are there?

– 7? 35? 1100?

• A version is a build number plus an install set:

– Windows XP SP 2 with 7 of the 49 security patches

– Win64 vs. Win32

– Office installed vs. not installed

– IDE v SCSI v SATA drivers

– Video adapter driver

– Etc.

CS 5150 15

Durability

• How long should you make the tests run?

– Once?

– Multiple times?

– A number calculated based on analysis?

• Determining MTBF is important because it is a measure of resilience (frailty). These tests frequently insert random events in the system as a part of the test harness.

CS 5150 16

Acting on the Results

• Once you run the test pass, what do you do?

– One bug for each failure

• Bug counts go up because

– Test coverage increased

– Code quality went down

– Major regression

– Code base change that requires change in the test pass

• Bug counts go down because:

– Test team is not testing or losing creativity

CS 5150 17

End Game

• The End Game: the process of balancing feedback from sales, marketing, executives, customers, engineering, testing, etc.

• Usually involves:

– A stabilization of bug find/fix ratios

– A code freeze

– A full test pass and subsequent analysis

– Discussions among the stakeholders that begin with:

Who would it effect if corporate customers couldn’t …

CS 5150 18

How to Decide When to Release

Imagine this scenario. What would you do?

The browser has a bug which causes rendering to be off by 3 pixels for certain combinations of tags within <DIV>s

The code works correctly on Windows Vista but not Windows 7.

Fixing the bug would require adding 2,000 lines of code to the rendering engine

Failure to fix will cause W3C standards advocates to complain.

CS 5150 19

How to Decide When to Release

Imagine this scenario. What would you do?

3 months prior to the scheduled release, you are in a code freeze.

Privacy advocates push Congress to pass legislation that would disable all cookies in the browser. The legislation passes.

The legislation makes it illegal to sell consumer products or publish web sites that use cookies.

CS 5150 20

How to Decide When to Release

Imagine this scenario. What would you do?

The Final Four is in two weeks. You run ESPN’s web site development and the bracket downloader fails on the version of Windows 7 systems that was released yesterday.

CS 5150 21

Hiring a Test Manager

Test Managers are different

(a) Some are good at hiring testers

(b) Some are good at thinking through business process

(c) Some have deep technical background in important areas

(d) Some have relationships with important players

Test Managers need to be able to articulate arguments in terms of the requirements. They also need to be able to argue for the importance of their employees during personnel reviews.

CS 5150 22

Thanks and Questions

• Thanks to the following people for assistance:

– Joel Maher (a great test engineer that I’ve worked with)

– Bill Walrond (a great PM who provided feedback)

• Questions?