Claire J. Caudry - StickyMinds€¦ · Claire J. Caudry is the Manager of Quality Assurance for Perceptive Software Inc. in Shawnee Kansas. She has been in the field of Quality Assurance

BIO PRESENTATION

International Conference On Software Test Analysis

And Review May 14-18, 2007 Orlando, FL USA

T11

5/17/2007 1:30:00 PM

"WHEN THERE IS TOO MUCH TO

TEST: ASK PARETO FOR HELP"

Claire Caudry Perceptive Software

Claire J. Caudry Claire J. Caudry is the Manager of Quality Assurance for Perceptive Software Inc. in Shawnee Kansas. She has been in the field of Quality Assurance for over 12 years working for such companies as Electronic Data Systems and UnitedHealth Group. She is certified in Object Oriented Analysis and Design, Software Quality Assurance and Project Management. Claire was born and raised in Fort Worth TX and graduated from the University of Central Arkansas. She has lived in Kansas City MO for the past 12 years and works with several quality organizations including ASQ, the NIST and the Association for Software Testing.

Copyright 2006 Perceptive Software | The documents you need. Now.

Applying the Paretto Principle to Defect ManagementPresenter: Claire J. Caudry

Presented at the 19th Annual Quality Management Conference

Dallas TX March 2, 2007


Creating vs Problem Solving

“The fundamental difference between creating and problem solving is simple. In problem solving we seek to make

something we do not like go away. In creating, we seek to make what we truly care about exist.”

Peter M. Senge, Creating Desired Futures in a Global Economy, Reflections The SoL Journal, 2003


What’s Paretto Got To Do With It?

● Explaining the title> The Paretto principle says that 80% of defects will be discovered in 20% of the

application. > Taking that a step further, 80% of your time will be spent on 20% of the issues> And taking it to the limit; 20% of those issues will be outside your realm of

prevention or control because they will be environmental in nature. The result of your customers loading another piece of software, an service pack or upgrading a platform version.

● Obligatory Disclaimer> Not everyone will be able to apply these principles> Make sure your company is in agreement before you decide to “scale back” your

testing


Agenda

● Discuss examples of recent environmental challenges● Admit the Obvious – we simply can’t predict or prevent everything

> Define what environments we will test> Define how much we will test on each environment> Prepare to support what we don’t test!

● Clarify the difference between Support and Certify for your customers and yourself> Support is what you say you do> Certify is what you actually do

● Tools to Help Determine What to Test> Collecting customer platforms> Surveying customer base> Web download collection gates> Industry trends

● Options for hitting environments with taking a financial hit● QA team skill sets● When all else fails – be willing and able to fix it in production!


Some Recent Examples


Examples of Recent Issues

● Microsoft Update 921883 > Prevented some Java from launching via MFC> MS “fix” was to have the customer click in the browser window real fast> There was a workaround script posted on the web within days but not from Java

or Microsoft> Why is “just remove the update or load this quick fix” not an option for

Customers?

● A fax board vendor changed their process by adding the boot ROM setup to the fax board firmware so there was no longer a need to “flash” the boards through the application. When the new boards started shipping, the flash command wiped out the boards firmware and rendered it unusable.

> Fax boards run about $10K each and it took 5 boards to figure what was happening

> 3 of those boards belonged to customers who were now not able to process faxes

> Vendor had failed to mention that there was a required code change.


● MS Security Update 912919 exposed print issue

Examples Cont.


When Customers Go Bad

● Customer reported that users could no longer log in after the 250th user, CPU was spiked and our server application would shut down> Customer was on an older version of our app which we knew had

performance issues but…> The application had previously worked fine with no issues and “nothing

had changed” in the customer environment● We put together a task force to work the issue and jumped through

numerous hoops to try to isolate the issue> After 4 weeks we called Microsoft and paid for a SQL consultant to go

onsite and optimize the SQL Database> Result: Within 15 minutes the MS consultant had located a “rogue

process” that was hogging resources. Someone had loaded a 3rd party software application to monitor the SQL server performance which was utilizing all the CPU. Once removed performance returned to normal.


Anyone Care to Share?


Admitting the Obvious


Admitting the Obvious

Depending on what platforms your software runs on, you will simply not be able to test everything fast enough or well enough.

● Step one: Define what you can test> What versions exist today?> What versions are expected for release during your development cycle? Are

there Beta versions that you could test on? > What configurations will you be testing on?

● Step two: Define how much you will test on the things you say you test> Do all configurations require a full test cycle?> Which ones can you cut back on based on risk analysis?

● Step three: Prepare to support all the things you didn’t test!> Do you have all the environments you claim in house?> Do you keep up with all updates, SP’s and patches during and after the test

cycle?


Define What You will TestTe

st

Env

ironm

ent (

s)P

latfo

rm

Rel

ease

Pro

duct

R

elea

se

What versions exist today?•Example timeline for product and Database update testing


Define What You will Test

2500500500500500500Ctree

2500500500500500500Oracle

2500500500500500500SQL

2500500500500500500DB2

15000LinuxAIXSolarisHP/UXWindowsUser Base

1000200200200200200JRE

1000200200200200200Tomcat

1000200200200200200WebLogic

1000200200200200200WebSphere

1000200200200200200OAS

What configurations will you be testing on?• Option 1: Test it all! In this example, you have 200 web app test cases and 500 database specific test cases. To do it all – would require 15,000 test passes!


T-Matrix Analysis

5841914194528OS Totals

2699393245Ctree

74981047Oracle

236236SQL

5131DB2

OS TotalsLinuxAIXSolarisHP/UXWindowsUser Base

88JRE

6281152Tomcat

22WebLogic

312WebSphere

312OAS

78902166Totals

● Option 2: Test around the platforms your customers are running> 1. Using a T-Matrix, identify the environments your customers are currently

running your application on to see where you should place emphasis.


T-Matrix Analysis of Customer Base

5841914194528OS Totals90.41%0.68%3.25%2.40%3.25%

2699393245Ctree41.95%0.51%1.54%0.51%1.54%46.06%

74981047Oracle8.05%1.71%1.37%1.54%12.67%

236236SQL40.41%40.41%

5131DB20.17%0.51%0.17%0.86%

OS TotalsLinuxAIXSolarisHP/UXWindowsUser BaseWindowsHP/UXSolarisAIXLinuxDB Totals

88JRE10.26%10.26%

6281152Tomcat66.67%1.28%1.28%10.26%79.49%

22WebLogic2.56%2.56%

312WebSphere2.56%1.28%3.85%

312OAS2.56%1.28%3.85%

78902166Totals84.62%1.28%2.56%0.00%11.54%

> 2. Calculate the percentages for individual OS’s, DB’s and Web Apps to see where the larger usages occur.


Final Configuration Test Plan

PartialPartialPartialPartialFullCtree

PartialPartialPartialFullOracle

FullSQL

PartialPartialPartialDB2

LinuxAIXSolarisHP/UXWindowsUser Base

PartialJRE

PartialPartialPartialFullTomcat

PartialWebLogic

PartialPartialWebSphere

PartialPartialOAS

> 3. Build your T-Matrix showing full and partial test suites for the various configurations used most by your customers – tests go down to a manageable 4300


What Are Customers Running?

● Tips for determining what your customers are running> Keep track of your customers platforms through support and sales by setting up

a database that tracks each customer environment at the time of a sale and maintenance renewal.

> Survey your customer base yearly to collect platform information. This can be done individually or can be combined with satisfaction surveys.

> If you are providing a website for customers to download the latest versions of your software, add a collection point to that mechanism as part of the download process

> If you are providing hosted solutions, be sure to view the web server reports for the traffic browsers and OS’s

> Don’t ignore platform vendor trends as they can give you an idea of the market shifts in platforms. In addition to vendor trends there are trends among individual industry sectors and even geographic regions. (The US may be Windows based, but not everyone is Windows based.)


Support vs. Certificationincludes Software and Hardware

Certify indicates:● Actively tested with new releases● Follows a defined test plan● Exists in-house for development,

testing and troubleshootingShould include:● All claimed Operating Systems● All claimed Web Application Servers● All claimed Database types● Specific hardware or software

configurations such as:>Scanner software and hardware configurations>Any peripherals that your software requires

Support indicates:● Not tested but there is no evidence to

indicate it will not work● Not available in-house for

troubleshooting but we will assist customers as needed

Can include:● Specific hardware components such

as:>Fax board models>Scanner models>Dual monitor setups

● Point or SP releases such as:>OS or Browser updates

● Specific configurations for software or hardware combinations


All the Bang Without All the Bucks!

Saving dollars each day, keeps the Boss away


Testing without the Financial Hit

● Options for high return on investment testing environments include:> Virtual machines> Hardware evaluation programs> Vendor labs


Virtual Machines

Pros● Isolation of applications from the host

machines ● Allows improved testing through snapshots

and rollbacks (especially helpful for installation testing to validate registry settings, drivers, etc.)

● Mobility is a plus as images can be used on any physical machine that has the VM player installed

● Quick and efficient troubleshooting of issues as you can create a library of image configurations mixing up versions and platforms

Cons● Images can take up a lot of space – large

storage disks will be required● Virtual machine servers for use by a group

of people can be costly and use of virtual machines will eat up a lot of memory on the host machine.

● There are performance penalties which vary from technology to technology and as a result this is not a solution for performance testing

● It supports only the hardware that both the virtual machine and the guest operating system support

● Licensing can be a bit tricky as most software requires licensing each individual image

●Per Wikipedia: In computer science, a virtual machine is software that creates a virtualized environment between the computer platform and its operating system, so that the end user can operate software on an abstract machine.


Hardware Evaluation Programs

Pros>Typically provide new hardware for 30 day evaluation periods>Allows access to new hardware for certification tests with your product>Allows you to address any issues before your customers start purchasing and using the newer hardware

Cons>Usually require vendor / partner relationships>Often lower on the priority ladder than new sales so hardware may not be available>Not always a lot of time to test all possible configurations

Most major hardware manufacturers have whats called Evaluation ProgramsWhere you can receive new hardware releases for a 30 day period for the purpose of certifying with your software.


Vendor Labs

● Most OS vendors provide labs where partners and customers can gain access to environments to test the compatibility of their applications with existing or new releases of software and/or hardware.

● The environments are “pristine” with no software other the OS.● Most provide 10/100 MB Ethernet, a variety of graphics adaptors and virtually

unlimited storage capabilities.● Some labs allow remote access while others do not. ● The biggest advantage of these labs is for certifying on hardware platforms without

having to purchase the hardware.


Pros and Cons of Vendor Labs

Pros● Provides early access to new releases of

operating systems or hardware platforms.● Provides access to all versions of operating

systems on applicable hardware eliminating the need to purchase all flavors of either the hardware or software.

● Allows the validation of numerous hardware platforms saving thousands of dollars.

● Provides early recognition of any platform specific issues customers might encounter.

● Provides a way to document best practice configurations for hardware / software configurations as they relate to your application.

Cons● There are usually several weeks of lead time

required for the scheduling of labs and times can be anywhere from 1 week to 14 days.

● Limited amount of time is blocked so you must know exactly what you are testing when you go in.

● There is limited support in terms of troubleshooting problems you have that are related to your software in their environment.

● Microsoft environments are provided through a third party, VeriTest, but all tests are conducted exclusively by them on released MS software.

● Each lab has it’s own policy regarding load testing or performance testing.


Vendor Lab Information Sources

● Sun Microsystems – Customer Environment Test Lab (CET)> Anyone can use the lab in return for specific marketing material about your

customers> http://www.sun.com/cet_lab/index.html

● Hewlett Packard – Application Migration and Testing Centers> Must be a member of HP’s Developer and Solution Partner Program (DSPP)> http://h21007.www2.hp.com

● IBM – Virtual Loaner Program> At the time of writing, IBM states that a true performance test could not be run in

the lab but could be done through fee-based services they offer through their Benchmark Center

> http://www-304.ibm.com/jct09002c/partnerworld/wps/pub/systems/vlp● Microsoft – VeriTest, a service of Lionbridge

> All Microsoft certifications are handled through VeriTest where they will exercise your application using your test cases, process and/or tools

> http://www.lionbridge.com/lionbridge/en-US/services/outsourced-testing/product-certification-programs/microsoft/default.htm


QA Team Skill Set

● In real estate, it’s location, location, location● In QA, it’s diversify, diversify, diversify

> Analysts don’t have to be DBA’s but someone should know how to build and manage the databases you support

> Analysts don’t have to web server guru’s but someone should know at least one supported server very well

> Analysts don’t have to be Unix administrators, but if you support Unix, you should have at least one person on the team who can navigate that environment well.

> Conduct a weekly review of any customer issues logged for that week with the QA team. Often referred to as the “acid bath” meeting by a previous team of mine, it is an excellent opportunity to identify and plug any holes in the testing process.


Customer Resolution Process Improvement

We can’t Test Everything, But we can Support Everyone!


Customer Resolution Process

● Step 1 – Strive to find the problems before the customer does> Load all new releases of all the platforms you support as soon as they are

available in Beta or release to use with day to day testing> Load all platform supported service packs, security updates, etc. as soon as they

are released (not usually posted for Beta testing) and conduct smoke tests> Work with your partners to participate in pre-release BETA programs when

available ● Step 2 – Make sure your production issue process is well defined and managed

> If it is a different group, work with the manager to define the Support to QA process

> Implement process changes to facilitate the quickest resolution of the most important issues

> Create a communication process for issues identified with new releases so customers can be informed before they update their environment

● Step 3 – Make sure someone is responsible for managing what is implemented!


Problem Cause Detailed Actions Detailed Causes Actions

Blue completed actionsOrange issue specificaction planWhite pending actions

Cause and Effect Diagram


Pointing the Finger

Should you point the finger at the third party vendor?● What does the Customer see?

> It is transparent to your customer who’s fault it is – all they know is what they are trying to do with your product isn’t working

> You chose the vendor, so saying it’s their fault is really saying you made a bad decision

> The customer bought something in good faith – it should do what you said it would do

● Tread lightly on your vendors, applying the following rules:> Take responsibility for the issue no matter who’s code it is in> Determine if there are any changes you can make to mask the issue until the

vendor can address it> Explain to the customer your limitations and what you are doing to get the fix

they require


Customer Communication

Proactively inform your customers about software / hardware conflicts with new releases: ● Information to provide in the customer notification

> Identification of the software/hardware release, including a link to any release notes the vendor has posted

> Description of the issue the update may cause> Areas of impact within your product> Actions you are recommending for your customers including possible

workarounds or patches already available> Actions you are taking to resolve the issues being caused and a timeframe for

resolution, if possible● How to get the information out (any or all of these methods may be done):

> Email blasts if you have customer information for the technical contact at the company

> Posting on your download website> Posting on your Support or Troubleshooting website

● Proactively follow up with customers when the issue is resolved in your software or a vendor patch


Presentation Recap

● It is not feasible to anticipate or prevent the effect platform vendor releases might have on your product in production

● To provide the most flexible platform support, you must determine the most effective and efficient testing process

> Know what your current customers are running in production> Plan full testing around the platforms most used by customers> Plan partial testing around the platforms that are less popular

● Keep on top of not only platform trends but also geographic and industry sector trends

● Create environments that are available when needed and are more cost efficient> Virtual machines> Vendor labs and evaluation programs

● Create and implement an issue resolution process that will allow you to focus on solving the problems of the lower impact environments

> Classification of issues> Management of the process

● Continual evaluation of your environment testing policy

Documents

Claire J. Caudry - StickyMinds€¦ · Claire J. Caudry is the Manager of Quality Assurance for Perceptive Software Inc. in Shawnee Kansas. She has been in the field of Quality Assurance