Test Progress Reporting using Functional Readiness

© 2001 Fannie Mae. All Rights Reserved.

Test Progress Reporting using Functional Readiness By Robyn Brilliant, Principal Technologist, Fannie Mae

Introduction One of the goals of testing is to find as many of the “big bugs” as possible while covering as much of the functionality as possible before the implementation date. On many projects the implementation date is not flexible and it is difficult to make a case to management for moving the date or at least considering the risk of implementing before making the decision. Although many projects review detailed defect reports at status meetings and make decisions on what has to be fixed on a daily basis, reviewing defects at this level of detail is often too much for management. They don’t get the “big picture” and detailed defect reports don’t always reflect risk. In order to help the project team and management assess risk and make decisions during the course of the project, there needs to be a way to express testing status in a comprehensive “big picture” view. Functional readiness is a way to simply describe test status in the context of risk. It is a combination of subjective and objective metrics that provides a clear view of perceived risk, test progress and resolution of defects. The Functional Readiness Matrix (FRM) is a decision-making tool that summarizes these metrics. An FRM is created for each phase of testing and the metrics contained in the FRM can be used as entrance criteria for the next phase of testing or for implementation. The FRM is used by the project team to: q Identify High Risk Functional Areas q Set Goals for Test Completeness and Defect Resolution Based on Risk Level q Obtain agreement on product quality goals q Track Actual Test Completeness and Defect Resolution Against the Set Goals q Support the Production Implementation Decision q Present the “big picture” to management Where did the FRM come from? In the early 1990s most Fannie Mae test leads were coordinating cross-functional projects involving multiple applications and organizations. This type of test was called an Integrated Project (IP) Test and the role was called IP Test Coordinator. One of the most difficult tasks of the IP Test Coordinator was communicating the status of the overall IP test to management across the organizations. Each application test team had its own way of reporting status. The FRM was developed to standardize the reporting both within and across applications. Using the FRM, the IP coordinator could display the status of the overall IP Test on a single page, at a high enough level for management to understand. Each application test lead could display status for that specific application in the same format. The FRM has gone through several changes over time to make it easier to use, understand and implement. The key benefit of the FRM remains the same: everyone sees the same type of information, in the same format across projects. This facilitates communication, helps pinpoint problem areas and allows comparison of status across multiple projects.

- 2 -


Summarizing functional information

Even if there are no requirements, features are usually known before coding begins. Functional readiness can be determined if the test cases can be categorized or traced to particular features or functions. Features or functions are listed in the leftmost column(s) of the functional readiness matrix. As an option for more detailed tracking Primary Functions can be broken down into lower-level requirements or Process Components.

Determining acceptable risk levels

Once the Primary Functions and Process Components are listed, the risk needs to be assessed. This is a subjective metric, the users of the product or whoever is responsible for

Primary Functions Process Components (Optional)

Function 1Requirement 1Requirement 2Requirement 3

Function 2Requirement 1Requirement 2Requirement 3Requirement 4Requirement 5Requirement 6Requirement 7Requirement 8

Functional Areas

Primary Functions Process Components (Optional)

Reliability Risk

Business Opportunity

RiskFunction 1

Requirement 1Requirement 2Requirement 3

Function 2Requirement 1Requirement 2Requirement 3Requirement 4Requirement 5Requirement 6Requirement 7Requirement 8

Risk LevelsFunctional Areas

- 3 -


prioritizing the requirements should determine the acceptable risk levels. Acceptable risk levels are determined for: 1. Reliability, or the risk of the function failing; and 2. Business Opportunity, or the risk of delivering the function late in relation to the business

opportunity. This risk information is used to set some guidelines for determining Test Completeness and Defect Resolution goals and can help the project manager prioritize coding and testing efforts. Using a scale from one to five, with one being the most severe and five the least severe, quantify the Reliability Risk as follows: 1 = Failure of this function would impact customers 2 = Failure of this function would have a significant impact to the company 3 = Failure of this function would cause a potential delay to customers 4 = Failure of this function would have a minor impact to the company 5 = Failure of this function would have no impact Using a scale from one to five, with one being the most severe and five the least severe, quantify the Business Opportunity Risk as follows: 1 = Untimely delivery of this function would lose a significant Business Opportunity for the

company 2 = Untimely delivery of this function would have significant impact on the Business

Opportunity 3 = Untimely delivery of this function would potentially delay the Business Opportunity 4 = Untimely delivery of this function would have minor impact on the Business Opportunity 5 = Untimely delivery of this function would have no impact on the Business Opportunity Test Completeness and Defect Resolution Test Completeness and Defect Resolution are the two metrics used in Functional Readiness to measure test progress. Test Completeness indicates the acceptable percentage of testing to be completed for each function prior to production implementation. This means that the test case was executed, not passed. The level of acceptable defects will be accounted for in defect resolution. Defect Resolution indicates the level of unresolved defects acceptable in a production product. Test Completeness is measured on a scale of 1-5 as follows: 1 = 0% - 20% of test cases executed at least once 2 = 21% - 50% of test cases executed at least once 3 = 51% - 80% of test cases executed at least once 4 = 81% - 95% of test cases executed at least once 5 = 96% - 100% of test cases executed at least once

- 4 -


Defect Resolution is measured on a scale of 1-5 as follows:

1 = Severe problems, non-recoverable 2 = Serious problems, performance substantially degraded 3 = Moderate problems, allows execution with errors 4 = Low and/or cosmetic, no significant impacts 5 = No problems outstanding

Setting testing goals based on risk

Goals are set to create reasonable acceptance criteria for implementation, based on the risk levels assigned. The goals reflect an acceptable level of quality at implementation time for each Primary Function and/or Process Component identified. There are no hard and fast rules about what goals are acceptable for a particular risk level, but the above table provides some guidelines that will help with analyzing what combination of test completeness and defect resolution might be acceptable for a particular combination of risk levels. If this information is collected and available before development begins, it can be used to prioritize the development effort and the test planning effort.

R e l i a b i l i t y R i s k

B u s i n e s s O p p o r t u n i t y

R i s k

5 4 3 2 1 5 4 3 2 11 1 ? X X X ? X X1 2 ? X X X ? X X1 3 X X X X ? X X1 4 X X X X X X X1 5 X X X X X X X2 1 X X X X X2 2 X X X X X2 3 X X X X ? X X2 4 X X X X X X X2 5 X X X X X X X3 1 X X X X3 2 X X X X3 3 X X X X3 4 X X X X3 5 X X X X4 1 ? X ? X4 2 ? X ? X4 3 X X X X4 4 X X X X4 5 X X X X5 1 ? ? ? ?5 2 ? ? ? ?5 3 ? ? ? ?5 4 ? ? ? ?5 5 ? ? ? ?

R e a s o n a b l e G o a l? P o s s i b l y a r e a s o n a b l e g o a l , b u t u s e c a u t i o nX N o t a R e a s o n a b l e G o a l

T e s t C o m p l e t e n e s s

G o a l

D e f e c t R e s o l u t i o n G o a l

- 5 -


To illustrate how goals might be set for an actual project, the FRM above lists four primary functions for a fictional loan processing system. The risk levels clearly show which functions are the core capabilities of the system. The Loan Data Entry and Monthly Payment Processing functions are critical. Based on the risk levels set by the users of the system, those two functions must be reliable and must be delivered quickly in order to take advantage of the business opportunity. Because Reliability has a heavier weight than Business Opportunity (you wouldn’t want to deliver something fast that doesn’t work!), the goal for test completeness reflects a need for thorough test coverage – 96 to 100% of the test cases must be executed at least once. In terms of defect resolution, the project team decided that they can live with a few low severity, cosmetic defects in the Loan Data Entry function, but the Monthly Payment Processing is so critical that it cannot be implemented until there are no defects. In contrast to the core functions, the Loan Views and Reporting functions are not as critical to the system. Test completeness and defect resolution goals reflect this as the project team decided they can implement Loan Views with only 81-95% of test cases executed and moderate defects remaining and Reporting with 51-80% of test cases executed and some serious defects remaining.

Tracking progress and calculating functional readiness The project team can determine an application or project's functional readiness by assessing risk and tracking actuals against goals for Test Completeness and Defect Resolution. Actuals are tracked using information from the Test Management and Defect Tracking systems. To calculate Test Completeness, count the number of test cases executed for the Primary Function and/or Process Component and divide it by the total number of test cases for the Primary Function and/or Process Component. Fit the resulting percentage into one of the 5 Test Completeness categories and enter it into the “current” column. To calculate Defect Resolution, review the open defects (by severity) for each Primary Function and/or Process Component. The Defect Resolution score is the severity of the most severe defect remaining open. For example, if there is one open severe, non-recoverable defect, the Defect Resolution score is 1. Enter the Defect Resolution value into the “current” column. The advantage of using automated Test Management and/or Defect Tracking systems is that the calculations can be done automatically. If Test Cases and Defects are tracked manually instead of in an automated system, the calculations have to be done manually as well. In order to determine if the application is at risk for missing its implementation date, functional readiness is assessed in terms of the project schedule. Test Schedule information is included at the top of the FRM. Enter a reporting date for the collected actual information. This will be the base date to determine business days exhausted by test. Enter the Scheduled test start date and scheduled test end date from the project schedule. The FRM is set up to determine a baseline of scheduled business days to test by subtracting the scheduled test start date from

Test Completeness Defect ResolutionReliability

RiskBusiness

Opportunity Risk

Goal Current Current as % of Goal


1 1 5 0% 4 0%3 5 4 0% 3 0%1 1 5 0% 5 0%4 5 3 0% 2 0%

Loan ViewsMonthly Payment ProcessingReporting

Risk LevelsFunctional Areas

Loan Data Entry

Primary Functions

- 6 -


the scheduled test end date. The FRM is also set up to determine the number of business days exhausted by subtracting the reporting date from the scheduled test start date. The business days exhausted is then divided by the scheduled business days to show the percentage of business days exhausted. This information can be compared to the progress toward the functional readiness goals.

The example above expands on the FRM created for the fictional Loan Processing System (LPS). The LPS team has been testing the system now for 60 days or 66.67% of the scheduled testing time. Notice that where testing goals were exceeded, the current percentage is capped at 100%. This avoids averaging based on percentages greater than 100% of the goal. Showing averages for all functions illustrates the danger in over-summarizing information. By looking at the individual functions rather than the overall summary, it is more obvious that there are several problems that could impact the implementation date. First, Loan Data Entry is in really bad shape. Although 60% of the Test Completeness Goals have been met, there are still severe defects. It looks like testing might at a standstill for Loan Data Entry – so where has the testing team concentrated their efforts? Certainly not on the other critical function, Monthly Payment processing – it is only at 40% of its Test Completeness goal. The testing team decided while Loan Data Entry was stuck, why not test the easiest, least critical function – Reporting. Reporting is exceeding both the Test Completeness and Defect Resolution goals. By having this information available throughout testing, the Project Manager and the Test Manager can adjust their team’s efforts before it is too late. In the example above, it is clear that the development team needs to focus on Loan Data Entry and while those fixes are being made, the testing team should shift their testing efforts to Monthly Payment Processing.

Presenting functional readiness information to management The FRM is used to present a clear picture of status throughout testing. If management is made aware of status early and often and in a way they can understand, it helps direct decisions about resource allocation, schedule impact and the state of the system. Using the FRM helps the test manager focus testing efforts and helps the development manager focus his team on fixing the most important defects first. It also shows senior management the state of the system in near real-time so that decisions can be made before the problems get too big.

Project Name: Loan Processing System (LPS)Reporting Date: 10/31/01Project Type: New Development Scheduled

Start DateScheduled End Date

Business Days

Scheduled

Business Days

Exhausted

% Business Days Exhausted

9/1/01 11/30/01 90 60 66.67%Test Completeness Defect Resolution

Reliability Risk

Business Opportunity

Risk



1 1 5 3 60% 4 1 25%3 5 4 2 50% 3 2 67%1 1 5 2 40% 5 4 80%4 5 3 5 100% 2 3 100%

4.25 3.00 62.50% 3.50 2.50 67.92%

Test Schedule Dates

Risk LevelsFunctional AreasPrimary Functions

Overall

Loan Data EntryLoan ViewsMonthly Payment ProcessingReporting

- 7 -


The FRM is a tool to help make decisions. The information included on the FRM should help to direct discussion on whether or not the system is ready for the next phase of testing or for implementation. It is important to note that as a decision making tool, the FRM should be flexible. If there is a need at any point in the project to change the risk assessment or the goals, that option should be available. Decisions to change the risk levels or goals should be made with adequate discussion and support from the project team. Conclusion One person or group cannot determine when a system is ready for release. It helps to have as much information as possible about risk and the expectations of the users in order to make that decision easier. Assessing Functional Readiness helps projects get closer to achieving the goal of knowing when the system is ready and provides senior management a simple way of looking at the whole project. Functional readiness provides management with the “big picture” of test progress and resolution of defects. It describes test status in the context of risk in order to facilitate realistic goal-setting and decision-making throughout the testing phase of the project. The Functional Readiness Matrix benefits projects by: q Providing useful metrics even if requirements are not well documented q Providing quality goals early in the project q Providing management with a “big picture” view of status q Serving as a baseline for decision making q Allowing comparison of status between multiple projects (or subsystems in a large project) q Helping to identify areas to focus on q Providing a simple, visual summarization of test progress

Robyn Brilliant Robyn Brilliant is a Principal Technologist in the Enterprise Architecture department at Fannie Mae. In her current position, she researches technology trends, helps define systems development processes and contributes to the definition of technology and process standards. She is a corporate expert in testing process and testing technology and is often consulted by project teams to assist in test management and test automation planning. Robyn’s past testing experience includes managing test planning and test automation for Y2K test efforts as well as test management, planning and execution for individual software projects. She has 10 years of software testing experience, including 8 years of experience with automated testing tools. Robyn has participated with other industry experts in the Austin Workshop on Test Automation (AWTA) and the Workshop on Heuristic and Exploratory Techniques (WHET). She is a Certified Software Test Engineer (CSTE) through the Quality Assurance Institue (QAI).

Documents

Test Progress Reporting using Functional Readiness