335Software Engineering. DOI: © 2012 Published by Elsevier Inc. All rights reserved.2013

http://dx.doi.org/10.1016/B978-0-12-407768-3.00020-3

Software Acceptance Testing 20

CHAPTER

CHAPTER OUTLINE

20.1 Products of software acceptance testing ......................................................... 33620.2 Software engineering (software acceptance testing stage) ............................... 33720.3 Software implementation organization (software acceptance testing stage) ....... 33820.4 Computing environment implementation organization

(software acceptance testing stage) ................................................................ 33920.5 Post-development process organization (software acceptance

testing stage) ................................................................................................. 33920.6 Software test and evaluation (software acceptance testing stage) ..................... 33920.7 Reviews and milestones (software acceptance testing stage) ........................... 34020.8 Establish the software product baseline ........................................................... 341

Acceptance testing is the formal testing activity that involves enterprise, customer, and stakeholder representatives to witness the readiness of the software product to be deployed. If a contract was the genesis for the software development program, then this activity represents a significant step in demonstrating that the software development program has fulfilled its contractual obligations. If the project was funded by internal enterprise resources, then this activity provides proof that the program requirements have been satisfied and the product is ready for deployment. Such products may be distributed internally in support of business processes or they may be marketed as consumer software packages.

Prior to software deployment, the software configuration items must be subjected to a final examination to ensure that the software data packages are complete. The architecture technical data package (TDP) must be audited to ensure that it accurately reflects the “as-built and tested” software configuration. The functional configuration audit (FCA) inspects software test results to ensure that the software product satisfies its specifications, as augmented by change proposals. The physical configuration audit (PCA) inspects the definitive software deployment data package (DDP) to ensure that the as-built and tested software configuration is properly reflected in its documenta-tion set. These configuration audits should be performed to establish the uniformity of the software product configuration to the architectural and configuration DDPs.

A deployment readiness review (DRR) should be conducted to present the results of acceptance testing, software configuration audits, and the status of each of

336 CHAPTER 20 Software Acceptance Testing

the post-development processes. The DRR is intended to ensure that the processes for software replication, distribution, training, and sustainment are operationally prepared to bolster customer and stakeholder demands to software product training and problem resolution. Figure 20.1 depicts the products of the acceptance testing stage of software development.

20.1 Products of software acceptance testing1. Acceptance test report. This report summarizes the results of software accept-

ance testing. It should provide the traceability to software problem reports generated to the software components and units of which the behavior did not satisfy the test procedure. Acceptance testing confirms that the software product (software configuration items and computing environment) satisfies the software specifications.

2. Software problem reports. Software problem reports should be generated for every discrepancy found during acceptance testing. The priority level, risks, workarounds, and resolution alternatives should be identified so the proper reso-lution alternative can be identified that will permit software deployment at the soonest opportunity. If dry-run testing was conducted, then there should not be any new software problem reports generated during acceptance testing.

3. Software deviations and waivers. Deviations and waivers should be generated for deficiencies in the software product that conflict with the software specifica-tions. Deviations represent nonconformity of the software product with software specifications. Deviations represent acknowledgment of these situations, and request authorization to proceed to deploy the software product in its current

Acceptance Testing Software Operations

Acceptance Test ReportSoftware Problem Reports

DeploymentReadiness

ReviewSoftware Deviations and Waivers

Software Product Baseline:

Software Replication OperationsSoftware Distribution OperationsSoftware Training OperationsSoftware Sustainment Operations

Customer Support Operations–– Software Support Operations

Final Architecture Technical Data Package (TDP)Definitivr Software Deployment Data Package (DDP)

Final Software Source Code Files–Final Software Build Procedures–Final Software Executable Files–

Definitive Post-development Process DDPSoftware Replication DDP–Software Distribution DDP–Software Training DDP–Software Sustainment DDP–

Customer Support DDP∗Software Support DDP∗

FIGURE 20.1

Acceptance testing stage.

33720.2 Software engineering

noncompliant state. The deviations provide temporary authorization to deploy the software product with a commitment to resolve the problems with a future patch or release. Waivers are for situations where the software does not satisfy contractual or software requirements, and by waiving the requirement, the soft-ware product can be deployed, as is, without a commitment to resolve the defi-ciency in future patches or releases.

4. Definitive software DDP. The software DDP should contain the implementation artifacts that will permit the consideration of future extensions and enhance-ments to the software architecture. The definitive software DDP should be accompanied with a list of all authorized changes assimilated into the software architecture and its design artifacts. There may be a separate software DDP for each software configuration item that comprise the software product.● Final software executables. The final software executables should be pre-

pared to establish a baseline for software replication and distribution.● Final software source files. The final set of source code files should be

prepared to be incorporated into the software product baseline. A software bill of material (SBOM) should document each of the source code files that make up the final software product configuration.

● Final build procedures. The final software build procedures should be docu-mented to explain how the software executable files are generated. These procedures will be utilized by the software sustainment organization to gen-erate patches or future releases of the software product.

● Final architecture TDP. The updated software architecture artifacts (require-ments baseline, functional and physical architectures, software nomenclature register, models, etc.) should be documented to provide a basis for post-development sustainment and software product and process improvement.

5. Definitive post-development process DDPs. The final deployment data packages for each post-development process should reflect the arrangement and layout of the facility, equipment, workstations, communications, networking and equip-ment connectivity, software tools and databases, etc. involved in facilitating each process. These DDPs provide the detailed design schematics and support documentation necessary to sustain each of the processes.

20.2 Software engineering (software acceptance testing stage)

1. Witness software acceptance testing. Representatives of the software engineer-ing integrated product team (SWE-IPT) should witness the conduct of accept-ance testing to ensure that the tests are conducted according to the software test procedures.

2. Witness post-development process qualification. Representatives of the SWE-IPT should witness the conduct of each post-development process qualification testing to ensure that the tests are conducted according to the test procedures.

338 CHAPTER 20 Software Acceptance Testing

3. Reevaluate the software architecture. The SWE-IPT should reevaluate the soft-ware architecture to understand the impact of software problems and assess potential solutions. The resolution of deficiencies at this late juncture in the soft-ware development effort may not be possible. Therefore, the SWE-IPT should prepare software deviation or waivers, as appropriate, to address the resolution of unresolved problem reports.

4. Support the software configuration audits. Representatives of the SWE-IPT should participate in the conduct of software FCAs and PCAs.

5. Prepare the final architecture technical data package. The SWE-IPT should prepare the updated architecture artifacts to support the software configuration audits. The definitive architecture data package should be accompanied with a list of all authorized changes assimilated into the software architecture and its artifacts (documentation, drawings, models, etc.).

20.3 Software implementation organization (software acceptance testing stage)

1. Monitor software acceptance testing. Representatives of the software implementation organization should witness the conduct of acceptance testing to provide insight into software behaviors and responses to test scenarios and conditions.

2. Evaluate software problem reports. The software implementation organization must determine the corrective action to be taken to resolve problems that arise during acceptance testing. Software problem reports generated as a result of acceptance testing should be evaluated to determine if they stem from software implementation defects. If necessary, a problem may be elevated to the SWE-IPT to be resolved by modifying the software architecture or pursuing a devia-tion or waiver.

3. Deliver the final software executables. The final software executables should be generated and delivered to the project configuration management organization to support the software replication process and software configuration audits.

4. Deliver the final software source files. The final set of source code files should be prepared to provide a basis for software problem isolation and resolution. The source code files also provide a basis for implementing future extensions and enhancement to the software product baseline.

5. Deliver the software build procedures. The final software build procedures should be prepared and delivered to the project configuration management organization to facilitate the software replication process. The software build procedures will be utilized by the software replication team to generate distribution media and accommodate patching and releasing future versions of the software product.

6. Support the software configuration audits. Representatives of the software implementation organization should participate in the conduct of software FCAs and PCAs.

33920.6 Software test and evaluation

20.4 Computing environment implementation organization (software acceptance testing stage)

1. Support the software acceptance testing. Representatives of the computing envi-ronment organization should participate in the conduct of acceptance testing to provide insight into the computing environment implementation behaviors. Confusion may arise among development team members concerning interpreta-tion of requirements or behavior specifications during testing. The computing environment is an element of the software product and is qualified as a result of software acceptance testing.

2. Deliver the computing environment DDP. The final computing environment DDP should be prepared to support the software configuration audits. The final computing environment DDP should contain a detailed specification of comput-ing environment physical characteristics and performance benchmarks.

3. Support the software configuration audits. Representatives of the computing environment organization should participate in the conduct of the software FCAs and PCAs.

20.5 Post-development process organization (software acceptance testing stage)

1. Finalize data processing workflow procedures. The post-development process organization should finalize the data processing workflow procedures for con-ducting software deployment or sustainment tasks. The workflow procedures should establish guidelines for how to perform typical tasks and contend with atypical situations.

2. Qualify the post-development processes. The post-development process organi-zation should prepare test procedures for the software replication, distribution, training, and support processes. Each post-development process should be quali-fied to ensure that it is ready to conduct software sustainment procedures.

3. Finalize the post-development process DDPs. The final deployment data packages for each post-development process should be prepared to provide the detailed design schematics and documentation necessary to sustain each of the processes.

20.6 Software test and evaluation (software acceptance testing stage)

1. Execute the acceptance test procedures. The software test and evaluation organization should execute acceptance test procedures to qualify the software configuration and computing environment. Any test failures must be analyzed to identify the source of the failure. Members of the software quality assurance

340 CHAPTER 20 Software Acceptance Testing

team should monitor the execution of each test to ensure that the test procedures were followed and that the results were properly captured or recorded.

2. Generate software problem reports. When a software test procedure does not generate the expected results, then a software problem report should be gener-ated to identify the problem and how it deviated from the expected results. Software problem reports should document each problem encountered in a manner that enables reconstructing the test results. Each problem report must be assigned to the appropriate software development organization for resolution.

3. Publish the acceptance test report. The software test and evaluation organiza-tion should prepare the acceptance test report to document the status of soft-ware acceptance testing. This report should identify any deficiency identified and establish the regression testing necessary to properly demonstrate that the repaired software configuration or computing environment characteristic ade-quately resolves the deficiency.

20.7 Reviews and milestones (software acceptance testing stage)

1. Functional configuration audit. The software engineering integrated product team leads the audit of the software configuration to ensure that requirements have been properly implemented, tested, and satisfied. Each requirement in the software specifications should be traced to the test results that confirmed the suitability of the software implementation. All authorized engineering change proposals (ECPs) and software problem reports should be evaluated to ensure that they have been resolved and assimilated into the software DDP.

2. Physical configuration audit. The software engineering integrated product team leads the audit of the software configuration to ensure that requirements have been properly implemented, tested, and satisfied. The audit should ensure that all authorized ECPs and software problem reports have been resolved.

3. Deployment readiness review. The deployment readiness review should be con-ducted to review the results of acceptance testing and configuration audits. The status of each of the post-development process qualifications must be reviewed to ensure their readiness to support software deployment. Once the deployment readiness review has been successfully completed, the software product is ready to transition to the software deployment stage of its life cycle. The software development project should be considered concluded unless an iterative or spiral development approach is being employed. A typical agenda for the deployment readiness review should address the following topics:1. Introduction

1.1. Agenda1.1.1. Goals of the Deployment Readiness Review1.1.2. Review Prerequisites1.1.3. Expected Outcomes

34120.8 Establish the software product baseline

1.2. Project Overview (Optional)1.2.1. Project Closure Criteria1.2.2. Transition to Software Operations

2. Software Development Status2.1. Software Engineering Status

2.1.1. Final Architecture TDP Status2.1.2. Outstanding Engineering Change Proposals2.1.3. Outstanding Waivers and Deviations

2.2. Software Implementation Readiness2.2.1. Software Implementation Readiness Status2.2.2. Software DDP Status2.2.3. Outstanding Software Problem Reports

2.3. Computing Environment Readiness2.3.1. Computing Environment Readiness Status2.3.2. Computing Environment DDP Status2.3.3. Outstanding Software Problem Reports

2.4. Test and Evaluation Status2.4.1. Acceptance Test Results2.4.2. Functional Configuration Audit Status2.4.3. Physical Configuration Audit Status

3. Post-development Process Readiness3.1. Software Replication Process Readiness3.2. Software Distribution Process Readiness3.3. Software Training Process Readiness3.4. Software Sustainment Process Readiness

4. Wrap-up4.1. Action Items and Assignments4.2. Conclusion4.3. Final Remarks

20.8 Establish the software product baselineThe software product baseline (SPB) should be established at the conclusion of the DRR. The SPB combines the final architecture TDP and software DDP into a com-plete baseline for the initial release of the software product. This product baseline should be provided to the software sustainment organization as a basis for software problem resolution and deriving engineering solutions for software product exten-sions and enhancements. If an incremental or evolutionary development concept is utilized, then the software product baseline should be provided to the next program organization taking responsibility for software product enhancements.