EVALUATION OF A CONTRACTOR’S SOFTWARE QUALITY …

1,.. .

NIL-HDBK-334

15 JULY lBB1

MILITARY HANDBOOK

EVALUATION OF A CONTRACTOR’SSOFTWARE

QUALITY ASSURANCE PROGRAM

QCIC

....

NO DELIVERABLE DATA S13QUIRED

BY THIS DOCUMENT

Downloaded from http://www.everyspec.com

MIL-HDBK-33415 JUL 1981

Efforts are underway within the Department of Defense to further refine thedefinition of Software Quality Assurance (SQA) and to propose changes to theparent document, MIL-S-52779A. Caution should be exercised in applying SQAprinciples espoused herein since the level and extent of SQA is highly dependenton the end-use function of the software.

i


DEPARTMENT OF DEFENSEWASHINGTON, D.C.

MiI.HDBK-334Evaluation of a Contractor Software Quality Aaaurence Program

15 JIM 1981

1.Thlaatandardization handbook wasdevaloped bytha Depatiment of Defenaa.i‘2. This publication waa approvedon 7 Decamber 1980 for printing and inclusion in the

military alandardizetion handbook seriaa.

3. Thiadocument provldee baaicand fundamental information andquidance to personnelconcernad with the evaluation of acontractor’s aoftwara quality assurance program, in con.naction with MIL-S-52779A. “Softwere Quality Aaeuranca Raquirementa.” The hendbook ianot. intended to ba raferencad in purchesa apeclficetiona, nor shell it supersede anyspecification requirements.

4. Eva~effoti haabeen madatoreflect thelataat information onthe evaluation of aeon.tractor’a software quatity aasurance ayatem. It ia the intent to review this handbookperiodically to insure ite completeness and accuracy. Beneficial comments (recommen.dationa, additions, deletions) and any pertinent data which may be used in improving thisdocumant should be addreaaed to:

CommanderLLS Army computer9@ems CommandATTN: ACW -OAA A (3TOPH.5)Ftklvoir,VA. 220S0

or by using the self-adtheeaed Standardization Document Improvement Proposal (DD Form1426) appearing at the end of this handbook.

ii

I


MIL-HDBK-334

15 JUL 1981

TABLE OF CONTENTS

● Paragraph

Introduction

1.01.11.21.3

2.02.12.2

3.03.13.23.2.13.2.23.2,33.2.43,2.53,2.63,2,73.2.83.2.93.3

4.04.14.2

5.0

6.06.16.26.2.16.2.2

SCOPE, . . . . . . . . . . . . . . . . . . . ., .“ - 0 ““ “ “ o ‘ “Applicability. . . . . . . . . . 0 . . . . . , . . , . . “ “ ‘ ‘ “ e “Contractual Intent . . , . . . ,. .,.. . . . . . . . . .

Relation to Other Contract Requ; rem&&: . , . . . . . . . . . , . . .

APPLICABLE DOCUMENTS. . . . . . . . . . . . . . . . . 0 . . . . . .Amendments and Revisions . . . . . . . . . . ~ . . 0 . . . . . . . . .Ordering Government Documents. . . . . . . . . . . . . , . . . . . . .

REQUIREMENTS. , . . . . . . . . . . . . . . . . . . . . . ~ . . . . .Software QAProgram. . . . . . . . . . . . . . . 0 . , . . , . . . “Software QA Program Requirements . . . . . . . . . . . . . . . . . . .Tools, Techniques, and Methodologies . . 0 . . . . . . . , ~ . “ “ .Computer Pro9ram Uesi9n . . . . . . . . . . . . . . . . . . . . ...”Work Certification .,..,.., . . . . . . . . . . . . . . . . . .Documentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . .Computer Program Library Controls. . . , , , . . . . . . . . . . . .Reviews and Audits.,..,., . . . . . . . . . . . . . . . . . . .Configuration Management (CM). . . . . , . . . . . . , , . . . , . . .Testing . . . . . . . . . . . . . . . . . . . . . . . . . . ...+-.Corrective Action . . . . . ...’. . . . . . . . . . . . . . . . . . .Subcontractor Control .!..... . . . . . . . . . . . . . . . . . .

QUALITY ASSURANCE PROVISIONS . . . . . . . . . . . . . . . . . . . . .Contractor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .Government Review at Contractor, Subcontractor or Vendor

Facilities . . . . . .

PREPARATION FOR DELIVERY

NOTES . . . . . . . . . .Intended Use . . . . . .Ordering Data. . . . . .Procurement RequirementsContract Data Requirements . . . . . . . . . . . . . . . . . . . . . .

Page

1112

333

33556678

i’111315

1616

16

17

1717171718

/

ii

. . .111


IMIL-HDBK-334

15 JUL 1981

INTRODUCTION

This document provides guidance topersonnel responsible for the evalwtion

of a contractor’s software quality prO-gram when Military Specification,MIL-SZ52779A, is invoked in the contract.MIL-S-52779A, “Software Quality AssuranceProgram Req”ireme”tse*, requires contra=-

tors to establish a Software QualityAssurance (SQA) Program which tiill assure

>Ogpliance with the requirements of theircontract. Since the contract will tailorthe application of MIL-s-52779A and other

specifications, care must be taken to__tailor the application of this dOc”nent

accordingly.

Both MIL-S-52779A and this document are

based on established Department ofDefense (DOD) concepts and policies which

provide that:

a. Contractors are solely respon-sible for the control of software quality

and for offering to the Government foracceptance only software determined bythem to conform to contractual require-ments.

b. Government representatives areresponsible for determining that contrac-tual requirements have, in fact, beencomplied with prior to acceptance of thesoftware.

c. Final decision of software

acceptability is solely the responsibil+ity of the Governinent.

The contractor, in accordance with MIL-

S-52779A, must design and maintain aneffective and economical software quality

program that includes procedures which

makes data available to the Government

adequate for use in establishing softwareacceptance criteria. Facilities and

management techniques vary so widely

within the broad pattern of Nationaleecurity and industrial establishmentsthat this evaluation document cannot pro-vide detailed checklists for all facetsof software quality assurance. Instead,

it reflects the software quality program

methods currently used in industry. Theemphasis throughout this document is on

the planning and execution of a compre-

hensive software quality program. The

evaluation of such a program depends on

how well decision criteria have beenselected, applied, and enforced. ●The Government’s evaluation plan should

apply tO all aspects of a contractor pro-gram. Thus , Government representativesshould be familiar with all requirements

of the prociifement to assure themselvesthat the contractor provides effectivequality control coverage throughout....theentire program. This may not be limitedrto the requirements ofwfl;-S-52779A\alone; many contracts include irequirements for software and hardware.

In the event that there is a combinationof hardware and software, Governinent

representatives should also be familiar,with the requirements of MIL-Q-9858A, ,“Quality Program Requirements”.

Quality programs are not intended tocorrect deficiencies in other contractual

requireinents. The contractor is notobligated to perform more than the

requirements specified in the contract.

Occasionally DOD Components contract forparallel development of software underthe concept of Independent Verificationand Validation. Even though this type

of an effort may be defined as a quality

assurance task, MILLS-52779A should be

imposed. Tailoring MIL-s-52779A (impos-

ing all, part or additions) should be

accomplished by the Primary Contracting

Officer.

I

●

A consistence format has been followed

throughout this document. In order to

relate the program evaluation suggestions

as directly as possible to the require-

ments of MIL-S-52779A, each subsection of

the specification is quoted verbatim andfollowed by appropriate comments, ~SfOllows :

SUBSECTION OF MIL-S-52779A

A. “Review of Requirement” -

Discussion “of the requirements

set forth in the subsection.

B. “Application” - Description and

examples of practices applied by

contractors in the past that are

●

iv

,,,, ,,, . ,— ,,,.,—,!


MIL-HDBK-33415 JUL 19B1

●tYPical and illustrative ratherthan all inclusive or mandatory.

c. “Criteria for Evaluation” -Questions which should be askedto evaluate that particular partof a contractor’s quality prO-gram.

It is important to note that the ques-

tions contained in the various “Criteriafor Evaluation” are essentially yeslnoquestions. Asking and answering themalone will not provide a thorough andcomplete evaluation of a contractor’squality program. The questions serveonly as indicators and reminders ofimportant points to cover; the evaluationis expected to cover them in appropriate

depth and detail to assure an effectiveand complete evaluation. Many questionsmay not be contractually required, and as

such, are to be considered self-deleting.The evaluation criteria contained in thisdocument are applicable to the QualityAssurance Plans/Programs/Procedures that

are designed to satisfy the requirementsof MIL-S-52779A during the development ormaintenance of computer software systems,i.e.:

1. Stand alone computer softwaresystems and subsystems.

2. Tactical systems that containembedded computer software.

3. Support software used to assistdevelopment and testing of deliverablesoftware.

4. Software used to maintain

deliverable foperational software.5. Software embedded in automatic

test equipment either as a deliverable oras a test and acceptance facility fordeliverable items.

v



1.0

1.1

CONTRACTOR

StOPE

QUALITY AND RELIABILITY ASSURANCE

SOFTWARE QUALITY ASSURANCE EVALUATION GUIDE

cation, contract, or order,this specification shallaPPIY to the acquisition ofsoftware (canputer programsand related data & documen-tation) where the acquisi-tion involves either soft-ware alone or softwareas a portion of a system orsubsystem. This specifica-tionto non%]]verj~~~ de~~~~test, support, and opera-tional software developedunder the contract, unlessspecifically exempted. .Forpurposes of this specifi-cation, the term softwareincludes firmware.

A. Review of Requirement. MIL-s-52779A

is applicable to computer Programs andsoftware systems to assure conformance tocontractual requirements through controlof the design, develOpent, and testingof the software. Unless otherwise

defined in a contract or order, flruwareis defined as hardware that contains acomputer program that cannot be alteredin its use environment. Examples are:

Programmab le-Read-On ly-Memory (PROM)devices, Read-Only-Memory (ROM) devicesand Erasable Programmable-Read-OnlY-

MeMOIY (EPROM) devices.

All computer programs that are, or will

be contained in firmware are classifiedas software.

The chip on which a computer program isburned in, is classified as hardware.

B. Application. Among the types of

software to which MIL-S-52779A may beapplied are:

1. Command and control computer pro-

grams (embedded), software systems, andoperational software end items.

2. Computer programs and softwaresystems (deliver ableinond eliverable)designed for acceptance testing, check-out, launch, or control weapon or spacesystems, or other aerospace systems.

3. Firmware in the above systems.Pirmware targeted software will be con-sidered the same as any other softwareunder this specification. The hardwarequality aspects of firmware are beyondthe scope of this specification andshould be specified in the contract.

C. Criteria for Evaluation.1. IS the procurement for computer

programslsof tware systems alone or cOm-puter programslsof tware systems that arepart of a larger system which alsoincludes hardware and/or firmware?

2. Does the contract or order spec-ify MIL-s-52779A for software qualityassurance program requirements ?

3. Has the contractor differentiatedbetween deliverable and non-deliverablesoftware?

4. IS the control of non-deliverablesoftware sufficient to insure productquality?

5. Has the contractor identifiedwhat software is to be delivered as firm-ware ?

1.2 Contractual Inten~. Thisspecification requires theestablishment and imple-mentation of a SoftwareQuality Assurance (SQA) Pro-gram (hereafter referred toas the “Program” ) by thecontractor. The purpose ofthe Program is to assurethat software developed,acquired, or otherwise pro-vided under the contractcomplies with the require-ments of the contract. Itis intended that the programbe effectively tailored andeconomically planned anddeveloped in consonancewith, or as an extensionof, the contractor’s otherquality assurance, admini-strative, and technical

1



programs. The term “Program”,as used herein identifiesthe collective requirementsof the specification. TheProgram shall require per-iodic assessment and, wherenecessary, realignment ofthe Program to conform tochanges in the acquisitionprogram. The Program issubject to disapproval bythe Government whenever itdoes not accomplish therequirements of this speci-fication.

A. Review of Requirement. MIL-S-52779Arequires contractors to establish and usea complete software quality program.This does not mean that the f“lfilhnentof the requirements of the specificationis the responsibility of any single con-tractor’s organization, function or per-son. This program must be designed toassure adequate controls throughout allareas of contract perfonnance, for exam-ple, software design, development a“dtesting. All or any part of a contrac-tor’s software quality program may bedisapproved by the Government when theprogram does not accomplish the require-ments of the contract.

B. Application.1. A complete Soft”are quality

assurance program is often the most com-prehensive and extensive activity of acontractor. Software development doesnot lend itself to the deficiency detec-tion and correction techniques of a hard-ware QA program because it is produced ina one-time program. Therefore, a con-tractor’s software quality assurance prO-gram should provide a system to detect adeficiency early and should provide

effective corrective action to assure

that the software complies with contractrequirements.

2. Since software relates co Devel-

opment over a periOd Of t~e, the initialplanning must be assessed periodically toassure continued and current application

of requirements.3. Completed software products which

do not conform to technical requirementsshall be rejected. In addition, when acontractor’s procedures are found to be

unsatisfactory, the procuring activity

will immediately notify the contractorand will disapprove all, or part, of the

I

—software quality program if timely,sffective corrective action is not taken.

4. It should be noted that paragraph1.2 states that the program shall be“effectively tailored and economicallyplanned”. The criteria for evaluationprovided in this handbook are to g.”idethe evaluator and should be tailored tothe specific contract requirements beingevaluated. Note that considerable dif-ferences may exist between software Qual-ity Programs for different contracts.Likely variations exist because of pro-ject size, the criticality of the missionapplication, and the program phase (Tech-nology Development , Validation, Engineer-ing Development, Full Scale Production,etc) .

c. Criteria for Evaluation.1. Does the contractor have a soft–

ware quality program which assures com-pliance with the requirements of the

contract?2. Are working le”el procedures

a“ailable, as well as the overall program

1.3

.

plan?3. Has the contractor documented a ●

management plan to periodically assessthe effectiveness of the quality program?

Relation to Other ContractRequirements. The contrac-tor is responsible for com-pliance with all provisionsof the contract and forfurnishing specified soft-ware which complies withal1 the requirements of thecontract. The SQA ProgramPlan shall reference otherplans; e.g. , configurationmanagement, test, develop-ment, etc. , specified underthe contract and shall becompatible and consistentwith them and not unneces-sarily duplicate their prov-isions. If any inconsis-tency exists between theterms of the contract andthis specification, theOrder of Precedence clauseof the contract shall gov-ern.

“A. Review of Requirement. The require-

ments of MIL-s-52779A are not intended

to cancel or conflict with any otherrequirements of a contract. Thus ,

I 2



MIL-s-52779A does not release contractorsof any of their contractual responsibil-ity. If there is an apparent conflictbetween the requirements of the contractand MIL-S-52779A, the contract require-ments prevail.

B. Application. Contractors usuallyreview with care all of the technicalrequirements of a contract to make cer-tain that all requirements are effec-tively covered by their quality programs.Though many requirements may be standardfrom contract to contract and from speci-fication to specification and can bedealt with by a standard response, mostcontractors insist on a total and thor-ough review because special or new co”-tract clauses may be included. Even infollow-on contracts for software previ-ously furnished, contractors may findspecifications requiring compliance tonew or different requirements. Thishandbook acknowledges that specificationsand standards such as: MIL-Q-9858A, Qual-ity Program Requirements; MIL-I-45208A,Inspection System Requirements; MIL-STD-1679 (USN), Weapon System Software Devel-opment; MIL-STD-J520A (USAF), CorrectiveAction and Disposition System for Non-conforming Material; and MIL-STD-1535A(USAF), Supplier Quality Assurance Pro-gram Requirements, interface with MIL-S-52779A, When they are contractuallyimposed, a“d should be used i“ conjunc-tion with MIL-s-52779A. The interfacebetween Government specifications whichhave been contractually imposed

should be described in the Software Qual-ity Assurance plan.

c. Criteria for Evaluation.1. Does MIL-s-52779A conflict with

any other requirements of the contract,

and has the contractor identified the

conflict and taken steps to eliminate it?2. Does the Software Quality Assur-

ance Plan show relationship to otherplans, specifications, and requirementsrather than duplicate them?

2.0 APPLICABLE DOCUMENTS

2.1 Amendments and Revisions.Mhenever this specificationis amended or revisedsubsequent to its contrac-tually effective date, the

2.2

3.0

3.1

contractor may follow, orauthorize his subcontrac-tors to follow, the amendedor revised document pro-vided no impact on sched-ule or increase in cost,price, or fee is required.The contractor shall not berequired to follow theamended or revised documentexcept as a formallyauthorized modification tothe contract. If the con-tractor elects to followthe amended or reviseddocument, he shall notifythe contracting officer inwriting of this elec-tion. When the contractorelects to follow the provi-sions of an amendment orrevision, he must followthem in full.

Ordering Government Docu--. Copies of specifi-cations, standards, anddocumentation reouired bvcontractors in c’onnectio;with specific procurementsmay be obtained from theprocuring agency, or asotherwise directed by thecontracting officer.

REQUIREMENTS

Software QA Program. Uponcontract award, the con-tractor shall Dlan.develoD.and implement a SQA Programwhich includes practicesand procedures to assurecompliance with all soft-ware requirements of thecontract. The Programactivities shall be a partof the management report-ing system throughout thelife of the contract. Thecontractor shal1 documentthe Program iflthe form ofa SQA Pla,: (hereafterreferred to ? t?e ‘“Plan”)which meets 2 require-ments of tt s specifica-tion. The Ian shallidentify organizational



responsibilities andauthorities for its execu-tion and the events crit-ical to its implementation.The Plan shall also iden-tify and make timely pro-visions for special needs(controls, tools, facili-ties, skills, etc. )required for the Programand shall provide fordetection, reporting, anal-ysis, and correction ofsoftware problems and defi-ciencies. Contractor per-sonnel performing qualityfunctions shall have theresponsibility, authority,and organizational freedomto evaluate software activ-ities, identify problems,and initiate or recommendcorrective action.

A. Review of Requirement.1. TO establish a software quality

program which fulfills the requirementsof MIL-s-52779A, contractors must iden-

tify the functions and activities thatdirectly affect software compliance andassign specific authority and responsi-bility for these functions. The assign-ment is made in terms of decisions andactions to identified elements at alllevels of organization.

2. The specification explicitlyrequires contractors to satisfy certainsoftware quality program requirements,but does not specify an organizational

arrangement of any kind for meeting theserequirements.

3. Software shall be developed in

a disciplined manner. h effective

check and balance shall be built into themanagement system for controlling all keysoftware development tasks. The contrac-tor*s quality organization shall beincluded in this process.

B. Application.1. Although MIL-s-52779A does not

dictate an organizational structure, nosingle department can satisfy all of thesoftware quality program requirements ofMIL-s-52779A. Contractors likely will

want to vest authority and responsibilityfor coordination and management of the

implementalion of MIL-s-52779A to aparticular organizational component (forexample, Software Quality AssuranceDepartment) . TYPicallY, the interactionof several departments of a contractor’sorganization (such as, engineering,programlproject office, test, and soft-ware quality assurance) is required toeffectively implement the software qual-ity program to which MIL-S-52779Aapplies.

2. A complete software quality pro-gram reflects a comprehensive andextensive activity of a contractor.Usually a contractor will have standardprocedures for the application ofMIL-s-52779A. In addition, relativeto each contract, these procedureswill be tailored as necessary toprovide assurance of compliance with con-tract requirements in an economicalmanner. The requirements of MIL-S-52779Adescribed in Section 3. which are con-tractually required, will be specified inthe Software Quality Pian. That Planwill likely reference the applicable com–pany procedures; however, the methods tobe utilized may be documented within thePlan itself. The Plan shall also identifyand make timely provisions for special

needs, controls, tools, facilities,skills, etc. , required for execution ofthe Plan. Or.hertypical documents likelyto be referenced include programmingstandards and conventions, manuals,computer handbooks and other formsof management manuals. The accomplish-ment of software quality functions

tYPically will be performed by personnelfrom several different organizations.

The key element is that the checks andbalances of the software quality programeffectively provide for delivery of con-tractually required software which com-plies with the design requirements.

c. Criteria for Evaluation.1. Does the established program

identify the organizational element

responsible for each of the various soft-ware quality efforts?

2. Do the personnel performing thesoftware quality functions have suffi-

cient authority, responsibility, andfreedom of action to evaluate SOftWaIedesign and production activities, and toinitiate and/or recommend changes?

9


MIL-HDBK-33415 JOL 1981

3. Do the personnel performing thesoftware quality functions have specificdocumented definitions of their assignedduties?

4. Are Software Quality AssuranceProEram activities included as a Dart ofthe- contractor’s management repcmti”gsystem?

5, Does the contractor delineate thevarious software quality efforts?

6. Is the program documented and issuch documentation available for Govern-ment review?

3.2 Software QA ProgramRequirements. The Planshall address the follow-ing requirements:

3.2.1 Tools, Techniques, andMethodo-. The Planshall identify the tools,techniques, methodologiesand records to be empleyedin the performance of thework which will support QAobjectives and describe howtheir use wil1 augment orsatisfy QA Program require-ments. Examples include:Operations Research -Systems Analysis tech-niques, functional andperformance requirementsanalysis, error analysis,software optimizationtools, specification trac-ing, and coding conven-tion.

A. Review of Requirement. MIL-S–52779Arequires contractors to identify in theirSoftware QA ?lan all of the tools, tech-niq”es, methodologies, and records thatthey propose to utilize during the lifeof the contract to verify the quality ofthe software. Additionally, the planwill describe how each tool, technique,

and methodology identified satisfies oraugments software requirements.

B. Application.1. During the life of a contract,

many individual functions are performedto ensu~e the quality of the software.

This includes the reviews of software

● documentation from the initial specifi-cation documents through the final test

reports. Contractors may in some caseshave developed automated test aids toverify the quality of the software. Sim-“latio” programs may be a part of thecontractors library or can he developedespecially for the testing of the soft-ware. The contractor describes how theuse of these programs actually verifiesthe quality of the software.

2. The contractor may include manua 1techniques and methods for such areas asdocument review. In this case detailedreview checklists or guidelines may beemployed. The identity of these check-lists are given along with how they willbe employed and recorded.

3. The contractor may include in thePlan structured design and coding tech-niques. These techniques in part can berepresented by standards and contentions.Also a method of how the contractor willverify the use of these standards andconventions should be identified.

c. Criteria for Evaluation.1. Has the contractor identified and

defined the systemfsoftware engineeringtechniques and methodologies planned foruse to support the requirements of qual-ity assurnnce? (They may he documentedin the Computer Program DevelopmentPlan.)

2. Are the contractor’s a“tomatedtools acceptable, or will they beaccepted prior to use?

3. Are the automated tools doc”-

mented and placed under configurationmanagement controls?

4. Does the contractor’s Plandescribe provisions for identifying,

document ing, controlling of revisions,

validating, and calibrating the appli-

cable sofcware support tools used to

ac;ually verify the quality of deliver-able software?

5. Does the contractor have avail-able documentation to support the use ofexisting test tools?

6. Has the contractor documented

those methods of analysis employed in theperform~~ce of their contract?

7. Do the above documents describethe initially known limitations of theiranalysis techniques and are the documentsupdated during the evolution of the pro-

gram? If required, are these techniquesconcurred with, as described in thecontract?

I


MIL-HDBK-33415 JUL 19B1

I

I

8. Are there records of allessential activities?

9. Are records available to Govern-ment personnel, and furnished whenrequired ?

10. Are there effective means forassuring the currency, ccnnplete”ess,andaccuracy of records?

11. Do records include only thenumbers and kinds of deficiencies? Isother essential data recorded? How andwhere?

12. Do records and work instructioncompliance records indicate the quantita-tive degree of acceptance or rejection ofproduct or “ork effort?

13. If rejection is recorded, dorecords show resulting action?

14. Do management actions reflect theanalysis and “se of records?

15. Does the contractor identifydesign and coding standards and convec-tions?

16. Does the contractor identify howdesign and coding standards and conven-tions are verified?

3.2.2 Computer Program Design.The Plan shall referenceor document the proceduresby which design documen-tation is reviewed toevaluate design logic, ful-fillment of requirements,completeness, and compli-ance with specified stand-ards. Design documentationshall be subjected toindependent review priorto its release for coding.

A. Review of Requirement. MIL-s-52779Arequires contractors to establish pro-cedures for the re”iew and evaluationof software design documentation. Thesereviews should emphasize review of the

design from the viewpoint that the designshould reflect the requirements and areto be accomplished prior to the com-mencement of coding. Adequate proceduresare necessary to assure that each design

document is complete, that all require-have been met, and that the logic of thedesign is described and substantiated.

The Software QA Plan should referenceprocedures for design review.

B. Application.1. Each computer program that will

be individually tested is coded to therequirements contained in a softwaredesign document. Independent (other thanthe designer) reviews of design documen-tation will be conducted prior to it’srelease for coding. Some contractorsestablisb software design review boaris.The boards are convened to conduct designreviews at predetermined points, ofteninde”tified as : System RequirementsReview (sRR), System Design Review (SDR),Software Segment Design Review, Prelim-inary Design Revie” (PDR), and CriticalDesign Review (CDR). The composition ofthese review boards should contain repre-sentatives from varied activities, suchas soft”are design, prograrmning, analy-sis, testing, and soft”are qualityassurance (refer to paragraph 3.2.6 forspecifics on reviews) .

2. Regardless of the methods

employed, the Software QA Plan includesprocedures for the evaluation of softwaredesign to verify that the design meetsthe requirements. The Plan includes pro-visions for assuring the effectivefollow-up on all action items resultingfrom tbe review.

c. Criteria for Evaluation.1. Do the co”tractcm’s procedures

address the conduct of design documenta-tion re”iews?

2. Are the informal and formal

reviews scheduled at critical decision

points during development ?3. Are design documentation reviews

conducted prior to release for coding?4. Does the contractor have a mech-

anism to determine if all software

requirements are satisfied by the design?5. Are design problems identified

and corrective action taken prior toapproval of design?

6. Ark design documentation reviewsconducted independently of the design

group?

3.2.3 Work Certifica@. ThePlan shal1 reference ordocument the contractor’sprocedures for formallY:[~roving or certifying

description, author-ization, and completion of

I 6

~ ,..


MIL-HDBK-33415 JUL 1981 I

work performed under thecontract. The Program shal1require monitoring toassure compliance, withthese procedures.

A. Review of Requirement. The contrac-tor’s procedures for issuing workinstructions should provide for defini-tion and authorization of tasks, trackingand reporting task progress, resourceallocation, and steps for closing outcompleted tasks . Procedures shouldidentify the method employed to monitorcompliance.

B. Application. Contractors usuallyhave a formal procedure for describingand authorizing work to he done as wellas to stop work in process whereappropriate. The procedure’s formalityand sophistication will depend on a con-tractor and the contract under considera-tion. The real significance is “hetherthe procedures provide adequate control.The procedures should depict the alloca-tion of quality resources.

c. Criteria for Evaluation.—1. Is the lowest acceptable organi-

zation level for QA involvement specifiedin the Plan?

2. Is the level of authorizationsufficient to provide management control?

3. Are there provisions for monitor-ing and tracking the progress of tasks?

4. Can the task progress be relatedto the approved project schedules?

5. Is the relationship between tasksand the Work Breakdown Structure (WBS)element visible, if contractuallyinvoked ?

6. Do the tasking procedures callfor a detailed description of the tasksrelated to the Statement of Work (SOW)?

7. Is the responsible manager foreach task identified?

8. Are plansfproced”resfrespons ibil-

ities defined to:-authorize tasks?-allocate resources?–close o“t completed tasks?

3.2.4 Documentation. Documentationstandards and programing

●conventions and practicesto be used for all softwareshall be referenced or

documented in the Pian.The Plan shal1 referenceor document the proceduresto be applied to assurecompliance with standards,practices, and conventionsand delivery of correctdocumentation and changeinformation to the Govern-ment. In addition, thePlan shall provide for theindependent review of docu-mentation and designationof contractor approvalauthority.

NOTE : For the purpose of placing properemphasis on this section, the Handbookwill treat the general subject of Docu-mentation and Programming Standards andlor Coding Conventions as separate items.

General Documentation.

A. Review of Requirement. MIL-s-52779A

requires that documentation standards bestated or referenced in the Plan. Themethod of incorporating changes will alsobe described in the Plan. The procedurefor the review and che contractor’sdesignated approval fdisapproval authorityshall also be defined, The method foraccomplishing required independentreviews shall be described.

B. Application. During the developmentof a software system, there will be dif-ferent software personnel preparing docu–ments for the computer programs that aretheir assigned responsibility. To pre-clude the publication of documentationthat varies in design from one wciter toanother and to simplify the reviewers andcustomers task, the contractor nl”st

establish the standards to be used. Thismay be accomplished by referring to DODStandard 7935.1-S, Automated Data SystemDocumentation Standard; MIL-STD-483, Con-fig”ratio” Management Practices for

Systems, Equipment, Munitions and Com-puter Programs; or MIL-STD-490, Specifi-cation Practices. For all documentssupporting deliverable software, the

contractor will reference or note the

prescribed standards to be utilized in“riting these documents. The methods forupdating andfor changing these documents,once they have heen approved a“d placed

7



under Config”ratfo” Control, are alsodescribed by the contractor.

c. Criteria for Evaluation.1. Does the contractor identify

standards to be followed when preparingthe required documentation?

2. Do the procedures call for inde-pendent technical review of documentationprior to release?

3. Do the procedures address thecontrol of changes to software documen-tation?

4. Do the procedures provide fortraceability of changes? (Documentationshould also be verified for traceabilityof requirements from one document toanother. )

5. Are there erovisions for info~-ing design personnel of the latestchanges in software documentaticm?

6. lk the procedures cover thereview of software documentation forcompleteness ?

7. Do the procedure.s cover thereview of software documentation forconsistency? (Consistency is a measureof understandability for both the userand customer which shows a strict anduuiform adherence to prescribed symbols,

notations and terminology.)8. Does the contractor designate the

software approval ldisapproval authority?

Programming Standards and Conventions.

A. Review of Requirement. The Planshall include or reference the proceduresdeveloped to assure uniformity throughprogramming standards and coding conven-tion. Under this subject, emphasis

should be placed on the audit functionthat will assure compliance to program-ming standards and coding conventions

prior to testing.

B. Application. Ourtug the developmentof Computer Programs, the contractor’sPlan shall identify the procedures andstandards that cover the methodology used

for encoding, structure design notations,flow charts, and the auditing of thesefunctions to assure conformance.

C. Criteria for Evaluation.1. Has the contractor established a

standard for naming conventions andabbreviations ?

2. Is there a standard set for entryto and exit from code segments?

3. IS there a coding standard forthe number of statements per source line?

4. Is there a standard for groupingof format statements?

5. Has a standard been establishedfor the grouping and arrangement of data?

6. Is there a standard focnnatoferror messages?

7. Are there rigorous standards forthe control and “se of patches?

B. Are there standards for the sizeof individual code segments? (For exam-ple, a code segment might be limited to50 lines of code.)

9. IS there a standard for the useof comments, their frequency. and theirclarity?

10. Is there a coding standard whichcontrols the use of loop variables?

3.2.5 Computer Program LibraryControls. The Plan shallreference or document thecontractor’s procedures andcontrols for the handlingof source code and objectcode and related data intheir various fotms andversions, from the timeof their initial approvalor acceptance unti1 theyhave been incorporated intothe final media. Theobjective of these controlsis to ensure that differentcomputer program versionsare accurately identifiedand documented, that nounauthorized niodificationsare made, that al1 approvedmodifications ar:n~roperlyincorporated, thatsoftware submitted fortesting is the correct ver-sion.

A. Review of Requirement. MIL-s-52779Arequires that contractors establish pos-

itive controls for the handling of sourceand object program materials. The inte-grity of these program materials is main-tained by the application of changecontrol procedures.

B. Application. An important functionfor a contractor is the maintenance and

1 8

,..



control of software materials. Thecontractor establishes controls toassure that when computer program mate-rials are produced, certified, and placedunder configuration control, they”are notaltered or changed without proper docu-mentation, aPPrOval and validation. Asecure Computer Program Library is estab-lished to maintain this integrity.

c. Criteria for Evaluation.1. Has the contractor established a

computer program library to be used forcontrolling program materials duringdevelopment and test?

2. Do the procedures identify howmaterials are approved and placed underlibrary controls?

3. DO the controls include formalrelease procedures for internally

approved design information?4. What safeguards have been estab-

lished to assure that unauthorizedalterations are not made to the

controlled material?5. Do the procedures provide direc-

tions for ensuring that all apprOvedmodifications are integrated?

6. Does the software library assignand track computer programs and documen-tation identification numbers, includingrevision codes?

7. Does the software library prop-erly store all released material “ithprovisions for accurate retrieval?

g. Does the library have the abilityto capture all Information essential toproduce distribution records and statusreports ?

9. Is the proper release authoriza-tion documented and followed?

10. Is there an authorized signaturelist for release documents?

3.2.6 Reviews and Audits. ThePlan shal1 reference ordocument the contractor’sprocedures for preparationand execution of reviewsand audits, for establish-ing the traceabi1ity ofinitial contract require-ments through the succes-sive baselines, and forensuring that the reviewsand audits are conducted inaccordance with the pre-scribed procedures.

The schedule for review and audits shallbe referenced or stated in the Plan.

A. Review of Requirement. ‘Ihespecifi-cation requires that the Software QA Planidentify tbe reviews and the proposedschedule for reviews. A schedule foraudits is also established. The reviewsand audits are conducted in accordance“ith the procedures described or refer-enced in the Software QA Plan, inresponse to contract requirements, (suchas MIL-STD-1521A, Technical Reviews andA“dits for Systems, Equipment and Com-puter Programs) .

B. Application.1. Sach individual computer program

is coded to the specifications containedin the software design document and pro-gram performance specification. Priorto detail design, a Preliminary DesignReview is held to provide an evaluationof the design and to verify that thedesign meets contractual requirements. ACritical Design Review is conducted whenthe design is essentially complete andthe detailed flow charts or other methodsof specifying detail design (e.g., PrO-gram Design Language (PDL)) are ready forcoding. The composition of boards orteams for these reviews should containexpertise from software design, pr0gr2m-ming, analysis, testing and softwarequality assurance.

2. The contractor performs Func-tional and Physical Configuration Auditswhen required by the procuring agency.These audits are performed to verify thatthe actual performance of the computerprogram complies with the DevelopmentSpecification. Detailed procedures forreviews are contained in MIL-STD-I 521A;when contractually imposed, if notimposed, it may be used as a guide.

C. Criteria for Evaluation.1. Does the contractor’s SofLware

QA Plan establish a schedule for reviewsand audits?

.2. Are the reviews an auditsclearly identified, scheduled, a. I proP-erly sequenced?

3. Does the Plan deline,... thespecialists within the QA or~;a .;:tionwho will participate in the revit+<~sandaudits?

9


4. Are the reviews and auditsconducted to the contractually imposedrequirements ?

5. Do the procedures define thetypes of information to be presented ateach review?

6. Are there agreements for follow-up action resulting from the reviews andaudits?

7. Will the results of the reviewsand audits be documented by the contrac-tor?

8. If MIL-STD-1521A is a contractualrequirement, does the contractor:

a. Conduct both the internal andformal reviews and audits on aschedule and within the guidelines ofMIL-STD-1521A?

b. Conduct software requirementreviews before tbe design starts? (Suchas the re”iew of the draft development

specification which can be reviewed atthe System Design Review or at tbe Soft-ware Segment Design Reviews. )

c. Conduct Preliminary DesignRevie”s (PDRS) prior to detailed design?

d. Conduct Critical DesignReviews (CDRS) prior to coding?

Conduct a Functionalfig”rati~~ Audit (FCA) to verify ~~actual performance of the Computer Pro-gram Configuration Item (CPCI) complieswith the Part I Specification?

f. Conduct a Physical Configu-ration Audit (PCA) to verify that theCPCI Part 11 document ation correctlyand fully describes the Computer Programproduct configuration baseline?

9. Are the results of reviews andcorrective actions adequately documentedto provide an audit trail?

3.2.7 Configuration Management~ The Plan shal1specify the relationshipsbetween the SQA and CMProqrams and shall ref-ereice or document theprocedures for assuringthat the objectives ofthe CM program are beingattained.

A. Review of Requirement. Governmentcontracts usually require a configura-tion Management Plan (cMY), as well asa Software Quality Assurance Plan.

Since tbeTe is a particularly strongrelationship between [he two disciplines,software quality cannot be assured with-out a disciplined CM Program.

●

B. Application. Since SQA and Cf.1Pt-o-grams are intertwined , their respectiveplans should be coordinated tc ensurethat all facets are identified a,,+ con–trols are generated. This will eli,,inateduplication of effort. Some of the co”-tents found in a CM Plan are:

1. Once a baseline has been estab-lished for a computer program or Sup-porting documentation, fOr t?xampk, thespecification, design, test plan, etc.,the integrity of the baseline or documen-tation is protected to ensure that thereare no unauthorized changes.

2. It is important that soft“areconfiguratio” plans identify the author-ity to enter material under configurationcontrol, and to identify the authorityfor removal of controlled items from theconfiguration management acti”ity.

3. The config”racion management planprovides explicit instructions for theidentification of baseline materials andsubsequent revisions or versions. The ●SQA Plan provides procedures that willpreclude the control facilities frombeing used as a repository for “nap-pro”ed, or uncontrolled computer programsand supporting document aticm.

4. Independent audits of the controlfacilities are performed by the organiza-tion designated in the Software QA Plan.The audits are documented to she” thedate of the audits, discrepancies fo“nd,and che completed corrective action.

c. Criteria for Evaluation.1. Does the SQA Plan depict the

relationship between it and the CffP?2. Do the SQA Plan procedures prO-

vide for the review of the CM internalcontrols to ensure that no unauthorized

changes occur to baseline specifications,supperting document ation or the CPCI?

3. Do the SQA Plan procedures pro-

“ide the methodology for CM to respond

to discrepancies found during QA audits?4. Is the contractor complying with

internal procedures for approval a“thor–ity, for placement of items under config-uration control, and for removal of con-trolled items from tbe cOntrOl facility? ●

10


MIL-HD3” 33415 JUL ’31

5. Are CM instructions foridentification of baseline items and sub-sequent revisions or versions beingfollowed?

6. Does the contractor ‘?. CM Planpreclude the control facilities frombeing used as a repository for unap-proved or fi”controlled computer programs,Softwal-etools , and supporting documen-tation?

7. Does the contractor’s CM Program/Plan identify membership to the softwareConfiguration Control Board (CCB)? AreQuality Assurance personnel participa-ting?

8. Does the SQA Plan require auditsof configuration management proceduresand practices?

9. Does the SQA Plan require thatthe results of the audits be documentedand available for Government review?

10. Does the SQA Plan require thatthe audits of CM be documented to she”the date of the audits, discrepanciesfo“nd and the completed correctiveaction?

11. Do the contractor’s proceduresassure that the CCB addresses all facets

● of interface, such aS specifications,manuals. desizn. test Procedures. etc?

with approved test plansand procedures.

e. Certification that testresults are the actualfindings of the tests.

f. Review and certifica-tion of test reports.

9. Ensuring that testrelated media and documen-tation are maintained toallow repeatability oftests.

h. The contractor shal1ensure that support soft-ware and cDmputer hardwareto be used to develop andtest software and hardwareunder the contract areacceptable to the Govern-ment.

A. Review of Requirement.1. The testing activities of the

software development process shouldprovide explicit ass”ra”ce that the soft-

ware Derfonns to its technical and-.

3.2.8 Testing. The Plan shallreference or document pro-cedures for assurinq theaccomplishment of th= fol-lowing:

a. Analysis of softwarerequirements to determinetestability.

b. Review of test require-ments and criteria foradequacy, feasibi1ity, andtraceability and satisfac-tion of requirements.

c. Review of test plans,procedures, and specifica-tions for compliance withcontractor and contractualrequirements and to insurethat al1 authorized andonly authorized changesare implemented.

d. Verification that testsare conducted in accordance

operational requirements. The qualityassurance provisions for this activity,therefore, should be aimed at the reali-zation of these objectives in an orderly,cohesive, clear and controlled fashion.The results of this activity will nor-mally provide the acceptability of thedelivered products.

2. MIL-S-52779A requires contractorsco address the software testing activ-ities in their Software QA Plan. Thisincludes che types of testing to beapplied to computer programs and softwaresystems . The organization that is

responsible for the preparation of testcriteria, test plans, test procedures,and test reports is identified. Themethods and requirements for review ofthe testing activities are defined aswell as the methods for tracking the pro-gre?- of each software component throughthe .esting cycle.

3. The Software QA Plan reflects howthe contractor will ensure that supportsoftware and related documents are

acceptable to the Government. .If there

is additional Suppert software or cOm-

puter hardware used in testing the

11


delitierable software, the contractoridentifies such items and shows howthese items are insured to be accept-able to the Government, prior to “se intesting and validation of the deliverablecomputer program.

B. Application.1. The qualification of software can

only be accomplished rhro”gh the applica-tion of stringent testing. Each phase ofthe development of a software system willnormally require testing and validationprior to continu5.ng to the succeedingstep. For example, some computer pro-grams are tested prior to integration ors“hsystem testing, and if modules areprcd.ced in a top down order, top downtesting can be employed.

2. The Software QA Plan identifiesthe individual types of testing to beutilized in the validation process (forexample, development test, verificationtest, validation test). The contractoridentifies the organization that isresponsible for the development and pre-paration of test plans, test procedures,test case data, test reports, and usermanuals.

3. Contractors are required to havea procedure to define their review oftesting activities to ensure co”fcmmanceto contractual provisions. These pro-cedures are not required to be a part of’the Software QA Plan if the contractorhas published a standard procedure thatis acceptable to the Government.

4. The identification and certifi-cation of tested software, support sOft-ware, and computer hardware is explainedin the Software QA Plan. Forms utilizedby quality assurance for test and entryinto configuration control are alsoidentified.

5. ManY of the difficulties incurredduring -the software development processha”e been due to the relegation of QAactivities to a formal test phase in thefinal stage of the process. This test-oriented approach to QA fails torecognize the contribution of lower leveltest activities to the test objectivesand ignores the fact that the finalproduct is only, at best, a reflection

of its specifications. An effective QAprogram for testing must begin with therequirements, and must address thetotality of the testing to be performed.

6. For the purpose of this documentthe definition of Verification and Vali-dation is:

a. Verification.(l). Computer program veri-

fication is the iterative process ofdetermining whether or not the prcduct ofeach step of the computer program acqui-sition process fulfills all requirementslevied by the previous step. These stepsare system specification verification,requirements verification, specificationverification and code verification.

(2). The process of deter-mining whether the results of execucingthe software product in a test environ-ment agree with the specifications.Verification is usually only concernedwith the software”’slogical correctness[i.e., satisfying the functional require-ments) and may be a manual or a computerbased process (i.e., testing software byexecuting it on a computer).

(3). The process of ensuringthat the system and its structure meetthe functional requirements of the base-line specification document.

b. Validation.(1). The process of deter-

mining whether exec“ting the system(i.e., software, hardware, user pro-cedures, personnel) in a user environmentcauses any operational difficulties. Theprocess includes ensuring that specificprogram functions meet their requirementsand specifications. Validation alsoincludes the prevention, detection, diag-nosis, recovery and correction of errors.

(2). Validation is more dif-ficult than the verification processsince it involves questions of thecompleteness of the specification andenvironment information. There are bothmanual and computer based validationtechniques.

(3). The process of ensuringthat specific program functions meettheir detailed design requirement speci-fication.

c. Criteria for Evaluation. The con-tractor’s test planning informationshould not be included or duplicated inthe SQA Plan. The contractor’s testplans and practices should be documentedor referenced in the Computer ProgramDevelopment Plan (CPDP) and in the Com-puter Program Configuration Item (CPCI),



D~elopment Test and Evaluation (DT&E)Plan. These documents should be reviewedwhen eval”acing the QA aspects of tbetest prwgram and answers to the followi-ng question determined:

1. Does the Software QA Plan iden-tify the contractors software testactivities?

2. Has testing responsibility bee”fdentifLed and assigned to a specificorganization?

3. Does the contractor have pro-cedures and documentation controllinghis internal Computer Program Test andEvaluation (cpT&E) activities?

4. Have the various levels of testbeen identified and scheduled as requiredby the contract?

5. Does the Software QA Plan pro-vide for review of test planslprocedureslspecifications for compliance with con-tract-al requirements?

6. Does the Software QA Plan pro-vide for review of test procedures forcompliance with the test specification,hard”are manuals, data item descriptionsand other contractual requirements?

7. Does the Software QA Plan pro-vide for monitories of tests and thecertification that test results are theactual ‘finding of the tests?

8. Is test-related documentationmaintained to allow repeatability oftests?

9. Is all support software and com-puter hardware that is used to developthe CPCI, acceptable to the Government?

3.2.9 Corrective Action. ThePlan shal1 reference ordocument procedures whichassure the prompt detec-tion, documentation, andcorrection of softwareproblems and deficiencies.Procedures shall include:

a. Documenting and report-ing problems and defi-ciencies to appropriatemanagement levels.

b. Analysis of data andexamination of problem anddeficiency reports todetermine their extent andcauses.

c. Analysis of trends inperformance of work to pre-vent the development ofnonccmpliant products.

d. Review of correctivemeasures to ensure thatproblems and deficiencieshave been resolved andcorrectly reflected in theappropriate documents.

e. Analysis or review asotherwise provided for inthe contract.

A. Review of Requirement.1. In the production of almost all

products, some nonconformaties willinevitably be discovered. Computerprograms are subject to errors, discrep-ancies, and nonconformances to theprocedures. The contractor describes inthe Software QA Plan the procedures to be

aPPlied in the detection and correctionof these problems. When software isproduced by a subcontractor, contractorsindicate how they will identify andensure that the subcontractor promptlycorrects all detected problems.

2. The method of reporting andanalyzing problems and implementing cor-rections is included in the Software QAPlan with additional procedures fortracking problems, trend analysis, andre”iews of the effectiveness of the cor–recti”e action program.

B. Application.1. The detection and correction of

“o”co”formaties to contractual require-ments i“ computer programs requires thecoordinated efforts of many departmentsof a contractor’s organization. A formalsystem is required by the contractor forreporting, tracking, analysis, andclosure of problems. The contractorsprocedures should require that problemsbe identified in writing to permittracking and resolution of the problems.This will assure that the necessary cor-rections are completed in a timelymanner. A discrepancy report form is“s”ally wed by contractors to report,record, and dispose of software problems.

2. Computer program problems thatare not usually considered reportable are

13


MIL-SIDBK-33415 JUL 1981

those that occur during the developmentand debugging phase. Once a program isunder configuration control and a base-line has been established, all problemsencountered are reported.

3. The methods of analyzing reportedproblems, recormnending sol”ticms, andcompleting corrective action are definedby the contractor to document nonconforma-nce tests and permit tracking andanalysis of trends during softwaretesting. Tbe requirements for correctiveaction and the reporting efforts of sub-contractors are defined by the co”crac-tor.

4. Error analysis is a veryimportant tool that should be used as ameans of evaluating computer programs,and directing attention to specific needsfor corrective action or recovery sys-tems. For the purpose of this handbookthe definition of error analysis is:

“The process of locating andassessing conceptual, syntatic, orclerical errors in software which cause(or could cause) the software to manifesta fault (or faults) during test oroperational use, or which could cause thesoftware to fail to perform its intendedf“netion. Errors may be identifiedthro”gb analysis of design documentation,analysis of test or execution documents(printouts), and through observation of

the failure of tbe software to execute inaccordance with test or operational pro-cedures. Software errors can be detectedduring testing on any level of assemblyfrom routine to full configuration item.The following are examples of softwareerror categories:

a. Requirementsb. Documentation (system devel-

opment or product level specification)c. Comp”cationald. Logicale. Data (input, o“cput, han-

dling)f. Interface

g. Data baseh. Environment (operating sys-

tem, support software, test materials andequipment, computer hardware)

i. Human (operator)

SOftware error analysis leads todetermination of the appropriate cor-rective action for each error and the

error data may be utilized to provide a“assessment of the operational readinessof the software.”

C. Criteria for Evaluation. The con-tractor is required to delineate proce-dures which will assure the detection,communication and correction of deficien-cies and errors. These procedures areintended to avoid “oncornpliantCPCI’s,and as such, any review should considerthe following:

1. Has the contractor identified tbeorganizational units involved in thecorrective action process? Are theirresponsibilities and interfaces defined?Is the independence of quality functionsmaintained ?

2. Is the organization responsiblefor administering the corrective actionprogram identified? Is it nested “iththe authority to enforce the correctiveaceion program?

3. Is the relationship clearlydefined between the corrective actionprogram, the overall quality program, tbeconfiguration system, and the programmanagement plan?

4. Does the contractor delineate acorrective action process that isresponsive to the req”iremencs ofMIL-S-52779A and other contractualrequirements?

5. IS there an established policyfor reporting and correcting deficienciesin software?

6. Are the products to be controlledidentified with specific software devel-opment and implementalion baselines?

7. Do the baselines permit a system-atic incorporation of controlled productinto the corrective action system,starting with soft”are documentation,and incorporating code and software media(e.g., storage procedures for disks,decks, and tapes) as coding and debuggingare completed?

g. Does the corrective action pro-cess involve distinct steps: identifyingthe discrepancy in writing, doc”me”ti“gthe proposed “fix”, independent review ofthe proposed “fix” for adequacy, and,when coding changes are required, retestof the affected code and all i“terfaci”gmodules and correction of the affecteddocumentation?

9. Does the corrective action systemestablish a mechanism for feed backof results of error analyses of

●

14

i

It



individual problems and recurrentproblems?

10. Is there a set of written pro–cedures for reporting, following up, andcorrecting software deficiencies, includ-ing forms with instructions for fillingthem out and transmitting them, analyzingthe data for error trends and specifyingthe nature of corrective action required?

11. Is the system for retrieving,analyzing, and reporting software defi-ciency data formalized?

12. Does the contractor’s correctiveaction system apply to discrepanciesgenerated by deliverable and nondeli”er–able software?

3,3 Subcontractor Control. ThePlan shall reference ordocument the procedures toassure that all softwareacquired from subcontrac-tors conforms to applicablerequirements of the con-tract and this specifica-tion. When the Governmentelects to perform reviewsat the subcontractor’sfacilities, such reviewsshall not be used bycontractors as evidence ofeffective control of qual-ity of subcontractors bythe contractor. It doesnot relieve the contractorof his responsibility forfurnishing software thatmeets al1 contract require-ments.

4. Review of Requirement.1. It is not enough for contractors

to Contro1 the quality of the computerprograms de”eloped a“d designed in theirown orga”izatio”. The contractor is alsoresponsible for assuring that all soft-ware , documentation and programmingmaterials procured from subcontractorsconform co the contract requirements.They also are required by MIL-S-52779A toassure control of the quality of softwarefurnished by their subcontractors. Thus ,contractors should choose subcontractorswho C.” maintain adequate q“.ality.Furthermore, contractors must develop anduse effectiva methods for communicating

●applicable Government requirements totheir snbcont~actors.

2. Contractors cannot depend onGovernment inspection at their subcon-tractor’s facilities; instead it isnecessary that they generate their ownknowledge and control of subcontractorquality. How often a contractor willassess the subcontractor’s quality systemdepends “pen the type and quantity of thesoftware purchases from that subcontrac-tor. The best evidence of subcontractorquality comes from the contractor’scontinuin8 evaluation of the software andthe services f“mished by the subcon-tractor. Any deficiencies which becomeknown to the contractor should be made

known in a timely fashion to the s“bco”-tractors for correction.

3. The Government reserves the rightto review products andfor services at thesubcontractor’s facility, although thecontractor is solely and exclusivelyresponsible for the quality of tbe soft–ware deli”ered regardless of the sourceof the software.

B. Application.1. The completeness with which con–

tractors control their software purchasesdetermines in a large measure the successof this phase of their quality program.In choosing their subcontractors, con-tractors should follow the same practicethat the Government follows when choosingbetween qualified competitors; the awardgoes to the best technically qualifiedresponsible offeror, price and otherfactors considered.

2. Various methods are used bycontractors to assure adequate subco”–tractor control. A few of the mostfrequently used are:

a. Participation in s“bcontrac-tor design reviews.

b. Monitoring the effectivenessof the subcontractor’s configurationcontrol methods.

c. Monitoring the subcontrac-tor’s discrepancy reporting and cor-rective action system.

d. Witnessing subcontractoracceptance testing to verify conformanceto test procedures.

3. A“ open, activ., comprehensiveflow of q“aliry information between sub-contractor and contractor can signifi–cantly reduce cost.

4. There are many ways to assurequality in purchased software. Selecting

15


MIL-SJDBK-33415 .lUL19B1

suppliers with a reputation for qualityis a good start. Participation in sub-contractor software design reviews,witnessIngfmonitoring subcontractoracceptance tests, and auditing subcon-tractor software quality system proce-dures and records are some of thetechniques used by contractors to assurequality of software. Of course, contrac-tor effort alone is not enough;subcontractors are expected to possessthe motivation, knowledge, and capabilityto control quality.

5. For some purchases, D3D requiresthe contractor to include on the purchaseorder a requirement for Government sub-contract inspection. When such actionsare deemed necessary by the GovernmentRepresentative, it is necessary thatspecific instructions be provided to theGovernment Representative at the subcon-tractor’s facility. The GovernmentRepresentative at the contractor’sfacility must advise the contractorconcerning any Government subcontractinspection plans as early as POSSible.

c. Criteria for Evaluation.1. Are there adequate procedures for

source selection?2. Are there adequate procedures for

source inspection?3. Does the contractor review their

subcontractor’s quality efforts at inter-vals consistent witb the complexity andquality of the software?

4. Do the contractor’s proceduresdestribe and mandate the methodology toassure that the applicable requirementsestablished in the prime contract arepassed down to subcontractors?

5. Does the contractor plan toaccomplish receiving inspection of pro-cured software to ensure that the properconfiguration of tbe subcontractor’sproduct was delivered?

6. Does the contractor require sub-contractors to prepare and maintain, asQA Plan, a CPDP, and a configurationmanagement plan?

7. Does the contractor review a“d

aPProve the subcontractor’s plaIIs?8. Does the contractor participate

in the subcentractor’s design reviewsand audits?

9. Does the contractor monitortesting performed by the subcontractor?

10. Are there procedures for assuringthat subcontractors correct all nOncOn-formances?

11. Is, there an established SYSternfor corrective attic.”with subcontractorsprior to, as well as subsequent todelivery of software? Is it included asa requirement in the subcontractor’squality assurance plan?

4.0 QUALITY ASSURANCE PROVISIONS.

4.1 Contractor. Nothing speci-fied herein relieves thecontractor from the obli-gation to submit, to theGovernment for acceptance,end products that conformto al1 contract require-ments.

4.2 Government Review at Con-tractor, Subcontractor, orVendor Facilities. TheGovernment reserves theright to review, at theirsources, al1 products orservices, including thosenot developed or performedat the contractor’s facil-ity, to determine theconformance of products orservices with contractrequirements.

A. Review of Requirement. A contractoris solely and exclusively responsible forthe quality of the software that isdelivered to the Government regardless ofthe sources of the sofa.are.

B. Application. Therefore, though theGovemmen t may conduct inspections atsubcontractor’s facilities, the primecontractor’s responsibilities remainmchanged. It should be noted that”onlyGovernment

/

representatives can auth rizeGovernment inspections at subcontractor’sfacilities. When such inspections arerequired, the Government Quality sur-ance Representative (QAR) at the con rac-tor’s facility shall handle allsubcontracts in accordance with heirrespective Procurement Quality AssuranceProgram (PQAP).

16


,:

t.fIL-HDBK-334~5 JLIL1981

C. Criteria for Evaluation.1. DO contractor purchasing docu-

ments require Government review ofsubcontractors only when the Governmentso requests?

5.0 PREPARATION FOR DEI.IVERY.The Plan shall referenceor document procedures forassuring integrity ofsoftware products duringhandling, storage, preser-vation, packaging andshipping,

A. Review of Requirement. Documentedwork instructions are necessary for boththe operation and the inspection of theshipping fmction. The material handlingaspect of shipping requires monitoredwork instructions Methods used to pre–serve and protect iterns must becompatible with the intended “se of theitems, yet protect the items againstdamage or deterioration in storage.Special requirements, such as a con-trolled storage en”iro”ment, must alsobe carefully de”ised, mai”tai”ed , andmonitored to assure full protection ofq“alicy. Labeling which clearlyindicates special handling and storagerequirements is imperative. Loadingpractices must conform with the require-ments of common carriers and with

specified Government (e.g., InterstateCommerce Commission, U. S. Post Office)or industry regulations, Contractualrequirements for the identification andmovement of shipments must be met. Thecontractor’s quality program mustestablish effective practices for pro-tecting quality during shipping. I“addition, all ha”dli”g, storage, anddeli”ery requirements must be covered bydocumented work i“str”ctions.

B. Application. Control during han-dling, storage, and delivery is animportant aspect of satisfactory qualityprograms. Ma””fact”rers and users ofproducts which are subject to damage anddeterioration when improperly handled andstored, carefully plan their preserva-tion, packaging, packing and storageefforts. They conduct regularlyscheduled inspections of all storedmaterial. Shipping and storage controldep.artme”ts usually de”elop documentedwork and inspection i“structio”s fo~

handling, storing, preset ..,g,packagi~g,packing, marking, and shipping materialsto prevent damage, 10ss, deterioration,substitution, degradation, or any otherquality defects.

C. Criteria for Evaluation.1. Are adequate work a“d inspection

instructions prepared and implemented forthe handling, storage and delivery ofmaterial?

2. Are handling, storage and deliv–ery procedures nm”itored in accordance“ith established quality program require-ments?

3. Are there procedures and regularschedules for tbe inspection of productsin storage, and are these proceduresadequate to pre”e”t deterioration ordamage?

4. Are .11 required critical e“vi-ro”ments maintained “ithin packaging?

5. Is all material to be storedor shipped properly identified andlabeled?

6. Are all shipments prepared andtransported in compliance with con-tractual requirements and applicableGovernment and carrier regulations?

6.0 m (The followinginformation is providedsolely f0r guidance inusing this specification.It has no contractualsignificance.)

6.1 Intended Use, This docu-ment will apply specifi-cally to the acquisitionof computer software wherethe acquisition involveseither software alone, Orsoftware as a portion of asystem or subsystem.

6.2 Ordering Data. The pro-curing activity shouldconsider specifying thefollowing:

6.2.1 Procurement Requirements.

a. T:tle, number, and dateof this specification.

b. Software QA ProgramPlan. Consideration ;houldbe given to requ,rinq the

17



contractor to deliver aSoftware QA Program Plan inresponse to the invitationfor bid, or request forproposal, or request forquotation and as a ContractData Requirements Listitem (see 6.2.2). The Planshould define the methodsand procedures which thecontractor proposes to usein fulfilling the require-ments of this specifica-tion. Note: The SoftwareQA Program Plan may beincluded as part of otherplans, see paragraph 1.3.

c. The application ofthis specification shouldbe carefullY tai1ored tomeet the minimal essentialneeds of the acquisition.

d. Consideration should begiven to citing currentstandards and specifica-tions f0r configurationmanagement, documentation,review, audit, development

\

practices, work breakdownstructures, etc.

e. Application of thisspecification to softwaremaintenance contracts isencouraged.

f. Rapidly changing tech-nology may require theacquiring activity to clar-ify use or application ofthe term “firmvare”.

6.2.2 Contract Data Requirements.Dlans, documentation,

and reports which arerequired to be deliveredto the Government willbe specifiedDD Form 1423, Co%rac~Data Requirements List(CDRL) or authorized equiv-alent. The format, type ofcopy, number of copies,degree of detai1 required,delivery schedules, andpurpose of submissionshould be specified on theCDRL.

18


II ., ,~!1,

Custodians:

Army - AD

Navy - OM

Air Force - 10

Review Activities:

Army - TM-MT-CR-ER-MI-SC

Navy - NM-OS-SH-AS

Air Force - 05-06-10-11-13-14-17-19-23

DLA - OH

User Activities:

Army

Navy - YD

Air Force - 01-02

MIL-HDBK-334 I

15 JUL 1981

Preparing Activity:

Army - USACSC

Project QCIC - 0002

19


,.

INSTRUCTIONS: In a continuingeffortto make ourstandardizationdocumentsbetter,theDoD providesthisform forusein

submittingcommenti and suggestionsforimprovements.AU usem of military standardization documents are invited to providesuggedtiom. TIIia form may be detached, folded along the lines indkated, taped along the loose edge (DO NOT STAPLE), and

mailed. In block5,be asspecificu possibleabut particularpmbleznareassuchM wordingwhtch required interpretation, wan

too rigid, restrictive, loose, ambiguous, or was incompatible, and give propoc-ed wording changes tiich would alleviate the

problems. Enterin block 6 any remarks not related to a specific paragraph of the document. If block 7 b fiiled out, an

acknowledgement till be mailed to You within 30 d8Y8to let You know that Your comments were received and are Ming

considered.

NOTE: Thii form may not b-e used to request copies of documents, nor to request waivers, deviations, or clarification of

specification requirement on cument contracts. Comments submitted on thk? form do not constitute or imply authorization

to waive anY portion of the referenced document(s) or to amend contractual requirements.

(Fold .1.”s thb 1(”.)

<Fold .1OW thic 1[”.)

DEPARTMENT OF THE ARMY

111111 l-l

lJblTEOSTATES

OFFICIAL BUSINESSPENALTY FOR PRIVATE USE WW BUSINESS REPLY MAIL

FIRST CLASS PEFIMITNO 12082 WASHINGTON 0 CPOSTAGE WILL BE PAIDBY THE DEPARTMENT OF THE ARMY’

Commander

U.S. Army Computer Systems Command

ATTN : ACSC-QAA-A (STOP H-5)

Ft. Belvoir, VA22060


I

I

-

IIII

II(II

II

III

/

I

STANDARDIZATION DOCUMENT IMPROVEMENT PROf@~L(SeeInstructions- Reverse Side) /’

DocuMENT NUMBER 2. DOCUMENT TITLE

NAME OF SUBMITTING 0nGAt41zATt0td 4.TYPE OF ORGANIZATION (workmu)

Q VEN121JR

❑ USEFIA00RE”&3.@hwt, City. 8WC, ZIP C&}

❑ MANUFACTURE.

❑ 04’HER (Smc,fy,:-

PBO%LEW AFIEAS~ [email protected] .nd Wovdintl:

b. moc.m.n.ndti WordinQ:

I

c. Ro~ll/Rstian.l. for RQcwnrrMndmion:

REMARKS

NAME OF SUBMITTER & t, Fimt, Ml) - q.t,onsl b. WORK TELEPHONE NUM8ER {J”cIU& Am.c.a&)- Olltie”d

WAILING AD DIIE2s(Street.City..SYatqZIPCode)- ‘OPtlond 8. DATE OF SUBMISSION (YYMMDD)

m FORM 4 Ah@ -- E,,,”,- .-, -,,.., #e,.-””, e--au 82-MARI*LD .-...,”.,- =“, , ,“.. .= “p”,-=, .s.

B

B

B
