TECHNICAL REVIEWS AND AUDITS - Middle East …ocw.metu.edu.tr/pluginfile.php/407/mod_resource/content/0/lecture... · Chapter 11 Technical Reviews and Audits 99 CHAPTER 11 ... The

Chapter 11 Technical Reviews and Audits

99

CHAPTER 11

TECHNICAL REVIEWSAND AUDITS

• Establishing a common configuration baselinefrom which to proceed to the next level ofdesign, and

• Recording design decision rationale in thedecision database.

Formal technical reviews are preceded by a seriesof technical interchange meetings where issues,problems and concerns are surfaced and addressed.The formal technical review is NOT the place forproblem solving, but to verify problem solving hasbeen done; it is a process rather than an event!

Planning

Planning for Technical Reviews must be extensiveand up-front-and-early. Important considerationsfor planning include the following:

• Timely and effective attention and visibility intothe activities preparing for the review,

• Identification and allocation of resourcesnecessary to accomplish the total review effort,

• Tailoring consistent with program risk levels,

• Scheduling consistent with availability ofappropriate data,

• Establishing event-driven entry and exit criteria,

• Where appropriate, conduct of incrementalreviews,

• Implementation by IPTs,

11.1 PROGRESS MEASUREMENT

The Systems Engineer measures design progressand maturity by assessing its development at keyevent-driven points in the development schedule.The design is compared to pre-established exitcriteria for the particular event to determine if theappropriate level of maturity has been achieved.These key events are generally known as TechnicalReviews and Audits.

A system in development proceeds through asequence of stages as it proceeds from concept tofinished product. These are referred to as “levelsof development.” Technical Reviews are done aftereach level of development to check design matu-rity, review technical risk, and determines whetherto proceed to the next level of development. Tech-nical Reviews reduce program risk and ease thetransition to production by:

• Assessing the maturity of the design/develop-ment effort,

• Clarifying design requirements,

• Challenging the design and related processes,

• Checking proposed design configurationagainst technical requirements, customer needs,and system requirements,

• Evaluating the system configuration at differentstages,

• Providing a forum for communication, coordi-nation, and integration across all disciplines andIPTs,

Systems Engineering Fundamentals Chapter 11

100

Figure 11-1. Technical Review Process

• Track actionitems andissues

• Track actionitem completiontrends

• Document anddistributeresults ofreview andaction itemcompletions

• Assignresponsibility

• Individual andteam reviews

• Facilitate andpace meeting

• Examine reviewdata andanalyses –record andclassify findings

• Address keyissues identi-fied by pre-review activity

• Assess severityof problems

• Identify actionitems

• Individual andteam reviews

• Examine data• Analyze data• Track and

documentanalysis

• Have overviewmeeting

• Identifyparticipants

• Assign rolesand tasks

• Establishguidelines andprocedures

• Establish anduse entrycriteria

• Establish exitcriteria basedon the event-driven schedule

Follow-up

Resolve

Review

Pre-review

Familiarize

Plan

Before During After

• Review of all system functions, and

• Confirmation that all system elements areintegrated and balanced.

The maturity of enabling products are reviewedwith their associated end product. Reviews shouldconsider the testability, producibility, training, andsupportability for the system, subsystem orconfiguration item being addressed.

The depth of the review is a function of the com-plexity of the system, subsystem, or configurationitem being reviewed. Where design is pushingstate-of-the-art technology the review will requirea greater depth than if it is for a commercial off-the-shelf item. Items, which are complex or anapplication of new technology, will require a moredetailed scrutiny.

Planning Tip: Develop a check list of pre-review,review, and post-review activities required. De-velop check lists for exit criteria and required levelof detail in design documentation. Include keyquestions to be answered and what informationmust be available to facilitate the review process.Figure 11-1 shows the review process with keyactivities identified.

11.2 TECHNICAL REVIEWS

Technical reviews are conducted at both the sys-tem level and at lower levels (e.g., sub-system).This discussion will focus on the primary system-level reviews. Lower-level reviews may be thoughtof as events that support and prepare for the sys-tem-level events. The names used in reference to


101

reviews is unimportant; however, it is importantthat reviews be held at appropriate points in pro-gram development and that both the contractor andgovernment have common expectations regardingthe content and outcomes.

Conducting Reviews

Reviews are event-driven, meaning that they areto be conducted when the progress of the productunder development merits review. Forcing a review(simply based on the fact that a schedule devel-oped earlier) projected the review at a point in timewill jeopardize the review’s legitimacy. Do thework ahead of the review event. Use the reviewevent as a confirmation of completed effort. Thedata necessary to determine if the exit criteria aresatisfied should be distributed, analyzed, andanalysis coordinated prior to the review. The typeof information needed for a technical reviewwould include: specifications, drawings, manuals,

schedules, design and test data, trade studies, riskanalysis, effectiveness analyses, mock-ups, bread-boards, in-process and finished hardware, testmethods, technical plans (Manufacturing, Test,Support, Training), and trend (metrics) data. Re-views should be brief and follow a prepared agendabased on the pre-review analysis and assessmentof where attention is needed.

Only designated participants should personallyattend. These individuals should be those that wereinvolved in the preparatory work for the reviewand members of the IPTs responsible for meetingthe event exit criteria. Participants should includerepresentation from all appropriate governmentactivities, contractor, subcontractors, vendors andsuppliers.

A review is the confirmation of a process. Newitems should not come up at the review. If signifi-cant items do emerge, it’s a clear sign the review is

Figure 11-2. Phasing of Technical Reviews

Tech Reviews

Documents

Baselines

Sys Perf Spec

Item Perf Specs

Item Detail/TDP

Functional

Allocated

Product

ASR SRR SFR PDR CDR SVRFCA

PCA

LRIP Block I

Demonstration

○ ○ ○ ○ ○ ○ ○

○ ○ ○ ○ ○ ○ ○

○ ○ ○ ○ ○ ○ ○ ○

Contractor Government

draft

A BMS

SystemDefinition

SysTech

Rqmts

ItemDesignRqmts

DetailedDesignRqmts

CI

SY

S

Prod Readiness Rate ProdIntegrationCADCE

CORD○ ○ ○


102

being held prematurely, and project risk has justincreased significantly. A poorly orchestrated andperformed technical review is a significantindicator of management problems.

Action items resulting from the review are docu-mented and tracked. These items, identified byspecific nomenclature and due dates, are preparedand distributed as soon as possible after the review.The action taken is tracked and results distributedas items are completed.

Phasing of Technical Reviews

As a system progresses through design and devel-opment, it typically passes from a given level ofdevelopment to another, more advanced level ofdevelopment. For example, a typical system willpass from a stage where only the requirements areknown, to another stage where a conceptualsolution has been defined. Or it may pass from astage where the design requirements for theprimary subsystems are formalized, to a stagewhere the physical design solutions for thoserequirements are defined. (See Figure 11-2.)

These stages are the “levels of development” re-ferred to in this chapter. System-level technicalreviews are generally timed to correspond to thetransition from one level of development to an-other. The technical review is the event at whichthe technical manager verifies that the technicalmaturity of the system or item under review is suf-ficient to justify passage into the subsequent phaseof development, with the concomitant commitmentof resources required.

As the system or product progresses throughdevelopment, the focus of technical assessmenttakes different forms. Early in the process, the pri-mary focus is on defining the requirements onwhich subsequent design and development activi-ties will be based. Similarly, technical reviewsconducted during the early stages of develop-ment are almost always focused on ensuring thatthe top-level concepts and system definitionsreflect the requirements of the user. Once system-level definition is complete, the focus turns to de-sign at sub-system levels and below. Technical re-views during these stages are typically design re-views that establish design requirements and then

Figure 11-3. Typical System-Level Technical Reviews

RequirementsReviews

DesignReviews

VerificationReviews

Alternative System Review

System Requirements Review

System Functional Review

Preliminary Design Review(includes System Software Specification Review)

Critical Design Review

Test Readiness Review

Production Readiness Review

Functional Configuration Audit

System Verification Review

Physical Configuration Audit


103

verify that physical solutions are consistent withthose requirements. In the final stages of develop-ment, technical reviews and audits are conductedto verify that the products produced meet the re-quirements on which the development is based.Figure 11-3 summarizes the typical schedule ofsystem-level reviews by type and focus.

Another issue associated with technical reviews,as well as other key events normally associatedwith executing the systems engineering process,is when those events generally occur relative tothe phases of the DoD acquisition life-cycleprocess. The timing of these events will vary some-what from program to program, based upon theexplicit and unique needs of the situation; how-ever, Figure 11-4 shows a generalized concept ofhow the technical reviews normal to systemsengineering might occur relative to the acquisitionlife-cycle phases.

Specific system-level technical reviews are knownby many different names, and different engi-neering standards and documents often use differ-ent nomenclature when referring to the samereview. The names used to refer to technicalreviews are unimportant; however, it is importantto have a grasp of the schedule of reviews that isnormal to system development and to have anunderstanding of what is the focus and purpose ofthose reviews. The following paragraphs outline aschedule of reviews that is complete in terms ofassessing technical progress from concept throughproduction. The names used were chosen becausethey seemed to be descriptive of the focus of theactivity. Of course, the array of reviews and thefocus of individual reviews is to be tailored to thespecific needs of the program under development,so not all programs should plan on conducting allof the following reviews.

Figure 11-4. Relationship of Systems Engineering Eventsto Acquisition Life Cycle Phases

SystemsEngineering

Activities

ASR

Fabric

ate, In

tegrat

e & Te

st

Design

TRR

PCAFCASVR

CDR

PDR

SFR

SRR

Systems Acquisition(Engineering Development, Demonstration,

LRIP and Production)

Pre-SystemsAcquisition

Sustainment andMaintenance

REQUIREMENTSREVIEW

Configuration Definition

Test

CE CAD Integration Sustainment

BA C

Demonstration LRIP Rate

Prototype Demos EDMs

Relationship to Requirements Process

MNS ORDAll validated by JROC


104

Alternative Systems Review (ASR)

After the concept studies are complete a preferredsystem concept is identified. The associated draftSystem Work Breakdown Structure, preliminaryfunctional baseline, and draft system specificationare reviewed to determine feasibility and risk.Technology dependencies are reviewed to ascer-tain the level of technology risk associated withthe proposed concepts. This review is conductedlate during the Concept Exploration stage of theConcept and Technology Development Phase ofthe acquisition process to verify that the preferredsystem concept:

• Provides a cost-effective, operationally-effectiveand suitable solution to identified needs,

• Meets established affordability criteria, and

• Can be developed to provide a timely solutionto the need at an acceptable level of risk.

The findings of this review are a significant inputto decision review conducted after ConceptExploration to determine where the system shouldenter in the life-cycle process to continue devel-opment. This determination is largely based ontechnology and system development maturity.

It is important to understand that the path of thesystem through the life-cycle process will bedifferent for systems of different maturities. Con-sequently, the decision as whether or not to conductthe technical reviews that are briefly described inthe following paragraphs is dependent on the extentof design and development required to bring thesystem to a level of maturity that justifies producingand fielding it.

System Requirements Review (SRR)

If a system architecture system must be developedand a top-down design elaborated, the system willpass through a number of well-defined levels ofdevelopment, and that being the case, a well-planned schedule of technical reviews is impera-tive. The Component Advanced Development stage(the second stage of Concept and Technology

Development in the revised acquisition life-cycleprocess) is the stage during which system-level ar-chitectures are defined and any necessary advanceddevelopment required to assess and control tech-nical risk is conducted. As the system passes intothe acquisition process, i.e., passes a Milestone Band enters System Development and Demonstra-tion, it is appropriate to conduct a SRR. The SRRis intended to confirm that the user’s requirementshave been translated into system specific techni-cal requirements, that critical technologies are iden-tified and required technology demonstrations areplanned, and that risks are well understood andmitigation plans are in place. The draft systemspecification is verified to reflect the operationalrequirements.

All relevant documentation should be reviewed,including:

• System Operational Requirements,

• Draft System Specification and any initial draftPerformance Item Specifications,

• Functional Analysis (top level block diagrams),

• Feasibility Analysis (results of technologyassessments and trade studies to justify systemdesign approach),

• System Maintenance Concept,

• Significant system design criteria (reliability,maintainability, logistics requirements, etc.),

• System Engineering Planning,

• Test and Evaluation Master Plan,

• Draft top-level Technical Performance Measure-ment, and

• System design documentation (layout drawings,conceptual design drawings, selected suppliercomponents data, etc.).

The SRR confirms that the system-level require-ments are sufficiently well understood to permit


105

the developer (contractor) to establish an initial sys-tem level functional baseline. Once that baseline isestablished, the effort begins to define the function-al, performance, and physical attributes of the itemsbelow system level and to allocate them to thephysical elements that will perform the functions.

System Functional Review (SFR)

The process of defining the items or elementsbelow system level involves substantial engineer-ing effort. This design activity is accompanied byanalysis, trade studies, modeling and simulation,as well as continuous developmental testing toachieve an optimum definition of the major ele-ments that make up the system, with associatedfunctionality and performance requirements. Thisactivity results in two major systems engineeringproducts: the final version of the system perfor-mance specification and draft versions of theperformance specifications, which describe theitems below system level (item performance speci-fications). These documents, in turn, define thesystem functional baseline and the draft allocatedbaseline. As this activity is completed, the systemhas passed from the level of a concept to a well-defined system design, and, as such, it is appropri-ate to conduct another in the series of technicalreviews.

The SFR will typically include the tasks listedbelow. Most importantly, the system technicaldescription (Functional Baseline) must be ap-proved as the governing technical requirementbefore proceeding to further technical development.This sets the stage for engineering design anddevelopment at the lower levels in the systemarchitecture. The government, as the customer,will normally take control of and manage thesystem functional baseline following successfulcompletion of the SFR.

The review should include assessment of the fol-lowing items. More complete lists are found instandards and texts on the subject.

• Verification that the system specificationreflects requirements that will meet userexpectations.

• Functional Analysis and Allocation of require-ments to items below system level,

• Draft Item Performance and some Item DetailSpecifications,

• Design data defining the overall system,

• Verification that the risks associated with thesystem design are at acceptable levels forengineering development,

• Verification that the design selections have beenoptimized through appropriate trade studyanalyses,

• Supporting analyses, e.g., logistics, human sys-tems integration, etc., and plans are identifiedand complete where appropriate,

• Technical Performance Measurement data andanalysis, and

• Plans for evolutionary design and developmentare in place and that the system design ismodular and open.

Following the SFR, work proceeds to complete thedefinition of the design of the items below systemlevel, in terms of function, performance, interfacerequirements for each item. These definitions aretypically captured in item performance specifica-tions, sometimes referred to as prime item devel-opment specifications. As these documents arefinalized, reviews will normally be held to verifythat the design requirements at the item level reflectthe set of requirements that will result in anacceptable detailed design, because all design workfrom the item level to the lowest level in the systemwill be based on the requirements agreed upon atthe item level. The establishment of a set of finalitem-level design requirements represents the defi-nition of the allocated baseline for the system.There are two primary reviews normally associ-ated with this event: the Software SpecificationReview (SSR), and the Preliminary Design Review(PDR).


106

Software Specification Review (SSR)

As system design decisions are made, typicallysome functions are allocated to hardware items,while others are allocated to software. A separatespecification is developed for software items todescribe the functions, performance, interfaces andother information that will guide the design anddevelopment of software items. In preparation forthe system-level PDR, the system softwarespecification is reviewed prior to establishing theAllocated Baseline. The review includes:

• Review and evaluate the maturity of softwarerequirements,

• Validation that the software requirements speci-fication and the interface requirements speci-fication reflect the system-level requirementsallocated to software,

• Evaluation of computer hardware and softwarecompatibility,

• Evaluation of human interfaces, controls, anddisplays

• Assurance that software-related risks have beenidentified and mitigation plans established,

• Validation that software designs are consistentwith the Operations Concept Document,

• Plans for testing, and

• Review of preliminary manuals.

Preliminary Design Review (PDR)

Using the Functional Baseline, especially theSystem Specification, as a governing requirement,a preliminary design is expressed in terms of designrequirements for subsystems and configurationitems. This preliminary design sets forth the func-tions, performance, and interface requirements thatwill govern design of the items below system level.Following the PDR, this preliminary design (Allo-cated Baseline) will be put under formal config-uration control [usually] by the contractor. The

Item Performance Specifications, including thesystem software specification, which form thecore of the Allocated Baseline, will be confirmedto represent a design that meets the SystemSpecification.

This review is performed during the SystemDevelopment and Demonstration phase. Reviewsare held for configuration items (CIs), or groupsof related CIs, prior to a system-level PDR. ItemPerformance Specifications are put under configu-ration control (Current DoD practice is for con-tractors to maintain configuration control over ItemPerformance Specifications, while the governmentexercises requirements control at the systemlevel). At a minimum, the review should includeassessment of the following items:

• Item Performance Specifications,

• Draft Item Detail, Process, and MaterialSpecifications,

• Design data defining major subsystems,equipment, software, and other systemelements,

• Analyses, reports, “ility” analyses, trade stud-ies, logistics support analysis data, and designdocumentation,

• Technical Performance Measurement data andanalysis,

• Engineering breadboards, laboratory models,test models, mockups, and prototypes used tosupport the design, and

• Supplier data describing specific components.

[Rough Rule of Thumb: ~15% of production draw-ings are released by PDR. This rule is anecdotaland only guidance relating to an “average” defensehardware program.]

Critical Design Review (CDR)

Before starting to build the production line thereneeds to be verification and formalization of the


107

mutual understanding of the details of the itembeing produced. Performed during the SystemDevelopment and Demonstration phase, this re-view evaluates the draft Production Baseline(“Build To” documentation) to determine if thesystem design documentation (Product Baseline,including Item Detail Specs, Material Specs, Pro-cess Specs) is satisfactory to start initial manufac-turing. This review includes the evaluation of allCIs. It includes a series of reviews conducted foreach hardware CI before release of design to fab-rication, and each computer software CI beforefinal coding and testing. Additionally, test plansare reviewed to assess if test efforts are develop-ing sufficiently to indicate the Test ReadinessReview will be successful. The approved detaildesign serves as the basis for final productionplanning and initiates the development of finalsoftware code.

[Rough Rule of Thumb: At CDR the design shouldbe at least 85% complete. Many programs usedrawing release as a metric for measuring designcompletion. This rule is anecdotal and only guid-ance relating to an “average” defense hardwareprogram.]

Test Readiness Review (TRR)

Typically performed during the System Demon-stration stage of the System Development andDemonstration phase (after CDR), the TRR as-sesses test objectives, procedures, and resourcestesting coordination. Originally developed as asoftware CI review, this review is increasinglyapplied to both hardware and software items. TheTRR determines the completeness of test proce-dures and their compliance with test plans anddescriptions. Completion coincides with theinitiation of formal CI testing.

Production Readiness Reviews (PRR)

Performed incrementally during the SystemDevelopment and Demonstration and during theProduction Readiness stage of the Production andDeployment phase, this series of reviews is heldto determine if production preparation for the sys-tem, subsystems, and configuration items is com-

plete, comprehensive, and coordinated. PRRs arenecessary to determine the readiness for produc-tion prior to executing a production go-aheaddecision. They will formally examine the pro-ducibility of the production design, the control overthe projected production processes, and adequacyof resources necessary to execute production.Manufacturing risk is evaluated in relationship toproduct and manufacturing process performance,cost, and schedule. These reviews support acqui-sition decisions to proceed to Low-Rate InitialProduction (LRIP) or Full-Rate Production.

Functional Configuration Audit/ SystemVerification Review (FCA)/(SVR)

This series of audits and the consolidating SVRre-examines and verifies the customer’s needs, andthe relationship of these needs to the system andsubsystem technical performance descriptions(Functional and Allocated Baselines). They deter-mine if the system produced (including produc-tion representative prototypes or LRIP units) iscapable of meeting the technical performancerequirements established in the specifications, testplans, etc. The FCA verifies that all requirementsestablished in the specifications, associated testplans, and related documents have been tested andthat the item has passed the tests, or correctiveaction has been initiated. The technical assessmentsand decisions that are made in SVR will be pre-sented to support the full-rate production go-aheaddecision. Among the issues addressed:

• Readiness issues for continuing design, continu-ing verifications, production, training, deploy-ment, operations, support, and disposal havebeen resolved,

• Verification is comprehensive and complete,

• Configuration audits, including completion of allchange actions, have been completed for all CIs,

• Risk management planning has been updatedfor production,

• Systems Engineering planning is updated forproduction, and


108

• Critical achievements, success criteria andmetrics have been established for production.

Physical Configuration Audit (PCA)

After full-rate production has been approved, fol-low-on independent verification (FOT&E) hasidentified the changes the user requires, and thosechanges have been corrected on the baseline docu-ments and the production line, then it is time toassure that the product and the product baselinedocumentation are consistent. The PCA will for-malize the Product Baseline, including specifica-tions and the technical data package, so that futurechanges can only be made through full configura-tion management procedures. Fundamentally, thePCA verifies the product (as built) is consistentwith the Technical Data Package which describesthe Product Baseline. The final PCA confirms:

• The subsystem and CI PCAs have beensuccessfully completed,

• The integrated decision database is valid andrepresents the product,

• All items have been baselined,

• Changes to previous baselines have beencompleted,

• Testing deficiencies have been resolved andappropriate changes implemented, and

• System processes are current and can beexecuted.

The PCA is a configuration management activityand is conducted following procedures establishedin the Configuration Management Plan.

11.3 TAILORING

The reviews described above are based on acomplex system development project requiringsignificant technical evaluation. There are also

cases where system technical maturity is moreadvanced than normal for the phase, for example,where a previous program or an Advanced Tech-nical Concept Demonstration (ACTD) has pro-vided a significant level of technical developmentapplicable to the current program. In some casesthis will precipitate the merging or even elimina-tion of acquisition phases. This does not justifyelimination of the technical management activi-ties grouped under the general heading of systemsanalysis and control, nor does it relieve thegovernment program manager of the responsibil-ity to see that these disciplines are enforced. It does,however, highlight the need for flexibility andtailoring to the specific needs of the program underdevelopment.

For example, a DoD acquisition strategy that pro-poses that a system proceed directly into the dem-onstration stage may skip a stage of the completeacquisition process, but it must not skip the for-mulation of an appropriate Functional Baseline andthe equivalent of an SFR to support the develop-ment. Nor should it skip the formulation of theAllocated Baseline and the equivalent of a PDR,and the formulation of the Product Baseline andthe equivalent of a CDR. Baselines must be devel-oped sequentially because they document differ-ent levels of design requirements and must buildon each other. However, the assessment of designand development maturity can be tailored as ap-propriate for the particular system. Tailored effortsstill have to deal with the problem of determiningwhen the design maturity should be assessed, andhow these assessments will support the formula-tion and control of baselines, which document thedesign requirements as the system matures.

In tailoring efforts, be extremely careful determin-ing the level of system complexity. The systemintegration effort, the development of a singleadvanced technology or complex sub-component,or the need for intensive software development maybe sufficient to establish the total system as a com-plex project, even though it appears simple becausemost subsystems are simple or off-the-shelf.


109

11.4 SUMMARY POINTS

• Each level of product development is evaluatedand progress is controlled by specification de-velopment (System, Item Performance, ItemDetail, Process, and Material specifications) andtechnical reviews and audits (ASR, SRR, SDR,SSR, PDR, CDR, TRR, PRR, FCA, SVR,PCA).

• Technical reviews assess development maturity,risk, and cost/schedule effectiveness to deter-mine readiness to proceed.

• Reviews must be planned, managed, andfollowed up to be effective as an analysis andcontrol tool.

• As the system progresses through the develop-ment effort, the nature of design reviews andaudits will parallel the technical effort. Initiallythey will focus on requirements and functions,and later become very product focused.

• After system level reviews establish the Func-tional Baseline, technical reviews tend to besubsystem and CI focused until late in devel-opment when the focus again turns to the sys-tem level to determine the system’s readinessfor production.


110