Jim Schultz 28-October-2008

28-May-200828-May-2008 11

Jim Schultz28-October-2008

Constellation Program Surveillance Strategy

28-May-200828-May-2008 22

CxP Surveillance StrategyCxP Surveillance Strategy

• Traditional Surveillance Strategies Traditional Surveillance Strategies consist primarily of Inspections consist primarily of Inspections and Audits.and Audits.

• Inspections and audits are Inspections and audits are necessary, however they are “after-necessary, however they are “after-the-fact” and should not be the the-fact” and should not be the sole surveillance approach.sole surveillance approach.

28-May-200828-May-2008 33

CxP Surveillance Strategy CxP Surveillance Strategy (cont.)(cont.)

• Constellation Projects are required to develop Constellation Projects are required to develop and document a surveillance strategy concurred and document a surveillance strategy concurred with by the Constellation Program QA manager.with by the Constellation Program QA manager.– The The level of risklevel of risk and the and the impact of failureimpact of failure are major are major

determinants in defining the type of surveillance determinants in defining the type of surveillance required. If the impact of failure is minor and the risk required. If the impact of failure is minor and the risk low, only a minimal amount of insight driven low, only a minimal amount of insight driven surveillance is needed. Conversely, if the failure surveillance is needed. Conversely, if the failure impact could be significant and the risk high, more impact could be significant and the risk high, more extensive surveillance (including possible oversight) is extensive surveillance (including possible oversight) is necessary. necessary.

– The focus should be on The focus should be on preventionprevention rather than rather than detection, i.e., emphasizing controlled processes and detection, i.e., emphasizing controlled processes and methods of operation, as opposed to relying solely methods of operation, as opposed to relying solely upon inspection to identify problems. upon inspection to identify problems.

28-May-200828-May-2008 44


• Identify specific risks and surveillance Identify specific risks and surveillance methods used to help track and mitigate methods used to help track and mitigate these risks.these risks.

• Examples of Surveillance MethodsExamples of Surveillance Methods– Metrics reviews (Metrics reviews (e.g., process escapes, audits e.g., process escapes, audits

scheduled, audits completed, audit findings, scheduled, audits completed, audit findings, nonconformances, corrective actions taken, cost of nonconformances, corrective actions taken, cost of quality, delivery performance, quality performance, quality, delivery performance, quality performance, process performance, cycle time, labor performance process performance, cycle time, labor performance (utilization, efficiency, productivity), lost (utilization, efficiency, productivity), lost workdays/recordable incidents, employee surveys, etc.workdays/recordable incidents, employee surveys, etc.) )

– Document/quality record reviewsDocument/quality record reviews– AuditsAudits

28-May-200828-May-2008 55


• Examples of Surveillance Methods (cont.)Examples of Surveillance Methods (cont.)– Management information (Management information (e.g., management e.g., management

review minutes (w/ focus on internal audit and review minutes (w/ focus on internal audit and corrective action effectiveness), development projects corrective action effectiveness), development projects status reports, board/meeting minutes, Contractor status reports, board/meeting minutes, Contractor weekly activity reports, weekly activity reports, NASA/Constellation ProjectNASA/Constellation Project ContractorContractor Management Management tag-ups, Special Issues tag-ups, Special Issues meetings, Flight Readiness Review chart, meetings, Flight Readiness Review chart, DataData RequirementsRequirements Documents (DRD), etc. Documents (DRD), etc.))

– Performance reviews (Performance reviews (e.g., witnessing the e.g., witnessing the performance of the task being surveyed such as flight performance of the task being surveyed such as flight controller simulation and real-time operations, controller simulation and real-time operations, process audits, Flight Readiness Reviews (FRR), real-process audits, Flight Readiness Reviews (FRR), real-time inline product inspection, tests, etc.time inline product inspection, tests, etc.) )

28-May-200828-May-2008 66


• Examples of Surveillance Methods (cont.)Examples of Surveillance Methods (cont.)– Quality tool implementation (Quality tool implementation (e.g., Design of e.g., Design of

Experiments (DOE), Multiple Environment Overstress Experiments (DOE), Multiple Environment Overstress Testing (MEOST), Environmental Stress Screening Testing (MEOST), Environmental Stress Screening (ESS), Quality Functional Deployment, Total (ESS), Quality Functional Deployment, Total Productivity Maintenance (TPM), Benchmarking, Self-Productivity Maintenance (TPM), Benchmarking, Self-Check Systems, Next Operation as Customer, Supply Check Systems, Next Operation as Customer, Supply Management, Total Value Engineering, Cycle Time Management, Total Value Engineering, Cycle Time Reduction, Lean Six Sigma, Fault Tree Analysis, Reduction, Lean Six Sigma, Fault Tree Analysis, Process Failure Modes and Effects Analysis (PFMEA), Process Failure Modes and Effects Analysis (PFMEA), Statistical Process Control (SPC), Probability Risk Statistical Process Control (SPC), Probability Risk Analysis (PRA), PRACA, FMEA/CIL, etc.Analysis (PRA), PRACA, FMEA/CIL, etc.))

– Customer feedback - formal and informal (Customer feedback - formal and informal (e.g., e.g., board representatives, Program Offices, all levels of board representatives, Program Offices, all levels of contractcontract management, customer surveys, field data, management, customer surveys, field data, etc.etc.))

28-May-200828-May-2008 77


• Examples of Surveillance Methods Examples of Surveillance Methods (cont.)(cont.)– Self-Check SystemsSelf-Check Systems

•Provide sensors (electrical, mechanical, or Provide sensors (electrical, mechanical, or visual) that warn an operator that a mistake visual) that warn an operator that a mistake has been or is about to be made. Keep the has been or is about to be made. Keep the solutions solutions simplesimple..

– In-Process Monitoring – address In-Process Monitoring – address condition before it becomes a problem condition before it becomes a problem (e.g., (e.g., Statistical Process Control (SPC), Statistical Process Control (SPC), Chemical Fingerprinting, cycle time, Chemical Fingerprinting, cycle time, precursor analysis, etc.precursor analysis, etc.))

28-May-200828-May-2008 88


• Examples of Surveillance Methods Examples of Surveillance Methods (cont.)(cont.)– Supplier PerformanceSupplier Performance

•Quality performanceQuality performance•Delivery performanceDelivery performance•Total recordable incident rate (TRIR)Total recordable incident rate (TRIR)•Lost work incident rate (LWIR)Lost work incident rate (LWIR)•Earnings Before Interest, Taxes, Earnings Before Interest, Taxes,

Depreciation, and AmortizationDepreciation, and Amortization•Mergers or changes in managementMergers or changes in management•Mergers or changes in major Mergers or changes in major

subcontractor managementsubcontractor management

28-May-200828-May-2008 99


• The surveillance strategy employed must not The surveillance strategy employed must not drive technical requirements. drive technical requirements.

• The surveillance strategy employed must not The surveillance strategy employed must not drive design; instead design, manufacturing, drive design; instead design, manufacturing, and assembly must drive surveillance and assembly must drive surveillance strategy employed.strategy employed.

• The surveillance strategy should change over The surveillance strategy should change over the life cycle of the item (vehicle, system, the life cycle of the item (vehicle, system, etc.). As the item ages and more risk etc.). As the item ages and more risk information is identified, changes may be information is identified, changes may be necessary or beneficial to reflect either an necessary or beneficial to reflect either an increase or decrease in risk. increase or decrease in risk.

• The surveillance strategy should be reviewed The surveillance strategy should be reviewed and adjusted at least annually.and adjusted at least annually.

28-May-200828-May-2008 1010

•Use a risk management approach and apply Use a risk management approach and apply Penetration Levels based on level of risk of each Penetration Levels based on level of risk of each area:area:

– High Risk Areas = Higher PenetrationHigh Risk Areas = Higher Penetration

– Low Risk Areas = Lower PenetrationLow Risk Areas = Lower Penetration

•Minimum acceptable Penetration Level is Minimum acceptable Penetration Level is contingent on ensuring:contingent on ensuring:

– Safety for the public. Safety for the public.

– Safety for astronauts and pilots.Safety for astronauts and pilots.– Safety for NASA workforce.Safety for NASA workforce.– Safety for high-value equipment and property. Safety for high-value equipment and property.


28-May-200828-May-2008 1111

•Feedback should be used Feedback should be used to adjust level of to adjust level of penetrationpenetration– Changes in risk Changes in risk

– NASA and Contractor metricsNASA and Contractor metrics

– Customer evaluationsCustomer evaluations

– Quality performance / delivery performanceQuality performance / delivery performance

– Cost of Quality (rework, repair, scrap, warranty, Cost of Quality (rework, repair, scrap, warranty, concessions, recalls/GIDEPs/NASA Alerts, returns, etc.)concessions, recalls/GIDEPs/NASA Alerts, returns, etc.)

– Inspection / test resultsInspection / test results

– Audit / survey / assessment resultsAudit / survey / assessment results

– Schedule aggressivenessSchedule aggressiveness

– Number and scope of requirements and design changesNumber and scope of requirements and design changes


28-May-200828-May-2008 1212

5 L M H H H

4 L M M H H

3 L M M M H

2 L L L M M

1 L L L L M

1 2 3 4 5Consequence of FailureConsequence of Failure

LLiikkeelliihhoooodd

Revie

w of

Imple

men

ta

tion

Did th

ey

do th

ings

right?

Incr

easin

g SR&QA P

enet

ratio

n

Did th

ey d

o

the

right

thin

gs?

Revie

w

of

Proce

sse

s

Level 5

Level 3 Level 2

Level 4

Level 1

Penetration

RiskHH = HighMM = MediumLL = Low

Typical Penetration Mapped on Risk Chart


28-May-200828-May-2008 1313

•Acceptable risk is the risk that is understood Acceptable risk is the risk that is understood and agreed to by the appropriate parties and agreed to by the appropriate parties sufficient to achieve the defined success sufficient to achieve the defined success criteria within the approved level of resources. criteria within the approved level of resources.

•Characterization of a primary risk as Characterization of a primary risk as "acceptable" must be supported by the "acceptable" must be supported by the rationale, with the concurrence of the senior rationale, with the concurrence of the senior management, that all reasonable mitigation management, that all reasonable mitigation options (within cost, schedule, and technical options (within cost, schedule, and technical constraints) have been instituted.constraints) have been instituted.


28-May-200828-May-2008 1414

Surveillance Activity must Add Surveillance Activity must Add ValueValue

• The surveillance strategy must add value to The surveillance strategy must add value to the output of the process.the output of the process.– Identify and correct problems (or potential Identify and correct problems (or potential

problems) as early as possible in the process.problems) as early as possible in the process.– Do not add unnecessary surveillance activities Do not add unnecessary surveillance activities

in the process.in the process.• The surveillance strategy should be The surveillance strategy should be

streamlined into the streamlined into the manufacturing/assembly processmanufacturing/assembly process– Steps performed within process during Steps performed within process during

operation.operation.– Avoid/rethink surveillance that inspects batches Avoid/rethink surveillance that inspects batches

of parts. Avoid surveillance bottle necks.of parts. Avoid surveillance bottle necks.

28-May-200828-May-2008 1515

ActivitActivityy

Is it (part of):Is it (part of):*adjusting?*approving?*doing work over?*expediting?*inspecting?*moving?*reviewing?*storing?*troubleshooting*waiting

Does the end customer care?(Is the end customer willing to pay for it?)

Does it convert input to output?(Is it required to produce output?)

Are we doing this for customer (vs. for us)?

NVANVANoNoNoNo NoNo NoNo

YesYesYesYes

YesYes

VAVA

YesYes

Surveillance Activity must Add Surveillance Activity must Add ValueValue

28-May-200828-May-2008 1616


• Summary:Summary:– The level of surveillance is determined by the The level of surveillance is determined by the

level of risklevel of risk and the and the impact of failureimpact of failure. . – There are many different surveillance methods; There are many different surveillance methods;

not just inspection.not just inspection.– Put an emphasis on surveillance methods that Put an emphasis on surveillance methods that

uncover conditions before they become uncover conditions before they become problems (e.g., precursor, self-check, SPC, problems (e.g., precursor, self-check, SPC, business/financial changes, etc.)business/financial changes, etc.)

– Characterize the acceptable risk with rationale Characterize the acceptable risk with rationale and concurrence from Senior Management.and concurrence from Senior Management.

– Surveillance methods must add value and be Surveillance methods must add value and be streamlined into processes.streamlined into processes.

28-May-200828-May-2008 1717

Questions?Questions?

28-May-200828-May-2008 1818

BackupBackup

28-May-200828-May-2008 1919

Penetration Penetration LevelLevel

Risk FactorsRisk Factors(Typical)(Typical)

Assurance Implementation Assurance Implementation (Typical)(Typical)

55

High risk to operational safety High risk to operational safety High risk to system safetyHigh risk to system safety Hardware is critical (Mission Critical and/or Hardware is critical (Mission Critical and/or

Criticality 1 / 2)Criticality 1 / 2) Contractor stability is poorContractor stability is poor Contractor capability is poorContractor capability is poor

NASA has in-line involvement. Performs a complete and independent evaluation of processes and products that satisfy NASA has in-line involvement. Performs a complete and independent evaluation of processes and products that satisfy SR&QA requirements. Approves all SR&QA processes and products. Approves all processes, including implementing SR&QA requirements. Approves all SR&QA processes and products. Approves all processes, including implementing instructions, that satisfy SR&QA requirements. Approves/disapproves or recommends approval/disapproval of request to instructions, that satisfy SR&QA requirements. Approves/disapproves or recommends approval/disapproval of request to depart from or permit noncompliance with SR&QA requirements. Mandatory NASA involvement, contractor cannot proceed depart from or permit noncompliance with SR&QA requirements. Mandatory NASA involvement, contractor cannot proceed without NASA “Go”. without NASA “Go”.

44

High risk to operational safetyHigh risk to operational safety High risk to system safetyHigh risk to system safety Hardware is critical (Mission Critical and/or Hardware is critical (Mission Critical and/or

Criticality 1 / 2)Criticality 1 / 2) Contractor stability is goodContractor stability is good Contractor capability is goodContractor capability is good

NASA has in-line involvement with processes and products associated with program accepted risks to SR&QA. Performs a NASA has in-line involvement with processes and products associated with program accepted risks to SR&QA. Performs a complete and independent evaluation, and approval of these processes and products. Reviews and approves all complete and independent evaluation, and approval of these processes and products. Reviews and approves all implementing work instructions. Approves/disapproves or recommends approval/disapproval of request to depart from or implementing work instructions. Approves/disapproves or recommends approval/disapproval of request to depart from or permit noncompliance with SR&QA requirements. Mandatory NASA involvement in accomplishment of processes and permit noncompliance with SR&QA requirements. Mandatory NASA involvement in accomplishment of processes and products associated with program accepted risks to SR&QA. Monitors, during occurrence, all other processes and products products associated with program accepted risks to SR&QA. Monitors, during occurrence, all other processes and products that satisfy SR&QA requirements. Contractor cannot proceed without NASA “Go”. that satisfy SR&QA requirements. Contractor cannot proceed without NASA “Go”.

33

Medium risk to operational safetyMedium risk to operational safety Medium risk to system safetyMedium risk to system safety Hardware is critical (Mission Essential Hardware is critical (Mission Essential

and/or Criticality 2)and/or Criticality 2) Contractor stability is very goodContractor stability is very good Contractor capability is very goodContractor capability is very good

NASA has in-line involvement with processes and products associated with program accepted risks to SR&QA. Reviews and NASA has in-line involvement with processes and products associated with program accepted risks to SR&QA. Reviews and approves the contractor completed evaluation of these processes and products, including implementing work instructions. approves the contractor completed evaluation of these processes and products, including implementing work instructions. Approves/disapproves or recommends approval/disapproval of request to depart from or permit noncompliance with SR&QA Approves/disapproves or recommends approval/disapproval of request to depart from or permit noncompliance with SR&QA requirements. Mandatory NASA involvement in accomplishment of processes and products associated with program requirements. Mandatory NASA involvement in accomplishment of processes and products associated with program accepted risks to SR&QA, contractor cannot proceed with these tasks without NASA “Go”. Monitors all other processes and accepted risks to SR&QA, contractor cannot proceed with these tasks without NASA “Go”. Monitors all other processes and products that satisfy SR&QA requirements, contractor proceeds unless NASA says “No Go”. products that satisfy SR&QA requirements, contractor proceeds unless NASA says “No Go”.

22

Some risk to operational safetySome risk to operational safety Some risk to system safetySome risk to system safety Hardware is critical (Mission Essential)Hardware is critical (Mission Essential) Contractor stability is excellentContractor stability is excellent Contractor capability is excellent Contractor capability is excellent

NASA has in-line involvement with the review and approval of processes and products associated with program accepted NASA has in-line involvement with the review and approval of processes and products associated with program accepted risks to SR&QA. Mandatory NASA involvement in the accomplishment of these tasks, contractor cannot proceed with these risks to SR&QA. Mandatory NASA involvement in the accomplishment of these tasks, contractor cannot proceed with these tasks without NASA “Go”. Monitors all other processes and products that satisfy SR&QA requirements, contractor proceeds tasks without NASA “Go”. Monitors all other processes and products that satisfy SR&QA requirements, contractor proceeds unless NASA says “No Go”. Approves/disapproves or recommends approval/disapproval of request to depart from or permit unless NASA says “No Go”. Approves/disapproves or recommends approval/disapproval of request to depart from or permit noncompliance with SR&QA requirements. noncompliance with SR&QA requirements.

11

Low risk to operational safetyLow risk to operational safety Low risk to system safetyLow risk to system safety Hardware is not criticalHardware is not critical Contractor stability is exceptionalContractor stability is exceptional Contractor capability is exceptionalContractor capability is exceptional

NASA has in-line involvement with the review and approval of products associated with program accepted risks to SR&QA NASA has in-line involvement with the review and approval of products associated with program accepted risks to SR&QA (i.e., Hazard reports, CILs, etc.). Mandatory NASA involvement may be required in the accomplishment of these tasks (i.e., Hazard reports, CILs, etc.). Mandatory NASA involvement may be required in the accomplishment of these tasks (GMIPs). Evaluates contractor data associated with all other processes and products to assure satisfaction of SR&QA (GMIPs). Evaluates contractor data associated with all other processes and products to assure satisfaction of SR&QA requirements, contractor proceeds unless NASA says “No Go”. Approves/disapproves or recommends approval/disapproval requirements, contractor proceeds unless NASA says “No Go”. Approves/disapproves or recommends approval/disapproval of request to depart from or permit noncompliance with SR&QA requirements.of request to depart from or permit noncompliance with SR&QA requirements.

The premise of risk-based surveillance is simply this: The higher the risk in terms of safety, hardware criticality, or contractor’s stability or capability, the deeper the surveillance penetration.

DRAFTAssurance Penetration LevelsAssurance Penetration LevelsProgram / Project LevelProgram / Project Level

28-May-200828-May-2008 2020

Penetration Level

Assurance Implementation

5In-line involvement with the supplier's processes and generally includes detailed participation in the process / task itself. NASA participation is normally serial to the process / task. (Witness / verify / approve.)

NASA retains and exercises the right to concur or nonconcur with the supplier's decisions. Nonconcurrence must be resolved before the supplier can proceed.

4In depth observation of the supplier's processes and generally includes detailed monitoring of the process / task itself. NASA participation is normally parallel to the process / task.

NASA retains and exercises the right to nonconcur with the supplier's decisions. Nonconcurrence must be resolved before the supplier can proceed.

3Task observation of the supplier's processes and generally includes sampling a minimum set of product or process data to assure product or process compliance. NASA participation is normally parallel to the process / task.

NASA intervenes based on analysis of data.

2Assessment of the supplier's processes and generally includes auditing processes / tasks to assure product or process compliance.

Supplier proceeds until non-compliance surfaces requiring NASA intervention.

1Evaluation of customer feedback; and assessment of the supplier's data and metrics, which generally includes reviewing a minimum set of information to assure compliance with requirements.

Supplier proceeds until an issue arises that comes to the attention of NASA for intervention.

Assurance Penetration LevelsAssurance Penetration LevelsProcess / Task LevelProcess / Task LevelDRAFT

28-May-200828-May-2008 2121

Level Involvement5 Mandatory NASA in-line participation.

4Mandatory NASA in-line participation in tasks involving technical risks. NASA monitors, real time, all other tasks.

3Mandatory NASA in-line participation in tasks imvolving moderate to high risks. NASA conducts statistically valid sample of contractor’s products/operation, NASA intervenes based on analysis of data.

2Mandatory NASA in-line participation in high risk tasks. NASA conducts audits to assure contractor’s products/operations are complying w ith requirements, NASA intervenes w hen non-compliance identif ied.

1

Mandatory NASA in-line participation in high risk tasks. NASA review s contractor-produced data and metrics to assure contractor processes are in control, NASA intervenes w hen evaluation indicates problem areas.

ASSURANCE LEVELS

PENETRATION

PENETRATION

Level 1 Level 2 Level 3 Level 4 Level 5

Cost Greater than 15% budget overrun acceptable.

10% to 15% overrun of budget permitted. 5% to 10% overrun of budget permitted. No more than 5% budget overrun permitted.

No overrun of budget acceptable.

Value of Asset </= $1,000 >$1,000 but </= $25,000 > $25,000 but </= $250,000 > $250,000 but </= $1,000,000 > $1,000,000

Mission Schedule Minimal affect on mission schedule. Delays achieving a processing schedule acceptable.

Delays achieving a Program Level II M ilestone acceptable.

Delays achieving a Program Level I M ilestone acceptable.

Extremely tight schedule; Program Level I M ilestone critical.

Project Schedule Minimal affect on project schedule. Greater that 1 year delay in achieving project completion milestone acceptable.

Up to 6 months delay in achieving project completion milestone acceptable.


Achieve project completion milestone critical.

Defined Processes All processes well defined and no findings on most recent audit

All processes well defined and no more than 2 findings on most recent audit

All processes adequately defined and only 1 finding on most recent audit

All processes adequately defined with 2 or more findings on most recent audit

Processes poorly defined regardless of audit results.

Design Complexity Most simplistic (one task / objective), proven concepts, and current technology

Less simplistic (six or less tasks / objectives, manual control).

Moderate complexity ( computer controlled activity, 1 system). New/proven technology, new/understood materials, and both new and proven processes.

More complex ( computer controlled activity and more than 3 interfacing systems). New and proven technology, understood materials, and proven processes.

More complex ( manual and computer controlled activity and more than 3 interfacing systems). Unproven technology, use of new materials, and new processes.

Task Complexity Most simplistic (one task / objective), proven concepts, and current technology. No critical skill required, non-hazardous process.

Average complexity, some critical tasks, some critical skills required, and history of previous task performance. Non-hazardous / hazardous process.

Complex, some critical tasks, critical skills required, and history of previous task performance. Non-hazardous / hazardous process.

Complex, numerous critical tasks, critical skills required, some risk to successful task accomplishment, and minimal history of task performance. Non-hazardous / hazardous process.

Highly complex, numerous critical tasks, critical skills required, high risk to successful task accomplishment, and no previous history of task performance. Non-hazardous / hazardous process.

Product Safety / Reliability

The product is designated as non-safety and non-reliability critical.

The product is designated as safety and/or reliability critical with no hazards or critical items identified.

The product is designated as safety and/or reliability critical with no hazards identified. Category 2 failure modes exists (CIL item).

The product is designated as safety and reliability critical. Hazards exist. Category 1 & 2 failure modes exists (CIL item). Category 2 risk mitigations are controlled operationally.

The product is designated as safety and reliability critical. Hazards exist. Category 1 & 2 failure modes exists (CIL item). Risk mitigations are controlled operationally.

Product Maintainability

The product may be easily maintained in accordance with prescribed requirements.

The product may be maintained in accordance with prescribed requirements. Requires some specialized maintenance skills and/or some difficulty in obtaining spares.

Product may be maintained with difficulty in accordance with prescribed requirements. Requires specialized maintenance skills and/or obtaining spares is challenging.

Product may be maintained with great difficulty. Requires specialized maintenance skills and/or obtaining spares is extremely challenging.

The product is not maintainable in accordance with prescribed requirements. There are no presently identified alternatives or solutions.

Product Quality The product is Commercial-Off-the-Shelf (COTS). A verifiable product pedigree is not required and latent defects are acceptable.

The product is simple to produce and delivered in accordance with its safety, performance, and design requirements, and approved processes are easily followed. Product pedigree is easily verifiable/not required and some latent defects are acceptable.

The product is moderately complex to produce and delivered in accordance with its safety, performance, and design requirements, and approved processes are reasonably easy to follow. Product pedigree is somewhat easy to verify; some attributes must be verified real time. Latent defects would have an undesired effect when they surface.

The product is complex to produce and delivered in accordance with its safety, performance, and design requirements, and approved processes can be followed. Product pedigree is, with challenges, verifiable, numerous attributes must be verified real time. Latent defects would have an most undesired effect when they surface.

The product is complicated and very complex to produce and delivered in accordance with its safety, performance, and design requirements, and approved processes can, with difficulty, be followed. Product pedigree is very difficult to verify / can’t be verified after the fact. Latent defects could be catastrophic when they surface.

Visibility Routine Management visibility important Politically sensitive Highly politically sensitive Highly politically volatile

Confidence in Vendor /

Contractor

5+ years experience with an excellent trust in the process implementer / product provider

3-5 years experience with and excellent trust in the process implementer / product provider

3-5 years experience with and average/moderate trust in the process implementer / product provider

Less than 3 years experience with and average / moderate trust in the process implementer / product provider.

Less than 3 years experience with and low / no trust in the process implementer / product provider.

Performance

Other Factors

FACTORS

Attributes

Cost

Schedule

Progra

m / Pro

ject Level

SR&QA Penetra

tion

28-May-200828-May-2008 2222

Level Involvement5 In-line involvement through w itness / verify / approve.4 In depth observation through monitoring.3 Task observation through sampling.2 Assessment through auditing

1Evaluating customer feedback and assessment through review ing management information systems

ASSURANCE LEVELS PENETRATION

PENETRATION

Level 1 Level 2 Level 3 Level 4 Level 5

Cost Greater than 15% budget overrun acceptable.

10% to 15% overrun of budget permitted. 5% to 10% overrun of budget permitted. No more than 5% budget overrun permitted.

No overrun of budget acceptable.

Value of Asset </= $1,000 >$1,000 but </= $25,000 > $25,000 but </= $250,000 > $250,000 but </= $1,000,000 > $1,000,000

Mission Schedule Minimal affect on mission schedule. Delays achieving a processing schedule acceptable.

Delays achieving a Program Level II M ilestone acceptable.

Delays achieving a Program Level I M ilestone acceptable.

Extremely tight schedule; Program Level I M ilestone critical.

Project Schedule Minimal affect on project schedule. Greater that 1 year delay in achieving project completion milestone acceptable.



Achieve project completion milestone critical.

Defined Processes All processes well defined and no findings on most recent audit

All processes well defined and no more than 2 findings on most recent audit

All processes adequately defined and only 1 finding on most recent audit

All processes adequately defined with 2 or more findings on most recent audit

Processes poorly defined regardless of audit results.

Design Complexity Most simplistic (one task / objective), proven concepts, and current technology

Less simplistic (six or less tasks / objectives, manual control).

Moderate complexity ( computer controlled activity, 1 system). New/proven technology, new/understood materials, and both new and proven processes.

More complex ( computer controlled activity and more than 3 interfacing systems). New and proven technology, understood materials, and proven processes.

More complex ( manual and computer controlled activity and more than 3 interfacing systems). Unproven technology, use of new materials, and new processes.

Task Complexity Most simplistic (one task / objective), proven concepts, and current technology. No critical skill required, non-hazardous process.

Average complexity, some critical tasks, some critical skills required, and history of previous task performance. Non-hazardous / hazardous process.

Complex, some critical tasks, critical skills required, and history of previous task performance. Non-hazardous / hazardous process.

Complex, numerous critical tasks, critical skills required, some risk to successful task accomplishment, and minimal history of task performance. Non-hazardous / hazardous process.

Highly complex, numerous critical tasks, critical skills required, high risk to successful task accomplishment, and no previous history of task performance. Non-hazardous / hazardous process.

Product Safety / Reliability

The product is designated as non-safety and non-reliability critical.

The product is designated as safety and/or reliability critical with no hazards or critical items identified.

The product is designated as safety and/or reliability critical with no hazards identified. Category 2 failure modes exists (CIL item).

The product is designated as safety and reliability critical. Hazards exist. Category 1 & 2 failure modes exists (CIL item). Category 2 risk mitigations are controlled operationally.

The product is designated as safety and reliability critical. Hazards exist. Category 1 & 2 failure modes exists (CIL item). Risk mitigations are controlled operationally.

Product Maintainability

The product may be easily maintained in accordance with prescribed requirements.

The product may be maintained in accordance with prescribed requirements. Requires some specialized maintenance skills and/or some difficulty in obtaining spares.

Product may be maintained with difficulty in accordance with prescribed requirements. Requires specialized maintenance skills and/or obtaining spares is challenging.

Product may be maintained with great difficulty. Requires specialized maintenance skills and/or obtaining spares is extremely challenging.

The product is not maintainable in accordance with prescribed requirements. There are no presently identified alternatives or solutions.

Product Quality The product is Commercial-Off-the-Shelf (COTS). A verifiable product pedigree is not required and latent defects are acceptable.

The product is simple to produce and delivered in accordance with its safety, performance, and design requirements, and approved processes are easily followed. Product pedigree is easily verifiable/not required and some latent defects are acceptable.

The product is moderately complex to produce and delivered in accordance with its safety, performance, and design requirements, and approved processes are reasonably easy to follow. Product pedigree is somewhat easy to verify; some attributes must be verified real time. Latent defects would have an undesired effect when they surface.

The product is complex to produce and delivered in accordance with its safety, performance, and design requirements, and approved processes can be followed. Product pedigree is, with challenges, verifiable, numerous attributes must be verified real time. Latent defects would have an most undesired effect when they surface.

The product is complicated and very complex to produce and delivered in accordance with its safety, performance, and design requirements, and approved processes can, with difficulty, be followed. Product pedigree is very difficult to verify / can’t be verified after the fact. Latent defects could be catastrophic when they surface.

Visibility Routine Management visibility important Politically sensitive Highly politically sensitive Highly politically volatile

Confidence in Vendor /

Contractor

5+ years experience with an excellent trust in the process implementer / product provider

3-5 years experience with and excellent trust in the process implementer / product provider

3-5 years experience with and average/moderate trust in the process implementer / product provider

Less than 3 years experience with and average / moderate trust in the process implementer / product provider.

Less than 3 years experience with and low / no trust in the process implementer / product provider.

Performance

Other Factors

FACTORS

Attributes

Cost

Schedule

Process / T

ask Level

SR&QA Penetra

tion

28-May-200828-May-2008 2323

Level 1 Level 2 Level 3 Level 4 Level 5 Witness/verification (GMIPs)

based on a risk of death / injury or vehicle/spacecraft loss.

Random observation of process / product tasks

Review contractor quality metrics/trending

Maintain insight of adverse trends, significant issues and significant problems

Accept nonconforming items for the Government

Perform process assessment of major activities

Witness/verification (GMIPs) based on an unacceptable level of risk of undetected nonconformances and risk of death / injury or vehicle/spacecraft loss / damage and supplemented by a Quality surveillance program.

GMIPS are adjusted based on change in risk / trends and metrics.

Perform surveillance of selected tasks based on risk level

Review contractor quality data/metrics/trending

Maintain insight of adverse trends, significant issues and significant problems

Develop quality data/metrics/trending of critical performance characteristics


Perform process assessment of critical activities

Witness/verification (GMIPs) based on an unacceptable level of risk of undetected nonconformances and risk of death / injury or vehicle/spacecraft loss / damage and supplemented by a Quality surveillance program.

GMIPs are adjusted based on change in risk / trends / metrics.

Perform surveillance of tasks based on risk level

Validate contractor quality data/metrics/trending

Maintain oversight of adverse trends, significant issues and significant problems

Develop quality data/metrics/trending of important performance characteristics


Perform process assessment of major activities

Review approve work procedures implementing program accepted risk

Witness/verification (GMIPs) when the task / test occurs

Assure satisfaction of design and operational quality requirements.

Assure defects are identified and screened out at each level of assembly prior to proceeding to the next level of assembly or operation.

GMIPs are adjusted based on change in risk / trends / metrics.

Review approve work procedures implementing program accepted risk

Approve nonconformance dispositions involving high risk


Perform surveillance of most activities


Maintain oversight of adverse trends, significant issues and significant problems

Develop quality data/metrics/trending of performance characteristics

Perform process assessment of most activities

Develop dissimilar models and conduct independent analysis of selected high risk areas

100% closure of all work procedures

100% task witness / verification

100% work procedure review and approval

100% nonconformance dispositions approval.



Perform an independent assessment of adverse trends, significant issues and significant problems

Develop quality data/metrics/trending of performance characteristics

Perform process assessment of most activities

Develop dissimilar models and conduct independent analysis of all high risk areas

DRAFTAssurance Levels - QualityAssurance Levels - Quality